MinIO: a high performance, distributed object storage system

Introduction:

MinIO is an object storage system, similar to the well known AWS S3. Objects can be documents, videos, PDF files etc. To store objects, MinIO offers a scalable, flexible, and efficient solution for storing, accessing and managing data. It's compatibility with AWS S3 API makes seamless integration with AWS S3 based applications.

Credit: Min.io

This is the basic architecture of MinIO, a distributed grid is a type of computer architecture that uses multiple nodes to execute a single task. The nodes are connected to each other by a network, which allows them to communicate with each other.

Need of MinIO:

Use case	Benefit
Big data storage	Scalable architecture can handle large datasets
Cloud-based applications	Compatible with Amazon S3 APIs, easy to integrate with cloud-based applications
Data backup and recovery	Distributed architecture provides high data durability
Object storage services	Similar APIs to major object storage services, users can benefit from the advantages of object storage.

Unlocking Data Analysis Potential: Integrating MinIO with Jupyter Notebook

In the field of data science and research, Jupyter notebook has emerged as a powerful tool, offering an interactive environment for data exploration, analysis, and visualization. As data sets grow in size and complexity, handling them efficiently becomes a daunting task for data scientists. Here, the need arises for a scalable and flexible storage solution to manage vast amounts of data seamlessly. This is where MinIO, a high-performance distributed object storage system, comes into play.

By connecting Jupyter notebook to MinIO, data scientists gain direct access to their stored objects, eliminating the hassles of manual data transfers and ensuring real-time data access. With compatibility to Amazon S3 APIs, the integration also facilitates effortless collaboration in cloud-based environments. Furthermore, MinIO's distributed architecture ensures high data durability, safeguarding valuable research and analysis results. In this section, we will demonstrate how to establish a connection between Jupyter notebook and MinIO, unlocking the potential for enhanced data manipulation and analysis in a scalable environment.

Steps to connect Minio with Jupyter Notebook

In this section, we will walk through a Python code snippet that demonstrates how to connect Jupyter notebook to MinIO, a high-performance distributed object storage system. The code leverages the MinIO Python SDK to establish a connection with MinIO, allowing seamless access to objects stored within the MinIO server directly from a Jupyter notebook environment. By following this step-by-step guide, data scientists and researchers can efficiently manage and analyze large datasets stored in MinIO, enhancing their data science workflow and collaboration.

1. Import MinIO Modules: The code begins by importing the required modules from the MinIO package. This includes the Minio class for creating a MinIO client and the S3Error class for handling errors.

from minio import Minio
from minio.error import S3Error

2. Define Access and Secret Keys: Before establishing the connection, replace the placeholders Enter Access Key and Enter Secret Key with the actual access and secret keys provided by MinIO. These credentials are required to authenticate and access MinIO resources.

Akey = 'Enter Access Key'
Skey = 'Enter Secret Key'

3. Create MinIO Client: The code creates a MinIO client object using the specified MinIO server address and the provided access and secret keys. This client will be used to interact with the MinIO server throughout the code.

client = Minio("s3.dsrs.illinois.edu",access_key=Akey,secret_key=Skey,)

4. List Buckets: The code lists all the buckets available on the MinIO server using the MinIO client object. Buckets are containers for storing objects within MinIO.

client.list_buckets()

5. Create Test Bucket: The code then creates a test bucket named 'test' if it does not already exist. The bucket_exists() method checks if the bucket exists, and if not, the make_bucket() method is used to create it.

bucket_name = "test"
found = client.bucket_exists(bucket_name)
if not found:
       client.make_bucket(bucket_name)
else:
       print(f"Bucket '{bucket_name}' already exists")

6. Upload and Download Objects: The code proceeds to upload and download objects to and from the 'test' bucket. First, it uploads a file named 'filename.csv' from the local filesystem to the 'test' bucket using the fput_object() method.

try:
       client.fput_object(bucket_name,'filename.csv','./filename.csv')
except ResponseError as err:
       print(err)

Then, it lists all objects within the 'test' bucket, including the newly uploaded 'filename.csv'.

objects = client.list_objects(bucket_name,recursive=True)
for obj in objects: 
       print(obj.bucket_name,obj.object_name.encode('utf-8'), obj.last_modified,
              obj.etag, obj.size, obj.content_type)

After that, it downloads the 'filename.csv' file from the 'test' bucket back to the local filesystem as 'filename_example.csv' using the fget_object() method.

try:
       client.fget_object(bucket_name, 'filename.csv', 'filename_example.csv')
except ResponseError as err:
       print(err)

7. Check for Differences: To ensure the accuracy of the file transfer, the code checks for differences between the original 'filename.csv' and the downloaded 'filename_example.csv' files using the diff command. Any differences between the files would indicate potential issues in the transfer process.

!diff filename.csv filename_example.csv

8. MD5 Hash Verification: Finally, the code verifies the integrity of the original 'filename.csv' file and the downloaded 'filename_example.csv' file by checking their MD5 hashes using the md5sum command. Matching MD5 hashes indicate that the files were transferred successfully without any data loss or corruption.

!md5sum filename.csv
!md5sum filename_example.csv

By incorporating this Python code into your Jupyter notebook, you can seamlessly interact with objects stored in MinIO, facilitating efficient data analysis and manipulation within a scalable and flexible storage environment.

Link to the code:

Accessing Minio using Jupyter