Amazon Web Services offers S3 (Simple Storage Service), a scalable object storage service used to store files, datasets, backups, and logs.<\/p>\n\n\n\n
Using Python, you can easily interact with Amazon S3 to upload, download, list, and delete files.<\/p>\n\n\n\n
Amazon S3 is an object storage service that stores data as:<\/p>\n\n\n\n
Example structure:<\/p>\n\n\n\n
my-bucket\/
data.csv
reports\/sales.xlsx<\/p>\n\n\n\n
S3 is highly scalable and cost-effective.<\/p>\n\n\n\n
Python uses boto3 to connect to AWS services.<\/p>\n\n\n\n
pip install boto3<\/pre>\n\n\n\nStep 2: Configure AWS Credentials<\/h1>\n\n\n\n
You need:<\/p>\n\n\n\n
Configure using:<\/p>\n\n\n\n
aws configure<\/pre>\n\n\n\nOr set environment variables.<\/p>\n\n\n\n
Step 3: Connect to S3 Using Python<\/h1>\n\n\n\n
import boto3s3 = boto3.client('s3')<\/pre>\n\n\n\nUpload a File to S3<\/h1>\n\n\n\n
s3.upload_file('local_file.csv', 'my-bucket', 'data\/local_file.csv')<\/pre>\n\n\n\nParameters:<\/p>\n\n\n\n
\n
- Local file path<\/li>\n\n\n\n
- Bucket name<\/li>\n\n\n\n
- S3 object key<\/li>\n<\/ul>\n\n\n\n
Download a File from S3<\/h1>\n\n\n\n
s3.download_file('my-bucket', 'data\/local_file.csv', 'downloaded.csv')<\/pre>\n\n\n\nList Files in a Bucket<\/h1>\n\n\n\n
response = s3.list_objects_v2(Bucket='my-bucket')for obj in response.get('Contents', []):
print(obj['Key'])<\/pre>\n\n\n\nDelete a File<\/h1>\n\n\n\n
s3.delete_object(Bucket='my-bucket', Key='data\/local_file.csv')<\/pre>\n\n\n\nReading CSV Directly from S3 Using Pandas<\/h1>\n\n\n\n
import pandas as pddf = pd.read_csv('s3:\/\/my-bucket\/data\/local_file.csv')<\/pre>\n\n\n\nFor this, you may need additional libraries like s3fs.<\/p>\n\n\n\n
Uploading DataFrame to S3<\/h1>\n\n\n\n
df.to_csv('output.csv', index=False)
s3.upload_file('output.csv', 'my-bucket', 'processed\/output.csv')<\/pre>\n\n\n\nBest Practices<\/h1>\n\n\n\n
\n
- Never hardcode credentials<\/li>\n\n\n\n
- Use IAM roles in production<\/li>\n\n\n\n
- Organize bucket folders properly<\/li>\n\n\n\n
- Enable versioning<\/li>\n\n\n\n
- Set proper access permissions<\/li>\n\n\n\n
- Use lifecycle policies for cost control<\/li>\n<\/ul>\n\n\n\n
Real-World Use Case Example<\/h1>\n\n\n\n
ETL Workflow:<\/p>\n\n\n\n
\n
- Extract API data<\/li>\n\n\n\n
- Save raw data to S3<\/li>\n\n\n\n
- Transform data<\/li>\n\n\n\n
- Store processed data back to S3<\/li>\n\n\n\n
- Load into data warehouse<\/li>\n<\/ol>\n\n\n\n
Interview Answer (Short Version)<\/h1>\n\n\n\n
Using Python with AWS S3 involves using the boto3 library to upload, download, and manage files in S3 buckets. It is commonly used in data engineering workflows for storing raw and processed datasets.<\/p>\n\n\n\n
Final Summary<\/h1>\n\n\n\n
Python with AWS S3 allows you to:<\/p>\n\n\n\n
\n
- Automate file uploads and downloads<\/li>\n\n\n\n
- Build cloud-based data pipelines<\/li>\n\n\n\n
- Store large datasets<\/li>\n\n\n\n
- Create scalable data lake architectures<\/li>\n<\/ul>\n\n\n\n
It is an essential skill for modern cloud-based Data Engineering projects.<\/p>\n\n\n