Cloud Computing for Data Science
352 words
2 min read
Visual companion
Python
Type and operator map
Python Week 1: the first filter for runtime behavior
View
Revision summary
What this note is really saying
Short form
# Cloud Computing for Data Science ## 🎯 Learning Objectives - Understand cloud service models (IaaS, PaaS, SaaS) - Provision cloud resources for data science workloads - Use cloud storage (S3, GCS, Blob) for data lakes - Deploy ML models on cloud platforms ## 📖 Core Content ### 7.1 Cloud Service Models *(Diagram)*...
Cloud Computing for Data Science
🎯 Learning Objectives
- Understand cloud service models (IaaS, PaaS, SaaS)
- Provision cloud resources for data science workloads
- Use cloud storage (S3, GCS, Blob) for data lakes
- Deploy ML models on cloud platforms
📖 Core Content
7.1 Cloud Service Models
(Diagram)
7.2 Cloud Services for Data Science
| Category | AWS | GCP | Azure |
|---|---|---|---|
| Compute | EC2, SageMaker | Compute Engine, Vertex AI | VMs, Azure ML |
| Storage | S3 | GCS (Cloud Storage) | Blob Storage |
| Serverless | Lambda | Cloud Functions | Functions |
| Data Warehouse | Redshift | BigQuery | Synapse |
| Orchestration | Step Functions | Composer (Airflow) | Data Factory |
| ML Platform | SageMaker | Vertex AI | Azure ML |
7.3 Cloud Storage for Data Science
python# runnable # Note: Requires boto3 (AWS) or google-cloud-storage (GCP) # AWS S3 example: # import boto3 # # s3 = boto3.client('s3') # # # Upload data # s3.upload_file('local_data.csv', 'my-bucket', 'data/raw/data.csv') # # # Download data # s3.download_file('my-bucket', 'data/processed/features.parquet', 'features.parquet') # # # List files # response = s3.list_objects_v2(Bucket='my-bucket', Prefix='data/') # for obj in response['Contents']: # print(obj['Key'])
7.4 Cost Management
| Strategy | Description |
|---|---|
| Reserved instances | Prepay for 1-3 years (up to 70% discount) |
| Spot instances | Use spare capacity (up to 90% discount, interruptible) |
| Auto-scaling | Scale down when not needed |
| Lifecycle policies | Move old data to cheaper storage tiers |
| Monitoring | Set budgets and alerts (don't get surprise bills) |
📝 Practice Questions
Q1: When would you use EC2 vs Lambda vs SageMaker?EC2: Full control, any workload, persistent. Lambda: Short-running (≤15 min), event-driven, no server management. SageMaker: Managed ML lifecycle (training, tuning, deployment). Use SageMaker for ML, Lambda for simple API triggers, EC2 for general compute. Q2: What is object storage and why is it good for data lakes?Object storage (S3, GCS) stores data as objects with metadata and unique IDs. It's: (1) infinitely scalable (exabytes), (2) cheap ($0.023/GB/month), (3) accessible via HTTP APIs, (4) durable (99.999999999% durability). Perfect for storing raw data before processing. Q3: How do you avoid cloud cost surprises?
- Set budget alerts (notify at 50%, 80%, 100% of budget)
- Use auto-scaling to reduce idle resources
- Clean up unused resources (forgotten EC2 instances, old snapshots)
- Use spot instances for non-critical training
- Monitor costs weekly; spikes usually mean a misconfiguration Join Discord PreviousMLflow & ExperimentsNextRegular Expressions