Big Data: Hadoop & Spark

🎯 Learning Objectives

• Explain the Hadoop ecosystem components
• Understand HDFS architecture and data replication
• Compare MapReduce and Spark processing models
• Use Spark DataFrames for distributed data processing

📖 Core Content

4.1 The Hadoop Ecosystem

4.2 HDFS Architecture

• Default block size: 128 MB
• Replication factor: 3 (default)
• Rack awareness: Replicates across racks

4.3 MapReduce vs Spark

4.4 Spark DataFrame API

python
Run Inline
# runnable
# Note: Requires Spark installation - this is reference code
# from pyspark.sql import SparkSession
# from pyspark.sql.functions import col, avg, count
# spark = SparkSession.builder.appName("analysis").getOrCreate()
# df = spark.read.parquet("sales_data.parquet")
#
# result = df.groupBy("category").agg(
#     avg("revenue").alias("avg_revenue"),
#     count("*").alias("transaction_count")
# ).filter(col("avg_revenue") > 1000)
#
# result.show()

📝 Practice Questions

Q1: Why does HDFS use 128 MB blocks instead of smaller sizes?

Large block sizes minimize seek time overhead (seek ~10ms per block). A 128 MB block takes ~1 second to read at 128 MB/s — only 1% overhead from seek. Smaller blocks would waste more time seeking. Also reduces metadata storage on NameNode. Q2: Why is Spark faster than MapReduce?

MapReduce writes intermediate results to disk (HDFS) between map and reduce stages. Spark keeps data in memory across operations. For iterative algorithms (ML, graph processing), this eliminates disk I/O bottlenecks. Spark also has a more efficient DAG execution engine vs MapReduce's rigid 2-stage pipeline. Q3: What is a Data Lake versus a Data Warehouse in the Big Data context?

Data Lake (Hadoop-based): Stores raw data in native format (CSV, JSON, Parquet). Schema-on-read. Used for exploratory analysis, data science, and machine learning. Data Warehouse: Stores processed, structured data. Schema-on-write. Used for BI reporting and dashboards. Many organizations use both: Data Lake stores everything, Data Warehouse stores the refined subset. Join Discord PreviousOLAP & CubesNextNoSQL Databases