Feature Stores: Feast, Feature Engineering, and Online/Offline Serving

🎯 Learning Objectives

• Understand the problems feature stores solve in ML production
• Design feature pipelines for batch and streaming data
• Implement point-in-time correct feature retrieval
• Use Feast for feature management, serving, and discovery
• Compare feature store solutions and choose the right one

📋 Prerequisites

• MLOps Lifecycle (Week 1): ML pipeline stages
• Data Engineering basics: Batch processing, ETL, SQL
• Docker basics (recommended): Container concepts

1. 📖 Core Content

1.1 Intuition: The Feature Problem

1. Inconsistency: Training features differ from serving features (training uses pandas, serving uses SQL)
2. Duplication: Every team builds the same features (average transaction amount)
3. Time travel: Training uses features from the past; serving uses real-time features. If not handled correctly, feature distributions differ.
4. Discovery: Data scientists don't know what features exist — they rebuild them
5. Serving latency: Batch features (daily aggregates) need different infrastructure than real-time features (seconds-old data) A feature store solves all of these by providing a centralized repository of features with consistent computation and serving.

1.2 Feature Store Architecture

• Offline Store: Column-oriented database (BigQuery, Redshift, Parquet) for batch feature computation and training data extraction. Stores historical feature values.
• Online Store: Low-latency database (Redis, DynamoDB, Cassandra) for serving real-time features. Stores the latest feature values.
• Feature Registry: Metadata catalog of all features, their sources, transformations, and owners.
• Feature Pipelines: Batch (Spark, Beam) and streaming (Kafka, Flink) pipelines that compute features from raw data.

1.3 Point-in-Time Correctness

sql
SELECT
  o.order_id,
  AVG(o_hist.amount) as avg_order_amount  -- includes FUTURE orders!
FROM orders o
JOIN orders o_hist ON o.user_id = o_hist.user_id
GROUP BY o.order_id

sql
SELECT
  o.order_id,
  AVG(o_hist.amount) as avg_order_amount
FROM orders o
JOIN orders o_hist
  ON o.user_id = o_hist.user_id
  AND o_hist.order_time < o.order_time  -- only past orders!
GROUP BY o.order_id

Worked Example 1: Point-in-Time Feature Computation

• Jan 1: $100
• Feb 1: $50
• Mar 1: $200 Training model to predict whether a user will order next month.

1.4 Feast: The Leading Open-Source Feature Store

1.4.1 Key Concepts

1.4.2 Feast Workflow

python
Run Inline
# 1. Define features (feature_view.py)
from feast import FeatureView, Field
from feast.types import Float32, Int64
from feast.value_type import ValueType
user_stats = FeatureView(
    name="user_transaction_stats",
    entities=["user_id"],
    ttl=timedelta(days=7),
    schema=[
        Field(name="avg_7d_amount", dtype=Float32),
        Field(name="num_7d_transactions", dtype=Int64),
    ],
    source=bigquery_source,
)
# 2. Apply to registry
# feast apply
# 3. Get training data
feature_service = FeatureService(
    name="fraud_detection_features",
    features=[user_stats]
)
training_df = fs.get_historical_features(
    entity_df=entity_df,  # user_ids + timestamps
    features=feature_service
).to_df()
# 4. Serve online features
feature_vector = fs.get_online_features(
    features=feature_service,
    entity_rows=[{"user_id": 123}]
).to_dict()

1.5 Feature Pipelines: Batch vs Streaming

1.5.1 Batch Features

python
Run Inline
# Daily batch feature computation (Spark)
def compute_user_features(spark, date):
    df = spark.table(f"orders")
    features = (df
        .filter(f"order_date < '{date}'")
        .groupBy("user_id")
        .agg(
            avg("amount").alias("avg_order_amount"),
            count("*").alias("num_orders"),
            avg("days_since_last_order").alias("avg_recency")
        )
    )
    features.write.mode("overwrite").save(f"features/user_features/dt={date}")

1.5.2 Streaming Features

python
Run Inline
# Streaming feature computation (Flink/Kafka)
from kafka import KafkaConsumer
import redis
consumer = KafkaConsumer('orders', bootstrap_servers='kafka:9092')
r = redis.Redis()
for msg in consumer:
    order = json.loads(msg.value)
    user_id = order['user_id']
    # Update running statistics
    r.hincrby(f"user:{user_id}", "txn_count", 1)
    r.hincrbyfloat(f"user:{user_id}", "amount_sum", order['amount'])

1.6 Feature Store Comparison

1.7 Edge Cases & Gotchas

• TTL Management: Old feature values must be expired. Set appropriate TTLs to avoid stale features.
• Feature Crossing: Some features are only useful when combined (user_age × product_category). Pre-compute crosses.
• Null Handling: New entities have no feature values. Decide on imputation strategy (zeros, mean, median).
• Feature Drift: Monitor feature distributions over time. A sudden shift can degrade models silently.
• Backfilling: When adding new features, you need to compute historical values for all past training dates. This is computationally expensive.

1.8 Why This Matters

• Uber (Michelangelo): Features as a service for hundreds of models
• Netflix: Feature store for recommendation and personalization
• Airbnb: Feature engineering platform for fraud, pricing, search Without a feature store, ML teams spend ~60% of their time on feature engineering and face chronic training-serving skew. A feature store cuts this dramatically.

2. 📐 Key Formulas / Concepts

3. ⚠️ Common Pitfalls

Pitfall 1: Not Handling Point-in-Time Correctness

Pitfall 2: Using the Same Features for Training and Serving Without Validation

Pitfall 3: Ignoring Feature TTL

4. 📝 Practice Questions

Q1: A fraud model uses "num_transactions_last_24h" as a feature. How would you compute this for training (with timestamps from 2023) vs. serving (real-time)?

For training (offline):

sql
SELECT
  t.transaction_id,
  COUNT(past.transaction_id) as num_transactions_last_24h
FROM transactions t
LEFT JOIN transactions past
  ON t.user_id = past.user_id
  AND past.timestamp BETWEEN t.timestamp - INTERVAL 24 HOUR AND t.timestamp
  AND past.transaction_id != t.transaction_id

For serving (online):

• Use a Redis counter that increments on each transaction
• Key: user:{user_id}:txn_24h
• TTL: 24 hours (auto-expires)
• On prediction: retrieve the counter value

The feature store handles both paths from a single feature definition. Q2: Explain the difference between a feature store and a data warehouse. When would you use each?

Data Warehouse: Stores raw data optimized for analytics queries. Contains all data at the most granular level. Queried by analysts and BI tools.

Feature Store: Stores processed features optimized for ML. Contains pre-computed aggregations. Queried by models (online) and training jobs (offline).

Use a data warehouse as the source for feature computation; use a feature store for ML-serving. Q3: Your team has 50 features used across 10 models. Currently, each team computes features independently. Calculate the time savings from adopting a feature store (assuming 60% of ML time is feature engineering).

Without feature store: 10 models × 100% feature engineering time = 10 person-units.

With feature store:

• Create shared feature registry: 2 units (one-time setup)
• Maintain shared pipelines: 1 unit/cycle
• Each model just uses features: 0 units for feature engineering

Feature engineering collaboration: 50 features built once, shared 10 ways → 50/10 = 5 units saved.

Total time savings: ~70% reduction in feature engineering time. This translates to about 42% overall ML project time savings (70% of 60%). Q4: A new feature "user_7d_avg_discount" needs backfilling for 2 years of training data. What approaches can speed this up?

1. Partitioned computation: Compute the feature day-by-day using incremental updates:

   python
   Run Inline
   for day in range(-730, 0):
       new_avg = (prev_avg * (count-1) + today_discount) / count
   

2. Window functions in SQL: Use ROWS BETWEEN 6 PRECEDING AND CURRENT ROW to compute rolling windows efficiently.
3. Incremental materialization: If the computation is additive (sum, count), maintain running totals and update rather than recompute from scratch.
4. Parallelism: Partition by entity (user_id hash) and compute in parallel.
5. Dimensional modeling: Pre-compute daily aggregates, then roll up to weekly:

   sql
   SELECT user_id, date,
          AVG(avg_daily_discount) OVER (
              PARTITION BY user_id
              ORDER BY date ROWS 6 PRECEDING
          ) as avg_7d_discount
   FROM daily_discounts
   

The fastest approach depends on data volume. For < 1B rows, optimized SQL is sufficient. For > 1B rows, use Spark with incremental updates.

5. 🔗 Cross-References

• Previous: Data Versioning (Week 3) — Data pipeline foundations
• Next: Containers & Docker (Week 5) — Deploying feature pipelines
• Related: Experiment Tracking (Week 2) — Logging feature engineering experiments
• External: Feast documentation (feast.dev) — Practical implementation guide Join Discord PreviousData VersioningNextContainers & Docker