Data Collection from Social Media APIs

🎯 Learning Objectives

• Understand how REST APIs work for social media data collection
• Implement OAuth authentication for accessing platform data
• Handle rate limits, pagination, and data formatting
• Evaluate ethical and legal considerations of data collection
• Apply sampling strategies for social media data

1. Social Media APIs: Overview

1.1 Intuition

1.2 What is a REST API?

1.3 Common Social Media APIs

2. Authentication: OAuth 2.0

2.1 Intuition

2.2 OAuth Flow

2.3 Token Types

2.4 Python Example: Twitter API v2

python
Run Inline
import requests
import json
# Your credentials from developer portal
bearer_token = "AAAAAAAAAAAAAAAAAAAA..."
def create_headers(bearer_token):
    return {"Authorization": f"Bearer {bearer_token}"}
def connect_to_endpoint(url, headers):
    response = requests.get(url, headers=headers)
    if response.status_code != 200:
        raise Exception(f"Request returned {response.status_code}: {response.text}")
    return response.json()
# Search for recent tweets about "machine learning"
url = "https://api.twitter.com/2/tweets/search/recent"
params = {
    "query": "machine learning",
    "max_results": 10,
    "tweet.fields": "created_at,public_metrics"
}
headers = create_headers(bearer_token)
response = requests.get(url, headers=headers, params=params)
data = response.json()
for tweet in data['data']:
    print(f"{tweet['created_at']}: {tweet['text']}")

pseudo
2024-01-15T10:30:00Z: Just finished a great course on machine learning!
2024-01-15T10:28:00Z: Machine learning is transforming healthcare diagnostics...

3. Rate Limiting & Pagination

3.1 Rate Limiting

pseudo
x-rate-limit-limit: 300       # Max requests allowed
x-rate-limit-remaining: 284    # Requests remaining in window
x-rate-limit-reset: 1643123456 # Unix timestamp when window resets

3.2 Handling Rate Limits

python
Run Inline
import time
import requests
def rate_limited_request(url, headers, params, max_retries=3):
    for attempt in range(max_retries):
        response = requests.get(url, headers=headers, params=params)
        if response.status_code == 429:  # Too Many Requests
            reset_time = int(response.headers.get('x-rate-limit-reset', 0))
            wait_time = max(reset_time - time.time(), 0) + 1
            print(f"Rate limited. Waiting {wait_time} seconds...")
            time.sleep(wait_time)
            continue
        if response.status_code == 200:
            remaining = response.headers.get('x-rate-limit-remaining', '?')
            print(f"Requests remaining: {remaining}")
            return response.json()
        print(f"Error {response.status_code}: {response.text}")
        return None
    print("Max retries exceeded")
    return None

3.3 Pagination

python
Run Inline
def collect_all_tweets(query, max_tweets=500):
    all_tweets = []
    next_token = None
    while len(all_tweets) < max_tweets:
        params = {
            "query": query,
            "max_results": 100,
            "tweet.fields": "created_at"
        }
        if next_token:
            params["next_token"] = next_token
        response = requests.get(url, headers=headers, params=params)
        data = response.json()
        if 'data' in data:
            all_tweets.extend(data['data'])
            print(f"Collected {len(all_tweets)} tweets so far")
        # Check for next page
        if 'meta' in data and 'next_token' in data['meta']:
            next_token = data['meta']['next_token']
        else:
            break
    return all_tweets[:max_tweets]

4. Data Formats & Processing

4.1 JSON Structure

json
{
  "data": {
    "id": "1234567890",
    "text": "Machine learning is amazing! #AI",
    "created_at": "2024-01-15T10:30:00.000Z",
    "author_id": "98765",
    "public_metrics": {
      "retweet_count": 15,
      "reply_count": 3,
      "like_count": 142,
      "quote_count": 2
    },
    "entities": {
      "hashtags": [{"tag": "AI"}],
      "mentions": [{"username": "ml_expert"}]
    }
  },
  "includes": {
    "users": [
      {
        "id": "98765",
        "name": "ML Enthusiast",
        "username": "ml_fan",
        "public_metrics": {
          "followers_count": 2500,
          "following_count": 180,
          "tweet_count": 3400
        }
      }
    ]
  }
}

4.2 Converting to Structured Data

python
Run Inline
import pandas as pd
def tweets_to_dataframe(tweets_data):
    """Convert raw API response to a pandas DataFrame."""
    rows = []
    for tweet in tweets_data.get('data', []):
        row = {
            'tweet_id': tweet['id'],
            'text': tweet['text'],
            'created_at': tweet['created_at'],
            'retweets': tweet['public_metrics']['retweet_count'],
            'likes': tweet['public_metrics']['like_count'],
            'replies': tweet['public_metrics']['reply_count'],
        }
        rows.append(row)
    df = pd.DataFrame(rows)
    df['created_at'] = pd.to_datetime(df['created_at'])
    return df

5. Sampling Strategies

5.1 Why Sample?

5.2 Sampling Methods

5.3 Worked Example: Stratified Sampling

python
Run Inline
countries = ['US', 'UK', 'IN', 'BR', 'JP']
tweets_per_country = 2000
collected = {}
for country in countries:
    query = f"politics lang:en place_country:{country}"
    collected[country] = collect_all_tweets(query, tweets_per_country)
    print(f"Collected {len(collected[country])} tweets from {country}")

6. Ethical & Legal Considerations

6.1 Platform Terms of Service

6.2 GDPR & Data Privacy

6.3 Ethical Guidelines

1. Minimize harm: Don't expose sensitive information
2. Be transparent: Disclose data collection methods
3. Respect privacy: Anonymize, aggregate where possible
4. Get consent: For non-public data
5. Follow platform rules: Terms of service matter

7. Common Pitfalls

Pitfall 1: Ignoring Rate Limits

Pitfall 2: Storing API Keys in Code

python
Run Inline
import os
from dotenv import load_dotenv
load_dotenv()
bearer_token = os.getenv("TWITTER_BEARER_TOKEN")

Pitfall 3: Not Handling Pagination

8. Key Concepts Reference

9. 📝 Practice Questions

Q1: You make 300 API requests in 10 minutes. Rate limit is 300 per 15 minutes. How long must you wait before the next request?

Answer: The window resets 15 minutes after the first request in the window. Since you made 300 requests in 10 minutes, you've exhausted your quota. You need to wait until the 15-minute window from your first request expires. If your first request was at t=0, you can request again at t=15 minutes (5 more minutes of waiting). Q2: What's the difference between a bearer token and a refresh token?

Answer: A bearer token is short-lived (1-2 hours) and is sent with each API request to authenticate. A refresh token is long-lived (days to months) and is used to obtain new bearer tokens without requiring the user to re-authenticate. Refresh tokens are stored securely on the server side and never sent with API requests. Q3: An API returns paginated results with next_token. The first page contains 100 results. How many API calls are needed to collect 500 results?

Answer: Each call returns up to 100 results (max_results=100). To get 500 results: 500/100 = 5 calls. Call 1 gets page 1 with next_token; call 2 uses that token for page 2; and so on until call 5 completes. Total: 5 API calls. Q4: Why should you use environment variables instead of hardcoding API keys?

Answer: Hardcoding API keys in source code exposes them to anyone with access to the codebase. If committed to version control, they become permanently visible. Environment variables keep secrets out of code, can be set per-deployment (dev/staging/production), and reduce the risk of accidental exposure. Q5: A platform returns HTTP 429 after 50 requests. Your rate limit was supposed to be 300. What might be wrong?

Answer: Several possibilities: (1) The rate limit might be per-endpoint, not global — you might be hitting a different endpoint than you registered. (2) Your app might be in a lower tier (e.g., Essential vs. Academic). (3) The rate limit might reset on a different schedule than you assumed (e.g., per minute, not per 15 minutes). (4) You might have accidentally triggered a separate rate limit (e.g., per-user instead of per-app). Check the response headers for x-rate-limit-limit to see the actual limit. Q6: What is the difference between cursor-based and offset-based pagination?

Answer: Cursor-based pagination uses a token pointing to a specific item, guaranteeing no duplicates even if new data is added (stable cursor). Offset-based pagination skips N items, which can cause duplicates or missed items if data changes between pages. Cursor-based is preferred for real-time data (like social media feeds), while offset-based is simpler for static datasets. Q7: Under GDPR, what must you do if a user requests deletion of their data from your research dataset?

Answer: You must (1) identify all records containing their data across your systems, (2) delete those records or anonymize them so the user is no longer identifiable, (3) confirm deletion to the user within the required timeframe (usually 30 days), (4) ensure no backups retain the identifiable data. You should have a process for handling such requests before starting data collection. Q8: You're collecting tweets about a sensitive health topic. What ethical considerations apply?

Answer: (1) Anonymize user handles and IDs before analysis/storage. (2) Consider whether users expected their tweets to be public for this purpose. (3) Aggregate findings rather than quoting individual tweets. (4) Store data securely with access controls. (5) Have an IRB/ethics review if possible. (6) Be prepared for potential media attention or platform scrutiny. Q9: An API returns JSON with data and includes fields. What are includes for?

Answer: The includes field contains expanded objects referenced in the data field. For example, a tweet in data might have author_id: "123", and the actual user object with name, username, and metrics is in includes.users. This is called "side-loading" or "embedding" and reduces the number of API calls needed to get related data. It's more efficient than making separate calls for each user. Q10: Compare snowball sampling with stratified sampling for social media research.

Answer: Snowball sampling starts with initial "seed" users and follows their connections/mentions, making it ideal for network analysis and hard-to-reach populations. Stratified sampling divides the population into relevant categories (by country, language, etc.) and samples each proportionally, making it better for comparative studies. Snowball can introduce selection bias (connected users are similar), while stratified requires knowing the categories in advance.

10. 🔗 Cross-References

• Week 1 - SNA Basics: Graph metrics from collected data
• Week 3 - Text Analysis: Processing collected text data
• Week 4 - Cyber Crime: API misuse patterns
• BSCS4024 (Computer Networks): HTTP, REST API fundamentals Join Discord PreviousSNA BasicsNextText Analysis