APIs & Web Scraping

🎯 Learning Objectives

• Consume REST APIs with Python requests
• Parse JSON responses into DataFrames
• Scrape HTML with BeautifulSoup
• Implement rate limiting and ethical scraping practices

📖 Core Content

5.1 Working with REST APIs

python
Run Inline
# runnable
import requests
import pandas as pd
# GET request
response = requests.get('https://api.github.com/repos/scikit-learn/scikit-learn')
if response.status_code == 200:
    data = response.json()
    print(f"Repository: {data['full_name']}")
    print(f"Stars: {data['stargazers_count']}")
    print(f"Language: {data['language']}")
# POST request
payload = {'username': 'test', 'password': 'secret'}
response = requests.post('https://httpbin.org/post', json=payload)
print(f"Response: {response.json()['json']}")

5.2 Web Scraping with BeautifulSoup

python
Run Inline
# runnable
# Note: Requires beautifulsoup4 and requests
# from bs4 import BeautifulSoup
# import requests
#
# url = 'https://example.com/articles'
# response = requests.get(url)
# soup = BeautifulSoup(response.text, 'html.parser')
#
# # Find all article titles
# titles = soup.find_all('h2', class_='article-title')
# for title in titles:
#     print(title.text.strip())

5.3 Ethical Scraping Guidelines

python
Run Inline
import time
import requests
def scrape_with_delay(urls, delay=1):
    """Scrape URLs with delay to be respectful."""
    results = []
    for url in urls:
        response = requests.get(url)
        results.append(response.text)
        time.sleep(delay)  # Be polite!
    return results

📝 Practice Questions

Q1: What is the difference between API scraping and web scraping?

API scraping: Uses structured endpoints (JSON/XML). Faster, reliable, official. Web scraping: Parses HTML. Fragile (site redesign breaks scraper). Slower. Use APIs when available — they're designed for programmatic access. Web scrape only when no API exists. Q2: Why check robots.txt before scraping?

robots.txt tells web crawlers which paths are allowed/disallowed. Ignoring it could: (1) violate the website's terms of service (legal risk), (2) waste resources scraping irrelevant pages, (3) get your IP blocked. Always respect Disallow: rules. Q3: How do you handle pagination when scraping?

Look for patterns: URL query parameters (?page=1, ?page=2), "Next" button links, or infinite scroll. Extract the next page URL from the response, increment and loop, or parse the API response which typically includes next, previous, count fields. Always include a delay between pages. Join Discord PreviousDockerNextMLflow & Experiments