Text Analysis of Social Media Data

🎯 Learning Objectives

• Preprocess social media text (tokenization, stemming, stop word removal)
• Perform sentiment analysis using lexicon-based and ML approaches
• Compute TF-IDF vectors for document comparison
• Apply topic modeling (LDA) to discover themes
• Use word embeddings for semantic analysis

1. Text Preprocessing

1.1 Intuition

1.2 Preprocessing Pipeline

1.3 Step-by-Step Example

pseudo
RT @ml_expert: Machine Learning is AMAZING!!! 🚀 Check out our new paper at https://example.com #AI #MachineLearning

1.4 Python Implementation

python
Run Inline
import re
import nltk
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
from nltk.stem import PorterStemmer
nltk.download('punkt')
nltk.download('stopwords')
def preprocess_tweet(text):
    """Clean and preprocess a single tweet."""
    # Lowercase
    text = text.lower()
    # Remove URLs
    text = re.sub(r'http\S+|www\S+|https\S+', '', text)
    # Remove @mentions
    text = re.sub(r'@\w+', '', text)
    # Keep hashtags but remove # symbol
    text = re.sub(r'#(\w+)', r'\1', text)
    # Remove special chars (keep letters, numbers, spaces)
    text = re.sub(r'[^a-zA-Z0-9\s]', '', text)
    # Tokenize
    tokens = word_tokenize(text)
    # Remove stop words
    stop_words = set(stopwords.words('english'))
    tokens = [t for t in tokens if t not in stop_words]
    # Stem
    stemmer = PorterStemmer()
    tokens = [stemmer.stem(t) for t in tokens]
    return tokens
# Example
tweet = "RT @ml_expert: Machine Learning is AMAZING!!! Check out https://example.com #AI"
print(preprocess_tweet(tweet))
# Output: ['machin', 'learn', 'amaz', 'check', 'ai']

2. Sentiment Analysis

2.1 Intuition

2.2 Lexicon-Based Approach

2.3 Worked Example: VADER Sentiment

python
Run Inline
from nltk.sentiment import SentimentIntensityAnalyzer
nltk.download('vader_lexicon')
sia = SentimentIntensityAnalyzer()
tweets = [
    "I love this new AI feature! It's amazing! 🎉",
    "This update is terrible. Worst experience ever.",
    "The meeting is at 3pm tomorrow.",
    "The product is okay, nothing special."
]
for tweet in tweets:
    scores = sia.polarity_scores(tweet)
    print(f"Tweet: {tweet[:40]}...")
    print(f"  Scores: {scores}")
    print(f"  Overall: {'Positive' if scores['compound'] > 0.05 else 'Negative' if scores['compound'] < -0.05 else 'Neutral'}")

2.4 ML-Based Sentiment

python
Run Inline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
# Training data (labeled tweets)
train_texts = [
    "This product is amazing",
    "I love using this app",
    "Terrible customer service",
    "Worst purchase ever",
]
train_labels = [1, 1, 0, 0]  # 1 = positive, 0 = negative
# Convert text to features
vectorizer = TfidfVectorizer()
X_train = vectorizer.fit_transform(train_texts)
# Train classifier
clf = MultinomialNB()
clf.fit(X_train, train_labels)
# Predict new tweet
new_tweet = ["This app is fantastic!"]
X_new = vectorizer.transform(new_tweet)
prediction = clf.predict(X_new)
print(f"Prediction: {'Positive' if prediction[0] == 1 else 'Negative'}")

3. TF-IDF Vectorization

3.1 Intuition

3.2 Formula

3.3 Worked Example

4. Topic Modeling with LDA

4.1 Intuition

4.2 How LDA Works

4.3 Python Example

python
Run Inline
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation
# Sample tweets
tweets = [
    "Just bought Bitcoin and Ethereum crypto investment",
    "Election results show voter turnout increase",
    "Bitcoin price drops after regulation news",
    "New healthcare bill passes Congress debate",
    "My crypto portfolio is up 50% this month",
    "Voting machines need better security measures"
]
# Vectorize
vectorizer = CountVectorizer(max_features=100, stop_words='english')
X = vectorizer.fit_transform(tweets)
# Run LDA with 2 topics
lda = LatentDirichletAllocation(n_components=2, random_state=42)
lda.fit(X)
# Display topics
feature_names = vectorizer.get_feature_names_out()
for topic_idx, topic in enumerate(lda.components_):
    top_words = [feature_names[i] for i in topic.argsort()[:-6:-1]]
    print(f"Topic {topic_idx}: {', '.join(top_words)}")

pseudo
Topic 0: bitcoin, crypto, ethereum, price, investment
Topic 1: election, voting, healthcare, bill, congress

5. Word Embeddings

5.1 Intuition

5.2 Using Pre-trained Embeddings

python
Run Inline
import gensim.downloader as api
# Load pre-trained GloVe embeddings (trained on Twitter data!)
# This is a 200MB download
glove = api.load("glove-twitter-25")
# Find similar words
similar = glove.most_similar("ai", topn=5)
for word, score in similar:
    print(f"{word}: {score:.3f}")

pseudo
machine: 0.85
intelligence: 0.82
deep: 0.76
learning: 0.74
data: 0.71

5.3 Visualizing Embeddings with PCA

python
Run Inline
import numpy as np
from sklearn.decomposition import PCA
import matplotlib.pyplot as plt
# Get vectors for some words
words = ['king', 'queen', 'man', 'woman', 'prince', 'princess',
         'computer', 'science', 'programming', 'data']
vectors = np.array([glove[w] for w in words])
# Reduce to 2D
pca = PCA(n_components=2)
vectors_2d = pca.fit_transform(vectors)
# Plot
plt.figure(figsize=(10, 8))
for i, word in enumerate(words):
    plt.scatter(vectors_2d[i, 0], vectors_2d[i, 1])
    plt.annotate(word, (vectors_2d[i, 0], vectors_2d[i, 1]))
plt.title("Word Embeddings Visualization")
plt.show()

6. Common Pitfalls

Pitfall 1: Removing All Punctuation (Including Hashtags)

Pitfall 2: Using Stop Words Lists Meant for Formal Text

Pitfall 3: Ignoring Emojis and Emoticons

python
Run Inline
import emoji
def demojize(text):
    return emoji.demojize(text)
tweet = "I love this! 😊🎉"
print(demojize(tweet))
# Output: I love this! :smiling_face_with_smiling_eyes: :party_popper:

7. Key Concepts Reference

8. 📝 Practice Questions

Q1: Tokenize and preprocess: "I CAN'T BELIEVE this!!! Check out http://bit.ly/abc #excited"

Answer: Step 1: Lowercase → "i can't believe this!!! check out http://bit.ly/abc #excited" Step 2: Remove URL → "i can't believe this!!! check out #excited" Step 3: Remove punctuation → "i cant believe this check out excited" Step 4: Tokenize → ['i', 'cant', 'believe', 'this', 'check', 'out', 'excited'] Step 5: Remove stop words → ['cant', 'believe', 'check', 'excited'] Step 6: Stem → ['cant', 'believ', 'check', 'excit'] Q2: Using VADER, a tweet scores compound = 0.0. What does this mean?

Answer: A compound score of 0.0 means the text is either perfectly neutral (equal positive and negative language) or contains no sentiment-bearing words at all. For example, "The meeting starts at 3pm" would score near 0.0. Compound scores between -0.05 and +0.05 are typically classified as neutral. Q3: A term appears in 5 out of 100 documents. What is its IDF?

Answer: IDF = log(N/df) = log(100/5) = log(20) = 2.996 (using base e) Or using base 10: log10(20) = 1.301 Or using base 2: log2(20) = 4.322 Most implementations use natural log or base 10. The exact value depends on the implementation, but the key point is that rare terms get higher IDF scores. Q4: Explain why TF-IDF is better than raw word counts for document comparison.

Answer: Raw word counts favor frequent words (like "the", "and", "is") regardless of their importance. TF-IDF downweights common words (high document frequency → low IDF) and upweights rare words that distinguish documents. For example, "blockchain" in a finance document would get high TF-IDF, while "the" would get near-zero. This makes TF-IDF much better at capturing document similarity based on meaningful content rather than function words. Q5: With K=3 topics in LDA, a tweet has topic distribution [0.7, 0.2, 0.1]. Interpret this.

Answer: The tweet is composed of approximately 70% Topic 1, 20% Topic 2, and 10% Topic 3. Since Topic 1 dominates, the tweet is most strongly associated with Topic 1. The mixture reflects how LDA models documents — they don't belong to a single topic but are mixtures of all topics. A threshold of 0.5 would classify this as Topic 1. Q6: Why might stemming reduce "universe" and "university" to similar roots?

Answer: The Porter stemmer reduces both to "univers" because it applies rules like removing common suffixes without considering meaning. This is a known limitation — words with different meanings but similar spelling get conflated. Lemmatization would handle this better by using dictionary lookups: "universe" → "universe", "university" → "university". Q7: In word embeddings, vector("Paris") - vector("France") + vector("Italy") ≈ ?

Answer: This should approximate vector("Rome"). The analogy is: Paris is to France as Rome is to Italy. The vector arithmetic computes: capital(Paris) - country(France) + country(Italy) = capital(Rome). This is the classic analogy-solving property of word embeddings — the vector space captures relational semantics. Q8: A sentiment classifier trained on movie reviews performs poorly on tweets. Why?

Answer: Domain mismatch: (1) Movie reviews are longer, more formal, and grammatically correct. Tweets are short, informal, full of slang/abbreviations. (2) Movie reviews use different vocabulary (plot, character, director vs. hashtags, @mentions, emojis). (3) Tweet sentiment is conveyed differently (caps, repeated punctuation, emojis). This is why domain-specific training data matters — a model trained on one domain doesn't transfer well. Q9: Calculate the TF for "crypto" in a 50-word tweet where "crypto" appears 3 times.

Answer: TF(crypto) = count of "crypto" / total terms = 3/50 = 0.06 This means 6% of the words in the tweet are "crypto" — a relatively high concentration indicating the tweet is likely about cryptocurrency. Q10: Why might you use word embeddings (like GloVe) instead of TF-IDF vectors?

Answer: (1) Embeddings capture semantic relationships (synonyms, analogies) while TF-IDF treats each word independently. (2) Embeddings are dense (300 dimensions) vs. TF-IDF being sparse (vocabulary-sized, often 10,000+). (3) Embeddings capture context — "bank" near "river" vs. "bank" near "money" would have different meanings, but TF-IDF treats them as the same term. (4) Embeddings are pre-trained on massive data and transfer well to new tasks.

9. 🔗 Cross-References

• Week 2 - Data Collection: Raw data comes from APIs
• Week 6 - Fake News: Text analysis for misinformation detection
• Week 8 - Privacy: Text anonymization techniques
• BSCS3031 (CSD): Sentiment analysis applications Join Discord PreviousData CollectionNextWeb Tracking