Word Embeddings: Word2Vec, GloVe, and FastText

🎯 Learning Objectives

• Explain distributional semantics ("You shall know a word by the company it keeps")
• Implement Word2Vec (CBOW and Skip-gram) training
• Understand GloVe's matrix factorization approach
• Compare FastText's subword n-grams with word-level embeddings

📋 Prerequisites

• Neural network fundamentals
• Softmax and negative sampling
• SVD / matrix factorization

---

1. 📖 Core Content

1.1 Distributional Semantics

• "He drank wine with dinner"
• "He drank beer with dinner"
• "He drank water with dinner" Wine, beer, and water appear in similar contexts → they're semantically related (beverages).

1.2 Word2Vec

Skip-gram Objective

Negative Sampling

$$P(D=1|w, c) = \sigma(v_w^T v_c) = \frac{1}{1 + \exp(-v_w^T v_c)}$$

python
Run Inline
# runnable
import numpy as np
class SkipGram:
    """Simple Skip-gram with negative sampling"""
    def __init__(self, vocab_size, embedding_dim=50):
        self.V = vocab_size
        self.D = embedding_dim
        np.random.seed(42)
        self.W_in = np.random.randn(vocab_size, embedding_dim) * 0.01  # Input vectors
        self.W_out = np.random.randn(vocab_size, embedding_dim) * 0.01  # Output vectors
    def forward(self, center_idx, context_idx, negative_idxs):
        """
        Forward pass for one training example
        center_idx: index of center word
        context_idx: index of actual context word (positive)
        negative_idxs: indices of negative samples
        """
        v_c = self.W_in[center_idx]  # Center word embedding
        v_pos = self.W_out[context_idx]  # Positive context embedding
        v_neg = self.W_out[negative_idxs]  # Negative context embeddings
        # Positive score
        pos_score = np.dot(v_c, v_pos)
        pos_prob = 1 / (1 + np.exp(-pos_score))
        # Negative scores
        neg_scores = np.dot(v_c, v_neg.T)
        neg_probs = 1 / (1 + np.exp(neg_scores))
        # Loss: -log(pos_prob) - sum(log(1 - neg_prob))
        loss = -np.log(pos_prob + 1e-10) - np.sum(np.log(1 - neg_probs + 1e-10))
        return loss
# Example
model = SkipGram(vocab_size=10000, embedding_dim=50)
center = 42
context = 87
negatives = np.random.randint(0, 10000, 5)
loss = model.forward(center, context, negatives)
print(f"Skip-gram loss: {loss:.4f}")
print(f"Embedding shape: {model.W_in.shape}")

1.3 GloVe

$$J = \sum_{i,j=1}^V f(X_{ij})(w_i^T \tilde{w}_j + b_i + \tilde{b}_j - \log X_{ij})^2$$

• $X_{ij}$: Co-occurrence count of words i and j
• $f(X_{ij})$: Weighting function (clips rare and frequent pairs)
• $w_i, \tilde{w}_j$: Word vectors
• $b_i, \tilde{b}_j$: Bias terms

1.4 FastText

$$v_{w} = \sum_{g \in G_w} v_g$$

1.5 Embedding Properties

python
Run Inline
# runnable
import numpy as np
def cosine_similarity(v1, v2):
    """Cosine similarity between two vectors"""
    return np.dot(v1, v2) / (np.linalg.norm(v1) * np.linalg.norm(v2) + 1e-10)
# Demonstrating analogy property with hypothetical vectors
embeddings = {
    'king': np.array([0.5, 0.3, 0.2]),
    'queen': np.array([0.5, 0.4, 0.1]),
    'man': np.array([0.3, 0.1, 0.4]),
    'woman': np.array([0.3, 0.2, 0.3]),
}
# king - man + woman ≈ queen
analogy = embeddings['king'] - embeddings['man'] + embeddings['woman']
queen_cos = cosine_similarity(analogy, embeddings['queen'])
print(f"Cosine similarity of (king - man + woman) with queen: {queen_cos:.4f}")
print("(Should be relatively high if embeddings capture analogies)")

---

📝 Practice Questions

Q1

<strong>Q1</strong>: Why does Skip-gram work better for rare words while CBOW works better for frequent words?

Skip-gram: Each center word predicts multiple context words. Rare words get more training updates (one center → multiple context pairs). This means rare words are "practiced" more during training.

CBOW: Multiple context words predict the center word. Frequent words appear in many contexts, giving them rich training signal. Rare words (rarely the center) get fewer updates.

This is why Skip-gram is preferred for smaller datasets or when rare words matter, and CBOW is preferred for large datasets with mostly common words. Q2

<strong>Q2

<strong>Q2</strong>: With a vocabulary of 100K and embedding dimension 300, how many parameters does Word2Vec have?

Word2Vec has two embedding matrices:

• Input embeddings: 100K × 300 = 30,000,000
• Output embeddings: 100K × 300 = 30,000,000
• Total: 60 million parameters

This is why negative sampling is essential — full softmax would require computing 100K scores per training example. Q3

<strong>Q3

<strong>Q3

<strong>Q3

<strong>Q3</strong>: FastText can generate embeddings for words not seen during training. How?

FastText represents each word as the sum of its character n-gram embeddings. For a new word, FastText:

1. Decomposes the word into all character n-grams
2. Looks up each n-gram's pre-trained embedding
3. Sums/averages them to create the word embedding

This works because character n-grams capture morphological patterns. An unseen word like "xylophonist" shares n-grams with known words like "xylophone" and "pianist", producing a reasonable embedding.

</details> * * * ## 🔗 Cross-References - **Next**: [Text Classification](/notes/04-degree-electives-bsda5005-nlp-week06-06-text-classification) - **Video**: BSDA5005 Week 5 transcripts [Join Discord](https://discord.gg/gE2m4Qrdqv) [Previous**Named Entity Recognition**](/notes/04-degree-electives-bsda5005-nlp-week04-04-named-entity-recognition)[Next**Text Classification**](/notes/04-degree-electives-bsda5005-nlp-week06-06-text-classification)