Transformers

The Big Picture

Before 2017, RNNs (LSTMs/GRUs) dominated sequence modeling. They process sequences one element at a time, which is:

• Sequential: Can't parallelize across time steps (bottleneck for training)
• Long-range memory: Information decays over long distances (vanishing gradients)

The 2017 paper "Attention Is All You Need" (Vaswani et al.) introduced the Transformer, which:

• Parallelizes completely: Process all tokens simultaneously (O(1) path length)
• Captures long-range dependencies: Direct connections between all pairs of tokens
• Enables scaling: Trains on billions of parameters (GPT-3: 175B params)

The key innovation: Self-Attention — a mechanism where each token dynamically learns to "pay attention" to relevant context.

Transformer Applications

Today, transformers power:

• Language Models: GPT-4, Claude, Gemini (next-token prediction)
• Classification: BERT, RoBERTa (text classification, sentiment analysis)
• Machine Translation: seq2seq transformers (source language → target language)
• Vision: Vision Transformers (ViT) for image classification
• Multimodal: CLIP, GPT-4V (image + text understanding)
• AdTech: User behavior prediction, bid optimization, ad ranking

AdTech Application: User Journey Modeling

In advertising, understanding user intent from their interaction sequence is critical for predicting conversion likelihood.

The Problem

A user's journey looks like:

• Search query: "running shoes"
• Browsed product page
• Clicked on competitor ad
• Returned to site, added item to cart
• Viewed checkout page

Which interaction matters most for predicting purchase? Do recent actions matter more? Is the search query still relevant after 3 clicks?

Transformer Approach

Self-attention solves this automatically. Instead of hand-crafting feature importance, the transformer learns:

Concrete Example: Given the sequence [search_query, product_page, competitor_click, cart_add, checkout], a transformer computes attention weights:

The model learns that the search query is highly predictive of conversion (0.45 weight). Without explicit coding, the transformer discovered this relationship.

Self-Attention Intuition

Let's start with a concrete example. Consider the sentence:

The pronoun "it" refers to the ad. How does a neural network know this?

The Query-Key-Value Insight

Self-attention works via three operations:

Step 1: The Token Sends Out a Query

When the model processes "it", this token asks: "What am I referring to? What should I attend to?"

This question is encoded as a Query vector (learned representation of the token's context need).

Step 2: All Other Tokens Advertise Their Identity

Each token in the sequence broadcasts a Key vector: "Here's what I represent. Here are my features."

The Key for "ad" might emphasize: nouns, objects, previous mentions.

The Key for "because" might emphasize: conjunctions, reasons.

Step 3: Compute Relevance via Similarity

The model computes similarity between the Query of "it" and the Keys of all tokens:

These similarities become attention weights (after softmax normalization).

Step 4: Retrieve Information via Values

Each token also has a Value vector: "If someone attends to me, here's the information I pass along."

The "it" token computes a weighted sum of all Values, weighted by attention:

Since "ad" has the highest attention weight (0.92), its information dominates the output.

Visual: Attention Weights in Action

The "it" token assigns 62% of its attention to "ad" — the model has learned to resolve the pronoun reference.

Q, K, V Matrices: The Three Learned Transformations

Now let's formalize self-attention. Each input token embedding x_i gets transformed into three vectors via learned weight matrices:

Intuition: The Library Lookup

Imagine searching a library:

Concrete Example with Numbers

Let's say we have 4 words with embeddings (d_model = 6):

Why Three Separate Matrices?

You might ask: why not just one transformation? The answer is that Q, K, and V serve different purposes:

• Q and K are for matching: We compute similarity Q·K^T to determine which tokens are relevant to each other
• V is for content: The actual information carried by each token
• Different representations: A word might be a good "query for action verbs" (Q direction) but a "provider of emotional context" (V direction). Separate matrices allow flexible role assignment.

In transformer jargon:

• Query and Key are typically smaller: d_k = d_model / num_heads (e.g., 512 / 8 = 64)
• Value is the same size: d_v = d_model / num_heads
• W_Q, W_K, W_V are learned during training via backpropagation

Scaled Dot-Product Attention

Now we have Query, Key, and Value vectors. The attention mechanism combines them via a four-step formula:

Step-by-Step Breakdown

Step 1: Compute Similarity Scores (QK^T)

Multiply queries by transpose of keys to compute pairwise similarities:

Result: A matrix where scores[i, j] = how much does position i (Query) match position j (Key)?

Step 2: Scale by √d_k

Divide by the square root of key dimension:

Why scale? As d_k increases, dot products grow larger. Large dot products push softmax into extreme regions (e.g., [0.999, 0.0001]), which have vanishing gradients. Scaling stabilizes training.

Example: If d_k = 64, we divide by √64 = 8.

Step 3: Apply Softmax to Get Attention Weights

Convert scores to probabilities (sum to 1):

Now weights[i, j] is between 0 and 1, and weights[i, :] sums to 1 (a proper probability distribution).

Step 4: Weighted Sum of Values

Multiply attention weights by Values:

This is a weighted combination of all value vectors, weighted by attention.

Numerical Walkthrough

Let's trace through with a toy example. 4 tokens, d_k = 2:

Query and Key matrices (simplified):

Step 1: QK^T

Step 2: Scale by √d_k = √2 ≈ 1.41

Step 3: Softmax (for first row)

Step 4: Weighted sum of V (for token 1)

Token 1's output is a blend of all values, with slightly more weight on positions 1 and 2.

Gradient Flow Benefit

Unlike RNNs with sequential multiplication of gradients (causing vanishing/exploding), attention has direct paths. Gradients flow directly from output to any input position. This is why transformers train faster and to better quality than RNNs.

Multi-Head Attention

One attention mechanism is good. Eight (or 12, or 16) in parallel is better. Multi-head attention allows the model to attend to different types of relationships simultaneously.

The Motivation

Different types of dependencies benefit from different attention patterns:

• Head 1: "Where are the nouns? Focus on syntactic roles."
• Head 2: "Which tokens are semantically related? Focus on word embeddings."
• Head 3: "Long-range dependencies. Focus on sentence structure."
• Head 4-8: Other specialized patterns.

A single attention head must compress all these patterns into one 64-dimensional representation. Multiple heads allow specialization.

The Mechanism

Multi-head attention uses h parallel attention mechanisms (typically h=8 or h=12):

Each head gets its own Q, K, V projections:

Each head operates on d_k = d_model / h dimensions (e.g., 512 / 8 = 64).

Concatenate all heads:

Visual: Multi-Head Architecture

Why This Works

• Specialization: Each head learns to focus on different linguistic phenomena
• Robustness: If one head fails to learn, others can compensate
• Interpretability: We can visualize each head's attention pattern separately
• Efficiency: Parallel computation (8 heads of 64-dim is faster than 1 head of 512-dim on GPUs)

Positional Encoding

Here's a critical insight: self-attention is permutation-invariant.

If you rearrange the input tokens, the attention weights change, but the mechanism itself treats all positions equally. The model has no inherent sense of "order" or "sequence position."

The Problem

Consider two sentences:

Word for word, they're identical. But the meaning is opposite because of word order. A transformer without positional information would treat them identically.

The Solution: Sinusoidal Positional Encoding

Before feeding tokens to the transformer, we add a positional encoding PE(pos, i) to each embedding:

Intuition

Different dimensions oscillate at different frequencies:

• Low frequencies (i=0): Oscillate slowly. Encode absolute position.
• High frequencies (i=255): Oscillate rapidly. Encode fine-grained relative position.

Key property: PE(pos+k) is a linear function of PE(pos). This allows the model to learn relative positions.

Numerical Example

d_model = 4, positions 0-3:

Notice: Dimension 0 (sin with period 2π): rapid oscillation. Dimension 2 (sin with longer period): slow oscillation.

Visual: Positional Encoding Heatmap

The heatmap shows how different frequency components encode position. The model learns to use these sinusoidal signals for understanding token order.

Full Transformer Architecture

Now let's assemble all pieces: self-attention, multi-head, positional encoding into the complete transformer stack.

Encoder-Decoder Structure

A transformer has two stacks:

• Encoder: Processes input (reads the entire source)
• Decoder: Generates output one token at a time (language model fashion)

Encoder Stack

The encoder repeats N times (typically N=6):

Feed-Forward Network (FFN):

Residual Connections: Each sub-layer computes: output = Layer(x) + x. These skip connections help gradients flow during backprop (same principle as ResNets).

Layer Normalization: Normalize activations to zero mean and unit variance. Stabilizes training.

Decoder Stack

The decoder also repeats N times, but with an extra attention mechanism:

Masked Self-Attention: When decoding token t, we can only attend to tokens 0 to t-1. This prevents "cheating" (looking at future tokens we're trying to predict). We enforce this by setting attention scores to -∞ for future positions before softmax.

Cross-Attention: Queries come from the decoder. Keys and Values come from the encoder. This allows each decoder token to attend to all encoder tokens (the full input).

Architecture Diagram

Key Design Choices

• Residual Connections: Help gradient flow (like ResNets in vision)
• Layer Norm: Applied after each sub-layer. Prevents activation explosions.
• FFN Expansion: Inner dimension is 4× model dimension. Provides "thinking space" for non-linear transformations.
• Dropout: Applied in attention and FFN to regularize

Training & Loss Functions

How do we train a transformer? The objective depends on the task.

Language Modeling (Next-Token Prediction)

For GPT-style models, the task is: given tokens [t_1, t_2, ..., t_n], predict the probability distribution over the next token t_{n+1}.

Loss Function: Cross-Entropy

Derivation: The transformer outputs logits (raw scores) from the final linear layer. We apply softmax to get probabilities. Cross-entropy loss compares this distribution to the ground truth (one-hot label for the correct next token).

Concrete example:

• Model outputs: [0.1, 0.6, 0.2, 0.1] for 4 tokens
• Ground truth: token 2 (one-hot: [0, 1, 0, 0])
• Cross-entropy: -log(0.6) ≈ 0.51

Teacher Forcing

During training, we feed the model the ground-truth tokens as input (not its own predictions). This is called "teacher forcing."

Why? Decoding one token at a time is slow. Teacher forcing lets us train in parallel: all positions are fed simultaneously with correct context.

Drawback: At inference time, the model must use its own predictions. This causes a "distribution shift" (exposure bias). The model hasn't seen its own mistakes during training.

Label Smoothing

Instead of a hard one-hot target, we soften it:

Label smoothing prevents overconfidence and regularizes the model.

Learning Rate Schedule

Transformers use a specific learning rate schedule:

Warm-up phase: Gradually increase learning rate for the first warmup_steps (e.g., 4000 steps).

Decay phase: Then decay proportional to 1/√step.

This schedule is crucial for stable transformer training.

Gradient Clipping

Transformers can suffer from gradient explosion. We clip gradients to a max norm:

Optimization: Adam or SGD + Momentum

Most transformers use Adam with β1=0.9, β2=0.98, ε=10^-9.

Transformer vs RNN: A Detailed Comparison

Why did transformers replace RNNs? Let's compare systematically.

The Key Win: O(1) Path Length

In RNNs, to propagate information from token 1 to token 1000, you must:

Each arrow is a matrix multiplication (potential for vanishing gradients).

In transformers, token 1000 can directly attend to token 1:

This is a direct computation in one step (O(1) path length).

The Trade-off: Quadratic Memory

Transformers have O(n²) memory for attention weights (n tokens × n attention heads). This limits sequence length.

RNNs have O(1) memory independent of sequence length.

Recent solutions: Linear attention (ALiBi), sparse attention (Longformer), efficient attention (FlashAttention) reduce memory.

When Do RNNs Win?

• Very long sequences: RNNs can handle arbitrary length (memory O(d) always)
• Streaming/online inference: RNN inference is O(1) per token (fixed hidden state size)
• Reinforcement learning: RNNs integrate naturally with recurrent policy networks

Python Implementation: From Scratch

Let's build core transformer components in NumPy (no frameworks). This is for understanding; production uses PyTorch/TensorFlow.

Scaled Dot-Product Attention

import numpy as np

def scaled_dot_product_attention(Q, K, V, mask=None):
    """
    Compute scaled dot-product attention.

    Args:
        Q: Query matrix (seq_len, d_k)
        K: Key matrix (seq_len, d_k)
        V: Value matrix (seq_len, d_v)
        mask: Optional mask for causal attention (seq_len, seq_len)

    Returns:
        output: Weighted values (seq_len, d_v)
        weights: Attention weights (seq_len, seq_len)
    """
    # Step 1: Compute similarity scores
    scores = Q @ K.T  # (seq_len, seq_len)

    # Step 2: Scale
    d_k = Q.shape[-1]
    scores = scores / np.sqrt(d_k)

    # Step 3: Apply mask (if causal)
    if mask is not None:
        scores = np.where(mask, scores, -1e9)

    # Step 4: Softmax
    weights = softmax(scores, axis=-1)

    # Step 5: Weighted sum of values
    output = weights @ V

    return output, weights


def softmax(x, axis=-1):
    """Numerically stable softmax."""
    x_max = np.max(x, axis=axis, keepdims=True)
    exp_x = np.exp(x - x_max)
    return exp_x / np.sum(exp_x, axis=axis, keepdims=True)
            

Positional Encoding

def positional_encoding(seq_len, d_model):
    """
    Create sinusoidal positional encodings.

    Args:
        seq_len: Sequence length
        d_model: Model dimension

    Returns:
        pe: Positional encoding (seq_len, d_model)
    """
    pos = np.arange(seq_len)[:, np.newaxis]
    i = np.arange(d_model)[np.newaxis, :]

    # Compute angle rates
    angle_rates = pos / (10000 ** (2 * (i // 2) / d_model))

    # Apply sin to even indices, cos to odd
    pe = np.zeros((seq_len, d_model))
    pe[:, 0::2] = np.sin(angle_rates[:, 0::2])
    pe[:, 1::2] = np.cos(angle_rates[:, 1::2])

    return pe

# Example:
pe = positional_encoding(seq_len=100, d_model=512)
print(pe.shape)  # (100, 512)
            

Multi-Head Attention

class MultiHeadAttention:
    def __init__(self, d_model, num_heads):
        """
        Multi-head attention layer.

        Args:
            d_model: Model dimension
            num_heads: Number of attention heads
        """
        assert d_model % num_heads == 0
        self.d_model = d_model
        self.num_heads = num_heads
        self.d_k = d_model // num_heads

        # Initialize weight matrices (in practice, learned)
        self.W_Q = np.random.randn(d_model, d_model) * 0.01
        self.W_K = np.random.randn(d_model, d_model) * 0.01
        self.W_V = np.random.randn(d_model, d_model) * 0.01
        self.W_O = np.random.randn(d_model, d_model) * 0.01

    def forward(self, Q, K, V, mask=None):
        """
        Args:
            Q: Query (seq_len, d_model)
            K: Key (seq_len, d_model)
            V: Value (seq_len, d_model)
            mask: Optional mask

        Returns:
            output: (seq_len, d_model)
        """
        seq_len = Q.shape[0]

        # Linear projections
        Q_proj = Q @ self.W_Q
        K_proj = K @ self.W_K
        V_proj = V @ self.W_V

        # Reshape for multi-head
        Q_heads = Q_proj.reshape(seq_len, self.num_heads, self.d_k)
        K_heads = K_proj.reshape(seq_len, self.num_heads, self.d_k)
        V_heads = V_proj.reshape(seq_len, self.num_heads, self.d_k)

        # Compute attention for each head
        outputs = []
        for h in range(self.num_heads):
            Q_h = Q_heads[:, h, :]
            K_h = K_heads[:, h, :]
            V_h = V_heads[:, h, :]

            output_h, _ = scaled_dot_product_attention(Q_h, K_h, V_h, mask)
            outputs.append(output_h)

        # Concatenate heads
        concat_output = np.concatenate(outputs, axis=-1)

        # Final linear projection
        output = concat_output @ self.W_O

        return output
            

Transformer Block

class TransformerBlock:
    def __init__(self, d_model, num_heads, d_ff=2048):
        """
        Single transformer encoder block.

        Args:
            d_model: Model dimension
            num_heads: Number of attention heads
            d_ff: Feed-forward dimension (default 4x d_model)
        """
        self.attention = MultiHeadAttention(d_model, num_heads)
        self.d_model = d_model
        self.d_ff = d_ff

        # Feed-forward weights
        self.W1 = np.random.randn(d_model, d_ff) * 0.01
        self.b1 = np.zeros(d_ff)
        self.W2 = np.random.randn(d_ff, d_model) * 0.01
        self.b2 = np.zeros(d_model)

    def forward(self, x, mask=None):
        """
        Args:
            x: Input (seq_len, d_model)
            mask: Optional attention mask

        Returns:
            output: (seq_len, d_model)
        """
        # Multi-head attention + residual + layer norm
        attn_output = self.attention.forward(x, x, x, mask)
        x = self.layer_norm(x + attn_output)

        # Feed-forward + residual + layer norm
        ff_output = self.feed_forward(x)
        x = self.layer_norm(x + ff_output)

        return x

    def feed_forward(self, x):
        """Feed-forward network with ReLU."""
        hidden = np.maximum(0, x @ self.W1 + self.b1)
        output = hidden @ self.W2 + self.b2
        return output

    def layer_norm(self, x, eps=1e-6):
        """Layer normalization."""
        mean = np.mean(x, axis=-1, keepdims=True)
        std = np.std(x, axis=-1, keepdims=True)
        return (x - mean) / (std + eps)

# Example:
x = np.random.randn(10, 512)  # (seq_len=10, d_model=512)
block = TransformerBlock(d_model=512, num_heads=8)
output = block.forward(x)
print(output.shape)  # (10, 512)
            

Using Real Data

import urllib.request
import os

# Download a small text dataset
url = "https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-2-v1.zip"
filename = "wikitext-2.zip"

if not os.path.exists(filename):
    print("Downloading dataset...")
    urllib.request.urlretrieve(url, filename)
    print("Download complete!")

# In practice, extract and read the text:
# with open("wikitext-2/train.txt", "r") as f:
#     text = f.read()
#
# # Build vocabulary
# vocab = set(text.split())
# word_to_idx = {word: i for i, word in enumerate(vocab)}
# idx_to_word = {i: word for word, i in word_to_idx.items()}
#
# # Tokenize and create sequences
# tokens = [word_to_idx[word] for word in text.split()]
#
# # Create training batches (context -> next word prediction)
# context_length = 50
# batch_size = 32
#
# for i in range(0, len(tokens) - context_length, batch_size):
#     context = tokens[i:i+context_length]
#     target = tokens[i+context_length]
#
#     # Forward pass through transformer
#     # Loss = cross_entropy(output, target)
#     # Backprop and update weights