BERT

The Big Picture

What Is BERT?

BERT (2018, Devlin et al.) stands for Bidirectional Encoder Representations from Transformers. It's a transformer-based model specifically designed for understanding text, not generating it.

Key distinctions:

• Encoder-only: BERT uses only the encoder stack from the transformer architecture (no decoder)
• Bidirectional: Unlike GPT (which reads left-to-right), BERT looks at both left AND right context simultaneously during pre-training
• Pre-trained + Fine-tune: Pre-train once on massive text corpus → fine-tune for specific downstream tasks (classification, NER, Q&A, etc.)

Why BERT Changed NLP

Before BERT (2018):

• Task-specific models: sentiment analysis → one model, NER → different model, Q&A → yet another model
• Lots of feature engineering and hand-crafted heuristics
• Pre-trained embeddings (Word2Vec, GloVe) were context-agnostic

After BERT:

• One pre-trained model → add simple task-specific head → fine-tune → state-of-the-art on nearly every NLP task
• Contextualized embeddings: the same word gets different representations depending on context
• Transfer learning worked at scale (110M parameters for BERT-Base, 340M for BERT-Large)

Transformer Foundation

BERT builds on the Transformer architecture. For transformer details (attention, Q/K/V matrices, multi-head attention), see the Transformer guide Section 4-6. Here we focus on what makes BERT unique.

AdTech Application: Ad Relevance Scoring

The Problem

You're running a search advertising platform. A user searches "best running shoes for marathons" and multiple ads are eligible to bid:

• Ad A: "Nike Marathon Elite Shoes - 20% Off"
• Ad B: "Running Gear for Beginners"
• Ad C: "Cheap Shoes Outlet"

Which ad is most relevant? You need a score from 0-1 to inform your auction. Users prefer relevant ads (better CTR), so relevance → higher quality scores → better auction dynamics.

BERT Approach

Input format: [CLS] search query [SEP] ad text [SEP]

For our example:

BERT encodes this as a sequence of embeddings. The [CLS] token (classification token) at the start gets a special role: its final hidden state representation summarizes the entire input pair and is used for the relevance classification.

Why BERT Wins Here

• Semantic matching: BERT learns that "Marathon Elite Shoes" matches "running shoes for marathons" semantically, even without exact words
• Context awareness: The word "shoes" is embedded differently in context of marathons vs. "Cheap Shoes Outlet"
• Pre-trained knowledge: BERT already learned language patterns from Wikipedia + BookCorpus, so it generalizes to new ads/queries

BERT Architecture (Detailed)

BERT-Base vs BERT-Large

Architecture Overview

BERT is a stack of transformer encoder blocks. No decoder. Each block contains:

Input Representation (Detailed)

Three Embeddings Summed Together

BERT's input representation is built by summing three types of embeddings for each token:

1. Token Embeddings (WordPiece)

BERT uses WordPiece tokenization, which breaks words into subword units. This allows BERT to handle:

• Rare words: "uncharacteristically" → ["un", "##character", "##istically"] (## means "continuation")
• OOV (out-of-vocab): Unknown words get broken into familiar subwords
• Morphology: Model learns "##ing" across many verbs

Example: "I love playing" → ["I", "love", "play", "##ing"]

Each token has a learned embedding vector (shape: 768 for BERT-Base).

2. Segment Embeddings

When processing two sentences (like query + ad), BERT marks which sentence each token belongs to:

• E_A: Embedding for tokens from sentence A (first input)
• E_B: Embedding for tokens from sentence B (second input)

These are learned embeddings (shape: 768), not hand-crafted. There are only 2 segment embeddings (for single-sentence and sentence-pair tasks).

3. Position Embeddings

Unlike the original Transformer (which uses sinusoidal position encodings), BERT uses learned position embeddings:

• Absolute positions: position 0, 1, 2, ..., up to max_seq_len (typically 512)
• Each position has a learned 768-dim vector
• Pre-trained on 512 tokens, so BERT handles sequences up to 512 tokens

Why learned instead of sinusoidal? Learned embeddings are more flexible and can capture task-specific positional patterns during fine-tuning.

Concrete Example

Input: [CLS] buy shoes online [SEP] Nike running shoes sale [SEP]

Tokenized (WordPiece): ["[CLS]", "buy", "shoes", "online", "[SEP]", "Nike", "running", "shoes", "sale", "[SEP]"]

For token "running" (position 6, segment B):

Special Tokens

Pre-Training: MLM & NSP (Detailed)

Two Pre-Training Objectives

BERT is pre-trained on two tasks simultaneously on a large corpus (Wikipedia + BookCorpus, 3.3B words):

Objective 1: Masked Language Model (MLM)

Idea: Hide some tokens, make BERT predict them. This forces BERT to learn bidirectional context.

Process:

• Step 1: Select 15% of tokens at random
• Step 2: For each selected token, replace it with:
  • 80% of the time: [MASK] token
  • 10% of the time: a random token from vocab
  • 10% of the time: keep original token unchanged
• Step 3: BERT encoder processes the modified sequence and predicts the original token at each masked position

Why 80/10/10 split? During fine-tuning, there are no [MASK] tokens. If BERT always sees [MASK], it won't learn to handle regular tokens. The 10% random and 10% unchanged force robustness.

Example:

Objective 2: Next Sentence Prediction (NSP)

Idea: Given sentence A and B, predict: is B the actual next sentence? Binary classification.

Process:

• 50% of the time: B is the actual next sentence (label=IsNext)
• 50% of the time: B is a random sentence from the corpus (label=NotNext)
• Use [CLS] token's hidden state → Linear(768→2) → softmax(IsNext, NotNext)

Why NSP? Helps BERT learn sentence-level relationships. Turned out later (RoBERTa) that NSP wasn't essential, but it didn't hurt.

Pre-Training Details

• Corpus: Wikipedia (2.5B words) + BookCorpus (800M words)
• Batch size: 256 sequences
• Max sequence length: 512 tokens
• Optimizer: Adam (lr=1e-4, warmup over 10k steps)
• Total training: 1M steps = ~1 epoch over corpus
• Hardware: 16 TPUs for 4 days (BERT-Base)

Training Walkthrough (Step-by-Step)

Here's exactly what happens during a single pre-training step:

Step 1: Tokenize

Convert text to WordPiece tokens:

Step 2: Create Two-Sentence Pair

Sample or construct sentence A and B, concatenate with special tokens:

Step 3: Apply MLM (Mask 15%)

Randomly select 15% of tokens, apply the 80/10/10 rule:

Step 4: Create Input Embeddings

For each token, sum three embeddings:

This creates: (seq_len=13, hidden_size=768) tensor

Step 5: Pass Through 12 Encoder Layers

The input tensor flows through the encoder stack. For each layer:

• Multi-Head Self-Attention: (seq_len, 768) → 12 attention heads, each computing attention (see Transformer guide Section 5)
• Add & LayerNorm: Residual connection + layer normalization
• FFN: Linear(768→3072) + GELU activation + Linear(3072→768)
• Add & LayerNorm: Residual + LayerNorm again
• Dropout: During training, randomly drop 10% of values

Key detail on GELU activation: GELU(x) = x · Φ(x), where Φ is the cumulative distribution function of standard normal. It's smoother than ReLU. See DNN guide Section 8 for activation functions.

Output shape after all 12 layers: (seq_len=13, hidden_size=768)

Step 6: MLM Head - Predict Masked Tokens

Take hidden states only at masked positions (positions 2, 4, 8). For each:

Step 7: NSP Head - Next Sentence Prediction

Use [CLS] token (position 0) to predict if B follows A:

Step 8: Combine Losses

Step 9: Backpropagation

Compute gradients with respect to all parameters (entire 12-layer stack + embeddings). See DNN guide Section 7 for backpropagation mechanics.

• Gradients flow back through all 12 layers (chain rule)
• Compute ∂L / ∂W for every matrix W
• Typical gradient shapes: attention W_Q: (768, 768), FFN W_1: (768, 3072), etc.

Step 10: Update Weights (Adam Optimizer)

BERT uses Adam with:

• Learning rate: 1e-4
• Warmup: Linear increase over first 10k steps, then decay
• β1=0.9, β2=0.999: Momentum and RMSprop parameters
• Weight decay: 0.01 (L2 regularization)

Fine-Tuning BERT

The Fine-Tuning Paradigm

After pre-training, BERT weights are frozen initially, and we add a small task-specific head on top:

For different tasks:

Fine-Tuning Hyperparameters

Two Approaches

Approach 1: Full Fine-Tuning

Update all BERT parameters + task head during training. This is typically best for moderate-sized datasets (100K+ examples).

Approach 2: Feature Extraction

Freeze BERT, train only the task head. Faster, uses less memory. Better for small datasets (<10K examples) or when compute is limited.

Extracting Embeddings from BERT (Detailed)

Which Hidden State to Use?

BERT produces 13 hidden state tensors (input embeddings + output from each of 12 layers). Which one should you use?

Concrete Example: Extracting Ad Relevance Embeddings

Input: "[CLS] running shoes [SEP] Nike Marathon Elite Shoes [SEP]"

Method 1: [CLS] token (sentence pair representation)

Method 2: Average pooling (sentence representation without special token)

Method 3: Concatenate last 4 layers (for token-level or richer representation)

Why Different Layers?

BERT layers learn progressively more abstract representations:

• Early layers (1-4): Syntax, morphology, local context
• Middle layers (5-8): Semantic phrases, relations
• Late layers (9-12): Task-specific high-level concepts

Concatenating multiple layers captures this hierarchy. For frozen embeddings (feature extraction), middle/late layers often outperform the very last layer because they're less task-specialized during pre-training.

Components Deep Dive

LayerNorm (Not BatchNorm)

BERT uses LayerNorm, not BatchNorm. Why?

LayerNorm formula:

Pros: Works with variable-length sequences. No batch dependency (important for NLP with padding).

Cons: Slightly slower than BatchNorm on GPUs.

Dropout

Dropout is applied after:

• Attention weights (before weighted sum)
• After FFN output
• On embeddings (less common)

BERT dropout: p=0.1 (10%)

Dropout prevents co-adaptation of neurons. See DNN guide Section 10 for details.

GELU Activation

BERT's FFN uses GELU instead of ReLU:

GELU is smoother than ReLU and often performs better. See DNN guide Section 8 for activation function comparisons.

Attention Mask for Padding

When sequences are padded to max_length, BERT must ignore [PAD] tokens in attention:

Loss Functions

Pre-Training Losses

Masked Language Model (MLM) Loss

Cross-entropy over predicted tokens at masked positions only:

Next Sentence Prediction (NSP) Loss

Binary cross-entropy on [CLS] classification:

Fine-Tuning Losses

Classification (Sentiment, Ad Relevance)

Token-Level (NER, POS)

Regression (Bid Prediction)

Ranking Loss (Ad Relevance Scoring)

For ranking ads by relevance, use ranking losses:

Python Implementation

Using HuggingFace Transformers

The easiest way to use BERT:

pip install transformers torch datasets

Load Pre-Trained BERT & Extract Embeddings

from transformers import AutoTokenizer, AutoModel
import torch

# Load pre-trained BERT-Base
model_name = "bert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name, output_hidden_states=True)

# Example: Ad relevance input
query = "best running shoes for marathons"
ad_text = "Nike Marathon Elite Shoes - 20% Off"

# Tokenize
inputs = tokenizer(
    query, ad_text,
    return_tensors="pt",
    padding=True,
    truncation=True,
    max_length=128
)

# Forward pass
with torch.no_grad():
    outputs = model(**inputs)

# Extract embeddings
last_hidden = outputs.last_hidden_state  # (1, seq_len, 768)
cls_embedding = last_hidden[:, 0, :]     # (1, 768) - CLS token

print(f"CLS embedding shape: {cls_embedding.shape}")
print(f"CLS embedding (first 5 values): {cls_embedding[0, :5]}")

Fine-Tune BERT for Ad Relevance Classification

from transformers import AutoTokenizer, AutoModelForSequenceClassification
from torch.optim import Adam
from torch.utils.data import DataLoader, TensorDataset
import torch

# Load BERT with classification head (2 classes: relevant/not-relevant)
model_name = "bert-base-uncased"
model = AutoModelForSequenceClassification.from_pretrained(
    model_name, num_labels=2
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Example fine-tuning data
queries = [
    "running shoes for marathons",
    "cheap office chairs",
    "best winter jackets"
]
ads = [
    "Nike Marathon Elite Shoes",
    "wooden desk chairs on sale",
    "Patagonia winter coats"
]
labels = torch.tensor([1, 1, 1])  # All relevant (1 = relevant)

# Tokenize
inputs = tokenizer(
    queries, ads,
    padding=True,
    truncation=True,
    return_tensors="pt"
)

# Create dataset
dataset = TensorDataset(
    inputs["input_ids"],
    inputs["attention_mask"],
    labels
)
loader = DataLoader(dataset, batch_size=2, shuffle=True)

# Fine-tune
optimizer = Adam(model.parameters(), lr=2e-5)
model.train()

for epoch in range(2):
    for batch_idx, (input_ids, attn_mask, batch_labels) in enumerate(loader):
        optimizer.zero_grad()

        outputs = model(
            input_ids,
            attention_mask=attn_mask,
            labels=batch_labels
        )
        loss = outputs.loss

        loss.backward()
        optimizer.step()

        if batch_idx % 10 == 0:
            print(f"Epoch {epoch}, Batch {batch_idx}, Loss: {loss.item():.4f}")

Inference on New Ads

model.eval()

test_query = "high performance running shoes"
test_ad = "Nike Vaporfly Elite Shoes - Record Breaking Speed"

inputs = tokenizer(
    test_query, test_ad,
    return_tensors="pt",
    padding=True,
    truncation=True
)

with torch.no_grad():
    outputs = model(**inputs)
    logits = outputs.logits
    probabilities = torch.softmax(logits, dim=-1)

relevant_score = probabilities[0, 1].item()  # P(relevant)
print(f"Relevance score: {relevant_score:.4f}")  # e.g., 0.92

Extract Different Embeddings

# Method 1: Last [CLS] (already shown above)
cls_emb = last_hidden[:, 0, :]  # shape: (batch, 768)

# Method 2: Average pooling
avg_emb = last_hidden.mean(dim=1)  # shape: (batch, 768)

# Method 3: Concatenate last 4 layers
all_hidden = outputs.hidden_states  # tuple of 13 tensors
# Take layers 9, 10, 11, 12 (last 4)
last_4 = torch.cat(all_hidden[-4:], dim=-1)  # (batch, seq_len, 3072)
cls_concat = last_4[:, 0, :]  # (batch, 3072)

print(f"CLS embedding: {cls_emb.shape}")
print(f"Avg pooling: {avg_emb.shape}")
print(f"Last 4 layers concat: {cls_concat.shape}")

Custom BERT Encoder Block (NumPy - Educational)

Here's a simplified transformer encoder block from scratch to understand the internals:

import numpy as np

class SimpleAttentionHead:
    """Single attention head (simplified)"""
    def __init__(self, d_model=768, d_k=64):
        self.d_model = d_model
        self.d_k = d_k
        # Initialize weights (Xavier init)
        self.W_q = np.random.randn(d_model, d_k) * np.sqrt(1 / d_model)
        self.W_k = np.random.randn(d_model, d_k) * np.sqrt(1 / d_model)
        self.W_v = np.random.randn(d_model, d_k) * np.sqrt(1 / d_model)

    def forward(self, X):
        """X shape: (seq_len, d_model)"""
        Q = X @ self.W_q  # (seq_len, d_k)
        K = X @ self.W_k
        V = X @ self.W_v

        # Scaled dot-product attention
        scores = (Q @ K.T) / np.sqrt(self.d_k)  # (seq_len, seq_len)
        attn_weights = softmax(scores, axis=1)  # (seq_len, seq_len)

        output = attn_weights @ V  # (seq_len, d_k)
        return output, attn_weights

class TransformerEncoderBlock:
    """One encoder block with attention + FFN"""
    def __init__(self, d_model=768, num_heads=12, d_ff=3072):
        self.d_model = d_model
        self.num_heads = num_heads
        self.d_k = d_model // num_heads

        # Multi-head attention (simplified: one head only)
        self.attn_head = SimpleAttentionHead(d_model, self.d_k)
        self.W_o = np.random.randn(self.d_k, d_model) * np.sqrt(1 / self.d_k)

        # FFN
        self.W_1 = np.random.randn(d_model, d_ff) * np.sqrt(1 / d_model)
        self.b_1 = np.zeros(d_ff)
        self.W_2 = np.random.randn(d_ff, d_model) * np.sqrt(1 / d_ff)
        self.b_2 = np.zeros(d_model)

        # LayerNorm parameters
        self.gamma_1 = np.ones(d_model)
        self.beta_1 = np.zeros(d_model)
        self.gamma_2 = np.ones(d_model)
        self.beta_2 = np.zeros(d_model)

    def layer_norm(self, X, gamma, beta, eps=1e-6):
        mean = X.mean(axis=1, keepdims=True)
        std = X.std(axis=1, keepdims=True)
        return gamma * (X - mean) / (std + eps) + beta

    def gelu(self, x):
        """GELU activation"""
        from scipy.stats import norm
        return x * norm.cdf(x)

    def forward(self, X):
        """X shape: (seq_len, d_model)"""
        # Multi-head attention (simplified: 1 head)
        attn_out, _ = self.attn_head.forward(X)
        attn_out = attn_out @ self.W_o  # project back to d_model

        # Residual + LayerNorm
        X = self.layer_norm(X + attn_out, self.gamma_1, self.beta_1)

        # FFN
        ffn_hidden = self.gelu(X @ self.W_1 + self.b_1)
        ffn_out = ffn_hidden @ self.W_2 + self.b_2

        # Residual + LayerNorm
        X = self.layer_norm(X + ffn_out, self.gamma_2, self.beta_2)

        return X

def softmax(x, axis=None):
    e_x = np.exp(x - np.max(x, axis=axis, keepdims=True))
    return e_x / np.sum(e_x, axis=axis, keepdims=True)

# Test the block
block = TransformerEncoderBlock(d_model=768, num_heads=12, d_ff=3072)
X = np.random.randn(10, 768)  # (seq_len=10, d_model=768)
output = block.forward(X)
print(f"Input shape: {X.shape}, Output shape: {output.shape}")