Two-Tower Models

The Big Picture

The core challenge in modern recommendation systems is retrieval at scale: given billions of items and millisecond latency constraints, how do you efficiently find the top-K items relevant to a user?

Two-Tower models (also called dual-encoder or siamese towers) solve this by learning separate embeddings for users and items, then using fast approximate nearest neighbor search to find matches. They're the backbone of production recommendation systems at companies with massive catalogs.

The Two-Tower Concept

The architecture is elegantly simple:

• User Tower: Takes user features (ID, demographics, behavior history) and produces a user embedding
• Item Tower: Takes item features (ID, content, metadata) and produces an item embedding
• Similarity: The two embeddings are in the same space. A dot product (or cosine similarity) between them scores how relevant an item is to a user

Where It Fits in the RecSys Pipeline

A complete recommendation system typically has three stages:

Key Components

A Concrete Example: Ad Platform Retrieval

Let's walk through a real-world scenario: an ad/e-commerce platform needs to retrieve relevant products for a user's search query in milliseconds.

User Features

When a user visits the platform, we collect their context:

User ID: user_12345
Demographic Features:
  - Age: 28
  - Gender: Female
  - Location: New York, NY
  - Device: Mobile

Behavior Features:
  - Last 5 items viewed: [item_001, item_042, item_999, ...]
  - Search history: ["summer dresses", "blue shoes", "workout gear"]
  - Purchase history: [purchase_1, purchase_2, ...]
  - Dwell time avg: 3.2 seconds
  - Click-through rate: 0.12

Item Features

Each item in the catalog has rich metadata:

Item ID: item_042
Content Features:
  - Title: "Lightweight Summer Dress - Navy Blue"
  - Description embedding: [0.2, -0.5, 0.8, ...] (from BERT)

Categorical Features:
  - Category: Apparel
  - Subcategory: Dresses
  - Brand: StyleCo
  - Color: Blue
  - Season: Summer

Popularity Features:
  - Total views: 50,000
  - Total clicks: 5,000
  - CTR: 0.10
  - Age: 15 days (newer items often get a boost)

Computing a Similarity Score

The two towers independently process their inputs:

Numeric Example

Simplified example with 4-dimensional embeddings:

User embedding (normalized):    u = [0.50, 0.25, -0.50, 0.50]
Item embedding (normalized):    v = [0.60, 0.20, -0.40, 0.60]

Dot product (similarity score):
  score = u · v
  = (0.50)(0.60) + (0.25)(0.20) + (-0.50)(-0.40) + (0.50)(0.60)
  = 0.30 + 0.05 + 0.20 + 0.30
  = 0.85

Interpretation: Score of 0.85 (out of max 1.0) = highly relevant match

Retrieval Pipeline at Query Time

User Tower Architecture

The user tower transforms heterogeneous user signals into a fixed-size dense embedding suitable for fast similarity computation.

Input Features

A user tower typically consumes:

• User ID embedding: Learned embedding for the user ID itself (~128-256 dims)
• Dense features: Age, income level, engagement metrics (normalized to [0,1])
• Categorical features: Gender, location, device type (converted to embeddings or one-hot)
• Sequence features: Last N items viewed, search queries (aggregated via pooling)
• Context features: Time of day, day of week, device type, network quality

Architecture Overview

Key Design Choices

Code Structure (Conceptual)

class UserTower:
    def __init__(self, output_dim=256):
        self.user_id_embedding = nn.Embedding(num_users, 128)
        self.fc1 = nn.Linear(1024, 512)  # input concat size
        self.fc2 = nn.Linear(512, 256)
        self.fc3 = nn.Linear(256, output_dim)
        self.bn1 = nn.BatchNorm1d(512)
        self.bn2 = nn.BatchNorm1d(256)
        self.dropout = nn.Dropout(0.2)

    def forward(self, user_id, demographics, behavior, context):
        # Lookup user embedding
        user_emb = self.user_id_embedding(user_id)

        # Concatenate all features
        x = torch.cat([user_emb, demographics, behavior, context], dim=1)

        # MLP
        x = self.fc1(x)
        x = self.bn1(x)
        x = F.relu(x)

        x = self.fc2(x)
        x = self.bn2(x)
        x = F.relu(x)
        x = self.dropout(x)

        # Output layer
        x = self.fc3(x)

        # L2 normalize
        x = F.normalize(x, p=2, dim=1)

        return x

Item Tower Architecture

The item tower encodes item identity and content features into a shared embedding space with the user tower.

Input Features

• Item ID embedding: Learned representation of item ID (~128-256 dims)
• Title/content embedding: Text embeddings from pre-trained models like BERT or averaged word2vec
• Categorical features: Category, brand, color, size (embeddings or one-hot)
• Popularity features: Total views, CTR, age since creation (normalized)
• Price & quality signals: Price range, rating, review count

Architecture Diagram

Shared Embedding Space

The user and item towers both output to the same d-dimensional space (e.g., 256 dims). This is crucial:

• Dot product between user and item embeddings is meaningful
• Enables efficient similarity search
• Allows for unified training on positive and negative pairs

Handling Text Content

For items with rich textual descriptions:

# Option 1: Pre-computed BERT embeddings
item_title_embedding = precomputed_bert_embeddings[item_id]  # Shape: (768,)

# Option 2: Average word embeddings
word_embeddings = [word2vec[word] for word in title.split()]
item_title_embedding = np.mean(word_embeddings, axis=0)  # Shape: (300,)

# Option 3: No text encoder, use category + popularity only
item_title_embedding = category_embedding + popularity_features

# In practice: concatenate with other features
item_input = np.concatenate([
    item_id_embedding,          # 128 dims
    item_title_embedding,       # 768 dims (BERT)
    category_embedding,         # 64 dims
    popularity_features         # 10 dims
])  # Total: ~970 dims

Serving Optimization: Precomputed Embeddings

At serving time, all item embeddings can be precomputed and indexed:

The Similarity Function

The core operation that makes two-tower efficient is the similarity function. After computing user and item embeddings, we need a fast way to score how well they match.

Dot Product vs Cosine Similarity

When embeddings are L2-normalized (unit vectors), dot product and cosine similarity are equivalent:

Dot Product:
  score = u · v = Σ(u_i * v_i)
  Range: [-1, 1] when L2-normalized

Cosine Similarity:
  score = (u · v) / (||u|| * ||v||)
  Range: [-1, 1]

When ||u|| = 1 and ||v|| = 1 (L2-normalized):
  Dot product = Cosine similarity

Benefit: Dot product is MUCH faster to compute than
cosine (no division/norm needed).

Why L2-Normalize?

L2 normalization is critical for production two-tower systems:

Temperature Scaling

In some cases, you might want softer or harder discrimination between similar items:

# Temperature scaling divides the score before softmax
# Lower temperature = sharper distribution
# Higher temperature = softer distribution

score_raw = u · v
score_scaled = score_raw / temperature

# In loss computation:
# temperature = 0.07 (common for contrastive learning)
# makes the model learn sharper distinctions

loss = -log(exp(score_pos / τ) / Σ exp(score_neg / τ))

Dot Product Geometry

Comparison to Cross-Encoders

Two-tower (dual-encoder) scores items independently. Cross-encoders score user-item pairs jointly:

Training: Negative Sampling

The critical challenge in training two-tower models is how to select negative examples. Positives (clicked items) are rare, so you must be strategic about which negatives to use.

The Negative Sampling Problem

In a typical e-commerce platform:

• User sees ~100 items
• User clicks ~1-2 items (positive examples)
• ~98 items are not clicked (potential negatives)

If you treat all non-clicks as negatives, the model learns that "almost everything is irrelevant," which is too easy. You need hard negatives.

In-Batch Negatives

The simplest and most efficient approach: treat other items in the batch as negatives.

Hard Negative Mining

In-batch negatives are fast but can include easy negatives. Hard negatives are items the model scores high but the user didn't interact with:

InfoNCE Loss (Contrastive Loss)

The standard loss for two-tower training:

InfoNCE Loss (Normalized Temperature-scaled Cross-Entropy):

L = -log(exp(s_pos / τ) / Σ_j exp(s_j / τ))

Where:
  s_pos = u · v_positive  (similarity with positive item)
  s_j = u · v_j           (similarity with negative items)
  τ = temperature (typically 0.07-0.1)

This is equivalent to softmax cross-entropy where:
  - Positive item should have high probability
  - Negative items should have low probability

As τ → 0: sharper distinction (harder training)
As τ → 1: softer distinction (easier training)

Why In-Batch Negatives Work

• Efficiency: No extra computation. One batch forward pass gives O(batch_size²) similarity scores.
• Statistical Effect: Larger batch sizes = more negatives = more challenging loss. Typical: batch size 256-4096.
• Implicit Hard Negative Mining: If someone clicks item A in one request and item B in another, they might appear in the same batch as negatives for each other, which is a natural hard negative.

Training Pipeline & Data

Building a production two-tower model requires careful data preparation, debiasing, and training infrastructure.

Positive Examples

What counts as a positive interaction?

• Click: User explicitly clicked an item. Binary signal.
• Dwell Time: User spent > N seconds on item. Stronger signal than click alone.
• Purchase: Strongest signal. User bought the item.
• Add to Wishlist: Implicit preference signal.
• Interaction Score: Weighted combination (e.g., 1 click + 2 dwell time + 5 purchase).

Negative Examples

• Impressions without clicks: Item was shown, user didn't click. "Easy" negatives.
• Hard negatives: Items with high model scores but no click. Require offline mining.
• Random negatives: Sample uniformly from catalog. Biased but simple.
• Popularity-weighted negatives: Sample negatives proportional to their popularity. Helps with debiasing.

Debiasing

Logged interactions have several biases:

Training Data Format

Typical training example:

{
    "user_id": 12345,
    "item_id": 42,
    "label": 1,  # 1 = positive (click), 0 = negative

    "user_features": {
        "age": 28,
        "gender": "F",
        "location": "NY",
        "device": "mobile",
        "historical_items": [item_1, item_2, ...],
        "historical_avg_dwell": 3.2
    },

    "item_features": {
        "title": "Summer Dress Blue",
        "title_embedding": [0.2, -0.5, ...],  # from BERT
        "category": "Apparel",
        "price": 49.99,
        "ctr": 0.10,
        "popularity_score": 0.85,
        "age_days": 15
    }
}

Online vs Offline Training

• Offline Training: Train on historical logs once per day/week. Fast, reproducible. But model sees stale data.
• Online Learning: Update model on live interactions continuously. Captures trends but adds complexity.
• Hybrid: Offline training + online fine-tuning on recent data. Common in production.

Practical Training Loop

def train_two_tower(user_tower, item_tower, train_loader, num_epochs):
    optimizer = Adam(lr=0.001)
    temperature = 0.07

    for epoch in range(num_epochs):
        total_loss = 0

        for batch_idx, batch in enumerate(train_loader):
            user_feats = batch['user_features']    # Shape: (batch_size, user_feat_dim)
            item_feats = batch['item_features']    # Shape: (batch_size, item_feat_dim)
            labels = batch['label']                # Shape: (batch_size,)

            # Forward pass
            user_emb = user_tower(user_feats)      # (batch_size, emb_dim)
            item_emb = item_tower(item_feats)      # (batch_size, emb_dim)

            # Compute all pairwise similarities (batch_size × batch_size)
            scores = torch.mm(user_emb, item_emb.t())  # Dot product

            # Scale by temperature
            scores = scores / temperature

            # Create labels: diagonal is 1 (positives), rest are 0 (in-batch negatives)
            batch_labels = torch.eye(batch_size)  # Ideally also matches actual labels

            # InfoNCE loss
            loss = F.cross_entropy(scores, batch_labels)

            # Backward
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

            total_loss += loss.item()

        print(f"Epoch {epoch} - Avg Loss: {total_loss / len(train_loader):.4f}")

Approximate Nearest Neighbor (ANN) Serving

After training, you have a user tower and item tower. At serving time, you need to find the top-K items most similar to a user's embedding. With 100M+ items, exact search is infeasible.

The Retrieval Challenge

At query time:

• User comes with features (ID, demographics, context)
• Encode into embedding: O(1) operation
• Find top-1000 items from 100M catalog
• Latency budget: ~10-50ms

Brute-force: compare user embedding with all 100M item embeddings

FAISS IVF-PQ: The Industry Standard

FAISS (Facebook AI Similarity Search) IVF-PQ is the most widely used ANN method:

• IVF (Inverted File): Cluster item embeddings into buckets. Search only relevant buckets.
• PQ (Product Quantization): Compress embeddings using learned codebooks. Reduces memory 16-32×.

HNSW (Hierarchical Navigable Small World)

Alternative to FAISS, increasingly popular for real-time systems:

• Graph-based: Items connected in hierarchical graph structure
• Search: Navigate graph from random start, greedily move toward user embedding
• Advantages: Fast search (O(log N)), dynamic updates (can add items without reindexing)
• Disadvantages: Higher memory than IVF-PQ, slower bulk insertion

ScaNN (Scalable Nearest Neighbors)

Google's recent ANN method, optimized for modern CPUs:

• Combines vector quantization with learnable distance metrics
• Faster than FAISS on modern hardware (AVX-512, GPU support)
• Good for hybrid retrieval (dense + sparse features)

Recall vs Latency Trade-off

Advanced Techniques

Beyond the basic two-tower architecture, several advanced techniques improve performance in production systems.

Asymmetric Loss

All positives are not equal. Items clicked by many users are less informative (everyone likes them). Rare items need stronger signal:

Popularity Debiasing

Correct for the fact that popular items were shown more often and received more clicks:

Feature Hashing for High-Cardinality Features

User IDs and item IDs can have billions of unique values. Embeddings are large. Solution: hash to fixed size:

Multi-Task Two-Tower

One tower, multiple objectives (CTR, CVR, latency):

Item Freshness & Time Decay

New items should get a boost. Add age-based features:

Hard Negative Mining Strategies

Offline Mining: After each training epoch, score all items for each user. Hard negatives are high-scoring items that weren't clicked.

Streaming Hard Negative Mining: During training, track items with high scores but label=0 and add them to future batches.

When to Use Two-Tower vs Alternatives

Two-tower is powerful, but not always the best choice. Here's how it compares to other recommendation approaches:

Comparison Table

Two-Tower Strengths

Two-Tower Limitations

• Dot Product Interactions: Can only model simple additive interactions. Can't capture complex cross-features (e.g., user age × item price).
• Static Item Embeddings: Item embeddings precomputed offline. Can't incorporate real-time item popularity or trending signals easily.
• Symmetry: User and item embeddings from different towers but in same space. Some recommend learning separate interaction matrices instead.

When to Use Alternatives

Typical Hybrid Pipeline

Python Implementation

A complete, practical two-tower implementation using NumPy and Pandas (no external ML libraries). This demonstrates the core concepts.

Complete Code

Key Implementation Notes

Running the Code