Learning to Rank (LTR)

The Big Picture

Ranking is the core problem in search engines, recommendation systems, and information retrieval. Given a query and a large corpus of documents, the goal is to order documents by relevance so the most useful results appear first.

From Manual to Learned Ranking

Manual Ranking (BM25): Traditional retrieval uses term-matching heuristics like BM25 (see BM25 & Inverted Index Guide). BM25 computes a static relevance score for each query-document pair based on term frequency and inverse document frequency. Simple and fast, but ignores semantic context and cannot adapt to different query intents.

Learned Ranking (LTR): Learning to Rank replaces manual heuristics with a learned function trained on labeled data. The system learns which signals (features) are predictive of relevance and how to combine them optimally.

The LTR Training Paradigm

LTR requires three components:

• Training Data: Query-document pairs labeled with relevance (0 = not relevant, 1 = relevant, 2 = highly relevant, etc.). Labels come from human judgments or click signals.
• Feature Extraction: For each query-document pair, compute numerical features (BM25 score, document length, embedding similarity, etc.).
• Ranking Function: A model (tree, neural network, linear) that maps features to a relevance score. Documents are ranked by this score.

Three Paradigms

The Ranking Pipeline

Features for Learning to Rank

The quality of an LTR system depends critically on the features used. Features are the "language" the ranking model uses to understand query-document relevance.

Categories of Features

Query Features

• Query length: Number of terms in the query.
• Query type: Is it a navigational query (looking for a specific site), informational (research), or transactional (ready to buy)? Encoded as a categorical feature.
• Reformulation rate: How often this query is reformulated by users (proxy for query clarity).
• Query language: Categorical: English, Spanish, etc.

Document Features

• BM25 score: Classic term-matching score. See BM25 & Inverted Index Guide.
• TF-IDF: Term frequency times inverse document frequency.
• Document length: Number of terms. Longer documents may have more relevant content but are also noisier.
• PageRank: Graph centrality score. Pages linked from many sources are often more authoritative.
• Document age: Freshness. Recent documents may be more relevant for current events.

Query-Document Features

• Exact match: Does the document contain the exact query phrase? Binary feature.
• BM25(query, document): Direct term matching between query and document.
• Embedding similarity: Cosine similarity between query embedding and document embedding. Captures semantic similarity.
• Anchor text match: How many incoming links have anchor text matching the query?
• Title match: Does the document title contain query terms?

Context Features

• Position in initial ranking: Documents retrieved higher are often more relevant (retrieval bias).
• Time of day: Query intent varies by time. Morning = informational, evening = entertainment.
• User history: CTR of documents similar to this one for the same user in the past.
• Device type: Mobile vs. desktop queries may have different relevant documents.

Feature Normalization

Features have different scales (BM25 ≈ 0–100, embedding similarity ≈ 0–1). Normalization ensures each feature contributes fairly to the model:

Feature Importance Analysis

Once trained, which features does the model rely on most? Use SHAP values (for tree models like LambdaMART) or attention weights (for neural models) to understand feature importance.

Example Feature Vector

For the query "python machine learning tutorial" and a document about scikit-learn:

Evaluation Metrics

How do you measure if ranking is good? Ranking metrics account for position: top positions matter more. A perfect ranking at position 10 is worse than a perfect ranking at position 1.

Normalized Discounted Cumulative Gain (NDCG)

NDCG is the most important LTR metric. It measures ranking quality considering both relevance and position.

DCG@k = Σ(i=1 to k) [ rel_i / log₂(i+1) ]

Where rel_i is the relevance label (0, 1, 2, 3, ...) of the document at position i.

IDCG@k = DCG of the ideal ranking (all most-relevant documents at top).

NDCG@k = DCG@k / IDCG@k. Normalized to [0, 1], where 1.0 is perfect.

Worked Example: Computing NDCG@3

Query: "how to bake bread"

Relevance labels: 0 = not relevant, 1 = somewhat relevant, 2 = highly relevant

Other Ranking Metrics

Choosing K

Should you report NDCG@5 or NDCG@10? It depends on your use case:

• K=5: Mobile search, where users see few results per screen.
• K=10: Desktop search, standard benchmark.
• K=1: Highly focused tasks (e.g., "best restaurant near me").

Pointwise Approaches

Pointwise methods treat ranking as a classification or regression problem on individual documents, ignoring their relative ordering.

Pointwise as Regression

Predict the relevance score of each document independently:

• Train a regression model (linear regression, neural network) on labeled query-document pairs.
• Loss: Mean Squared Error (MSE) = (y_true - y_pred)²
• At ranking time, score all candidate documents and rank by predicted score.

Pointwise as Classification

Treat relevance as a discrete label:

• Binary: Relevant (1) vs. Not Relevant (0). Loss: Binary cross-entropy.
• Multi-class: Graded (0 = bad, 1 = fair, 2 = good, 3 = excellent). Loss: Cross-entropy.

Comparison with Ranking Objectives

When to Use Pointwise

Pointwise methods are rarely used for production ranking, but they're useful:

• As a baseline or sanity check.
• When you have massive data but limited compute (simple models train fast).
• For warm-start: pre-train a pointwise model, then fine-tune with pairwise/listwise.

For a comparison with logistic regression, see Logistic Regression Guide.

Pairwise Approaches: RankNet & LambdaRank

Pairwise methods learn from document pairs: which document is more relevant? This directly optimizes relative ordering.

RankNet

Core idea: For each query, compare pairs of documents. For each pair (doc_i, doc_j):

P(doc_i ranked above doc_j | features) = σ(s_i - s_j)

where σ is the sigmoid function and s_i, s_j are model scores.

Loss for a pair: Binary cross-entropy on pairwise preference.

• If doc_i is more relevant than doc_j (rel_i > rel_j), we want P(doc_i > doc_j) = 1.
• Loss = -[y_ij * log(P) + (1 - y_ij) * log(1 - P)]
• where y_ij = 1 if rel_i > rel_j, else 0.

Pair Sampling

For n documents per query, there are O(n²) pairs. Sampling strategies:

LambdaRank: Incorporating Ranking Metrics

The insight: Not all pairs matter equally. Swapping doc_A and doc_B changes NDCG differently than swapping doc_K and doc_L.

Lambda: A weighting factor for each pair gradient proportional to the NDCG change from swapping them.

λ_ij = (∂C/∂s_i) × |ΔNDCG_ij|

where ΔNDCG_ij = change in NDCG if docs i and j swap positions.

Algorithm:

1. Compute pairwise RankNet gradients.
2. For each pair, multiply gradient by the NDCG change from swapping them.
3. Pairs that change NDCG more get higher gradient weight.
4. Update model in gradient direction.

Implementation Notes

LambdaRank is typically implemented inside a Gradient Boosting framework (LambdaMART, see Section 7). For a standalone neural implementation, you'd compute:

1. Score documents with neural network.
2. For each pair (i, j) with rel_i > rel_j, compute RankNet loss.
3. Multiply gradient by |ΔNDCG_ij|.
4. Backpropagate weighted gradients.

Listwise Approaches

Listwise methods optimize the entire ranking list directly, not individual documents or pairs. This directly aligns training with ranking metrics like NDCG.

ListNet

Idea: Model the probability of entire rankings. For each query, we have a true ranking (ideal ordering) and a predicted ranking. Minimize KL divergence between them.

Given documents with relevance labels, compute:

• True distribution: P(ranking | true labels) = probability of an ordering proportional to relevance.
• Predicted distribution: P(ranking | model scores) = probability proportional to model scores.
• Loss: KL divergence = Σ P_true(π) * log(P_true(π) / P_pred(π)) summed over all permutations π.

Since there are n! permutations, this is intractable for large n. ListNet uses a top-k approximation: only consider top-K documents and top-K permutations.

ListMLE (Maximum Likelihood Estimation)

Idea: Model the probability of the ideal permutation directly using a factorized distribution.

For the ideal ranking π = (i₁, i₂, ..., i_k):

P(π) = P(i₁ first) × P(i₂ second | i₁ first) × ... (chain rule)

Each factor uses a softmax over remaining documents:

P(i_j | i₁, ..., i_{j-1}) = exp(s_ij) / Σ_k≠{i₁...i_{j-1}} exp(s_k)

Loss: Negative log-likelihood of the true ranking.

ListMLE is more efficient than ListNet and handles variable-length lists naturally.

SoftRank

Idea: Use a differentiable approximation to permutation operations. Instead of hard sorting, use soft permutation matrices that approximate sorting.

Enables direct gradient flow through ranking operations, allowing end-to-end optimization of NDCG.

Listwise vs. Pointwise/Pairwise

Advantages & Challenges

Advantages:

• Directly optimizes ranking metrics (NDCG, etc.).
• Accounts for position importance naturally.
• Often outperforms pointwise/pairwise in practice.

Challenges:

• Exponential permutation space: expensive even with approximations.
• Harder to implement than pointwise/pairwise.
• Requires careful truncation (top-K) to be tractable.

LambdaMART: The Industrial Standard

LambdaMART = LambdaRank gradients + MART (Multiple Additive Regression Trees). This combination dominates production ranking systems because it's fast, accurate, and interpretable.

MART (Gradient Boosting)

MART is gradient boosting on regression trees. It's equivalent to XGBoost (see XGBoost Guide). Algorithm:

1. Start with an initial prediction (usually zero or constant).
2. Compute residuals: true_label - current_prediction.
3. Fit a regression tree to predict these residuals.
4. Add the tree's predictions to the overall model (with learning rate shrinkage).
5. Repeat until convergence.

LambdaMART Algorithm

Replace the residuals in step 2 with LambdaRank gradients:

1. For each query, compute model scores on current documents.
2. Compute pairwise RankNet gradients (like section 5).
3. Weight each gradient by NDCG change: λ = gradient × |ΔNDCG|.
4. Fit a regression tree to these weighted gradients.
5. Add tree to the ensemble.
6. Repeat for n_estimators iterations.

Why this works: Trees naturally find feature non-linearities and interactions. Each tree corrects ranking errors in the previous ensemble, weighted by how much they hurt NDCG.

Hyperparameters

Why LambdaMART Dominates

• Speed: Training is fast (seconds to minutes for millions of query-document pairs). Inference is <1ms per query.
• Accuracy: Listwise optimization (NDCG weighting) + tree expressiveness. Hard to beat in practice.
• Interpretability: Feature importance is clear. Feature interactions learned automatically.
• Robustness: Trees handle outliers and missing values naturally.
• Maturity: Many implementations available (LightGBM, XGBoost, custom solutions at big companies).

Neural Learning to Rank

Neural networks can replace trees for ranking. Advantages: end-to-end learning, semantic understanding. Disadvantages: slower inference, more data hungry.

DSSM (Deep Structured Semantic Model)

Architecture:

• Query encoder: MLP that maps query text/features to embedding.
• Document encoder: MLP that maps document text/features to embedding.
• Relevance score = cosine similarity between embeddings.

Training: Pointwise or pairwise loss on embeddings.

Inference: Pre-compute document embeddings once. For a query, compute embedding once and score all cached docs via dot product (very fast).

BERT Cross-Encoder Reranker

Architecture:

[CLS] query [SEP] document [SEP] → BERT → relevance score

• BERT sees query and document together, enabling interaction awareness.
• Output: [CLS] token passed through MLP → relevance score (0–1).

Training: Pairwise loss: margin loss = max(0, 1 - score_positive + score_negative).

Inference latency: ~50–200ms per query (depends on document count). Slower than LambdaMART but more accurate for small result sets.

Production use: Use LambdaMART to rank top-1000 candidates, then BERT cross-encoder to rerank top-10 for display (cascade).

Distillation

Idea: Train a small, fast model to mimic a large, accurate model.

Process:

1. Train a large BERT cross-encoder on labeled data (teacher).
2. Run teacher on all unlabeled data. Soft labels = model confidence.
3. Train a small model (smaller BERT, neural net) on soft labels (student).
4. Student learns to mimic teacher's scoring without full BERT overhead.

Achieves 80-90% of teacher's accuracy at 10-20% of latency.

Neural vs. LambdaMART

See Hybrid Search & Reranking Guide for more on BERT-based reranking.

Training Data: Judgments, Clicks & Debiasing

The quality of training data determines the quality of the ranking model. Two sources: human judgments and click signals. Each has pros and cons.

Human Judgments

Process:

• Curate representative queries.
• Retrieve top documents per query (maybe 100-1000).
• Human judges rate relevance: 0 (bad), 1 (fair), 2 (good), 3 (perfect).
• Resolve disagreements via consensus or majority vote.

Advantages: High quality, no position bias, explicit labels.

Disadvantages: Expensive ($1000s for decent set), time-consuming, limited coverage (only labeled queries).

Click Data

Process: Log user clicks from search engine. Assume clicked documents are relevant, unclicked = not relevant.

Advantages: Free, massive scale, real user feedback.

Critical flaw: Position bias. Users are more likely to click documents shown higher, regardless of relevance. A bad document at position 1 gets more clicks than an excellent document at position 10.

Click Models for Debiasing

Position Bias Model (PBM): Assumes click probability factors into examination probability and relevance:

P(click_i) = P(exam_i) × P(relevant_i)

Where P(exam_i) depends only on position (position bias), and P(relevant_i) depends on the document.

Algorithm: EM iterations to estimate both biases. Once estimated, reweight clicks by 1/P(exam_i) (Inverse Propensity Weighting, IPS).

Dynamic Bayesian Network (DBN): More sophisticated. Models user behavior: examine position 1, if not satisfied examine position 2, etc. Captures examination bias more accurately.

Unbiased LTR

Idea: Train models to predict true relevance, not click probability. Use debiased click labels:

Unbiased label = P(relevant | click, position) ∝ P(click | relevant) × P(relevant)

This inverts position bias, giving true relevance estimates independent of position.

Online Evaluation: Interleaving

Problem: Offline metrics don't tell you if a new ranking model is better in production. Users might interact differently.

Interleaving: Show results from model A and model B interleaved (alternating positions). Count wins: how often users click on docs from A vs. B?

Advantage: Directly measures user preference without explicit labels.

Disadvantage: Slower than offline evaluation. Need thousands of interleaved sessions for statistical significance.

When to Use What: Comparison & Architecture

Choosing an LTR approach depends on data size, latency constraints, and accuracy requirements.

Method Comparison

Production Architecture: The Cascade

Most production systems use a cascade: fast rough ranking → slow accurate ranking.

Decision Tree: Choosing Your Approach

Start with: LambdaMART on BM25 features. Train on human judgments or debiased clicks. Evaluate on NDCG@10.

If latency allows (>50ms per query): Add BERT cross-encoder on top-100 candidates.

If you have massive data (millions of queries): Consider neural methods (DSSM, BERT) for better semantic understanding.

For cold-start (no training data): Use rule-based features + pointwise regression to bootstrap. Collect human judgments. Graduate to pairwise/listwise.

Benchmarks: LETOR & MSLR-WEB

LETOR: Public benchmark. Queries from Microsoft Bing. Small (~30K documents per query). Standard for academic papers.

MSLR-WEB10K & WEB30K: Larger benchmarks from Microsoft. Millions of query-document pairs. More representative of production. NDCG@10 is standard metric.

Typical performance:

• Random baseline (NDCG@10): ≈0.40
• BM25 baseline: ≈0.60
• LambdaMART state-of-art: ≈0.74

Python Implementation: Complete LTR Pipeline

Build a complete learning-to-rank system from scratch using NumPy and Pandas. This code demonstrates the concepts without heavy dependencies.

NDCG Evaluator

import numpy as np
from typing import List, Tuple

class NDCGEvaluator:
    """Compute DCG, NDCG@K, and MRR for ranked lists."""

    @staticmethod
    def dcg(relevances: np.ndarray, k: int = 10) -> float:
        """Discounted Cumulative Gain @ K.

        Args:
            relevances: array of relevance scores (0, 1, 2, 3, ...)
            k: cutoff position

        Returns:
            DCG@K value
        """
        rel_k = relevances[:k]
        positions = np.arange(1, len(rel_k) + 1)
        discounts = 1.0 / np.log2(positions + 1)
        return np.sum(rel_k * discounts)

    @staticmethod
    def idcg(relevances: np.ndarray, k: int = 10) -> float:
        """Ideal DCG (best possible ranking)."""
        ideal_rel = np.sort(relevances)[::-1]
        return NDCGEvaluator.dcg(ideal_rel, k)

    @staticmethod
    def ndcg(relevances: np.ndarray, k: int = 10) -> float:
        """Normalized DCG @ K."""
        dcg_val = NDCGEvaluator.dcg(relevances, k)
        idcg_val = NDCGEvaluator.idcg(relevances, k)
        return dcg_val / idcg_val if idcg_val > 0 else 0.0

    @staticmethod
    def mrr(relevances: np.ndarray) -> float:
        """Mean Reciprocal Rank of first relevant doc."""
        for i, rel in enumerate(relevances):
            if rel > 0:
                return 1.0 / (i + 1)
        return 0.0

    @staticmethod
    def evaluate_ranking(
        true_relevances: np.ndarray,
        predicted_scores: np.ndarray,
        k: int = 10
    ) -> dict:
        """Full ranking evaluation.

        Args:
            true_relevances: ground truth relevance labels
            predicted_scores: model predictions
            k: cutoff

        Returns:
            dict with NDCG, MRR, and ranking
        """
        # Sort by predicted scores
        ranking_idx = np.argsort(predicted_scores)[::-1]
        ranked_rel = true_relevances[ranking_idx]

        return {
            'ndcg@10': NDCGEvaluator.ndcg(ranked_rel, 10),
            'ndcg@5': NDCGEvaluator.ndcg(ranked_rel, 5),
            'mrr': NDCGEvaluator.mrr(ranked_rel),
            'ranked_indices': ranking_idx,
            'ranked_relevances': ranked_rel
        }

Pointwise LTR: Logistic Regression

class PointwiseLTR:
    """Pointwise ranking via logistic regression.

    Treats each document independently. Predicts relevance probability.
    """

    def __init__(self, learning_rate: float = 0.01, iterations: int = 100):
        self.lr = learning_rate
        self.iterations = iterations
        self.weights = None
        self.bias = 0.0

    def sigmoid(self, z: np.ndarray) -> np.ndarray:
        return 1.0 / (1.0 + np.exp(-np.clip(z, -500, 500)))

    def fit(self, X: np.ndarray, y: np.ndarray):
        """Train on query-document feature vectors.

        Args:
            X: feature matrix (n_samples, n_features)
            y: binary relevance labels {0, 1}
        """
        n_samples, n_features = X.shape
        self.weights = np.zeros(n_features)

        for _ in range(self.iterations):
            # Forward pass
            z = X @ self.weights + self.bias
            predictions = self.sigmoid(z)

            # Gradient
            error = predictions - y
            grad_w = (X.T @ error) / n_samples
            grad_b = np.mean(error)

            # Update
            self.weights -= self.lr * grad_w
            self.bias -= self.lr * grad_b

    def predict(self, X: np.ndarray) -> np.ndarray:
        """Score documents."""
        z = X @ self.weights + self.bias
        return self.sigmoid(z)

Pairwise LTR: RankNet-Style

class PairwiseLTR:
    """Pairwise ranking: learn from document pairs.

    Loss: binary cross-entropy on whether doc_i > doc_j.
    """

    def __init__(self, learning_rate: float = 0.01, iterations: int = 100):
        self.lr = learning_rate
        self.iterations = iterations
        self.weights = None
        self.bias = 0.0

    def sigmoid(self, z: np.ndarray) -> np.ndarray:
        return 1.0 / (1.0 + np.exp(-np.clip(z, -500, 500)))

    def create_pairs(self, X: np.ndarray, y: np.ndarray) -> Tuple:
        """Generate pairwise training examples.

        For each query, create pairs (i, j) where rel_i > rel_j.
        """
        pairs = []
        labels = []

        for i in range(len(y)):
            for j in range(i + 1, len(y)):
                if y[i] != y[j]:
                    pairs.append((X[i], X[j]))
                    labels.append(1 if y[i] > y[j] else 0)

        return pairs, labels

    def fit(self, X: np.ndarray, y: np.ndarray):
        """Train on pairwise examples.

        Args:
            X: feature matrix
            y: relevance labels (0, 1, 2, 3, ...)
        """
        pairs, pair_labels = self.create_pairs(X, y)
        n_features = X.shape[1]
        self.weights = np.zeros(n_features)

        for _ in range(self.iterations):
            for (x_i, x_j), label in zip(pairs, pair_labels):
                # Score difference
                s_i = x_i @ self.weights + self.bias
                s_j = x_j @ self.weights + self.bias
                s_diff = s_i - s_j

                # Sigmoid probability
                p_ij = self.sigmoid(s_diff)

                # Loss gradient: cross-entropy on pairwise preference
                error = p_ij - label

                # Update weights
                grad = error * (x_i - x_j)
                self.weights -= self.lr * grad / len(pairs)

    def predict(self, X: np.ndarray) -> np.ndarray:
        """Score documents."""
        return X @ self.weights + self.bias

Simplified LambdaMART-style GBDT

class SimplifiedLambdaMART:
    """Gradient boosting with LambdaRank gradients.

    Simplified: uses decision stumps instead of full trees.
    Demonstrates the core LambdaRank concept.
    """

    def __init__(self, n_trees: int = 50, learning_rate: float = 0.1):
        self.n_trees = n_trees
        self.lr = learning_rate
        self.trees = []
        self.predictions = None

    def compute_lambda_gradients(
        self,
        X: np.ndarray,
        y: np.ndarray,
        current_scores: np.ndarray
    ) -> np.ndarray:
        """Compute LambdaRank gradients.

        For each document pair, weight gradient by NDCG change.
        Simplified: use 1 / (1 + exp(-(s_i - s_j))) as gradient.
        """
        n = len(y)
        gradients = np.zeros(n)

        # For each pair of documents with different labels
        for i in range(n):
            for j in range(i + 1, n):
                if y[i] != y[j]:
                    # RankNet gradient
                    s_diff = current_scores[i] - current_scores[j]
                    p_ij = 1.0 / (1.0 + np.exp(-np.clip(s_diff, -500, 500)))

                    # Target: 1 if y_i > y_j, else 0
                    target = 1.0 if y[i] > y[j] else 0.0
                    error = p_ij - target

                    # Approximate NDCG change (simplified)
                    ndcg_change = abs(y[i] - y[j]) / (n * np.log2(10))
                    lambda_ij = error * ndcg_change

                    # Accumulate gradients
                    if y[i] > y[j]:
                        gradients[i] += lambda_ij
                        gradients[j] -= lambda_ij
                    else:
                        gradients[i] -= lambda_ij
                        gradients[j] += lambda_ij

        return gradients

    def fit(self, X: np.ndarray, y: np.ndarray):
        """Train ensemble.

        Args:
            X: feature matrix (n_samples, n_features)
            y: relevance labels
        """
        self.predictions = np.zeros(len(y))

        for t in range(self.n_trees):
            # Compute lambda gradients
            gradients = self.compute_lambda_gradients(X, y, self.predictions)

            # Fit a simple linear model to gradients
            feature_idx = np.random.randint(X.shape[1])
            x_feat = X[:, feature_idx]

            # Threshold based on feature
            threshold = np.median(x_feat)
            direction = 1.0 if np.corrcoef(x_feat, gradients)[0, 1] > 0 else -1.0

            # Tree prediction: left/right based on threshold
            tree_pred = np.where(x_feat > threshold, direction, -direction)

            # Update ensemble
            self.predictions += self.lr * tree_pred
            self.trees.append((feature_idx, threshold, direction))

    def predict(self, X: np.ndarray) -> np.ndarray:
        """Score documents."""
        scores = np.zeros(len(X))
        for feature_idx, threshold, direction in self.trees:
            tree_pred = np.where(
                X[:, feature_idx] > threshold, direction, -direction
            )
            scores += self.lr * tree_pred
        return scores

Complete LTR Pipeline

class LTRPipeline:
    """End-to-end learning to rank pipeline.

    Retrieval → Feature Extraction → Ranking Model → Evaluation
    """

    def __init__(self, ranker_type: str = 'lambdamart'):
        """
        Args:
            ranker_type: 'pointwise', 'pairwise', or 'lambdamart'
        """
        self.ranker_type = ranker_type
        if ranker_type == 'pointwise':
            self.ranker = PointwiseLTR()
        elif ranker_type == 'pairwise':
            self.ranker = PairwiseLTR()
        else:
            self.ranker = SimplifiedLambdaMART()

        self.evaluator = NDCGEvaluator()
        self.feature_scaler_mean = None
        self.feature_scaler_std = None

    def normalize_features(self, X: np.ndarray, fit: bool = False) -> np.ndarray:
        """Z-score normalization."""
        if fit:
            self.feature_scaler_mean = np.mean(X, axis=0)
            self.feature_scaler_std = np.std(X, axis=0) + 1e-8

        return (X - self.feature_scaler_mean) / self.feature_scaler_std

    def train(
        self,
        X_train: np.ndarray,
        y_train: np.ndarray
    ):
        """Train ranking model.

        Args:
            X_train: training features (n_samples, n_features)
            y_train: relevance labels
        """
        X_normalized = self.normalize_features(X_train, fit=True)
        self.ranker.fit(X_normalized, y_train)

    def rank_candidates(
        self,
        X_candidates: np.ndarray
    ) -> Tuple[np.ndarray, np.ndarray]:
        """Rank candidate documents.

        Returns:
            (ranked_indices, ranked_scores)
        """
        X_normalized = self.normalize_features(X_candidates, fit=False)
        scores = self.ranker.predict(X_normalized)

        # Sort by score (descending)
        ranking_idx = np.argsort(scores)[::-1]
        return ranking_idx, scores[ranking_idx]

    def evaluate(
        self,
        X_test: np.ndarray,
        y_test: np.ndarray
    ) -> dict:
        """Evaluate on test set.

        Returns:
            dict with metrics
        """
        ranking_idx, scores = self.rank_candidates(X_test)
        ranked_rel = y_test[ranking_idx]

        return {
            'ndcg@10': self.evaluator.ndcg(ranked_rel, 10),
            'ndcg@5': self.evaluator.ndcg(ranked_rel, 5),
            'mrr': self.evaluator.mrr(ranked_rel),
            'ranking': ranking_idx,
            'scores': scores
        }

Complete Working Example

# Synthetic dataset: 10 queries, 100 docs each
np.random.seed(42)
n_queries = 10
docs_per_query = 100
n_features = 5

# Generate synthetic data
X_all = []
y_all = []

for q in range(n_queries):
    # Features: BM25, length, embedding similarity, etc.
    X_q = np.random.randn(docs_per_query, n_features)

    # Relevance: correlated with first feature
    y_q = (
        (X_q[:, 0] > 0).astype(int) +  # BM25 match
        (X_q[:, 1] > 0.5).astype(int) * 2  # Long doc
    )
    y_q = np.clip(y_q, 0, 3)  # Graded relevance

    X_all.append(X_q)
    y_all.append(y_q)

# Flatten for training
X_train = np.vstack(X_all[:7])
y_train = np.hstack(y_all[:7])
X_test = np.vstack(X_all[7:])
y_test = np.hstack(y_all[7:])

# Train all three methods
methods = ['pointwise', 'pairwise', 'lambdamart']
results = {}

for method in methods:
    pipeline = LTRPipeline(ranker_type=method)
    pipeline.train(X_train, y_train)
    metrics = pipeline.evaluate(X_test, y_test)
    results[method] = metrics

    print(f"{method.upper()}:")
    print(f"  NDCG@10: {metrics['ndcg@10']:.4f}")
    print(f"  NDCG@5:  {metrics['ndcg@5']:.4f}")
    print(f"  MRR:     {metrics['mrr']:.4f}")
    print()

# Compare improvements
print("Comparison vs. Pointwise:")
pointwise_ndcg = results['pointwise']['ndcg@10']
for method in ['pairwise', 'lambdamart']:
    improvement = (
        (results[method]['ndcg@10'] - pointwise_ndcg) / pointwise_ndcg * 100
    )
    print(f"  {method}: +{improvement:.2f}%")

Key Takeaways from Implementation

• NDCGEvaluator: Always compute NDCG@K to evaluate ranking quality.
• PointwiseLTR: Simple baseline. Ignores ordering. Usually beats random but loses to pairwise/listwise.
• PairwiseLTR: Better than pointwise because it learns preferences. Scales to large candidate sets.
• LambdaMART: Industry standard combines gradient boosting with LambdaRank gradients. Hard to beat.
• Feature normalization: Critical for all methods. Z-score normalization or rank-based normalization.
• Ensemble importance: LambdaMART uses many weak learners. Each tree corrects previous mistakes.

Next Steps

This code is educational. For production:

• Use LightGBM or XGBoost with objective='rank:ndcg' for LambdaMART.
• Handle variable-length query result sets (different query → different number of docs).
• Implement group-level metrics (NDCG averaged across queries).
• Add cross-validation for hyperparameter tuning.
• Integrate feature generation from actual search index (BM25, embeddings, etc.).