XGBoost

The Big Picture — What XGBoost Does & Why

XGBoost stands for "eXtreme Gradient Boosting." It's an ensemble learning method that combines many weak learners (typically shallow decision trees) into a single strong predictor. The fundamental idea is sequential correction: each new tree learns from the mistakes of all previous trees.

Why XGBoost for AdTech?

In advertising prediction tasks (CTR prediction, conversion prediction, bid valuation), data is often:

• High-dimensional with many categorical features (browser type, device, user segment, creative ID, etc.)
• Imbalanced (conversions are rare events, maybe 1-5% of impressions convert)
• Noisy (user behavior is stochastic; same user context can lead to different outcomes)
• Structured with natural interactions (certain device × browser combinations perform differently)

XGBoost excels here because it:

• Captures non-linearities via tree splits—handles complex feature interactions naturally
• Reduces bias iteratively—starts with a weak initial prediction and keeps adding corrective trees
• Regularizes aggressively—avoids overfitting via L1/L2 penalties on leaf weights and tree complexity
• Handles missing values automatically—learns best direction for missing features
• Provides feature importance—tells us which ad features drive conversions

The XGBoost Algorithm at 30,000 Feet

1. Initialize: Start with a simple prediction (e.g., log-odds of base conversion rate)
2. Loop M times:
   • Compute residuals = (actual outcome) - (current ensemble prediction)
   • Fit a new tree to predict these residuals
   • Add the tree to the ensemble (scaled by learning rate)
3. Output: Sum of all tree predictions = final prediction

The "gradient" and "extreme" parts make this algorithm powerful and practical at scale—we'll dive deep into both.

AdTech Example: Conversion Prediction for Bing Ads

Let's use a concrete example: predicting whether a user will convert (purchase) after clicking an ad on Bing.

Why This Matters in AdTech

Advertisers pay based on performance. If our model correctly identifies high-converting ads, we can:

• Bid more aggressively on high-conversion-probability impressions (win more high-value inventory)
• Allocate budget efficiently to creatives with higher predicted conversion rates
• Reduce waste by deprioritizing low-conversion contexts

Features in Our Example

Throughout this guide, we'll use this example to illustrate how XGBoost learns to predict conversions from these features.

What is "Gradient" in Gradient Boosting — Role of Residuals

The Core Idea: Gradient Descent in Function Space

"Gradient boosting" is called that because it uses gradient descent to optimize the ensemble. But unlike standard gradient descent (which moves parameters in parameter space), boosting performs gradient descent in function space—the space of possible prediction functions.

Here's the conceptual insight:

• In standard ML, we minimize loss by adjusting weights (parameter space)
• In gradient boosting, we minimize loss by adding new trees (function space)

Gradient = Residual (for most loss functions)

The gradient of the loss function tells us in which direction to move to reduce loss. For boosting, each new tree fits the negative gradient of the loss function—which is often the residual.

Example 1: Regression (Mean Squared Error)

In AdTech context: If we're predicting conversion rate and our current ensemble predicts 0.3 for a user who actually converts (y=1), the residual is 0.7. The next tree learns to predict this 0.7 error.

Example 2: Binary Classification (Log Loss / Cross-Entropy)

This is why in XGBoost's classification mode, the residuals are label minus probability, not label minus prediction in odds space.

Visual: The Iterative Gradient Descent Process

Why This is Called "Gradient" Boosting

The term "gradient" emphasizes that we're following the gradient of the loss function to minimize it. Each iteration, we add a tree in the direction that reduces loss the most (locally). This is exactly what gradient descent does, but applied to the space of functions instead of parameters.

What is "Extreme" in XGBoost — Regularized Objective & Systems Optimizations

"eXtreme" in XGBoost refers to two dimensions: extreme regularization (preventing overfitting) and extreme computational efficiency (speed and scalability).

Part 1: Regularized Objective (Extreme Regularization)

Standard gradient boosting minimizes:

This can lead to overfitting—especially with many trees.

XGBoost adds regularization terms to the objective:

What this means:

• γT term: Penalizes trees with many leaves. High γ → simpler trees with fewer splits.
• ½λ||w||² term: Shrinks leaf weights toward 0. High λ → smaller adjustments per tree, more conservative updates.
• α||w||₁ term: Can drive leaf weights to exactly 0, pruning less important leaves.

Comparison: Standard vs XGBoost Objective

Second-Order Taylor Approximation (Why XGBoost Uses Hessian)

When building a new tree, XGBoost doesn't just look at gradients (first derivative); it also uses the Hessian (second derivative) of the loss function. This enables:

• More accurate loss approximation: Taylor expansion to 2nd order is more precise than 1st order
• Closed-form split gains: XGBoost can compute the optimal leaf weight analytically
• Faster convergence: Information about curvature helps the algorithm make bigger, smarter steps

Part 2: Extreme Computational Efficiency

Beyond regularization, "extreme" also means extreme efficiency. XGBoost is faster than naive gradient boosting because of:

How the First Tree Gets Created — Initial Prediction & First Residuals

Step 1: Compute Initial Prediction

XGBoost begins by making a simple, global prediction on all samples. This is the starting point for the boosting process.

For Regression (MSE):

Start with the average of all target values.

For Binary Classification (Log Loss):

In AdTech terms: if your historical conversion rate is 3%, the initial prediction is log(0.03 / 0.97) ≈ -3.48 in log-odds space. This is then transformed to probability via sigmoid: p = 1 / (1 + e^(-ŷ)) = 0.03.

Step 2: Compute Residuals (Pseudo-Residuals)

Now compute how far off the initial prediction is for each sample:

For Regression:

For Binary Classification:

AdTech Example: Conversion Prediction

p = 30 / 1000 = 0.03
ŷ₀ = log(0.03 / 0.97) = -3.48 (log-odds)
p̂₀ = 1 / (1 + e^3.48) = 0.03 (probability)

Step 3: Grow First Tree to Fit Residuals

The first decision tree is grown to predict the residuals, not the original labels. At each split, XGBoost chooses the feature and threshold that best reduce the residual variance (or more precisely, that maximize the gain formula we'll see in Section 7).

For the conversion example:

• Tree 1 might find: "Users on mobile (device='mobile') have different residuals than desktop users"
• Another split: "Users from US convert differently than other countries"
• The tree constructs leaf predictions that, when added to ŷ₀, give better predictions

Step 4: Add Tree Output to Ensemble

Once Tree 1 is built, we update the ensemble prediction:

Example: If ŷ₀ = -3.48 (prob 0.03) and Tree₁ predicts +0.5 for a mobile user with learning_rate=0.1:

ŷ₁ = -3.48 + 0.1 × 0.5 = -3.43
p̂₁ = sigmoid(-3.43) ≈ 0.032  ← slightly higher conversion prob

The tree pushed the probability from 3% to 3.2%. Small, cautious steps (controlled by learning_rate) prevent huge swings.

Visual: First Tree Creation Process

How Subsequent Trees Are Grown — Fitting Residuals Iteratively

The Boosting Loop (Rounds 2, 3, 4, ...)

After Tree 1 is added to the ensemble, the process repeats. Each new tree corrects the errors of all previous trees combined.

Round 2:

# Current predictions from Ensemble (Tree 0 + Tree 1)
ŷ₁ = ŷ₀ + η × Tree₁(x)
p̂₁ = sigmoid(ŷ₁)

# New residuals
residual₂ = y - p̂₁

# Grow Tree 2 to fit residual₂
# (Tree 2 finds different splits than Tree 1, focusing on remaining errors)

# Update ensemble
ŷ₂ = ŷ₁ + η × Tree₂(x)
p̂₂ = sigmoid(ŷ₂)

Key insight: Trees are not independent. Tree 2 only exists because of Tree 1. It specializes in fixing what Tree 1 missed.

Why This Reduces Bias

Bias is how far the average prediction is from the true value. Boosting reduces bias iteratively:

• Round 1: Start with base rate prediction (high bias, simple model)
• Round 2: Add Tree 1 corrections → bias reduces
• Round 3: Add Tree 2 corrections → bias reduces more
• Round k: Add Tree k corrections → bias keeps reducing

With enough rounds, the ensemble can eventually express very complex prediction functions.

Role of Learning Rate (Shrinkage)

The learning rate η (often called learning_rate or eta) controls step size:

Why shrink?

• Prevents overfitting: Takes small steps instead of giant leaps, leaving room for future trees to refine
• Improves generalization: Each tree has less power to overfit individual training samples
• Requires more trees: Lower η means you need more rounds (n_estimators) to converge, but the model is more robust

Round 1: ŷ₁ = -3.48 + 0.3 × 0.5 = -3.33  → p = 0.035
Round 2: ŷ₂ = -3.33 + 0.3 × 0.4 = -3.21  → p = 0.040
Round 3: ŷ₃ = -3.21 + 0.3 × 0.2 = -3.15  → p = 0.043
Round 4: Converges quickly but may overfit

Round 1: ŷ₁ = -3.48 + 0.05 × 0.5 = -3.465  → p = 0.0305
Round 2: ŷ₂ = -3.465 + 0.05 × 0.4 = -3.445 → p = 0.0310
Round 3: ŷ₃ = -3.445 + 0.05 × 0.2 = -3.435 → p = 0.0313
Round 4: Slower convergence but more robust

Convergence and Early Stopping

With infinite rounds, the training loss would eventually go to 0 (the model would memorize the training data). In practice, we stop when:

• n_estimators rounds are completed (e.g., stop after 100 trees)
• Validation loss stops improving for N consecutive rounds (early stopping)

Early stopping is critical in AdTech: with billions of samples, even tiny prediction adjustments matter. Stopping too late means overfitting to noise in the training data.

Visual: How Prediction Improves Over Iterations

Splitting Metrics — Gain Formula with Regularization

The XGBoost Gain Formula

When building a tree, we need to choose which feature to split on and where. XGBoost uses a specific gain formula that incorporates regularization:

This formula answers: "How much does this split improve the objective?"

Breaking Down the Formula

Part 1: Score Before Split

Part 2: Score After Split (Left + Right)

Part 3: Gain (Improvement)

Effect of Regularization Parameters

λ (L2 Regularization)

High λ → larger denominator → lower gain → fewer splits. Trees stay simple.

• λ = 0: Full splits allowed (risk: overfitting)
• λ = 1 (default): Moderate regularization
• λ = 10: Strong regularization, very simple trees

γ (Minimum Gain Threshold)

Before accepting a split, its gain must exceed γ.

• γ = 0 (default): Every split accepted if it helps training loss
• γ = 1: Only splits with gain > 1 are accepted (prunes weak splits)
• γ = 5: Very aggressive pruning

Comparison: Standard Gini/Entropy vs XGBoost Gain

AdTech Example: Evaluating a Split

Gradients in leaf: [0.97, -0.03, 0.97, -0.03, ...] (conversions and non-conversions)
GL = sum of gradients in left (mobile) = 25
GR = sum of gradients in right (non-mobile) = -5

HL = sum of Hessians in left (mobile) = 30   (probability * (1-probability))
HR = sum of Hessians in right (non-mobile) = 28

λ = 1.0 (default L2 regularization)
γ = 0 (no minimum gain threshold)

Score before split:
(GL+GR)² / (HL+HR+λ) = 20² / (58+1) = 400 / 59 = 6.78

Score after split:
GL²/(HL+λ) + GR²/(HR+λ) = 25²/(30+1) + 5²/(28+1)
                        = 625/31 + 25/29
                        = 20.16 + 0.86 = 21.02

Gain = ½(21.02 - 6.78) - 0 = 7.12

Key Parameters to Tune

XGBoost has 20+ tunable hyperparameters. Here are the most important ones, organized by category:

Category 1: Tree Structure (Control Model Complexity)

Practical Interpretation:

• High bias problem? → Increase max_depth or decrease min_child_weight
• High variance (overfitting)? → Decrease max_depth, increase min_child_weight or gamma

Category 2: Sampling (Add Stochasticity to Reduce Overfitting)

Why Sampling Helps:

By using only 80% of data/features per tree, each tree is slightly different. Similar to Random Forest, but applied to boosting. Reduces variance without sacrificing bias reduction much (since we have many trees).

Category 3: Regularization (Direct Penalty on Complexity)

Difference Between L1 and L2:

• L2 (lambda): Shrinks weights gradually. Rarely zeros them out. Good for most cases.
• L1 (alpha): Can drive weights to exactly 0, effectively pruning leaves. Useful for sparse feature spaces.

Category 4: Learning (Control Boosting Speed)

Learning Rate & n_estimators Trade-off:

learning_rate = 0.01
n_estimators = 500

Each tree adds very small adjustments (1% of residuals).
Need 500 rounds to converge.
More robust, less likely to overfit.
Slower training.

learning_rate = 0.3
n_estimators = 50

Each tree adds large adjustments (30% of residuals).
Only 50 rounds needed.
Faster training.
Higher risk of overfitting.

Tuning Strategy (Practical Workflow)

1. Start simple: Use defaults, train model, check train/val loss gap
   • If gap is large (overfitting): increase regularization (lambda, gamma, min_child_weight)
   • If gap is small and both losses are high (underfitting): decrease regularization, increase max_depth
2. Tune tree structure: Use GridSearch or RandomSearch on max_depth, min_child_weight
   • Typical: max_depth ∈ [3, 5, 7, 10], min_child_weight ∈ [1, 3, 5, 10]
3. Tune regularization: Sweep lambda, gamma, alpha
   • Typical: lambda ∈ [0, 0.5, 1, 5, 10]
4. Tune sampling: Add subsample and colsample_bytree to break ties
   • Typical: subsample ∈ [0.7, 0.8, 0.9, 1.0]
5. Tune learning rate: Final fine-tuning, often paired with n_estimators
   • Lower η → increase n_estimators proportionally

Parameter Summary Table

Bias-Variance Tradeoff — Why Boosting Reduces Bias

Core Concepts Refresher

Bias: How far the average prediction is from the true value. High bias = underfitting = model is too simple to capture the pattern.

Variance: How much predictions change with different training data. High variance = overfitting = model is too sensitive to training noise.

Total Error ≈ Bias² + Variance + Irreducible Noise

How Boosting Reduces Bias Iteratively

Key insight: Boosting starts with a weak, high-bias learner and iteratively reduces bias by adding corrective trees.

Visual: Bias-Variance Curves for Boosting

Boosting vs Random Forest: Different Bias-Variance Profiles

How to Tell If You Have a Bias or Variance Problem

XGBoost vs Random Forest vs Decision Trees — Comprehensive Comparison

Three Learning Paradigms

When to Use Each

Empirical Performance: Typical Accuracy Ranking

On balanced tabular data with good features:

1. XGBoost (or LightGBM, CatBoost): ~95% accuracy (best)
2. Random Forest: ~90% accuracy
3. Single Decision Tree: ~80% accuracy
4. Logistic Regression: ~75% accuracy

On imbalanced AdTech data (3% conversion rate):

1. XGBoost with proper class weights: Best AUC-ROC
2. Random Forest: Can struggle with class imbalance
3. Single Tree: Easily biased toward majority class

Advanced Features

1. Missing Value Handling

Unlike many algorithms, XGBoost doesn't require missing values to be imputed. Instead, it learns the best direction to send missing values at each split.

How It Works:

At each split, XGBoost tries both directions:
- Missing values go LEFT
- Missing values go RIGHT

It picks whichever direction reduces loss more.
This is learned during training, not preset.

Example:

Suppose we have a feature "user_browser" with 10% missing values. At the root split:

• Option A: Send missing browser values to the left child (with Chrome users)
• Option B: Send missing browser values to the right child (with Firefox users)

XGBoost evaluates both and picks the one with higher gain. If Option A has gain=5.2 and Option B has gain=4.1, XGBoost chooses Option A.

2. Column Sampling (Subsampling Features)

Similar to Random Forest's random feature selection, XGBoost has feature subsampling at multiple levels:

• colsample_bytree: Fraction of features to consider when building each tree. E.g., 0.8 means 80% of features per tree.
• colsample_bylevel: Fraction of features to consider at each depth level within a tree. Fine-grained control.

Benefits: Reduces overfitting, increases diversity of trees, sometimes improves generalization.

3. Early Stopping

Instead of training a fixed n_estimators trees, early stopping monitors validation loss and stops when it stops improving:

model.fit(
    X_train, y_train,
    eval_set=[(X_val, y_val)],
    early_stopping_rounds=10,  # Stop if val loss doesn't improve for 10 rounds
    verbose=False
)

This prevents overfitting: you train just enough trees to minimize validation error, not more.

4. Feature Importance

XGBoost provides multiple ways to measure feature importance:

A. Gain (Average Improvement)

B. Split Count (Frequency)

C. Cover (Hessian Sum)

AdTech Example: Feature Importance in Conversion Prediction

5. Custom Loss Functions

XGBoost allows custom loss functions if you define the gradient and Hessian:

def custom_loss(y_true, y_pred):
    gradient = # your custom gradient
    hessian = # your custom Hessian
    return gradient, hessian

# Use in training
model.fit(X, y, obj=custom_loss)

This enables domain-specific optimization. For example, in ads, you might care more about losing a high-value conversion than gaining a low-value one.

6. Scale_pos_weight (Handling Class Imbalance)

For imbalanced classification (e.g., 3% conversion rate), set scale_pos_weight to the ratio of negative to positive samples:

# If 97% non-conversions and 3% conversions:
scale_pos_weight = (1.0 - 0.03) / 0.03 = 97 / 3 ≈ 32.3

model = XGBClassifier(scale_pos_weight=32.3)

This automatically weights the loss so the minority class (conversions) gets appropriate weight in training. Without this, the model might predict 0% conversion rate for everything (because that minimizes training loss).

Python Implementation from Scratch

Here's a complete, class-based implementation of gradient boosting using only numpy and pandas. This shows the core algorithm without XGBoost's optimizations but with the key ideas: gradient descent, residual fitting, regularization.

import numpy as np
import pandas as pd
from sklearn.tree import DecisionTreeRegressor
from sklearn.metrics import mean_squared_error, log_loss

class SimpleDecisionTree:
    """Simplified decision tree for boosting (regression stumps or small trees)"""
    def __init__(self, max_depth=3):
        self.tree = DecisionTreeRegressor(max_depth=max_depth)

    def fit(self, X, y):
        self.tree.fit(X, y)

    def predict(self, X):
        return self.tree.predict(X)


class GradientBoostedTrees:
    """
    Gradient Boosting from scratch.
    Supports regression (MSE) and binary classification (log loss).
    """
    def __init__(self, n_estimators=100, learning_rate=0.1, max_depth=3,
                 loss='mse', random_state=None):
        """
        Args:
            n_estimators: number of boosting rounds
            learning_rate: shrinkage parameter (0 < eta <= 1)
            max_depth: depth of each tree
            loss: 'mse' for regression, 'logloss' for binary classification
            random_state: for reproducibility
        """
        self.n_estimators = n_estimators
        self.learning_rate = learning_rate
        self.max_depth = max_depth
        self.loss = loss
        self.random_state = random_state
        self.trees = []
        self.init_pred = None
        self.feature_importance_ = None

    def _sigmoid(self, x):
        """Sigmoid function for classification"""
        return 1 / (1 + np.exp(-np.clip(x, -500, 500)))

    def _log_loss_gradient(self, y, y_pred_proba):
        """Gradient of log loss: (y - p)"""
        return y - y_pred_proba

    def _mse_gradient(self, y, y_pred):
        """Gradient of MSE: (y - ŷ)"""
        return y - y_pred

    def fit(self, X, y, eval_set=None, early_stopping_rounds=None, verbose=False):
        """
        Train the gradient boosted ensemble.

        Args:
            X: feature matrix (n_samples, n_features)
            y: target vector (n_samples,)
            eval_set: optional (X_val, y_val) for early stopping
            early_stopping_rounds: stop if val loss doesn't improve for N rounds
            verbose: print progress
        """
        n_samples, n_features = X.shape

        # Initialize prediction
        if self.loss == 'mse':
            self.init_pred = np.mean(y)
            F = np.full(n_samples, self.init_pred)  # current prediction
        else:  # logloss
            p_pos = np.mean(y)
            # log odds
            self.init_pred = np.log(p_pos / (1 - p_pos + 1e-10))
            F = np.full(n_samples, self.init_pred)  # log odds

        # Track feature importance
        self.feature_importance_ = np.zeros(n_features)

        # Early stopping setup
        best_val_loss = np.inf
        best_round = 0
        patience = early_stopping_rounds or self.n_estimators

        for t in range(self.n_estimators):
            # Compute residuals (gradients)
            if self.loss == 'mse':
                residuals = y - F
            else:  # logloss
                proba = self._sigmoid(F)
                residuals = y - proba

            # Fit tree to residuals
            tree = SimpleDecisionTree(max_depth=self.max_depth)
            tree.fit(X, residuals)
            self.trees.append(tree)

            # Update predictions
            tree_pred = tree.predict(X)
            F = F + self.learning_rate * tree_pred

            # Track feature importance (from sklearn tree's feature_importances_)
            if hasattr(tree.tree, 'feature_importances_'):
                self.feature_importance_ += tree.tree.feature_importances_

            # Validation and early stopping
            if eval_set is not None:
                X_val, y_val = eval_set
                F_val = self._predict_raw(X_val)

                if self.loss == 'mse':
                    val_loss = mean_squared_error(y_val, F_val)
                else:
                    proba_val = self._sigmoid(F_val)
                    val_loss = log_loss(y_val, np.clip(proba_val, 1e-15, 1-1e-15))

                if verbose and (t + 1) % 10 == 0:
                    print(f"Round {t+1}: val_loss = {val_loss:.6f}")

                if val_loss < best_val_loss:
                    best_val_loss = val_loss
                    best_round = t
                elif t - best_round >= patience:
                    if verbose:
                        print(f"Early stopping at round {t}")
                    break

        # Normalize feature importance
        if np.sum(self.feature_importance_) > 0:
            self.feature_importance_ /= np.sum(self.feature_importance_)

    def _predict_raw(self, X):
        """Predict raw values (log odds for classification, raw value for regression)"""
        F = np.full(X.shape[0], self.init_pred)
        for tree in self.trees:
            F += self.learning_rate * tree.predict(X)
        return F

    def predict(self, X):
        """
        Predict values.
        For regression: returns raw predictions
        For classification: returns probabilities
        """
        F = self._predict_raw(X)
        if self.loss == 'logloss':
            return self._sigmoid(F)
        else:
            return F

    def predict_proba(self, X):
        """For classification: return [proba(0), proba(1)]"""
        if self.loss != 'logloss':
            raise ValueError("predict_proba only available for classification")
        proba = self.predict(X)
        return np.column_stack([1 - proba, proba])


# ============================================================================
# EXAMPLE: AdTech Conversion Prediction
# ============================================================================

def create_dummy_adtech_data(n_samples=10000, random_state=42):
    """Create synthetic ad impression data with conversion labels"""
    np.random.seed(random_state)

    # Features
    device = np.random.choice(['mobile', 'desktop'], n_samples)
    browser = np.random.choice(['chrome', 'safari', 'firefox'], n_samples)
    user_age = np.random.normal(35, 15, n_samples).astype(int)
    user_age = np.clip(user_age, 18, 80)
    ad_creative_id = np.random.choice(range(100), n_samples)
    country = np.random.choice(['US', 'UK', 'DE', 'FR', 'JP'], n_samples)
    hour_of_day = np.random.randint(0, 24, n_samples)

    # Create feature matrix (one-hot encoding)
    df = pd.DataFrame({
        'device_mobile': (device == 'mobile').astype(int),
        'browser_safari': (browser == 'safari').astype(int),
        'browser_firefox': (browser == 'firefox').astype(int),
        'user_age': user_age,
        'ad_creative_id': ad_creative_id,
        'country_uk': (country == 'UK').astype(int),
        'country_de': (country == 'DE').astype(int),
        'country_fr': (country == 'FR').astype(int),
        'country_jp': (country == 'JP').astype(int),
        'hour_of_day': hour_of_day,
    })

    # Generate labels with some signal
    X = df.values

    # Conversion probability depends on features
    prob = 0.03  # base rate
    prob += 0.01 * X[:, 0]  # mobile users slightly different
    prob += 0.005 * X[:, 3] / 50  # older users
    prob += 0.002 * X[:, 4] / 50  # creative ID (weak signal)
    prob = np.clip(prob, 0.01, 0.1)

    y = (np.random.rand(n_samples) < prob).astype(int)

    return df, y


if __name__ == '__main__':
    # Create data
    print("Creating synthetic AdTech dataset...")
    X, y = create_dummy_adtech_data(n_samples=5000)
    print(f"Dataset shape: {X.shape}, Positive rate: {y.mean():.2%}\n")

    # Split into train/val
    n_train = int(0.8 * len(X))
    X_train, X_val = X[:n_train], X[n_train:]
    y_train, y_val = y[:n_train], y[n_train:]

    # Train gradient boosted model
    print("Training gradient boosted model...")
    model = GradientBoostedTrees(
        n_estimators=50,
        learning_rate=0.1,
        max_depth=3,
        loss='logloss',  # binary classification
        random_state=42
    )

    model.fit(
        X_train, y_train,
        eval_set=(X_val, y_val),
        early_stopping_rounds=5,
        verbose=True
    )
    print()

    # Predictions
    y_pred = model.predict(X_val)
    val_loss = log_loss(y_val, np.clip(y_pred, 1e-15, 1-1e-15))
    print(f"Final validation log loss: {val_loss:.6f}")
    print(f"Avg predicted probability: {y_pred.mean():.4f}")
    print(f"Actual positive rate: {y_val.mean():.4f}\n")

    # Feature importance
    print("Feature Importance:")
    feature_names = X.columns
    importances = model.feature_importance_
    sorted_idx = np.argsort(importances)[::-1]
    for idx in sorted_idx[:5]:
        print(f"  {feature_names[idx]}: {importances[idx]:.4f}")