Wide & Deep + DIN

User ID: 12345
Age: 28
Country: US
Device: mobile
Context Time: 14:30 (2 PM)
Candidate Item ID: 98765
Item Category: Electronics
Item Price Range: $100-500
Item Language: English
User History: [item_56, item_23, item_91, item_77]
Clicked? YES (1)

Feature Engineering:
  user_country:item_language = US:English
  user_age_bucket:item_price_bucket = 25-30:mid
  device:time_of_day = mobile:afternoon

y_wide = σ(w^T [x, φ(x)] + b)

Where:
  x          = raw input features (sparse and dense)
  φ(x)       = cross-product transformation features
  w          = learned weights
  b          = bias term
  σ(·)       = sigmoid activation (for binary CTR)

Deep Model:
  Input: sparse categorical features (user_id, item_id, category, ...)

  Embedding Layer:
    user_embedding = embed(user_id)      ∈ R^d_user
    item_embedding = embed(item_id)      ∈ R^d_item
    category_embedding = embed(category) ∈ R^d_category
    ...
    concatenated = [user_emb, item_emb, cat_emb, ...]  ∈ R^D

  Hidden Layers:
    a^(1) = ReLU(W^(1) · concatenated + b^(1))
    a^(2) = ReLU(W^(2) · a^(1) + b^(2))
    a^(3) = ReLU(W^(3) · a^(2) + b^(3))
    ...

  Output:
    y_deep = σ(w_out^T · a^(L) + b_out)

y = σ(y_wide + y_deep)
  = σ(w_wide^T [x, φ(x)] + b_wide + w_deep^T a^(L) + b_deep)

Alternative formulation:
y = σ(w_wide^T [x, φ(x)] + w_deep^T a^(L) + b)

The outputs are summed before sigmoid, so both contribute equally
to the final decision boundary.

Forward Pass:
  y_wide = σ(w_wide^T [x, φ(x)] + b_wide)
  y_deep = σ(w_deep^T a^(L) + b_deep)
  y_pred = σ(y_wide + y_deep)

Loss:
  L = -[y_true · log(y_pred) + (1 - y_true) · log(1 - y_pred)]

Backpropagation:
  ∂L/∂(y_wide) = ∂L/∂y_pred · ∂y_pred/∂(y_wide)
  ∂L/∂(y_deep) = ∂L/∂y_pred · ∂y_pred/∂(y_deep)

Both components receive gradient signals!

y_pred = σ(y_wide + y_deep)

L = -1/N Σ[y_i · log(y_pred_i) + (1 - y_i) · log(1 - y_pred_i)]

Where:
  N = batch size
  y_i = true label (0 or 1: user clicked or not)
  y_pred_i = predicted probability from joint model

Gradient of L w.r.t. y_pred:
  ∂L/∂y_pred = y_pred - y_true

This gradient is then routed to both wide and deep components.

w_i = softmax(f(e_behavior_i, e_candidate))

Where:
  e_behavior_i       = embedding of behavior item i
  e_candidate        = embedding of candidate item
  f(·, ·)            = attention network (small MLP)
  softmax(·)         = ensures Σ w_i = 1

Then the interest representation is:
  u_interest = Σ w_i × e_behavior_i

Attention Network (Local Activation Unit):
  Input: concat(e_behavior_i, e_candidate)       ∈ R^(2d)

  Layer 1: h = ReLU(W^(1) · input + b^(1))       ∈ R^80
  (80 is a typical hidden size for attention)

  Layer 2: logits = W^(2) · h + b^(2)           ∈ R^1
  (Single output: raw attention score)

  Output (per-item): w_i = exp(logits_i) / Σ_j exp(logits_j)
                           (softmax across all history items)

Standard ReLU:
  y = max(0, x)
  Problem: Hard cutoff; always zero for negative inputs

Dice (Data-Adaptive PReLU):
  y = p(x) · x + (1 - p(x)) · α · x
  Where p(x) = sigmoid(α_p · (x - μ(x)) / σ(x))

Benefits:
  - Adaptive: threshold depends on batch statistics
  - Smoother: learns α parameter (slope for negatives)
  - Data-adaptive: personalized per mini-batch
  - Reduces information loss in deep networks

L2 Regularization in DIN:
  L_total = L_loss + λ · L_reg

  Where L_reg accounts for:
    - Feature frequency in the mini-batch
    - Embedding magnitude
    - Sparse feature sparsity patterns

  This prevents overfitting to frequent features
  while allowing rare features to contribute.

L = -1/N Σ[y_i · log(ŷ_i) + (1 - y_i) · log(1 - ŷ_i)]

Where:
  ŷ_i = σ(concat(interest_representation, other_features) → DNN output)
  y_i = ground truth label (click: 1, no-click: 0)

Gradient flows through:
  1. Output sigmoid
  2. DNN layers
  3. Attention weights (how much each history item contributes)
  4. Embedding layer
  5. Regularization terms

import numpy as np
import pandas as pd
from sklearn.preprocessing import LabelEncoder

def generate_ctr_dataset(n_samples=50000):
    """Generate synthetic CTR data with user features, item features, and behavior."""
    np.random.seed(42)

    data = {
        'user_id': np.random.randint(0, 5000, n_samples),
        'item_id': np.random.randint(0, 10000, n_samples),
        'user_country': np.random.choice(['US', 'UK', 'JP', 'DE', 'CN'], n_samples),
        'item_category': np.random.choice(['Electronics', 'Clothing', 'Books',
                                            'Home', 'Sports'], n_samples),
        'user_age_bucket': np.random.choice(['18-24', '25-34', '35-44', '45-54', '55+'],
                                             n_samples),
        'item_price_range': np.random.choice(['cheap', 'mid', 'expensive'], n_samples),
        'device_type': np.random.choice(['mobile', 'desktop', 'tablet'], n_samples),
        'time_of_day': np.random.choice(['morning', 'afternoon', 'evening', 'night'],
                                         n_samples),
    }

    df = pd.DataFrame(data)

    # Generate synthetic labels with some structure
    # e.g., US users are more likely to buy Electronics
    click_prob = 0.1  # base rate

    # Country effect
    click_prob += np.where(df['user_country'] == 'US', 0.05, -0.01)

    # Category effect
    click_prob += np.where(df['item_category'] == 'Electronics', 0.08, -0.02)

    # Price effect
    click_prob += np.where(df['item_price_range'] == 'mid', 0.03, -0.01)

    # Country:Category cross (memorization pattern)
    click_prob += np.where((df['user_country'] == 'JP') &
                           (df['item_category'] == 'Electronics'), 0.15, 0)

    click_prob = np.clip(click_prob, 0.01, 0.95)
    df['clicked'] = np.random.binomial(1, click_prob)

    return df

df = generate_ctr_dataset(50000)
print(f"Dataset shape: {df.shape}")
print(f"Click rate: {df['clicked'].mean():.2%}")
print(df.head())

class WideAndDeepModel:
    """
    Wide & Deep Learning model for CTR prediction.

    Wide component: Linear model with manually engineered cross features
    Deep component: MLP on dense embeddings
    """

    def __init__(self, embedding_dim=32, hidden_sizes=(128, 64), learning_rate=0.01):
        self.embedding_dim = embedding_dim
        self.hidden_sizes = hidden_sizes
        self.learning_rate = learning_rate

        # Embeddings for categorical features
        self.embeddings = {}  # feature_name -> {value: embedding_vector}
        self.feature_names = ['user_id', 'item_id', 'user_country', 'item_category',
                              'user_age_bucket', 'item_price_range', 'device_type',
                              'time_of_day']

        # Initialize embeddings randomly
        for feat in self.feature_names:
            self.embeddings[feat] = {}

        # Wide component weights (for cross features)
        self.wide_weights = {}  # cross_feature_name -> weight
        self.wide_bias = 0.0

        # Deep component weights
        self.deep_weights = []  # list of weight matrices for hidden layers
        self.deep_biases = []

        # Initialize deep weights
        input_dim = len(self.feature_names) * embedding_dim
        prev_dim = input_dim
        for hidden_dim in hidden_sizes:
            self.deep_weights.append(np.random.randn(prev_dim, hidden_dim) * 0.01)
            self.deep_biases.append(np.zeros(hidden_dim))
            prev_dim = hidden_dim

        # Output layer
        self.output_weight = np.random.randn(prev_dim, 1) * 0.01
        self.output_bias = 0.0

    def get_embedding(self, feature_name, value):
        """Get or create embedding for a feature value."""
        if value not in self.embeddings[feature_name]:
            self.embeddings[feature_name][value] = (
                np.random.randn(self.embedding_dim) * 0.01
            )
        return self.embeddings[feature_name][value]

    def forward(self, sample):
        """Forward pass: compute y_wide and y_deep, then combine."""
        # Extract embeddings
        embeddings_list = []
        cross_features = {}

        for feat in self.feature_names:
            emb = self.get_embedding(feat, sample[feat])
            embeddings_list.append(emb)

        # Concatenate embeddings for deep component
        deep_input = np.concatenate(embeddings_list)  # shape: (embedding_dim * len(features),)

        # DEEP PATH: Forward through MLP
        deep_activation = deep_input
        for W, b in zip(self.deep_weights, self.deep_biases):
            deep_activation = np.dot(deep_activation, W) + b
            deep_activation = np.maximum(deep_activation, 0)  # ReLU

        y_deep = (np.dot(deep_activation, self.output_weight) +
                  self.output_bias)[0]

        # WIDE PATH: Manually engineered cross features
        cross_feat_key = f"{sample['user_country']}:{sample['item_category']}"
        if cross_feat_key not in self.wide_weights:
            self.wide_weights[cross_feat_key] = np.random.randn() * 0.01

        y_wide = self.wide_weights[cross_feat_key]

        # COMBINE: y = sigmoid(y_wide + y_deep)
        y_combined = 1.0 / (1.0 + np.exp(-(y_wide + y_deep)))

        return y_combined, y_wide, y_deep

    def train(self, df, epochs=5, batch_size=32):
        """Train the model on a DataFrame."""
        losses = []

        for epoch in range(epochs):
            epoch_loss = 0.0
            n_batches = len(df) // batch_size

            for batch_idx in range(n_batches):
                start = batch_idx * batch_size
                end = start + batch_size
                batch_df = df.iloc[start:end]

                batch_loss = 0.0
                for _, row in batch_df.iterrows():
                    y_true = row['clicked']
                    y_pred, y_wide, y_deep = self.forward(row.to_dict())

                    # Binary cross-entropy loss
                    eps = 1e-7
                    loss = -(y_true * np.log(y_pred + eps) +
                            (1 - y_true) * np.log(1 - y_pred + eps))
                    batch_loss += loss

                    # Simple gradient update (simplified; real implementation uses backprop)
                    # Gradient of loss w.r.t. predictions
                    grad = (y_pred - y_true)

                    # Update wide component
                    cross_feat_key = (f"{row['user_country']}:{row['item_category']}")
                    if cross_feat_key in self.wide_weights:
                        self.wide_weights[cross_feat_key] -= (
                            self.learning_rate * grad * 0.1
                        )

                    # Update bias (simplified)
                    self.output_bias -= self.learning_rate * grad * 0.01

                epoch_loss += batch_loss / batch_size

            avg_loss = epoch_loss / n_batches
            losses.append(avg_loss)
            print(f"Epoch {epoch+1}/{epochs} - Loss: {avg_loss:.4f}")

        return losses

    def predict_batch(self, df):
        """Predict on a batch of samples."""
        predictions = []
        for _, row in df.iterrows():
            y_pred, _, _ = self.forward(row.to_dict())
            predictions.append(y_pred)
        return np.array(predictions)


# Train Wide & Deep model
print("\n=== WIDE & DEEP MODEL ===")
model_wd = WideAndDeepModel(embedding_dim=16, hidden_sizes=(64, 32), learning_rate=0.01)
losses_wd = model_wd.train(df, epochs=3, batch_size=256)

# Evaluate
test_sample = df.iloc[0].to_dict()
pred, y_w, y_d = model_wd.forward(test_sample)
print(f"\nSample prediction: {pred:.4f}")
print(f"  Wide component: {y_w:.4f}")
print(f"  Deep component: {y_d:.4f}")
print(f"  Label: {test_sample['clicked']}")

class DINModel:
    """
    Deep Interest Network (DIN) for CTR prediction.

    Key feature: Attention-weighted pooling of user behavior history.
    """

    def __init__(self, embedding_dim=32, attention_hidden=64, hidden_sizes=(128, 64),
                 max_history_len=20, learning_rate=0.01):
        self.embedding_dim = embedding_dim
        self.attention_hidden = attention_hidden
        self.hidden_sizes = hidden_sizes
        self.max_history_len = max_history_len
        self.learning_rate = learning_rate

        # Embeddings
        self.embeddings = {}
        self.feature_names = ['user_id', 'item_id', 'user_country', 'item_category',
                              'user_age_bucket', 'item_price_range']

        for feat in self.feature_names:
            self.embeddings[feat] = {}

        # Attention network weights
        self.attention_w1 = np.random.randn(embedding_dim * 2, attention_hidden) * 0.01
        self.attention_b1 = np.zeros(attention_hidden)
        self.attention_w2 = np.random.randn(attention_hidden, 1) * 0.01
        self.attention_b2 = 0.0

        # Deep network weights (after attention pooling)
        input_dim = len(self.feature_names) * embedding_dim + embedding_dim
        prev_dim = input_dim
        self.deep_weights = []
        self.deep_biases = []

        for hidden_dim in hidden_sizes:
            self.deep_weights.append(np.random.randn(prev_dim, hidden_dim) * 0.01)
            self.deep_biases.append(np.zeros(hidden_dim))
            prev_dim = hidden_dim

        # Output layer
        self.output_weight = np.random.randn(prev_dim, 1) * 0.01
        self.output_bias = 0.0

    def get_embedding(self, feature_name, value):
        """Get or create embedding for a feature value."""
        if value not in self.embeddings[feature_name]:
            self.embeddings[feature_name][value] = (
                np.random.randn(self.embedding_dim) * 0.01
            )
        return self.embeddings[feature_name][value]

    def attention_unit(self, behavior_emb, candidate_emb):
        """
        Compute attention score between a behavior item and candidate.

        Args:
            behavior_emb: embedding of historical item (shape: embedding_dim)
            candidate_emb: embedding of candidate item (shape: embedding_dim)

        Returns:
            score: scalar attention score
        """
        # Concatenate
        concat_emb = np.concatenate([behavior_emb, candidate_emb])

        # MLP: concat_emb -> attention_hidden -> 1
        hidden = np.dot(concat_emb, self.attention_w1) + self.attention_b1
        hidden = np.maximum(hidden, 0)  # ReLU

        score = np.dot(hidden, self.attention_w2) + self.attention_b2
        return score[0]

    def forward(self, sample, user_history):
        """
        Forward pass with attention-weighted history.

        Args:
            sample: dict with current features
            user_history: list of historical item_ids (most recent first)

        Returns:
            y_pred: predicted CTR probability
        """
        # Get candidate embedding
        candidate_emb = self.get_embedding('item_id', sample['item_id'])

        # Get context embeddings (user, country, etc., except item which changes)
        context_embeddings = []
        for feat in ['user_id', 'user_country', 'user_age_bucket', 'item_price_range']:
            if feat in sample:
                emb = self.get_embedding(feat, sample[feat])
                context_embeddings.append(emb)

        # Get item_category embedding
        if 'item_category' in sample:
            emb = self.get_embedding('item_category', sample['item_category'])
            context_embeddings.append(emb)

        # Attention pooling over user history
        if user_history and len(user_history) > 0:
            # Truncate history to max length
            history = user_history[:self.max_history_len]

            # Get embeddings for historical items
            history_embeddings = [self.get_embedding('item_id', item_id)
                                 for item_id in history]

            # Compute attention scores
            scores = [self.attention_unit(hist_emb, candidate_emb)
                     for hist_emb in history_embeddings]

            # Softmax
            scores = np.array(scores)
            scores = np.exp(scores) / (np.sum(np.exp(scores)) + 1e-10)

            # Weighted sum
            interest_emb = np.zeros(self.embedding_dim)
            for score, hist_emb in zip(scores, history_embeddings):
                interest_emb += score * hist_emb
        else:
            # No history: use zero vector
            interest_emb = np.zeros(self.embedding_dim)

        # Concatenate: context + attention-pooled interest
        deep_input = np.concatenate(context_embeddings + [interest_emb])

        # Deep network forward pass
        deep_activation = deep_input
        for W, b in zip(self.deep_weights, self.deep_biases):
            deep_activation = np.dot(deep_activation, W) + b
            deep_activation = np.maximum(deep_activation, 0)  # ReLU

        # Output
        y_pred = np.dot(deep_activation, self.output_weight) + self.output_bias
        y_pred = 1.0 / (1.0 + np.exp(-y_pred[0]))  # Sigmoid

        return y_pred

    def train(self, df, user_histories, epochs=3, batch_size=32):
        """
        Train DIN model.

        Args:
            df: DataFrame with samples
            user_histories: dict {user_id: [list of historical item_ids]}
            epochs: number of training epochs
            batch_size: batch size
        """
        losses = []

        for epoch in range(epochs):
            epoch_loss = 0.0
            n_batches = len(df) // batch_size

            for batch_idx in range(n_batches):
                start = batch_idx * batch_size
                end = start + batch_size
                batch_df = df.iloc[start:end]

                batch_loss = 0.0
                for _, row in batch_df.iterrows():
                    user_id = row['user_id']
                    history = user_histories.get(user_id, [])

                    y_true = row['clicked']
                    y_pred = self.forward(row.to_dict(), history)

                    # Binary cross-entropy loss
                    eps = 1e-7
                    loss = -(y_true * np.log(y_pred + eps) +
                            (1 - y_true) * np.log(1 - y_pred + eps))
                    batch_loss += loss

                    # Gradient update (simplified)
                    grad = (y_pred - y_true)
                    self.output_bias -= self.learning_rate * grad * 0.01

                epoch_loss += batch_loss / batch_size

            avg_loss = epoch_loss / n_batches
            losses.append(avg_loss)
            print(f"Epoch {epoch+1}/{epochs} - Loss: {avg_loss:.4f}")

        return losses


# Create user histories for DIN training
print("\n=== CREATING USER HISTORIES ===")
user_histories = {}
for user_id in df['user_id'].unique():
    user_items = df[df['user_id'] == user_id]['item_id'].tolist()
    user_histories[user_id] = user_items[:20]  # Keep last 20 items per user

print(f"Created histories for {len(user_histories)} users")

# Train DIN model
print("\n=== DIN MODEL ===")
model_din = DINModel(embedding_dim=16, attention_hidden=32, hidden_sizes=(64, 32),
                     learning_rate=0.01)
losses_din = model_din.train(df, user_histories, epochs=3, batch_size=256)

# Evaluate
test_user_id = df.iloc[0]['user_id']
test_sample = df.iloc[0].to_dict()
test_history = user_histories.get(test_user_id, [])
pred_din = model_din.forward(test_sample, test_history)
print(f"\nSample DIN prediction: {pred_din:.4f}")
print(f"  History length: {len(test_history)}")
print(f"  Label: {test_sample['clicked']}")

from sklearn.metrics import roc_auc_score, log_loss

# Create test set
test_df = df.tail(5000).reset_index(drop=True)

print("\n=== MODEL COMPARISON ===\n")

# Wide & Deep predictions
print("Wide & Deep Model:")
pred_wd = model_wd.predict_batch(test_df)
auc_wd = roc_auc_score(test_df['clicked'], pred_wd)
ll_wd = log_loss(test_df['clicked'], pred_wd)
print(f"  AUC: {auc_wd:.4f}")
print(f"  Log Loss: {ll_wd:.4f}")

# DIN predictions
print("\nDIN Model:")
pred_din = []
for _, row in test_df.iterrows():
    user_id = row['user_id']
    history = user_histories.get(user_id, [])
    y = model_din.forward(row.to_dict(), history)
    pred_din.append(y)
pred_din = np.array(pred_din)

auc_din = roc_auc_score(test_df['clicked'], pred_din)
ll_din = log_loss(test_df['clicked'], pred_din)
print(f"  AUC: {auc_din:.4f}")
print(f"  Log Loss: {ll_din:.4f}")

print("\nBaseline (always predict mean CTR):")
baseline_pred = np.full(len(test_df), test_df['clicked'].mean())
auc_baseline = roc_auc_score(test_df['clicked'], baseline_pred)
ll_baseline = log_loss(test_df['clicked'], baseline_pred)
print(f"  AUC: {auc_baseline:.4f}")
print(f"  Log Loss: {ll_baseline:.4f}")

# Summary
print("\n=== SUMMARY ===")
print(f"Wide & Deep AUC improvement: {(auc_wd / auc_baseline - 1) * 100:.1f}%")
print(f"DIN AUC improvement: {(auc_din / auc_baseline - 1) * 100:.1f}%")
print(f"DIN vs Wide & Deep: {(auc_din / auc_wd - 1) * 100:+.1f}%")