Deep Neural Networks

z = w₁·x₁ + w₂·x₂ + ... + wₙ·xₙ + b
a = σ(z)  (apply activation function)

z⁽¹⁾ = w⁽¹⁾·x + b⁽¹⁾
a⁽¹⁾ = σ(z⁽¹⁾)

z⁽²⁾ = w⁽²⁾·a⁽¹⁾ + b⁽²⁾
a⁽²⁾ = σ(z⁽²⁾)

...

z⁽ᴸ⁾ = w⁽ᴸ⁾·a⁽ᴸ⁻¹⁾ + b⁽ᴸ⁾
ŷ = σ(z⁽ᴸ⁾)  [output activation]

Loss(ŷ, y) = -[y·log(ŷ) + (1-y)·log(1-ŷ)]

If y=1 (conversion happened):
  Loss = -log(ŷ)
  • If ŷ=0.99: Loss ≈ 0.01 (good prediction)
  • If ŷ=0.01: Loss ≈ 4.6 (terrible prediction)

If y=0 (no conversion):
  Loss = -log(1-ŷ)
  • If ŷ=0.01: Loss ≈ 0.01 (good)
  • If ŷ=0.99: Loss ≈ 4.6 (terrible)

Loss(ŷ, y) = -Σᵢ yᵢ·log(ŷᵢ)

where ŷ comes from softmax: ŷᵢ = e^(zᵢ) / Σⱼ e^(zⱼ)

Loss(ŷ, y) = (ŷ - y)²
Cost = (1/N) Σᵢ (ŷᵢ - yᵢ)²

Layer 1: z⁽¹⁾ = w⁽¹⁾·x + b⁽¹⁾, a⁽¹⁾ = ReLU(z⁽¹⁾)
Layer 2: z⁽²⁾ = w⁽²⁾·a⁽¹⁾ + b⁽²⁾, a⁽²⁾ = ReLU(z⁽²⁾)
Output:  z_out = w_out·a⁽²⁾ + b_out, ŷ = σ(z_out)
Loss:    L = -[y·log(ŷ) + (1-y)·log(1-ŷ)]

∂L/∂ŷ = -[y/ŷ - (1-y)/(1-ŷ)] = (ŷ - y) / [ŷ(1-ŷ)]

∂ŷ/∂z_out = σ'(z_out) = ŷ(1-ŷ)

∂L/∂z_out = ∂L/∂ŷ · ∂ŷ/∂z_out = (ŷ - y)

∂L/∂w_out = ∂L/∂z_out · ∂z_out/∂w_out = (ŷ - y) · a⁽²⁾ᵀ
∂L/∂b_out = ∂L/∂z_out = (ŷ - y)

∂L/∂a⁽²⁾ = ∂L/∂z_out · ∂z_out/∂a⁽²⁾ = (ŷ - y) · w_outᵀ

∂L/∂z⁽²⁾ = ∂L/∂a⁽²⁾ · ∂a⁽²⁾/∂z⁽²⁾ = (ŷ - y) · w_outᵀ · ReLU'(z⁽²⁾)
           where ReLU'(z) = 1 if z > 0, else 0

∂L/∂w⁽²⁾ = ∂L/∂z⁽²⁾ · ∂z⁽²⁾/∂w⁽²⁾ = ∂L/∂z⁽²⁾ · a⁽¹⁾ᵀ
∂L/∂b⁽²⁾ = ∂L/∂z⁽²⁾

∂L/∂a⁽¹⁾ = ∂L/∂z⁽²⁾ · ∂z⁽²⁾/∂a⁽¹⁾ = ∂L/∂z⁽²⁾ · w⁽²⁾ᵀ

∂L/∂z⁽¹⁾ = ∂L/∂a⁽¹⁾ · ∂a⁽¹⁾/∂z⁽¹⁾ = ∂L/∂a⁽¹⁾ · ReLU'(z⁽¹⁾)

∂L/∂w⁽¹⁾ = ∂L/∂z⁽¹⁾ · xᵀ
∂L/∂b⁽¹⁾ = ∂L/∂z⁽¹⁾

w ← w - α · ∂L/∂w
b ← b - α · ∂L/∂b

where α is the learning rate (e.g., 0.001)

σ(z) = 1 / (1 + e^(-z))
σ'(z) = σ(z)·(1-σ(z))

Maximum value: σ'(0) = 0.25
For z far from 0 (saturated region): σ'(z) ≈ 0

σ(z) = 1 / (1 + e^(-z))
σ'(z) = σ(z)·(1 - σ(z)), max value 0.25

Output range: (0, 1)
When to use: Output layer for binary classification
Pros: Outputs probability, good historical choice
Cons: Saturates, causes vanishing gradients

tanh(z) = (e^z - e^(-z)) / (e^z + e^(-z)) = 2·σ(2z) - 1
tanh'(z) = 1 - tanh(z)², max value 1.0

Output range: (-1, 1)
When to use: Hidden layers in older models
Pros: Zero-centered (helps training), steeper gradient than sigmoid
Cons: Still saturates and causes vanishing gradients

ReLU(z) = max(0, z)
ReLU'(z) = 1 if z > 0, else 0

Output range: [0, ∞)
When to use: Hidden layers (default choice for deep networks)
Pros: No saturation for z>0, fast to compute, avoids vanishing gradients
Cons: "Dying ReLU" (dead neurons if learning rate too high)

LeakyReLU(z) = max(αz, z), where α is small (e.g., 0.01)
LeakyReLU'(z) = 1 if z > 0, else α

Output range: (-∞, ∞)
When to use: Hidden layers (improvement over ReLU)
Pros: No dying neurons, still efficient
Cons: Minimal practical improvement over ReLU in most cases

ELU(z) = z if z > 0, else α(e^z - 1)
ELU'(z) = 1 if z > 0, else α·e^z

Output range: (-α, ∞)
Pros: Smooth, near zero-centered for negative values
Cons: Slower to compute (exponential)

GELU(z) = z · Φ(z), where Φ is the standard normal CDF

Output range: approx (-0.17, z+0.84)
When to use: State-of-the-art language models, vision transformers
Pros: Smooth, better empirical performance than ReLU
Cons: More expensive to compute, less well-understood

Approach 1: Dropout during training, scale at inference
  During training:  a_dropped = a * (Bernoulli(1-p)) / (1-p)
  During inference: a (no dropout)

Approach 2: Inverted Dropout (more common)
  During training:  a_dropped = a * (Bernoulli(1-p)) / (1-p)
  During inference: a (no change needed)

During training (using batch statistics):
  z_normalized = (z - batch_mean) / sqrt(batch_var + ε)
  z_scaled = γ·z_normalized + β   [learnable parameters]

During inference (using running statistics):
  z_normalized = (z - running_mean) / sqrt(running_var + ε)
  z_scaled = γ·z_normalized + β

Effect: Reduces internal covariate shift, allows higher learning rates

L_total = L_original + λ·Σ(w²)

Gradient update:
  w ← w - α·∂L/∂w - 2αλ·w

Effect: Shrinks weights toward zero, prevents large weights

import torch

model = MyDNN()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

# TRAINING LOOP
model.train()  # Set to training mode
for epoch in range(100):
    for batch_x, batch_y in train_loader:
        # Forward pass
        predictions = model(batch_x)
        loss = criterion(predictions, batch_y)

        # Backward pass
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

# INFERENCE (serving)
model.eval()  # Set to inference mode
with torch.no_grad():  # Disable gradient computation for speed
    for batch_x in test_loader:
        predictions = model(batch_x)
        # Use predictions (no gradients)

import numpy as np
from scipy.special import softmax
import matplotlib.pyplot as plt

class DeepNeuralNetwork:
    """
    A fully-connected deep neural network with configurable architecture.
    Supports ReLU and Sigmoid activations, dropout, and L2 regularization.
    """

    def __init__(self, layer_sizes, activation='relu', dropout_rate=0.2, l2_lambda=0.0001):
        """
        Args:
            layer_sizes: list of ints, e.g. [784, 128, 64, 10]
                         First element is input size, last is output size
            activation: 'relu' or 'sigmoid' for hidden layers
            dropout_rate: probability of dropping a neuron (0 = no dropout)
            l2_lambda: L2 regularization coefficient
        """
        self.layer_sizes = layer_sizes
        self.activation = activation
        self.dropout_rate = dropout_rate
        self.l2_lambda = l2_lambda
        self.training_mode = True

        # Initialize weights and biases
        self.weights = []
        self.biases = []
        for i in range(len(layer_sizes) - 1):
            w = np.random.randn(layer_sizes[i], layer_sizes[i+1]) * 0.01
            b = np.zeros((1, layer_sizes[i+1]))
            self.weights.append(w)
            self.biases.append(b)

    def relu(self, z):
        return np.maximum(0, z)

    def relu_derivative(self, z):
        return (z > 0).astype(float)

    def sigmoid(self, z):
        return 1 / (1 + np.exp(-np.clip(z, -500, 500)))

    def sigmoid_derivative(self, z):
        s = self.sigmoid(z)
        return s * (1 - s)

    def forward(self, X):
        """
        Forward propagation. Returns cache for backprop.
        X shape: (batch_size, input_size)
        """
        self.cache = []
        A = X

        # Hidden layers
        for i in range(len(self.weights) - 1):
            Z = np.dot(A, self.weights[i]) + self.biases[i]
            if self.activation == 'relu':
                A_new = self.relu(Z)
            else:  # sigmoid
                A_new = self.sigmoid(Z)

            # Dropout
            if self.training_mode and self.dropout_rate > 0:
                mask = np.random.binomial(1, 1 - self.dropout_rate, A_new.shape)
                A_new = A_new * mask / (1 - self.dropout_rate)
            else:
                mask = None

            self.cache.append((A, Z, A_new, mask))
            A = A_new

        # Output layer (sigmoid for binary, softmax for multi-class)
        Z_out = np.dot(A, self.weights[-1]) + self.biases[-1]
        A_out = self.sigmoid(Z_out)

        self.cache.append((A, Z_out, A_out, None))
        return A_out

    def backward(self, X, Y, learning_rate):
        """
        Backpropagation. Computes gradients and updates weights.
        """
        m = X.shape[0]
        gradients_w = [np.zeros_like(w) for w in self.weights]
        gradients_b = [np.zeros_like(b) for b in self.biases]

        # Output layer gradient
        A_out = self.cache[-1][2]
        dZ = A_out - Y  # Binary cross-entropy gradient
        dA_prev = self.cache[-2][2]

        gradients_w[-1] = np.dot(dA_prev.T, dZ) / m
        gradients_b[-1] = np.sum(dZ, axis=0, keepdims=True) / m

        # Hidden layers (backpropagate)
        dA = dZ
        for i in range(len(self.weights) - 2, -1, -1):
            dA = np.dot(dA, self.weights[i + 1].T)

            # Apply dropout mask from forward pass
            if self.cache[i][3] is not None:
                dA = dA * self.cache[i][3] / (1 - self.dropout_rate)

            # Activation derivative
            Z = self.cache[i][1]
            if self.activation == 'relu':
                dA = dA * self.relu_derivative(Z)
            else:
                dA = dA * self.sigmoid_derivative(Z)

            A_prev = self.cache[i][0]
            gradients_w[i] = np.dot(A_prev.T, dA) / m
            gradients_b[i] = np.sum(dA, axis=0, keepdims=True) / m

        # Update weights with L2 regularization
        for i in range(len(self.weights)):
            self.weights[i] -= learning_rate * (gradients_w[i] + self.l2_lambda * self.weights[i])
            self.biases[i] -= learning_rate * gradients_b[i]

    def train_mode(self):
        self.training_mode = True

    def eval_mode(self):
        self.training_mode = False

    def predict(self, X):
        return self.forward(X)

    def loss(self, Y_pred, Y_true):
        """Binary cross-entropy loss"""
        m = Y_true.shape[0]
        epsilon = 1e-15
        Y_pred = np.clip(Y_pred, epsilon, 1 - epsilon)
        loss = -np.mean(Y_true * np.log(Y_pred) + (1 - Y_true) * np.log(1 - Y_pred))
        return loss

# ============================================================================
# TRAINING ON MNIST
# ============================================================================

from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

# Load MNIST digits (8x8 images = 64 features)
digits = load_digits()
X = digits.data / 16.0  # Normalize to [0, 1]
y = digits.target

# For simplicity, binary classification: digit 0 vs others
Y = (y == 0).astype(int).reshape(-1, 1)

# Split data
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.2, random_state=42)

# Standardize
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

# Create model: 64 inputs -> 128 hidden -> 64 hidden -> 1 output
model = DeepNeuralNetwork(
    layer_sizes=[64, 128, 64, 1],
    activation='relu',
    dropout_rate=0.3,
    l2_lambda=0.0001
)

# Training parameters
epochs = 100
learning_rate = 0.01
batch_size = 32

train_losses = []
val_losses = []

print("Training Deep Neural Network on MNIST (0 vs others)...")
for epoch in range(epochs):
    model.train_mode()

    # Mini-batch training
    indices = np.random.permutation(X_train.shape[0])
    epoch_loss = 0
    num_batches = 0

    for i in range(0, X_train.shape[0], batch_size):
        batch_indices = indices[i:i+batch_size]
        X_batch = X_train[batch_indices]
        Y_batch = Y_train[batch_indices]

        # Forward pass
        Y_pred = model.forward(X_batch)
        loss = model.loss(Y_pred, Y_batch)
        epoch_loss += loss
        num_batches += 1

        # Backward pass
        model.backward(X_batch, Y_batch, learning_rate)

    train_losses.append(epoch_loss / num_batches)

    # Validation
    model.eval_mode()
    Y_val_pred = model.forward(X_test)
    val_loss = model.loss(Y_val_pred, Y_test)
    val_losses.append(val_loss)

    if (epoch + 1) % 10 == 0:
        accuracy = np.mean((Y_val_pred > 0.5) == Y_test)
        print(f"Epoch {epoch+1:3d} | Train Loss: {train_losses[-1]:.4f} | "
              f"Val Loss: {val_loss:.4f} | Accuracy: {accuracy:.4f}")

print("Training complete!")

# ============================================================================
# RESULTS
# ============================================================================

model.eval_mode()
Y_pred = model.forward(X_test)
accuracy = np.mean((Y_pred > 0.5) == Y_test)
print(f"\nFinal Test Accuracy: {accuracy:.4f}")
print(f"Final Train Loss: {train_losses[-1]:.4f}")
print(f"Final Val Loss: {val_losses[-1]:.4f}")