AI/ML Deep Dive: Deep Learning Foundations

Backpropagation is the neural network's "distributed attribution." A forward pass is like a request flowing through a microservice call chain, producing a final response (prediction) and an error (loss). Backprop is like tracing blame backward through that chain after an incident: computing how much responsibility each service (parameter) bears for the final error, then fixing each in proportion. No magic—it's just one backward traversal of a computational graph.

Pain point: a network has hundreds of millions of parameters. How do you know which direction and how much to nudge each one to lower the loss? Brute force—perturbing each parameter one at a time—means hundreds of millions of forward passes. Impossible. Backprop uses the chain rule to compute gradients for all parameters in a single backward traversal, at roughly the cost of one forward pass.

The core math is one formula. For a path w → z → y → L, the sensitivity of the loss to weight w is:

Intuition: ∂L/∂w reads as "nudge w a tiny bit—how much does L move?" You can't compute it directly, but every local derivative is easy (z w.r.t. w, y w.r.t. z, each a single step). The chain rule decomposes the global problem into a product of local derivatives along the path—the same idea as distributed tracing splitting end-to-end latency into per-hop times. Why go backward? Compute ∂L/∂y near the loss first, then at each layer going back, reuse the previous layer's result, avoiding recomputation.

import numpy as np
# Hand-rolled backprop for one neuron: y = sigmoid(w*x + b)
x, y_true = 1.5, 1.0
w, b = 0.3, 0.0

# --- Forward: cache intermediates, the backward pass reuses them ---
z = w * x + b
y = 1 / (1 + np.exp(-z))          # sigmoid activation
L = 0.5 * (y - y_true) ** 2          # mean-squared error loss

# --- Backward: chain rule, segment by segment ---
dL_dy = (y - y_true)                 # ∂L/∂y
dy_dz = y * (1 - y)                  # sigmoid derivative
dz_dw = x                            # ∂z/∂w
grad_w = dL_dy * dy_dz * dz_dw       # product of three = ∂L/∂w

w -= 0.1 * grad_w                    # update via gradient (next concept)
print(f"grad_w={grad_w:.4f}  new_w={w:.4f}")

Gradient descent = descending a hill blindfolded. You can't see the global landscape, but you can feel which direction is steepest underfoot (the gradient), so you step in the steepest downhill direction, repeating until you reach the valley (minimum loss). Backend analogy: a feedback control loop—measure error, adjust parameters toward it, measure again, converging iteratively. "How big a step" is the learning rate.

Backprop gives each parameter's gradient (its blame direction), but a gradient only tells you where to go, not how far or how steadily. The optimizer handles that. The simplest update rule:

The gradient ∇L is "a vector of partial derivatives," pointing in the direction of steepest ascent, so we negate it to go down. Three key evolutions:

• SGD (stochastic gradient descent): instead of the full dataset (too expensive), estimate the gradient from a mini-batch each step. Like estimating a full-table aggregate by sampling—100x cheaper, at the cost of noisy gradients (a jittery path). But the noise actually helps you escape shallow traps;
• Momentum: give the descent some inertia. Take an exponential moving average (EMA) of past gradients—the same trick you use to smooth a monitoring metric—damping jitter and accelerating across flat regions;
• Adam: today's default optimizer. Maintains both an EMA of the gradient (1st moment m, momentum for direction) and an EMA of squared gradients (2nd moment v, each parameter's "volatility"), updating via m̂/(√v̂+ε)—giving each parameter an adaptive step size: volatile parameters take small steps, stable ones take large steps. Similar to TCP congestion control adapting its window from feedback.

import torch
# Fit y = 2x with PyTorch; compare the "descent" of SGD vs Adam
x = torch.linspace(-1, 1, 64).unsqueeze(1)
y = 2 * x
w = torch.zeros(1, 1, requires_grad=True)

opt = torch.optim.Adam([w], lr=0.1)  # swap for SGD([w], lr=0.1) to compare speed
for step in range(50):
    pred = x @ w
    loss = ((pred - y) ** 2).mean()  # MSE
    opt.zero_grad()                  # clear last step's grads (else they accumulate)
    loss.backward()                   # ← this is backprop: auto-computes gradients
    opt.step()                        # update w (θ ← θ − η·∇L)
print(f"learned w={w.item():.3f}  (target=2.0)")

An activation function is the non-linear "gate" wedged between layers. Without it, even a deep network collapses into a single layer—like a chain of pure pass-through proxies: no matter how many you stack, the whole thing is still one linear transform. The activation is like a transistor (a non-linear switch) in a circuit: that bit of non-linearity is exactly what lets the network express if/else-style complex logic, not just y = ax + b.

Key fact: a composition of linear functions is still linear. Two layers without activation, W₂(W₁x) = (W₂W₁)x, is two matrices multiplied into one—equivalent to a single layer. Stacking 100 layers is the same: pointless. Activations insert a non-linear kink after each layer, so the network can approximate arbitrarily complex functions. Three common ones:

• Sigmoid σ(x)=1/(1+e⁻ˣ): squashes input into (0,1). Problem: its derivative peaks at just 0.25, so in a deep net the chain rule multiplies a string of numbers ≤0.25 → gradient approaches 0, i.e. vanishing gradient, and deep layers stop learning. This is exactly the question left at the end of the last section;
• ReLU max(0,x): the modern default. Derivative is exactly 1 in the positive region, so products don't decay, curing vanishing gradients; computation is trivial (one comparison). Cost: in the negative region both output and derivative are 0, so a neuron stuck there can "die" permanently (dead ReLU) and never update again;
• GELU / SwiGLU: mainstream in the Transformer era. Shaped like a "smooth ReLU"—instead of a hard cut to 0, the negative region transitions smoothly, easing the dying problem with smoother gradients.

import numpy as np
# Intuition check: sigmoid derivatives vanish, ReLU's don't
sigmoid = lambda x: 1 / (1 + np.exp(-x))
d_sigmoid = lambda x: sigmoid(x) * (1 - sigmoid(x))
relu = lambda x: np.maximum(0, x)
d_relu = lambda x: (x > 0).astype(float)

# Simulate a 6-layer net: multiply each layer's activation derivative
xs = np.array([0.5, -0.3, 1.2, 0.8, -1.0, 0.6])
print("sigmoid product:", np.prod(d_sigmoid(xs)))  # → ~0.0007, nearly vanished
print("relu product:   ", np.prod(d_relu(xs)))     # → 1.0 (all positive) or 0 (a dead neuron)
# Conclusion: deep nets default to ReLU-family not by taste, but by math

Dropout is Chaos Engineering for neural networks. Netflix's Chaos Monkey randomly kills production nodes, forcing the whole system to not depend on any single point and to learn redundant fault tolerance. Dropout randomly "kills" a fraction of neurons during training, forcing the network to not rely on any single neuron to hold an answer—so it learns robust, distributed features instead of brittle rote memorization.

The pain is overfitting: the model scores near-perfect on the training set but collapses on real data—it memorized the training samples (noise included) rather than learning a generalizable rule. This is the machine version of Goodhart's law: "optimize a metric too hard and the metric loses meaning." Two go-to remedies:

• L2 regularization / weight decay: add a term λ·‖w‖² to the loss, fining large weights. After differentiation this is equivalent to multiplying weights by a number slightly less than 1 each step (decay). Intuition: a tax on "resource usage"—forcing the model to use small, distributed weights rather than a few huge ones that memorize specific samples. Small weights = smooth function = more likely to generalize;
• Dropout: during training each neuron is randomly zeroed with probability p (often 0.1–0.5), so every batch is a different "thinned network." Equivalent to training an exponential number of sub-networks at once and ensembling them at test time. At test time nothing is dropped, but outputs are scaled to keep the expectation consistent.

import torch.nn as nn
# Both regularizers: a Dropout layer + the optimizer's weight_decay (=L2)
model = nn.Sequential(
    nn.Linear(784, 256),
    nn.ReLU(),
    nn.Dropout(p=0.3),          # drop 30% of neurons during training
    nn.Linear(256, 10),
)
# weight_decay is the L2 strength λ (fines large weights)
opt = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-2)

model.train()   # train mode: Dropout active
# ... training loop ...
model.eval()    # eval mode: Dropout off, use the full network (critical! don't forget)

• Rumelhart, Hinton & Williams 1986 — Learning representations by back-propagating errors (Nature) — the founding backprop paper that launched modern neural nets
• Karpathy — Yes you should understand backprop — why you must grasp backprop by hand (with intuition for dead ReLUs and vanishing gradients), highly recommended
• Kingma & Ba 2014 — Adam: A Method for Stochastic Optimization — the original paper for today's default optimizer
• Srivastava et al. 2014 — Dropout (JMLR) — the pioneering work on Dropout for overfitting
• 3Blue1Brown — Neural Networks series — visualizes gradient descent and backprop beautifully, top-tier geometric intuition
• Goodfellow, Bengio & Courville — Deep Learning — the classic textbook; chapters 6–8 cover everything in this issue