diff --git a/phases/01-math-foundations/01-linear-algebra-intuition/docs/en.md b/phases/01-math-foundations/01-linear-algebra-intuition/docs/en.md
index 255124ee17..44a04e57c8 100644
--- a/phases/01-math-foundations/01-linear-algebra-intuition/docs/en.md
+++ b/phases/01-math-foundations/01-linear-algebra-intuition/docs/en.md
@@ -32,7 +32,7 @@ A vector is just a list of numbers. But those numbers mean something -- they're
|---|---|-------|
| 3 | 2 | The vector points from origin (0,0) to (3, 2) on the plane |
-The vector has magnitude sqrt(3^2 + 2^2) = sqrt(13) and points up and to the right.
+The vector has magnitude $\sqrt{3^2 + 2^2} = \sqrt{13}$ and points up and to the right.
In AI, vectors represent everything:
- A word → a vector of 768 numbers (its "meaning" in embedding space)
@@ -71,13 +71,17 @@ In AI, matrices ARE the model:
The dot product of two vectors tells you how similar they are.
-```
-a · b = a₁×b₁ + a₂×b₂ + ... + aₙ×bₙ
+$$
+\mathbf{a} \cdot \mathbf{b} = a_1 b_1 + a_2 b_2 + \cdots + a_n b_n
+$$
-Same direction: a · b > 0 (similar)
-Perpendicular: a · b = 0 (unrelated)
-Opposite direction: a · b < 0 (dissimilar)
-```
+$$
+\begin{aligned}
+\text{Same direction:} \quad & \mathbf{a} \cdot \mathbf{b} > 0 \quad (\text{similar}) \\
+\text{Perpendicular:} \quad & \mathbf{a} \cdot \mathbf{b} = 0 \quad (\text{unrelated}) \\
+\text{Opposite direction:} \quad & \mathbf{a} \cdot \mathbf{b} < 0 \quad (\text{dissimilar})
+\end{aligned}
+$$
This is literally how search engines, recommendation systems, and RAG work -- find vectors with high dot products.
@@ -95,25 +99,25 @@ v2 = [0, 1, 0]
v3 = [2, 1, 0] # v3 = 2*v1 + v2
```
-v1 and v2 are independent -- neither is a scalar multiple or combination of the other. But v3 = 2*v1 + v2, so {v1, v2, v3} is a dependent set. These three vectors all lie in the xy-plane. No matter how you combine them, you cannot reach [0, 0, 1]. You have three vectors but only two dimensions of freedom.
+v1 and v2 are independent -- neither is a scalar multiple or combination of the other. But $\mathbf{v}_3 = 2\mathbf{v}_1 + \mathbf{v}_2$, so $\{\mathbf{v}_1, \mathbf{v}_2, \mathbf{v}_3\}$ is a dependent set. These three vectors all lie in the xy-plane. No matter how you combine them, you cannot reach $[0, 0, 1]$. You have three vectors but only two dimensions of freedom.
-In a dataset: if feature_3 = 2*feature_1 + feature_2, adding feature_3 gives the model zero new information. Worse, it makes the normal equations singular -- there is no unique solution for the weights.
+In a dataset: if $\text{feature}_3 = 2 \cdot \text{feature}_1 + \text{feature}_2$, adding feature_3 gives the model zero new information. Worse, it makes the normal equations singular -- there is no unique solution for the weights.
### Basis and Rank
A basis is a minimal set of linearly independent vectors that span the entire space. The number of basis vectors is the dimension of the space.
-The standard basis for 3D space is {[1,0,0], [0,1,0], [0,0,1]}. But any three independent vectors in 3D form a valid basis. The choice of basis is a choice of coordinate system.
+The standard basis for 3D space is $\{[1,0,0], [0,1,0], [0,0,1]\}$. But any three independent vectors in 3D form a valid basis. The choice of basis is a choice of coordinate system.
-Rank of a matrix = number of linearly independent columns = number of linearly independent rows. If rank < min(rows, cols), the matrix is rank-deficient. This means:
+Rank of a matrix = number of linearly independent columns = number of linearly independent rows. If $\text{rank} < \min(\text{rows}, \text{cols})$, the matrix is rank-deficient. This means:
- The system has infinitely many solutions (or none)
- Information is lost in the transformation
- The matrix cannot be inverted
| Situation | Rank | What it means for ML |
|-----------|------|---------------------|
-| Full rank (rank = min(m, n)) | Maximum possible | Unique least-squares solution exists. Model is well-conditioned. |
-| Rank deficient (rank < min(m, n)) | Below maximum | Features are redundant. Infinitely many weight solutions. Regularization needed. |
+| Full rank ($\text{rank} = \min(m, n)$) | Maximum possible | Unique least-squares solution exists. Model is well-conditioned. |
+| Rank deficient ($\text{rank} < \min(m, n)$) | Below maximum | Features are redundant. Infinitely many weight solutions. Regularization needed. |
| Rank 1 | 1 | Every column is a scaled copy of one vector. All data lies on a line. |
| Near rank-deficient (small singular values) | Numerically low | Matrix is ill-conditioned. Tiny input noise causes large output changes. Use SVD truncation or ridge regression. |
@@ -121,11 +125,11 @@ Rank of a matrix = number of linearly independent columns = number of linearly i
Projecting vector **a** onto vector **b** gives the component of **a** in the direction of **b**:
-```
-proj_b(a) = (a dot b / b dot b) * b
-```
+$$
+\text{proj}_{\mathbf{b}}(\mathbf{a}) = \frac{\mathbf{a} \cdot \mathbf{b}}{\mathbf{b} \cdot \mathbf{b}} \, \mathbf{b}
+$$
-The residual (a - proj_b(a)) is perpendicular to b. This orthogonal decomposition is the foundation of least-squares fitting.
+The residual $\mathbf{a} - \text{proj}_{\mathbf{b}}(\mathbf{a})$ is perpendicular to $\mathbf{b}$. This orthogonal decomposition is the foundation of least-squares fitting.
Projection is everywhere in ML:
- Linear regression minimizes the distance from observations to the column space -- the solution IS a projection
@@ -143,9 +147,11 @@ graph LR
end
```
-**Example:** a = [3, 4], b = [1, 0]
+**Example:** $\mathbf{a} = [3, 4]$, $\mathbf{b} = [1, 0]$
-proj_b(a) = (3*1 + 4*0) / (1*1 + 0*0) * [1, 0] = 3 * [1, 0] = [3, 0]
+$$
+\text{proj}_{\mathbf{b}}(\mathbf{a}) = \frac{3 \cdot 1 + 4 \cdot 0}{1 \cdot 1 + 0 \cdot 0} \, [1, 0] = 3 \, [1, 0] = [3, 0]
+$$
The projection drops the y-component. This is dimensionality reduction in its simplest form -- throw away the directions you don't care about.
@@ -159,19 +165,17 @@ The algorithm:
3. Take the third vector, subtract its projections onto all previous vectors, normalize
4. Repeat for remaining vectors
-```
-Input: v1, v2, v3, ... (linearly independent)
-
-u1 = v1 / |v1|
-
-w2 = v2 - (v2 dot u1) * u1
-u2 = w2 / |w2|
-
-w3 = v3 - (v3 dot u1) * u1 - (v3 dot u2) * u2
-u3 = w3 / |w3|
-
-Output: u1, u2, u3, ... (orthonormal basis)
-```
+$$
+\begin{aligned}
+\text{Input:} \quad & \mathbf{v}_1, \mathbf{v}_2, \mathbf{v}_3, \ldots \ (\text{linearly independent}) \\[4pt]
+\mathbf{u}_1 &= \frac{\mathbf{v}_1}{\lVert \mathbf{v}_1 \rVert} \\[4pt]
+\mathbf{w}_2 &= \mathbf{v}_2 - (\mathbf{v}_2 \cdot \mathbf{u}_1)\, \mathbf{u}_1 \\
+\mathbf{u}_2 &= \frac{\mathbf{w}_2}{\lVert \mathbf{w}_2 \rVert} \\[4pt]
+\mathbf{w}_3 &= \mathbf{v}_3 - (\mathbf{v}_3 \cdot \mathbf{u}_1)\, \mathbf{u}_1 - (\mathbf{v}_3 \cdot \mathbf{u}_2)\, \mathbf{u}_2 \\
+\mathbf{u}_3 &= \frac{\mathbf{w}_3}{\lVert \mathbf{w}_3 \rVert} \\[4pt]
+\text{Output:} \quad & \mathbf{u}_1, \mathbf{u}_2, \mathbf{u}_3, \ldots \ (\text{orthonormal basis})
+\end{aligned}
+$$
This is how QR decomposition works internally. Q is the orthonormal basis, R captures the projection coefficients. QR decomposition is used in:
- Solving linear systems (more stable than Gaussian elimination)
diff --git a/phases/01-math-foundations/04-calculus-for-ml/docs/en.md b/phases/01-math-foundations/04-calculus-for-ml/docs/en.md
index 055fb57b5f..fa89a8aedb 100644
--- a/phases/01-math-foundations/04-calculus-for-ml/docs/en.md
+++ b/phases/01-math-foundations/04-calculus-for-ml/docs/en.md
@@ -9,7 +9,7 @@
## Learning Objectives
-- Compute numerical and analytical derivatives for common ML functions (x^2, sigmoid, cross-entropy)
+- Compute numerical and analytical derivatives for common ML functions ($x^2$, sigmoid, cross-entropy)
- Implement gradient descent from scratch to minimize a loss function in 1D and 2D
- Derive the gradient of a linear regression model and train it via manual weight updates
- Explain the Hessian matrix, Taylor series approximations, and their connection to optimization methods
@@ -24,11 +24,11 @@ Without calculus, training a neural network would mean trying random changes and
### What is a derivative?
-A derivative measures the rate of change. For a function y = f(x), the derivative f'(x) tells you: if you nudge x by a tiny amount, how much does y change?
+A derivative measures the rate of change. For a function $y = f(x)$, the derivative $f'(x)$ tells you: if you nudge $x$ by a tiny amount, how much does $y$ change?
Geometrically, the derivative is the slope of the tangent line at a point.
-**f(x) = x^2:**
+**$f(x) = x^2$:**
| x | f(x) | f'(x) (slope) |
|---|------|---------------|
@@ -37,15 +37,13 @@ Geometrically, the derivative is the slope of the tangent line at a point.
| 2 | 4 | 4 (tangent line slope at this point) |
| 3 | 9 | 6 |
-At x=2, the slope is 4. If you move x a tiny bit to the right, y increases by about 4 times that amount. At x=0, the slope is 0. You are at the bottom of the bowl.
+At $x=2$, the slope is 4. If you move $x$ a tiny bit to the right, $y$ increases by about 4 times that amount. At $x=0$, the slope is 0. You are at the bottom of the bowl.
The formal definition:
-```
-f'(x) = lim f(x + h) - f(x)
- h->0 -----------------
- h
-```
+$$
+f'(x) = \lim_{h \to 0} \frac{f(x + h) - f(x)}{h}
+$$
In code, you skip the limit and just use a very small h. That is the numerical derivative.
@@ -53,12 +51,13 @@ In code, you skip the limit and just use a very small h. That is the numerical d
Real functions have many inputs. A neural network loss depends on thousands of weights. A partial derivative holds all variables constant except one, then takes the derivative with respect to that one.
-```
-f(x, y) = x^2 + 3xy + y^2
-
-df/dx = 2x + 3y (treat y as a constant)
-df/dy = 3x + 2y (treat x as a constant)
-```
+$$
+\begin{aligned}
+f(x, y) &= x^2 + 3xy + y^2 \\
+\frac{\partial f}{\partial x} &= 2x + 3y \quad \text{(treat $y$ as a constant)} \\
+\frac{\partial f}{\partial y} &= 3x + 2y \quad \text{(treat $x$ as a constant)}
+\end{aligned}
+$$
Each partial derivative answers: if I nudge just this one weight, how does the loss change?
@@ -66,43 +65,43 @@ Each partial derivative answers: if I nudge just this one weight, how does the l
The gradient collects every partial derivative into one vector. For a function f(x, y, z), the gradient is:
-```
-grad f = [ df/dx, df/dy, df/dz ]
-```
+$$
+\nabla f = \left[ \frac{\partial f}{\partial x},\ \frac{\partial f}{\partial y},\ \frac{\partial f}{\partial z} \right]
+$$
The gradient points in the direction of steepest ascent. To minimize a function, go in the opposite direction.
-**Contour plot of f(x,y) = x^2 + y^2:**
+**Contour plot of $f(x,y) = x^2 + y^2$:**
The function forms a bowl shape with concentric circles as contour lines. The minimum is at (0, 0).
-| Point | grad f | -grad f (descent direction) |
+| Point | $\nabla f$ | $-\nabla f$ (descent direction) |
|-------|--------|----------------------------|
-| (1, 1) | [2, 2] (points uphill, away from minimum) | [-2, -2] (points downhill, toward minimum) |
-| (0, 0) | [0, 0] (flat, at the minimum) | [0, 0] |
+| $(1, 1)$ | $[2, 2]$ (points uphill, away from minimum) | $[-2, -2]$ (points downhill, toward minimum) |
+| $(0, 0)$ | $[0, 0]$ (flat, at the minimum) | $[0, 0]$ |
This is gradient descent in a picture. Compute the gradient, negate it, take a step.
### The connection to optimization
-Training a neural network is optimization. You have a loss function L(w1, w2, ..., wn) that measures how wrong the model is. You want to minimize it.
+Training a neural network is optimization. You have a loss function $L(w_1, w_2, \ldots, w_n)$ that measures how wrong the model is. You want to minimize it.
-```
Gradient descent update rule:
- w_new = w_old - learning_rate * dL/dw
+$$
+w_{\text{new}} = w_{\text{old}} - \text{learning\_rate} \cdot \frac{\partial L}{\partial w}
+$$
For every weight:
- 1. Compute the partial derivative of loss with respect to that weight
- 2. Subtract a small multiple of it from the weight
- 3. Repeat
-```
+1. Compute the partial derivative of loss with respect to that weight
+2. Subtract a small multiple of it from the weight
+3. Repeat
The learning rate controls step size. Too big and you overshoot. Too small and you crawl.
**Loss landscape (1D slice):**
-The loss function L(w) forms a curve with peaks and valleys as the weight w varies.
+The loss function $L(w)$ forms a curve with peaks and valleys as the weight $w$ varies.
| Feature | Description |
|---------|-------------|
@@ -116,19 +115,17 @@ Gradient descent follows the slope downhill. It can get stuck in local minima, b
There are two ways to compute a derivative.
-Analytical: apply calculus rules by hand. For f(x) = x^2, the derivative is f'(x) = 2x. Exact. Fast.
+Analytical: apply calculus rules by hand. For $f(x) = x^2$, the derivative is $f'(x) = 2x$. Exact. Fast.
-Numerical: approximate using the definition. Compute f(x+h) and f(x-h) for a tiny h, then use the difference.
+Numerical: approximate using the definition. Compute $f(x+h)$ and $f(x-h)$ for a tiny $h$, then use the difference.
-```
Numerical (central difference):
-f'(x) ~= f(x + h) - f(x - h)
- -----------------------
- 2h
+$$
+f'(x) \approx \frac{f(x + h) - f(x - h)}{2h}
+$$
-h = 0.0001 works well in practice
-```
+$h = 0.0001$ works well in practice.
Numerical derivatives are slower but work for any function. Analytical derivatives are fast but require you to derive the formula. Neural network frameworks use a third approach: automatic differentiation, which computes exact derivatives mechanically. You will see that in Phase 3.
@@ -148,7 +145,7 @@ f(x) = ln(x) f'(x) = 1/x Cross-entropy loss
f(x) = 1/(1+e^-x) f'(x) = f(x)(1-f(x)) Sigmoid activation
```
-For f(x) = x^2:
+For $f(x) = x^2$:
```
f(x) = x^2 f'(x) = 2x
@@ -161,27 +158,29 @@ f(x) = x^2 f'(x) = 2x
2 4 4 slope tilts right (increasing)
```
-For f(w) = wx + b with x=3, b=1:
+For $f(w) = wx + b$ with $x=3$, $b=1$:
-```
-f(w) = 3w + 1 f'(w) = 3
+$$
+f(w) = 3w + 1 \qquad f'(w) = 3
+$$
-The derivative with respect to w is just x.
-If x is big, a small change in w causes a big change in output.
-```
+The derivative with respect to $w$ is just $x$. If $x$ is big, a small change in $w$ causes a big change in output.
### The chain rule
When functions are composed, the chain rule tells you how to differentiate.
-```
-If y = f(g(x)), then dy/dx = f'(g(x)) * g'(x)
+If $y = f(g(x))$, then $\frac{dy}{dx} = f'(g(x)) \cdot g'(x)$.
-Example: y = (3x + 1)^2
- outer: f(u) = u^2 f'(u) = 2u
- inner: g(x) = 3x + 1 g'(x) = 3
- dy/dx = 2(3x + 1) * 3 = 6(3x + 1)
-```
+Example: $y = (3x + 1)^2$
+
+$$
+\begin{aligned}
+\text{outer:} \quad f(u) &= u^2 & f'(u) &= 2u \\
+\text{inner:} \quad g(x) &= 3x + 1 & g'(x) &= 3 \\
+\frac{dy}{dx} &= 2(3x + 1) \cdot 3 = 6(3x + 1)
+\end{aligned}
+$$
Neural networks are chains of functions: input -> linear -> activation -> linear -> activation -> loss. Backpropagation is the chain rule applied repeatedly from output to input. That is the entire algorithm.
@@ -189,18 +188,17 @@ Neural networks are chains of functions: input -> linear -> activation -> linear
The gradient tells you the slope. The Hessian tells you the curvature.
-The Hessian is the matrix of second-order partial derivatives. For a function f(x1, x2, ..., xn), entry (i, j) of the Hessian is:
+The Hessian is the matrix of second-order partial derivatives. For a function $f(x_1, x_2, \ldots, x_n)$, entry $(i, j)$ of the Hessian is:
-```
-H[i][j] = d^2f / (dx_i * dx_j)
-```
+$$
+H[i][j] = \frac{\partial^2 f}{\partial x_i\, \partial x_j}
+$$
-For a 2-variable function f(x, y):
+For a 2-variable function $f(x, y)$:
-```
-H = | d^2f/dx^2 d^2f/dxdy |
- | d^2f/dydx d^2f/dy^2 |
-```
+$$
+H = \begin{bmatrix} \dfrac{\partial^2 f}{\partial x^2} & \dfrac{\partial^2 f}{\partial x\, \partial y} \\[1ex] \dfrac{\partial^2 f}{\partial y\, \partial x} & \dfrac{\partial^2 f}{\partial y^2} \end{bmatrix}
+$$
**What the Hessian tells you at a critical point (where gradient = 0):**
@@ -210,49 +208,51 @@ H = | d^2f/dx^2 d^2f/dxdy |
| Negative definite (all eigenvalues < 0) | Local maximum | Bowl pointing down |
| Indefinite (mixed eigenvalues) | Saddle point | Horse saddle shape |
-**Example:** f(x, y) = x^2 - y^2 (a saddle function)
+**Example:** $f(x, y) = x^2 - y^2$ (a saddle function)
-```
-df/dx = 2x df/dy = -2y
-d^2f/dx^2 = 2 d^2f/dy^2 = -2 d^2f/dxdy = 0
+$$
+\begin{aligned}
+\frac{\partial f}{\partial x} &= 2x & \frac{\partial f}{\partial y} &= -2y \\
+\frac{\partial^2 f}{\partial x^2} &= 2 & \frac{\partial^2 f}{\partial y^2} &= -2 & \frac{\partial^2 f}{\partial x\, \partial y} &= 0
+\end{aligned}
+$$
-H = | 2 0 |
- | 0 -2 |
+$$
+H = \begin{bmatrix} 2 & 0 \\ 0 & -2 \end{bmatrix}
+$$
-Eigenvalues: 2 and -2 (one positive, one negative)
---> Saddle point at (0, 0)
-```
+Eigenvalues: $2$ and $-2$ (one positive, one negative) $\to$ saddle point at $(0, 0)$.
-Compare with f(x, y) = x^2 + y^2 (a bowl):
+Compare with $f(x, y) = x^2 + y^2$ (a bowl):
-```
-H = | 2 0 |
- | 0 2 |
+$$
+H = \begin{bmatrix} 2 & 0 \\ 0 & 2 \end{bmatrix}
+$$
-Eigenvalues: 2 and 2 (both positive)
---> Local minimum at (0, 0)
-```
+Eigenvalues: $2$ and $2$ (both positive) $\to$ local minimum at $(0, 0)$.
**Why the Hessian matters in ML:**
Newton's method uses the Hessian to take better optimization steps than gradient descent. Instead of just following the slope, it accounts for curvature:
-```
-Newton's update: w_new = w_old - H^(-1) * gradient
-Gradient descent: w_new = w_old - lr * gradient
-```
+$$
+\begin{aligned}
+\text{Newton's update:} \quad w_{\text{new}} &= w_{\text{old}} - H^{-1} \cdot \text{gradient} \\
+\text{Gradient descent:} \quad w_{\text{new}} &= w_{\text{old}} - \text{lr} \cdot \text{gradient}
+\end{aligned}
+$$
Newton's method converges faster because the Hessian "rescales" the gradient -- steep directions get smaller steps, flat directions get larger steps.
-The catch: for a neural network with N parameters, the Hessian is N x N. A model with 1 million parameters would need a 1 trillion-entry matrix. That is why we use approximations.
+The catch: for a neural network with $N$ parameters, the Hessian is $N \times N$. A model with 1 million parameters would need a 1 trillion-entry matrix. That is why we use approximations.
| Method | What it uses | Cost | Convergence |
|--------|-------------|------|-------------|
-| Gradient descent | First derivatives only | O(N) per step | Slow (linear) |
-| Newton's method | Full Hessian | O(N^3) per step | Fast (quadratic) |
-| L-BFGS | Approximate Hessian from gradient history | O(N) per step | Medium (superlinear) |
-| Adam | Per-parameter adaptive rates (diagonal Hessian approx) | O(N) per step | Medium |
-| Natural gradient | Fisher information matrix (statistical Hessian) | O(N^2) per step | Fast |
+| Gradient descent | First derivatives only | $O(N)$ per step | Slow (linear) |
+| Newton's method | Full Hessian | $O(N^3)$ per step | Fast (quadratic) |
+| L-BFGS | Approximate Hessian from gradient history | $O(N)$ per step | Medium (superlinear) |
+| Adam | Per-parameter adaptive rates (diagonal Hessian approx) | $O(N)$ per step | Medium |
+| Natural gradient | Fisher information matrix (statistical Hessian) | $O(N^2)$ per step | Fast |
In practice, Adam is the default optimizer for deep learning. It approximates second-order information cheaply by tracking the running mean and variance of gradients per parameter.
@@ -260,17 +260,17 @@ In practice, Adam is the default optimizer for deep learning. It approximates se
Any smooth function can be approximated locally by a polynomial:
-```
-f(x + h) = f(x) + f'(x)*h + (1/2)*f''(x)*h^2 + (1/6)*f'''(x)*h^3 + ...
-```
+$$
+f(x + h) = f(x) + f'(x)\,h + \frac{1}{2}\,f''(x)\,h^2 + \frac{1}{6}\,f'''(x)\,h^3 + \cdots
+$$
The more terms you include, the better the approximation -- but only near the point x.
**Why Taylor series matter for ML:**
-- **First-order Taylor = gradient descent.** When you use f(x + h) ~ f(x) + f'(x)*h, you are making a linear approximation. Gradient descent minimizes this linear model to choose h = -lr * f'(x).
+- **First-order Taylor = gradient descent.** When you use $f(x + h) \approx f(x) + f'(x)\,h$, you are making a linear approximation. Gradient descent minimizes this linear model to choose $h = -\text{lr} \cdot f'(x)$.
-- **Second-order Taylor = Newton's method.** Using f(x + h) ~ f(x) + f'(x)*h + (1/2)*f''(x)*h^2, you get a quadratic model. Minimizing it gives h = -f'(x)/f''(x) -- Newton's step.
+- **Second-order Taylor = Newton's method.** Using $f(x + h) \approx f(x) + f'(x)\,h + \frac{1}{2}\,f''(x)\,h^2$, you get a quadratic model. Minimizing it gives $h = -f'(x)/f''(x)$ -- Newton's step.
- **Loss function design.** MSE and cross-entropy are smooth, which means their Taylor expansions are well-behaved. This is not an accident. Smooth losses make optimization predictable.
@@ -291,28 +291,28 @@ Derivatives tell you rates of change. Integrals compute accumulations -- area un
In ML, you rarely compute integrals by hand, but the concept is everywhere:
-**Probability.** For a continuous random variable with density p(x):
-```
-P(a < X < b) = integral from a to b of p(x) dx
-```
+**Probability.** For a continuous random variable with density $p(x)$:
+$$
+P(a < X < b) = \int_a^b p(x)\, dx
+$$
The area under the probability density curve between a and b is the probability of landing in that range.
**Expected value.** The average outcome weighted by probability:
-```
-E[f(X)] = integral of f(x) * p(x) dx
-```
+$$
+E[f(X)] = \int f(x) \cdot p(x)\, dx
+$$
The expected loss over a data distribution is an integral. Training minimizes an empirical approximation of this.
**KL divergence.** Measures how different two distributions are:
-```
-KL(p || q) = integral of p(x) * log(p(x) / q(x)) dx
-```
+$$
+\text{KL}(p \parallel q) = \int p(x) \cdot \log\!\left(\frac{p(x)}{q(x)}\right) dx
+$$
Used in VAEs, knowledge distillation, and Bayesian inference.
**Normalization constants.** In Bayesian inference:
-```
-p(w | data) = p(data | w) * p(w) / integral of p(data | w) * p(w) dw
-```
+$$
+p(w \mid \text{data}) = \frac{p(\text{data} \mid w) \cdot p(w)}{\int p(\text{data} \mid w) \cdot p(w)\, dw}
+$$
The denominator is an integral over all possible parameter values. It is often intractable, which is why we use approximations like MCMC and variational inference.
| Integral concept | Where it appears in ML |
@@ -353,16 +353,16 @@ This is all backpropagation is: the chain rule applied systematically through a
When a function maps a vector to a vector (like a neural network layer), its derivative is a matrix. The Jacobian contains every partial derivative of every output with respect to every input.
-For f: R^n -> R^m, the Jacobian J is an m x n matrix:
+For $f: \mathbb{R}^n \to \mathbb{R}^m$, the Jacobian $J$ is an $m \times n$ matrix:
-| | x1 | x2 | ... | xn |
+| | $x_1$ | $x_2$ | ... | $x_n$ |
|---|---|---|---|---|
-| f1 | df1/dx1 | df1/dx2 | ... | df1/dxn |
-| f2 | df2/dx1 | df2/dx2 | ... | df2/dxn |
+| $f_1$ | $\partial f_1/\partial x_1$ | $\partial f_1/\partial x_2$ | ... | $\partial f_1/\partial x_n$ |
+| $f_2$ | $\partial f_2/\partial x_1$ | $\partial f_2/\partial x_2$ | ... | $\partial f_2/\partial x_n$ |
| ... | ... | ... | ... | ... |
-| fm | dfm/dx1 | dfm/dx2 | ... | dfm/dxn |
+| $f_m$ | $\partial f_m/\partial x_1$ | $\partial f_m/\partial x_2$ | ... | $\partial f_m/\partial x_n$ |
-You will not compute Jacobians by hand for neural networks. PyTorch handles it. But knowing it exists helps you understand shapes in backpropagation: if a layer maps R^n to R^m, its Jacobian is m x n. The gradient flows backward through the transpose of this matrix.
+You will not compute Jacobians by hand for neural networks. PyTorch handles it. But knowing it exists helps you understand shapes in backpropagation: if a layer maps $\mathbb{R}^n$ to $\mathbb{R}^m$, its Jacobian is $m \times n$. The gradient flows backward through the transpose of this matrix.
### Why this matters for neural networks
@@ -531,7 +531,7 @@ for h in [0.1, 0.5, 1.0, 2.0]:
print(f"h={h:.1f} sin(h)={true_val:.4f} order1={t1:.4f} order2={t2:.4f}")
```
-Near x0=0, sin(x) ~ x (first-order Taylor). The approximation is excellent for small h but breaks down for large h. This is why gradient descent works best with small learning rates -- each step assumes the linear approximation is accurate.
+Near $x_0=0$, $\sin(x) \approx x$ (first-order Taylor). The approximation is excellent for small $h$ but breaks down for large $h$. This is why gradient descent works best with small learning rates -- each step assumes the linear approximation is accurate.
### Step 8: Why this matters for a neural network
@@ -600,9 +600,9 @@ You just built gradient descent from scratch. PyTorch automates the gradient com
## Exercises
-1. Implement `numerical_second_derivative(f, x)` using `numerical_derivative` called twice. Verify that the second derivative of x^3 at x=2 is 12.
-2. Use gradient descent to find the minimum of f(x, y) = (x - 3)^2 + (y + 1)^2. Start from (0, 0). The answer should converge to (3, -1).
-3. Add momentum to the gradient descent loop: maintain a velocity vector that accumulates past gradients. Compare convergence speed with and without momentum on f(x) = x^4 - 3x^2.
+1. Implement `numerical_second_derivative(f, x)` using `numerical_derivative` called twice. Verify that the second derivative of $x^3$ at $x=2$ is 12.
+2. Use gradient descent to find the minimum of $f(x, y) = (x - 3)^2 + (y + 1)^2$. Start from $(0, 0)$. The answer should converge to $(3, -1)$.
+3. Add momentum to the gradient descent loop: maintain a velocity vector that accumulates past gradients. Compare convergence speed with and without momentum on $f(x) = x^4 - 3x^2$.
## Key Terms
@@ -613,12 +613,12 @@ You just built gradient descent from scratch. PyTorch automates the gradient com
| Gradient | "Direction of steepest ascent" | A vector of all partial derivatives. Points in the direction that increases the function fastest. |
| Gradient descent | "Go downhill" | Subtract the gradient (times a learning rate) from the parameters to reduce the loss. The core of neural network training. |
| Learning rate | "Step size" | A scalar that controls how big each gradient descent step is. Too large: diverge. Too small: converge slowly. |
-| Chain rule | "Multiply the derivatives" | The rule for differentiating composed functions: df/dx = df/dg * dg/dx. The mathematical basis of backpropagation. |
+| Chain rule | "Multiply the derivatives" | The rule for differentiating composed functions: $\frac{df}{dx} = \frac{df}{dg} \cdot \frac{dg}{dx}$. The mathematical basis of backpropagation. |
| Jacobian | "Matrix of derivatives" | When a function maps vectors to vectors, the Jacobian is the matrix of all partial derivatives of outputs with respect to inputs. |
| Numerical derivative | "Finite differences" | Approximating a derivative by evaluating the function at two nearby points and computing the slope between them. |
| Backpropagation | "Reverse-mode autodiff" | Computing gradients layer by layer from output to input using the chain rule. How neural networks learn. |
| Hessian | "Matrix of second derivatives" | The matrix of all second-order partial derivatives. Describes the curvature of a function. Positive definite Hessian at a critical point means local minimum. |
-| Taylor series | "Polynomial approximation" | Approximating a function near a point using its derivatives: f(x+h) ~ f(x) + f'(x)h + (1/2)f''(x)h^2 + ... The basis for understanding why gradient descent and Newton's method work. |
+| Taylor series | "Polynomial approximation" | Approximating a function near a point using its derivatives: $f(x+h) \approx f(x) + f'(x)h + \frac{1}{2}f''(x)h^2 + \cdots$ The basis for understanding why gradient descent and Newton's method work. |
| Integral | "Area under the curve" | The accumulation of a quantity over a range. In ML, integrals define probabilities, expected values, and KL divergence. |
## Further Reading
diff --git a/phases/01-math-foundations/05-chain-rule-and-autodiff/docs/en.md b/phases/01-math-foundations/05-chain-rule-and-autodiff/docs/en.md
index 0b8ce49485..0318fe8418 100644
--- a/phases/01-math-foundations/05-chain-rule-and-autodiff/docs/en.md
+++ b/phases/01-math-foundations/05-chain-rule-and-autodiff/docs/en.md
@@ -28,30 +28,32 @@ This is how PyTorch, TensorFlow, and JAX work. You will build a miniature versio
### The Chain Rule
-If `y = f(g(x))`, the derivative of `y` with respect to `x` is:
+If $y = f(g(x))$, the derivative of $y$ with respect to $x$ is:
-```
-dy/dx = dy/dg * dg/dx = f'(g(x)) * g'(x)
-```
+$$
+\frac{dy}{dx} = \frac{dy}{dg} \cdot \frac{dg}{dx} = f'(g(x)) \cdot g'(x)
+$$
Multiply the derivatives along the chain. Each link contributes its local derivative.
-Example: `y = sin(x^2)`
+Example: $y = \sin(x^2)$
-```
-g(x) = x^2 g'(x) = 2x
-f(g) = sin(g) f'(g) = cos(g)
-
-dy/dx = cos(x^2) * 2x
-```
+$$
+\begin{aligned}
+g(x) &= x^2 & g'(x) &= 2x \\
+f(g) &= \sin(g) & f'(g) &= \cos(g) \\
+\frac{dy}{dx} &= \cos(x^2) \cdot 2x
+\end{aligned}
+$$
For deeper compositions, the chain extends:
-```
-y = f(g(h(x)))
-
-dy/dx = f'(g(h(x))) * g'(h(x)) * h'(x)
-```
+$$
+\begin{aligned}
+y &= f(g(h(x))) \\
+\frac{dy}{dx} &= f'(g(h(x))) \cdot g'(h(x)) \cdot h'(x)
+\end{aligned}
+$$
Every layer in a neural network is one link in this chain.
@@ -88,47 +90,49 @@ The backward pass applies the chain rule at every node, propagating gradients fr
There are two ways to apply the chain rule through a graph.
-**Forward mode** starts at the inputs and pushes derivatives forward. It computes `dx/dx = 1` and propagates through each operation. Good when you have few inputs and many outputs.
-
-```
-Forward mode: seed dx/dx = 1, propagate forward
+**Forward mode** starts at the inputs and pushes derivatives forward. It computes $dx/dx = 1$ and propagates through each operation. Good when you have few inputs and many outputs.
- x = 2 (dx/dx = 1)
- a = x^2 (da/dx = 2x = 4)
- y = sin(a) (dy/dx = cos(a) * da/dx = cos(4) * 4 = -2.615)
-```
+$$
+\begin{aligned}
+&\text{Forward mode: seed } dx/dx = 1, \text{ propagate forward} \\
+x &= 2 & (dx/dx &= 1) \\
+a &= x^2 & (da/dx &= 2x = 4) \\
+y &= \sin(a) & \left(\frac{dy}{dx} = \cos(a) \cdot \frac{da}{dx} = \cos(4) \cdot 4 = -2.615\right)
+\end{aligned}
+$$
-**Reverse mode** starts at the output and pulls gradients backward. It computes `dy/dy = 1` and propagates through each operation in reverse. Good when you have many inputs and few outputs.
+**Reverse mode** starts at the output and pulls gradients backward. It computes $dy/dy = 1$ and propagates through each operation in reverse. Good when you have many inputs and few outputs.
-```
-Reverse mode: seed dy/dy = 1, propagate backward
-
- y = sin(a) (dy/dy = 1)
- a = x^2 (dy/da = cos(a) = cos(4) = -0.654)
- x = 2 (dy/dx = dy/da * da/dx = -0.654 * 4 = -2.615)
-```
+$$
+\begin{aligned}
+&\text{Reverse mode: seed } dy/dy = 1, \text{ propagate backward} \\
+y &= \sin(a) & (dy/dy &= 1) \\
+a &= x^2 & (dy/da &= \cos(a) = \cos(4) = -0.654) \\
+x &= 2 & \left(\frac{dy}{dx} = \frac{dy}{da} \cdot \frac{da}{dx} = -0.654 \cdot 4 = -2.615\right)
+\end{aligned}
+$$
Neural networks have millions of inputs (weights) and one output (loss). Reverse mode computes all gradients in one backward pass. This is why backpropagation uses reverse mode.
| Mode | Seed | Direction | Best when |
|------|------|-----------|-----------|
-| Forward | `dx_i/dx_i = 1` | Input to output | Few inputs, many outputs |
-| Reverse | `dy/dy = 1` | Output to input | Many inputs, few outputs (neural nets) |
+| Forward | $dx_i/dx_i = 1$ | Input to output | Few inputs, many outputs |
+| Reverse | $dy/dy = 1$ | Output to input | Many inputs, few outputs (neural nets) |
### Dual Numbers for Forward Mode
-Forward mode can be implemented elegantly with dual numbers. A dual number has the form `a + b*epsilon` where `epsilon^2 = 0`.
+Forward mode can be implemented elegantly with dual numbers. A dual number has the form $a + b\epsilon$ where $\epsilon^2 = 0$.
-```
-Dual number: (value, derivative)
-
-(2, 1) means: value is 2, derivative w.r.t. x is 1
-
-Arithmetic rules:
- (a, a') + (b, b') = (a+b, a'+b')
- (a, a') * (b, b') = (a*b, a'*b + a*b')
- sin(a, a') = (sin(a), cos(a)*a')
-```
+$$
+\begin{aligned}
+&\text{Dual number: } (\text{value}, \text{derivative}) \\
+&(2, 1) \text{ means: value is 2, derivative w.r.t. } x \text{ is 1} \\
+&\text{Arithmetic rules:} \\
+(a, a') + (b, b') &= (a+b,\ a'+b') \\
+(a, a') \cdot (b, b') &= (a \cdot b,\ a' \cdot b + a \cdot b') \\
+\sin(a, a') &= (\sin(a),\ \cos(a) \cdot a')
+\end{aligned}
+$$
Seed the input variable with derivative 1. The derivative propagates automatically through every operation.
@@ -300,11 +304,11 @@ The basic Value class handles addition, multiplication, and relu. A real autogra
| Operation | Backward rule | Used in |
|-----------|--------------|---------|
| `__sub__` | Reuses add + neg | Loss computation (pred - target) |
-| `__pow__` | n * x^(n-1) | Polynomial activations, MSE (error^2) |
+| `__pow__` | $n \cdot x^{n-1}$ | Polynomial activations, MSE ($\text{error}^2$) |
| `__truediv__` | Reuses mul + pow(-1) | Normalization, learning rate scaling |
-| `exp` | exp(x) * upstream | Softmax, log-likelihood |
-| `log` | (1/x) * upstream | Cross-entropy loss, log probabilities |
-| `tanh` | (1 - tanh^2) * upstream | Classic activation function |
+| `exp` | $\exp(x) \cdot \text{upstream}$ | Softmax, log-likelihood |
+| `log` | $(1/x) \cdot \text{upstream}$ | Cross-entropy loss, log probabilities |
+| `tanh` | $(1 - \tanh^2) \cdot \text{upstream}$ | Classic activation function |
The clever part: `__sub__` and `__truediv__` are defined in terms of existing operations. They get correct gradients for free because the chain rule composes through the underlying add/mul/pow operations.
@@ -350,7 +354,7 @@ class MLP:
return [p for layer in self.layers for p in layer.parameters()]
```
-A `Neuron` computes `tanh(w1*x1 + w2*x2 + ... + b)`. A `Layer` is a list of neurons. An `MLP` stacks layers. Every weight is a `Value`, so calling `loss.backward()` propagates gradients to every parameter.
+A `Neuron` computes $\tanh(w_1 x_1 + w_2 x_2 + \dots + b)$. A `Layer` is a list of neurons. An `MLP` stacks layers. Every weight is a `Value`, so calling `loss.backward()` propagates gradients to every parameter.
**Training on XOR:**
@@ -442,8 +446,8 @@ print(f"dy/dx1 = {x1.grad}") # 3.0 (= x2)
print(f"dy/dx2 = {x2.grad}") # 2.0 (= x1)
```
-Manual check: `y = relu(x1*x2 + 1)`. Since `x1*x2 + 1 = 7 > 0`, relu is identity.
-`dy/dx1 = x2 = 3`. `dy/dx2 = x1 = 2`. The engine matches.
+Manual check: $y = \text{relu}(x_1 x_2 + 1)$. Since $x_1 x_2 + 1 = 7 > 0$, relu is identity.
+$dy/dx_1 = x_2 = 3$. $dy/dx_2 = x_1 = 2$. The engine matches.
## Use It
@@ -489,11 +493,11 @@ The Value class built here is the foundation for the neural network training loo
## Exercises
-1. Add `__pow__` to the Value class so you can compute `x ** n`. Verify that `d/dx(x^3)` at `x=2` equals `12.0`.
+1. Add `__pow__` to the Value class so you can compute $x^n$. Verify that $\frac{d}{dx}(x^3)$ at $x=2$ equals $12.0$.
-2. Add `tanh` as an activation function. Verify that `tanh'(0) = 1` and `tanh'(2) = 0.0707` (approx).
+2. Add `tanh` as an activation function. Verify that $\tanh'(0) = 1$ and $\tanh'(2) = 0.0707$ (approx).
-3. Build a computation graph for a single neuron: `y = relu(w1*x1 + w2*x2 + b)`. Compute all five gradients and verify against PyTorch.
+3. Build a computation graph for a single neuron: $y = \text{relu}(w_1 x_1 + w_2 x_2 + b)$. Compute all five gradients and verify against PyTorch.
4. Implement forward-mode autodiff using dual numbers. Create a `Dual` class and verify it gives the same derivatives as your reverse-mode engine.
@@ -506,13 +510,13 @@ The Value class built here is the foundation for the neural network training loo
| Forward mode | "Push derivatives forward" | Autodiff that propagates derivatives from inputs to outputs. One pass per input variable. |
| Reverse mode | "Backpropagation" | Autodiff that propagates gradients from outputs to inputs. One pass per output variable. |
| Autograd | "Automatic gradients" | A system that records operations on values, builds a graph, and computes exact gradients via the chain rule |
-| Dual numbers | "Value plus derivative" | Numbers of the form a + b*epsilon (epsilon^2 = 0) that carry derivative information through arithmetic |
+| Dual numbers | "Value plus derivative" | Numbers of the form $a + b\epsilon$ ($\epsilon^2 = 0$) that carry derivative information through arithmetic |
| Topological sort | "Dependency order" | Ordering graph nodes so every node comes after all its dependencies. Required for correct gradient propagation. |
| Gradient accumulation | "Add, don't replace" | When a value feeds into multiple operations, its gradient is the sum of all incoming gradient contributions |
| Dynamic graph | "Define by run" | A computation graph rebuilt on every forward pass, allowing Python control flow inside models (PyTorch style) |
| Gradient checking | "Numerical verification" | Comparing autodiff gradients against numerical finite-difference gradients to verify correctness. Essential for debugging. |
| MLP | "Multi-layer perceptron" | A neural network with one or more hidden layers of neurons. Each neuron computes a weighted sum plus bias, then applies an activation function. |
-| Neuron | "Weighted sum + activation" | The basic unit: output = activation(w1*x1 + w2*x2 + ... + b). The weights and bias are learnable parameters. |
+| Neuron | "Weighted sum + activation" | The basic unit: $\text{output} = \text{activation}(w_1 x_1 + w_2 x_2 + \dots + b)$. The weights and bias are learnable parameters. |
## Further Reading
diff --git a/phases/01-math-foundations/06-probability-and-distributions/docs/en.md b/phases/01-math-foundations/06-probability-and-distributions/docs/en.md
index 3c1d981487..d527945a37 100644
--- a/phases/01-math-foundations/06-probability-and-distributions/docs/en.md
+++ b/phases/01-math-foundations/06-probability-and-distributions/docs/en.md
@@ -24,44 +24,52 @@ Every prediction a model makes is a probability distribution. Every loss functio
### Events, Sample Spaces, and Probability
-The sample space S is the set of all possible outcomes. An event is a subset of the sample space. Probability maps events to numbers between 0 and 1.
-
-```
-Coin flip:
- S = {H, T}
- P(H) = 0.5, P(T) = 0.5
-
-Single die roll:
- S = {1, 2, 3, 4, 5, 6}
- P(even) = P({2, 4, 6}) = 3/6 = 0.5
-```
+The sample space $S$ is the set of all possible outcomes. An event is a subset of the sample space. Probability maps events to numbers between 0 and 1.
+
+$$
+\begin{aligned}
+&\text{Coin flip:} \\
+&\quad S = \{H, T\} \\
+&\quad P(H) = 0.5, \quad P(T) = 0.5 \\
+\\
+&\text{Single die roll:} \\
+&\quad S = \{1, 2, 3, 4, 5, 6\} \\
+&\quad P(\text{even}) = P(\{2, 4, 6\}) = \tfrac{3}{6} = 0.5
+\end{aligned}
+$$
Three axioms define all of probability:
-1. P(A) >= 0 for any event A
-2. P(S) = 1 (something always happens)
-3. P(A or B) = P(A) + P(B) when A and B cannot both occur
+1. $P(A) \geq 0$ for any event $A$
+2. $P(S) = 1$ (something always happens)
+3. $P(A \text{ or } B) = P(A) + P(B)$ when $A$ and $B$ cannot both occur
Everything else (Bayes' theorem, expectations, distributions) follows from these three rules.
### Conditional Probability and Independence
-P(A|B) is the probability of A given that B happened.
+$P(A \mid B)$ is the probability of A given that B happened.
-```
-P(A|B) = P(A and B) / P(B)
+$$
+P(A \mid B) = \frac{P(A \text{ and } B)}{P(B)}
+$$
-Example: deck of cards
- P(King | Face card) = P(King and Face card) / P(Face card)
- = (4/52) / (12/52)
- = 4/12 = 1/3
-```
+$$
+\begin{aligned}
+\text{Example: deck of cards} \\
+P(\text{King} \mid \text{Face card}) &= \frac{P(\text{King and Face card})}{P(\text{Face card})} \\
+&= \frac{4/52}{12/52} \\
+&= \frac{4}{12} = \frac{1}{3}
+\end{aligned}
+$$
Two events are independent when knowing one tells you nothing about the other:
-```
-Independent: P(A|B) = P(A)
-Equivalent to: P(A and B) = P(A) * P(B)
-```
+$$
+\begin{aligned}
+\text{Independent:} &\quad P(A \mid B) = P(A) \\
+\text{Equivalent to:} &\quad P(A \text{ and } B) = P(A) \cdot P(B)
+\end{aligned}
+$$
Coin flips are independent. Drawing cards without replacement is not.
@@ -69,28 +77,32 @@ Coin flips are independent. Drawing cards without replacement is not.
Discrete random variables have a probability mass function (PMF). Each outcome has a specific probability that you can read off directly.
-```
-PMF: P(X = k)
-
-Fair die:
- P(X = 1) = 1/6
- P(X = 2) = 1/6
- ...
- P(X = 6) = 1/6
-
- Sum of all probabilities = 1
-```
+$$
+\begin{aligned}
+&\text{PMF: } P(X = k) \\
+\\
+&\text{Fair die:} \\
+&\quad P(X = 1) = \tfrac{1}{6} \\
+&\quad P(X = 2) = \tfrac{1}{6} \\
+&\quad \dots \\
+&\quad P(X = 6) = \tfrac{1}{6} \\
+\\
+&\quad \text{Sum of all probabilities} = 1
+\end{aligned}
+$$
Continuous random variables have a probability density function (PDF). The density at a single point is not a probability. Probability comes from integrating the density over an interval.
-```
-PDF: f(x)
-
-P(a <= X <= b) = integral of f(x) from a to b
-
-f(x) can be greater than 1 (density, not probability)
-integral from -inf to +inf of f(x) dx = 1
-```
+$$
+\begin{aligned}
+&\text{PDF: } f(x) \\
+\\
+&P(a \leq X \leq b) = \int_a^b f(x)\,dx \\
+\\
+&f(x) \text{ can be greater than 1 (density, not probability)} \\
+&\int_{-\infty}^{+\infty} f(x)\,dx = 1
+\end{aligned}
+$$
This distinction matters in ML. Classification outputs are PMFs (discrete choices). VAE latent spaces use PDFs (continuous).
@@ -98,79 +110,95 @@ This distinction matters in ML. Classification outputs are PMFs (discrete choice
**Bernoulli:** one trial, two outcomes. Models binary classification.
-```
-P(X = 1) = p
-P(X = 0) = 1 - p
-Mean = p, Variance = p(1-p)
-```
+$$
+\begin{aligned}
+&P(X = 1) = p \\
+&P(X = 0) = 1 - p \\
+&\text{Mean} = p, \quad \text{Variance} = p(1-p)
+\end{aligned}
+$$
**Categorical:** one trial, k outcomes. Models multi-class classification (softmax output).
-```
-P(X = i) = p_i, where sum of p_i = 1
-Example: P(cat) = 0.7, P(dog) = 0.2, P(bird) = 0.1
-```
+$$
+\begin{aligned}
+&P(X = i) = p_i, \quad \text{where } \sum_i p_i = 1 \\
+&\text{Example: } P(\text{cat}) = 0.7, \quad P(\text{dog}) = 0.2, \quad P(\text{bird}) = 0.1
+\end{aligned}
+$$
**Uniform:** all outcomes equally likely. Used for random initialization.
-```
-Discrete: P(X = k) = 1/n for k in {1, ..., n}
-Continuous: f(x) = 1/(b-a) for x in [a, b]
-```
+$$
+\begin{aligned}
+&\text{Discrete: } P(X = k) = \tfrac{1}{n} \text{ for } k \in \{1, \dots, n\} \\
+&\text{Continuous: } f(x) = \tfrac{1}{b-a} \text{ for } x \in [a, b]
+\end{aligned}
+$$
-**Normal (Gaussian):** the bell curve. Parameterized by mean (mu) and variance (sigma^2).
+**Normal (Gaussian):** the bell curve. Parameterized by mean ($\mu$) and variance ($\sigma^2$).
-```
-f(x) = (1 / sqrt(2*pi*sigma^2)) * exp(-(x - mu)^2 / (2*sigma^2))
+$$
+f(x) = \frac{1}{\sqrt{2\pi\sigma^2}} \cdot \exp\left(-\frac{(x - \mu)^2}{2\sigma^2}\right)
+$$
-Standard normal: mu = 0, sigma = 1
- 68% of data within 1 sigma
- 95% within 2 sigma
- 99.7% within 3 sigma
-```
+$$
+\begin{aligned}
+&\text{Standard normal: } \mu = 0, \; \sigma = 1 \\
+&\quad 68\% \text{ of data within } 1\sigma \\
+&\quad 95\% \text{ within } 2\sigma \\
+&\quad 99.7\% \text{ within } 3\sigma
+\end{aligned}
+$$
**Poisson:** counts of rare events in a fixed interval. Models event rates.
-```
-P(X = k) = (lambda^k * e^(-lambda)) / k!
-Mean = lambda, Variance = lambda
-```
+$$
+\begin{aligned}
+&P(X = k) = \frac{\lambda^k \cdot e^{-\lambda}}{k!} \\
+&\text{Mean} = \lambda, \quad \text{Variance} = \lambda
+\end{aligned}
+$$
### Expected Value and Variance
Expected value is the weighted average outcome.
-```
-Discrete: E[X] = sum of x_i * P(X = x_i)
-Continuous: E[X] = integral of x * f(x) dx
-```
+$$
+\begin{aligned}
+\text{Discrete:} &\quad E[X] = \sum_i x_i \cdot P(X = x_i) \\
+\text{Continuous:} &\quad E[X] = \int x \cdot f(x)\,dx
+\end{aligned}
+$$
Variance measures spread around the mean.
-```
-Var(X) = E[(X - E[X])^2] = E[X^2] - (E[X])^2
-Standard deviation = sqrt(Var(X))
-```
+$$
+\begin{aligned}
+&\text{Var}(X) = E[(X - E[X])^2] = E[X^2] - (E[X])^2 \\
+&\text{Standard deviation} = \sqrt{\text{Var}(X)}
+\end{aligned}
+$$
In ML, expected value appears as the loss function (average loss over the data distribution). Variance tells you about model stability. High variance in gradients means noisy training.
### Joint and Marginal Distributions
-A joint distribution P(X, Y) describes two random variables together.
+A joint distribution $P(X, Y)$ describes two random variables together.
-Joint PMF example (X = weather, Y = umbrella):
+Joint PMF example ($X$ = weather, $Y$ = umbrella):
-| | Y=0 (no umbrella) | Y=1 (umbrella) | Marginal P(X) |
+| | $Y=0$ (no umbrella) | $Y=1$ (umbrella) | Marginal $P(X)$ |
|---|---|---|---|
-| X=0 (sun) | 0.40 | 0.10 | P(X=0) = 0.50 |
-| X=1 (rain) | 0.05 | 0.45 | P(X=1) = 0.50 |
-| **Marginal P(Y)** | P(Y=0) = 0.45 | P(Y=1) = 0.55 | 1.00 |
+| $X=0$ (sun) | 0.40 | 0.10 | $P(X=0) = 0.50$ |
+| $X=1$ (rain) | 0.05 | 0.45 | $P(X=1) = 0.50$ |
+| **Marginal $P(Y)$** | $P(Y=0) = 0.45$ | $P(Y=1) = 0.55$ | 1.00 |
The marginal distribution sums out the other variable:
-```
-P(X = x) = sum over all y of P(X = x, Y = y)
-```
+$$
+P(X = x) = \sum_{y} P(X = x, Y = y)
+$$
The row and column totals in the table above are the marginals.
@@ -196,23 +224,27 @@ This is why:
Raw probabilities cause numerical problems. Multiplying many small probabilities together quickly underflows to zero.
-```
-P(sentence) = P(word1) * P(word2) * ... * P(word_n)
- = 0.01 * 0.003 * 0.02 * ...
- -> 0.0 (underflow after ~30 terms)
-```
+$$
+\begin{aligned}
+P(\text{sentence}) &= P(\text{word}_1) \cdot P(\text{word}_2) \cdot \dots \cdot P(\text{word}_n) \\
+&= 0.01 \cdot 0.003 \cdot 0.02 \cdot \dots \\
+&\to 0.0 \quad (\text{underflow after } \sim 30 \text{ terms})
+\end{aligned}
+$$
Log probabilities fix this. Multiplications become additions.
-```
-log P(sentence) = log P(word1) + log P(word2) + ... + log P(word_n)
- = -4.6 + -5.8 + -3.9 + ...
- -> finite number (no underflow)
-```
+$$
+\begin{aligned}
+\log P(\text{sentence}) &= \log P(\text{word}_1) + \log P(\text{word}_2) + \dots + \log P(\text{word}_n) \\
+&= -4.6 + -5.8 + -3.9 + \dots \\
+&\to \text{finite number (no underflow)}
+\end{aligned}
+$$
Rules:
-- log(a * b) = log(a) + log(b)
-- log probabilities are always <= 0 (since 0 < P <= 1)
+- $\log(a \cdot b) = \log(a) + \log(b)$
+- log probabilities are always $\leq 0$ (since $0 < P \leq 1$)
- More negative = less likely
- Cross-entropy loss is the negative log probability of the correct class
@@ -220,27 +252,29 @@ Rules:
Neural networks output raw scores (logits). Softmax converts them into a valid probability distribution.
-```
-softmax(z_i) = exp(z_i) / sum(exp(z_j) for all j)
+$$
+\text{softmax}(z_i) = \frac{\exp(z_i)}{\sum_j \exp(z_j)}
+$$
Properties:
- - All outputs are in (0, 1)
- - All outputs sum to 1
- - Preserves relative ordering of inputs
- - exp() amplifies differences between logits
-```
+- All outputs are in $(0, 1)$
+- All outputs sum to 1
+- Preserves relative ordering of inputs
+- $\exp()$ amplifies differences between logits
The softmax trick: subtract the max logit before exponentiating to prevent overflow.
-```
-z = [100, 101, 102]
-exp(102) = overflow
-
-z_shifted = z - max(z) = [-2, -1, 0]
-exp(0) = 1 (safe)
-
-Same result, no overflow.
-```
+$$
+\begin{aligned}
+&z = [100, 101, 102] \\
+&\exp(102) = \text{overflow} \\
+\\
+&z_{\text{shifted}} = z - \max(z) = [-2, -1, 0] \\
+&\exp(0) = 1 \quad (\text{safe}) \\
+\\
+&\text{Same result, no overflow.}
+\end{aligned}
+$$
Log-softmax combines softmax and log for numerical stability. PyTorch uses this internally for cross-entropy loss.
@@ -434,20 +468,20 @@ You built these from scratch. Now you know what the library calls are doing.
| Term | What people say | What it actually means |
|------|----------------|----------------------|
-| Sample space | "All the possibilities" | The set S of every possible outcome of an experiment |
+| Sample space | "All the possibilities" | The set $S$ of every possible outcome of an experiment |
| PMF | "The probability function" | A function that gives the exact probability of each discrete outcome, summing to 1 |
| PDF | "The probability curve" | A density function for continuous variables. Integrate it over an interval to get probability |
-| Conditional probability | "Probability given something" | P(A\|B) = P(A and B) / P(B). The foundation of Bayesian thinking and Bayes' theorem |
-| Independence | "They don't affect each other" | P(A and B) = P(A) * P(B). Knowing one event tells you nothing about the other |
+| Conditional probability | "Probability given something" | $P(A \mid B) = P(A \text{ and } B) / P(B)$. The foundation of Bayesian thinking and Bayes' theorem |
+| Independence | "They don't affect each other" | $P(A \text{ and } B) = P(A) \cdot P(B)$. Knowing one event tells you nothing about the other |
| Expected value | "The average" | The probability-weighted sum of all outcomes. The loss function is an expected value |
| Variance | "How spread out" | The expected squared deviation from the mean. High variance = noisy, unstable estimates |
-| Normal distribution | "The bell curve" | f(x) = (1/sqrt(2*pi*sigma^2)) * exp(-(x-mu)^2/(2*sigma^2)). Appears everywhere due to the CLT |
+| Normal distribution | "The bell curve" | $f(x) = (1/\sqrt{2\pi\sigma^2}) \cdot \exp(-(x-\mu)^2/(2\sigma^2))$. Appears everywhere due to the CLT |
| Central Limit Theorem | "Averages become normal" | The mean of many independent samples converges to a normal distribution regardless of the source |
-| Joint distribution | "Two variables together" | P(X, Y) describes the probability of every combination of X and Y outcomes |
-| Marginal distribution | "Sum out the other variable" | P(X) = sum_y P(X, Y). Recovers one variable's distribution from the joint |
-| Log probability | "Log of the probability" | log P(x). Turns products into sums, preventing numerical underflow in long sequences |
-| Softmax | "Turn scores into probabilities" | softmax(z_i) = exp(z_i) / sum(exp(z_j)). Maps real-valued logits to a valid probability distribution |
-| Cross-entropy | "The loss function" | -sum(p_true * log(p_predicted)). Measures how different two distributions are. Lower is better |
+| Joint distribution | "Two variables together" | $P(X, Y)$ describes the probability of every combination of $X$ and $Y$ outcomes |
+| Marginal distribution | "Sum out the other variable" | $P(X) = \sum_y P(X, Y)$. Recovers one variable's distribution from the joint |
+| Log probability | "Log of the probability" | $\log P(x)$. Turns products into sums, preventing numerical underflow in long sequences |
+| Softmax | "Turn scores into probabilities" | $\text{softmax}(z_i) = \exp(z_i) / \sum_j \exp(z_j)$. Maps real-valued logits to a valid probability distribution |
+| Cross-entropy | "The loss function" | $-\sum p_{\text{true}} \cdot \log(p_{\text{predicted}})$. Measures how different two distributions are. Lower is better |
| Logits | "Raw model outputs" | Unnormalized scores before softmax. Named after the logistic function |
| Sampling | "Drawing random values" | Generating values according to a probability distribution. How models generate output |
diff --git a/phases/01-math-foundations/07-bayes-theorem/docs/en.md b/phases/01-math-foundations/07-bayes-theorem/docs/en.md
index 432bd93bf0..7a696ede70 100644
--- a/phases/01-math-foundations/07-bayes-theorem/docs/en.md
+++ b/phases/01-math-foundations/07-bayes-theorem/docs/en.md
@@ -30,25 +30,25 @@ If you build ML systems without understanding this, you will misinterpret model
You already know from Lesson 06 that conditional probability is:
-```
-P(A|B) = P(A and B) / P(B)
-```
+$$
+P(A \mid B) = \frac{P(A \text{ and } B)}{P(B)}
+$$
And symmetrically:
-```
-P(B|A) = P(A and B) / P(A)
-```
-
-Both expressions share the same numerator: P(A and B). Set them equal and rearrange:
-
-```
-P(A and B) = P(A|B) * P(B) = P(B|A) * P(A)
+$$
+P(B \mid A) = \frac{P(A \text{ and } B)}{P(A)}
+$$
-Therefore:
+Both expressions share the same numerator: $P(A \text{ and } B)$. Set them equal and rearrange:
-P(A|B) = P(B|A) * P(A) / P(B)
-```
+$$
+\begin{aligned}
+P(A \text{ and } B) &= P(A \mid B) \cdot P(B) = P(B \mid A) \cdot P(A) \\
+\text{Therefore:} \\
+P(A \mid B) &= \frac{P(B \mid A) \cdot P(A)}{P(B)}
+\end{aligned}
+$$
That is Bayes' theorem. Four quantities, one equation.
@@ -56,36 +56,36 @@ That is Bayes' theorem. Four quantities, one equation.
| Part | Name | What it means |
|------|------|---------------|
-| P(A\|B) | Posterior | Your updated belief about A after seeing evidence B |
-| P(B\|A) | Likelihood | How probable the evidence B is if A is true |
-| P(A) | Prior | Your belief about A before seeing any evidence |
-| P(B) | Evidence | Total probability of seeing B under all possibilities |
+| $P(A \mid B)$ | Posterior | Your updated belief about A after seeing evidence B |
+| $P(B \mid A)$ | Likelihood | How probable the evidence B is if A is true |
+| $P(A)$ | Prior | Your belief about A before seeing any evidence |
+| $P(B)$ | Evidence | Total probability of seeing B under all possibilities |
-The evidence term P(B) acts as a normalizer. You can expand it using the law of total probability:
+The evidence term $P(B)$ acts as a normalizer. You can expand it using the law of total probability:
-```
-P(B) = P(B|A) * P(A) + P(B|not A) * P(not A)
-```
+$$
+P(B) = P(B \mid A) \cdot P(A) + P(B \mid \lnot A) \cdot P(\lnot A)
+$$
### Medical test example
A disease affects 1 in 10,000 people. The test is 99% accurate (catches 99% of sick people, gives false positives 1% of the time).
-```
-P(sick) = 0.0001 (prior: disease is rare)
-P(positive|sick) = 0.99 (likelihood: test catches it)
-P(positive|healthy) = 0.01 (false positive rate)
-
-P(positive) = P(positive|sick) * P(sick) + P(positive|healthy) * P(healthy)
- = 0.99 * 0.0001 + 0.01 * 0.9999
- = 0.000099 + 0.009999
- = 0.010098
-
-P(sick|positive) = P(positive|sick) * P(sick) / P(positive)
- = 0.99 * 0.0001 / 0.010098
- = 0.0098
- = 0.98%
-```
+$$
+\begin{aligned}
+P(\text{sick}) &= 0.0001 \quad \text{(prior: disease is rare)} \\
+P(\text{positive} \mid \text{sick}) &= 0.99 \quad \text{(likelihood: test catches it)} \\
+P(\text{positive} \mid \text{healthy}) &= 0.01 \quad \text{(false positive rate)} \\[6pt]
+P(\text{positive}) &= P(\text{positive} \mid \text{sick}) \cdot P(\text{sick}) + P(\text{positive} \mid \text{healthy}) \cdot P(\text{healthy}) \\
+&= 0.99 \cdot 0.0001 + 0.01 \cdot 0.9999 \\
+&= 0.000099 + 0.009999 \\
+&= 0.010098 \\[6pt]
+P(\text{sick} \mid \text{positive}) &= \frac{P(\text{positive} \mid \text{sick}) \cdot P(\text{sick})}{P(\text{positive})} \\
+&= \frac{0.99 \cdot 0.0001}{0.010098} \\
+&= 0.0098 \\
+&= 0.98\%
+\end{aligned}
+$$
Less than 1%. The prior dominates. When a condition is rare, even accurate tests produce mostly false positives. This is why doctors order confirmation tests.
@@ -93,19 +93,19 @@ Less than 1%. The prior dominates. When a condition is rare, even accurate tests
You receive an email containing the word "lottery". Is it spam?
-```
-P(spam) = 0.3 (30% of email is spam)
-P("lottery"|spam) = 0.05 (5% of spam emails contain "lottery")
-P("lottery"|not spam) = 0.001 (0.1% of legitimate emails contain "lottery")
-
-P("lottery") = 0.05 * 0.3 + 0.001 * 0.7
- = 0.015 + 0.0007
- = 0.0157
-
-P(spam|"lottery") = 0.05 * 0.3 / 0.0157
- = 0.955
- = 95.5%
-```
+$$
+\begin{aligned}
+P(\text{spam}) &= 0.3 \quad \text{(30\% of email is spam)} \\
+P(\text{"lottery"} \mid \text{spam}) &= 0.05 \quad \text{(5\% of spam emails contain "lottery")} \\
+P(\text{"lottery"} \mid \text{not spam}) &= 0.001 \quad \text{(0.1\% of legitimate emails contain "lottery")} \\[6pt]
+P(\text{"lottery"}) &= 0.05 \cdot 0.3 + 0.001 \cdot 0.7 \\
+&= 0.015 + 0.0007 \\
+&= 0.0157 \\[6pt]
+P(\text{spam} \mid \text{"lottery"}) &= \frac{0.05 \cdot 0.3}{0.0157} \\
+&= 0.955 \\
+&= 95.5\%
+\end{aligned}
+$$
One word shifts the probability from 30% to 95.5%. A real spam filter applies Bayes across hundreds of words simultaneously.
@@ -113,58 +113,57 @@ One word shifts the probability from 30% to 95.5%. A real spam filter applies Ba
Naive Bayes extends this to multiple features by assuming all features are conditionally independent given the class:
-```
-P(class | feature_1, feature_2, ..., feature_n)
- = P(class) * P(feature_1|class) * P(feature_2|class) * ... * P(feature_n|class)
- / P(feature_1, feature_2, ..., feature_n)
-```
+$$
+P(\text{class} \mid \text{feature}_1, \text{feature}_2, \ldots, \text{feature}_n)
+= \frac{P(\text{class}) \cdot P(\text{feature}_1 \mid \text{class}) \cdot P(\text{feature}_2 \mid \text{class}) \cdots P(\text{feature}_n \mid \text{class})}{P(\text{feature}_1, \text{feature}_2, \ldots, \text{feature}_n)}
+$$
The "naive" part is the independence assumption. In text, word occurrences are not independent ("New" and "York" are correlated). But the assumption works surprisingly well in practice because the classifier only needs to rank classes, not produce calibrated probabilities.
Since the denominator is the same for all classes, you can skip it and just compare numerators:
-```
-score(class) = P(class) * product of P(feature_i | class)
-```
+$$
+\text{score}(\text{class}) = P(\text{class}) \cdot \prod_i P(\text{feature}_i \mid \text{class})
+$$
Pick the class with the highest score.
### Maximum likelihood estimation (MLE)
-How do you get P(feature|class) from training data? Count.
+How do you get $P(\text{feature} \mid \text{class})$ from training data? Count.
-```
-P("free"|spam) = (number of spam emails containing "free") / (total spam emails)
-```
+$$
+P(\text{"free"} \mid \text{spam}) = \frac{\text{number of spam emails containing "free"}}{\text{total spam emails}}
+$$
This is MLE: choose the parameter values that make the observed data most likely. You are maximizing the likelihood function, which for discrete counts reduces to relative frequency.
Problem: if a word never appears in spam during training, MLE gives it probability zero. One unseen word kills the entire product. Fix this with Laplace smoothing:
-```
-P(word|class) = (count(word, class) + 1) / (total_words_in_class + vocabulary_size)
-```
+$$
+P(\text{word} \mid \text{class}) = \frac{\text{count}(\text{word}, \text{class}) + 1}{\text{total\_words\_in\_class} + \text{vocabulary\_size}}
+$$
Adding 1 to every count ensures no probability is ever zero.
### Maximum a posteriori (MAP)
-MLE asks: what parameters maximize P(data|parameters)?
+MLE asks: what parameters maximize $P(\text{data} \mid \text{parameters})$?
-MAP asks: what parameters maximize P(parameters|data)?
+MAP asks: what parameters maximize $P(\text{parameters} \mid \text{data})$?
By Bayes' theorem:
-```
-P(parameters|data) proportional to P(data|parameters) * P(parameters)
-```
+$$
+P(\text{parameters} \mid \text{data}) \propto P(\text{data} \mid \text{parameters}) \cdot P(\text{parameters})
+$$
MAP adds a prior over the parameters themselves. If you believe parameters should be small, you encode that as a prior that penalizes large values. This is identical to L2 regularization in ML. The "ridge" penalty in ridge regression is literally a Gaussian prior on the weights.
| Estimation | Optimizes | ML equivalent |
|------------|-----------|---------------|
-| MLE | P(data\|params) | Unregularized training |
-| MAP | P(data\|params) * P(params) | L2 / L1 regularization |
+| MLE | $P(\text{data} \mid \text{params})$ | Unregularized training |
+| MAP | $P(\text{data} \mid \text{params}) \cdot P(\text{params})$ | L2 / L1 regularization |
### Bayesian vs frequentist: the practical difference
@@ -189,7 +188,7 @@ The connection is deeper than analogy:
**Priors are regularization.** A Gaussian prior on weights is L2 regularization. A Laplace prior is L1. Every time you add a regularization term, you are making a Bayesian statement about what parameter values you expect.
-**Posteriors are uncertainty.** A single predicted probability tells you nothing about how confident the model is in that estimate. Bayesian methods give you a distribution: "I think P(spam) is between 0.8 and 0.95."
+**Posteriors are uncertainty.** A single predicted probability tells you nothing about how confident the model is in that estimate. Bayesian methods give you a distribution: "I think $P(\text{spam})$ is between 0.8 and 0.95."
**Bayes updates are online learning.** Today's posterior becomes tomorrow's prior. When your model sees new data, it updates its beliefs incrementally instead of retraining from scratch.
@@ -345,27 +344,29 @@ When the prior and posterior belong to the same family of distributions, the pri
| Likelihood | Conjugate Prior | Posterior | Example |
|-----------|----------------|-----------|---------|
-| Bernoulli | Beta(a, b) | Beta(a + successes, b + failures) | Coin flip bias estimation |
-| Normal (known variance) | Normal(mu_0, sigma_0) | Normal(weighted mean, smaller variance) | Sensor calibration |
-| Poisson | Gamma(a, b) | Gamma(a + sum of counts, b + n) | Modeling arrival rates |
-| Multinomial | Dirichlet(alpha) | Dirichlet(alpha + counts) | Topic modeling, language models |
+| Bernoulli | $\text{Beta}(a, b)$ | $\text{Beta}(a + \text{successes}, b + \text{failures})$ | Coin flip bias estimation |
+| Normal (known variance) | $\text{Normal}(\mu_0, \sigma_0)$ | $\text{Normal}(\text{weighted mean, smaller variance})$ | Sensor calibration |
+| Poisson | $\text{Gamma}(a, b)$ | $\text{Gamma}(a + \text{sum of counts}, b + n)$ | Modeling arrival rates |
+| Multinomial | $\text{Dirichlet}(\alpha)$ | $\text{Dirichlet}(\alpha + \text{counts})$ | Topic modeling, language models |
Why this matters: without conjugate priors, you need Monte Carlo sampling or variational inference to approximate the posterior. With conjugate priors, you just update two numbers.
-The Beta distribution is the most common conjugate prior in practice. Beta(a, b) represents your belief about a probability parameter. The mean is a/(a+b). The larger a+b, the more concentrated (confident) the distribution.
+The Beta distribution is the most common conjugate prior in practice. $\text{Beta}(a, b)$ represents your belief about a probability parameter. The mean is $a/(a+b)$. The larger $a+b$, the more concentrated (confident) the distribution.
Special cases of the Beta prior:
-- Beta(1, 1) = uniform. You have no opinion about the parameter.
-- Beta(10, 10) = peaked at 0.5. You strongly believe the parameter is near 0.5.
-- Beta(1, 10) = skewed toward 0. You believe the parameter is small.
+- $\text{Beta}(1, 1)$ = uniform. You have no opinion about the parameter.
+- $\text{Beta}(10, 10)$ = peaked at 0.5. You strongly believe the parameter is near 0.5.
+- $\text{Beta}(1, 10)$ = skewed toward 0. You believe the parameter is small.
The update rule is dead simple:
-```
-Prior: Beta(a, b)
-Data: s successes, f failures
-Posterior: Beta(a + s, b + f)
-```
+$$
+\begin{aligned}
+\text{Prior:} \quad & \text{Beta}(a, b) \\
+\text{Data:} \quad & s \text{ successes}, \ f \text{ failures} \\
+\text{Posterior:} \quad & \text{Beta}(a + s, b + f)
+\end{aligned}
+$$
No integrals. No sampling. Just addition.
@@ -376,19 +377,19 @@ Bayesian inference is naturally sequential. Today's posterior becomes tomorrow's
Concrete example: estimating whether a coin is fair.
**Day 1: No data yet.**
-Start with Beta(1, 1) -- a uniform prior. You have no opinion.
+Start with $\text{Beta}(1, 1)$ -- a uniform prior. You have no opinion.
- Prior mean: 0.5
-- Prior is flat across [0, 1]
+- Prior is flat across $[0, 1]$
**Day 2: Observe 7 heads, 3 tails.**
-Posterior = Beta(1 + 7, 1 + 3) = Beta(8, 4)
-- Posterior mean: 8/12 = 0.667
+Posterior $= \text{Beta}(1 + 7, 1 + 3) = \text{Beta}(8, 4)$
+- Posterior mean: $8/12 = 0.667$
- Evidence suggests the coin is biased toward heads
**Day 3: Observe 5 more heads, 5 more tails.**
Use yesterday's posterior as today's prior.
-Posterior = Beta(8 + 5, 4 + 5) = Beta(13, 9)
-- Posterior mean: 13/22 = 0.591
+Posterior $= \text{Beta}(8 + 5, 4 + 5) = \text{Beta}(13, 9)$
+- Posterior mean: $13/22 = 0.591$
- The balanced new data pulled the estimate back toward 0.5
```mermaid
@@ -398,7 +399,7 @@ graph LR
C -->|"5H, 5T"| D["Posterior 2 Beta(13,9) mean = 0.59"]
```
-The order of observations does not matter. Beta(1,1) updated with all 12 heads and 8 tails at once gives Beta(13, 9) -- the same result. Sequential updating and batch updating are mathematically equivalent. But sequential updating lets you make decisions at each step without storing raw data.
+The order of observations does not matter. $\text{Beta}(1,1)$ updated with all 12 heads and 8 tails at once gives $\text{Beta}(13, 9)$ -- the same result. Sequential updating and batch updating are mathematically equivalent. But sequential updating lets you make decisions at each step without storing raw data.
This is the foundation of online learning in production ML systems. Thompson sampling for bandits, incremental recommendation systems, and streaming anomaly detectors all use this pattern.
@@ -410,22 +411,22 @@ Setup: you are testing two button colors. Variant A (blue) and variant B (green)
The Bayesian A/B test:
-1. **Prior.** Start with Beta(1, 1) for both variants. No prior preference.
+1. **Prior.** Start with $\text{Beta}(1, 1)$ for both variants. No prior preference.
2. **Data.** Variant A: 50 clicks out of 1000 views. Variant B: 65 clicks out of 1000 views.
3. **Posteriors.**
- - A: Beta(1 + 50, 1 + 950) = Beta(51, 951). Mean = 0.051
- - B: Beta(1 + 65, 1 + 935) = Beta(66, 936). Mean = 0.066
-4. **Decision.** Compute P(B > A) -- the probability that B's true conversion rate is higher than A's.
+ - A: $\text{Beta}(1 + 50, 1 + 950) = \text{Beta}(51, 951)$. Mean $= 0.051$
+ - B: $\text{Beta}(1 + 65, 1 + 935) = \text{Beta}(66, 936)$. Mean $= 0.066$
+4. **Decision.** Compute $P(B > A)$ -- the probability that B's true conversion rate is higher than A's.
-Computing P(B > A) analytically is hard. But Monte Carlo makes it trivial:
+Computing $P(B > A)$ analytically is hard. But Monte Carlo makes it trivial:
-```
+```text
1. Draw 100,000 samples from Beta(51, 951) -> samples_A
2. Draw 100,000 samples from Beta(66, 936) -> samples_B
3. P(B > A) = fraction of samples where B > A
```
-If P(B > A) > 0.95, you ship variant B. If it is between 0.05 and 0.95, you keep collecting data. If P(B > A) < 0.05, you ship variant A.
+If $P(B > A) > 0.95$, you ship variant B. If it is between 0.05 and 0.95, you keep collecting data. If $P(B > A) < 0.05$, you ship variant A.
Advantages over frequentist A/B testing:
- You get a direct probability statement: "there is a 97% chance B is better"
@@ -435,35 +436,35 @@ Advantages over frequentist A/B testing:
| Aspect | Frequentist A/B | Bayesian A/B |
|--------|----------------|--------------|
-| Output | p-value | P(B > A) |
+| Output | p-value | $P(B > A)$ |
| Interpretation | "How surprising is this data if A=B?" | "How likely is B better than A?" |
| Early stopping | Inflates false positives | Safe at any point (given a well-chosen prior and correctly specified model) |
| Prior knowledge | Not used | Encoded as Beta prior |
-| Decision rule | p < 0.05 | P(B > A) > threshold |
+| Decision rule | $p < 0.05$ | $P(B > A) > \text{threshold}$ |
## Exercises
-1. **Multiple tests.** A patient tests positive twice on independent tests (both 99% accurate, disease prevalence 1 in 10,000). What is P(sick) after both tests? Use the posterior from the first test as the prior for the second.
+1. **Multiple tests.** A patient tests positive twice on independent tests (both 99% accurate, disease prevalence 1 in 10,000). What is $P(\text{sick})$ after both tests? Use the posterior from the first test as the prior for the second.
2. **Smoothing impact.** Run the spam classifier with smoothing values of 0.01, 0.1, 1.0, and 10.0. How do the top word probabilities change? What happens with smoothing=0 and a word that appears only in ham?
-3. **Add features.** Extend the NaiveBayes class to also use message length (short/long) as a feature alongside word counts. Estimate P(short|spam) and P(short|ham) from the training data and fold it into the prediction score.
+3. **Add features.** Extend the NaiveBayes class to also use message length (short/long) as a feature alongside word counts. Estimate $P(\text{short} \mid \text{spam})$ and $P(\text{short} \mid \text{ham})$ from the training data and fold it into the prediction score.
-4. **MAP by hand.** Given observed data (7 heads in 10 coin flips), compute the MAP estimate of the bias using a Beta(2,2) prior. Compare it to the MLE estimate (7/10).
+4. **MAP by hand.** Given observed data (7 heads in 10 coin flips), compute the MAP estimate of the bias using a $\text{Beta}(2,2)$ prior. Compare it to the MLE estimate ($7/10$).
## Key Terms
| Term | What people say | What it actually means |
|------|----------------|----------------------|
-| Prior | "My initial guess" | P(hypothesis) before observing evidence. In ML: the regularization term. |
-| Likelihood | "How well the data fits" | P(evidence\|hypothesis). How probable the observed data is under a specific hypothesis. |
-| Posterior | "My updated belief" | P(hypothesis\|evidence). The prior multiplied by the likelihood, then normalized. |
-| Evidence | "The normalizing constant" | P(data) across all hypotheses. Ensures the posterior sums to 1. |
+| Prior | "My initial guess" | $P(\text{hypothesis})$ before observing evidence. In ML: the regularization term. |
+| Likelihood | "How well the data fits" | $P(\text{evidence} \mid \text{hypothesis})$. How probable the observed data is under a specific hypothesis. |
+| Posterior | "My updated belief" | $P(\text{hypothesis} \mid \text{evidence})$. The prior multiplied by the likelihood, then normalized. |
+| Evidence | "The normalizing constant" | $P(\text{data})$ across all hypotheses. Ensures the posterior sums to 1. |
| Naive Bayes | "That simple text classifier" | A classifier that assumes features are independent given the class. Works well despite the false assumption. |
| Laplace smoothing | "Add-one smoothing" | Adding a small count to every feature to prevent zero probabilities from unseen data. |
-| MLE | "Just use the frequencies" | Choose parameters that maximize P(data\|parameters). No prior. Can overfit with small data. |
-| MAP | "MLE with a prior" | Choose parameters that maximize P(data\|parameters) * P(parameters). Equivalent to regularized MLE. |
-| Log-probability | "Work in log space" | Using log(P) instead of P to avoid floating-point underflow when multiplying many small numbers. |
+| MLE | "Just use the frequencies" | Choose parameters that maximize $P(\text{data} \mid \text{parameters})$. No prior. Can overfit with small data. |
+| MAP | "MLE with a prior" | Choose parameters that maximize $P(\text{data} \mid \text{parameters}) \cdot P(\text{parameters})$. Equivalent to regularized MLE. |
+| Log-probability | "Work in log space" | Using $\log(P)$ instead of $P$ to avoid floating-point underflow when multiplying many small numbers. |
| False positive | "A wrong alarm" | The test says positive, but the true state is negative. Drives the base rate fallacy. |
## Further Reading
diff --git a/phases/01-math-foundations/08-optimization/docs/en.md b/phases/01-math-foundations/08-optimization/docs/en.md
index 2474924615..92fb2fa502 100644
--- a/phases/01-math-foundations/08-optimization/docs/en.md
+++ b/phases/01-math-foundations/08-optimization/docs/en.md
@@ -28,19 +28,17 @@ Every optimizer in deep learning is an answer to the same question: how do you g
Optimization is finding the input values that minimize (or maximize) a function. In machine learning, the function is the loss. The inputs are the model's weights. Training is optimization.
-```
-minimize L(w) where:
- L = loss function
- w = model weights (could be millions of parameters)
-```
+$$
+\text{minimize } L(w) \quad \text{where } L = \text{loss function}, \; w = \text{model weights (could be millions of parameters)}
+$$
### Gradient descent (vanilla)
The simplest optimizer. Compute the gradient of the loss with respect to every weight. Move each weight in the opposite direction of its gradient. Scale the step by the learning rate.
-```
-w = w - lr * gradient
-```
+$$
+w = w - \text{lr} \cdot \text{gradient}
+$$
That is the entire algorithm. One line.
@@ -94,10 +92,12 @@ The noise in SGD and mini-batch is not a bug. It helps escape shallow local mini
Vanilla gradient descent only looks at the current gradient. If the gradient zigzags (common in narrow valleys), progress is slow. Momentum fixes this by accumulating past gradients into a velocity term.
-```
-v = beta * v + gradient
-w = w - lr * v
-```
+$$
+\begin{aligned}
+v &= \beta \cdot v + \text{gradient} \\
+w &= w - \text{lr} \cdot v
+\end{aligned}
+$$
The analogy: a ball rolling downhill. It does not stop and restart at every bump. It builds speed in consistent directions and dampens oscillations.
@@ -116,7 +116,7 @@ graph TD
end
```
-`beta` (typically 0.9) controls how much history to keep. Higher beta means more momentum, smoother paths, but slower response to direction changes.
+$\beta$ (typically 0.9) controls how much history to keep. Higher $\beta$ means more momentum, smoother paths, but slower response to direction changes.
### Adam: adaptive learning rates
@@ -127,17 +127,17 @@ Adam (Adaptive Moment Estimation) tracks two things per weight:
1. First moment (m): running average of gradients (like momentum)
2. Second moment (v): running average of squared gradients (gradient magnitude)
-```
-m = beta1 * m + (1 - beta1) * gradient
-v = beta2 * v + (1 - beta2) * gradient^2
-
-m_hat = m / (1 - beta1^t) bias correction
-v_hat = v / (1 - beta2^t) bias correction
-
-w = w - lr * m_hat / (sqrt(v_hat) + epsilon)
-```
+$$
+\begin{aligned}
+m &= \beta_1 \cdot m + (1 - \beta_1) \cdot \text{gradient} \\
+v &= \beta_2 \cdot v + (1 - \beta_2) \cdot \text{gradient}^2 \\[4pt]
+\hat{m} &= m / (1 - \beta_1^t) \quad &&\text{bias correction} \\
+\hat{v} &= v / (1 - \beta_2^t) \quad &&\text{bias correction} \\[4pt]
+w &= w - \text{lr} \cdot \hat{m} / (\sqrt{\hat{v}} + \epsilon)
+\end{aligned}
+$$
-The division by `sqrt(v_hat)` is the key insight. Weights with large gradients get divided by a large number (small effective step). Weights with small gradients get divided by a small number (large effective step). Each weight gets its own adaptive learning rate.
+The division by $\sqrt{\hat{v}}$ is the key insight. Weights with large gradients get divided by a large number (small effective step). Weights with small gradients get divided by a small number (large effective step). Each weight gets its own adaptive learning rate.
Default hyperparameters: `lr=0.001, beta1=0.9, beta2=0.999, epsilon=1e-8`. These defaults work well for most problems.
@@ -149,14 +149,14 @@ Common schedules:
| Schedule | Formula | Use case |
|----------|---------|----------|
-| Step decay | lr = lr * factor every N epochs | Simple, manual control |
-| Exponential decay | lr = lr_0 * decay^t | Smooth reduction |
-| Cosine annealing | lr = lr_min + 0.5 * (lr_max - lr_min) * (1 + cos(pi * t / T)) | Transformers, modern training |
+| Step decay | $\text{lr} = \text{lr} \cdot \text{factor}$ every $N$ epochs | Simple, manual control |
+| Exponential decay | $\text{lr} = \text{lr}_0 \cdot \text{decay}^t$ | Smooth reduction |
+| Cosine annealing | $\text{lr} = \text{lr}_{\min} + 0.5 \cdot (\text{lr}_{\max} - \text{lr}_{\min}) \cdot (1 + \cos(\pi \cdot t / T))$ | Transformers, modern training |
| Warmup + decay | Linear ramp up, then decay | Large models, prevents early instability |
### Convex vs non-convex
-A convex function has one minimum. Gradient descent always finds it. A quadratic like `f(x) = x^2` is convex.
+A convex function has one minimum. Gradient descent always finds it. A quadratic like $f(x) = x^2$ is convex.
Neural network loss functions are non-convex. They have many local minima, saddle points, and flat regions.
@@ -205,9 +205,9 @@ gradient-descent
The Rosenbrock function is a classic optimization benchmark. Its minimum is at (1, 1) inside a narrow curved valley that is easy to find but hard to follow.
-```
-f(x, y) = (1 - x)^2 + 100 * (y - x^2)^2
-```
+$$
+f(x, y) = (1 - x)^2 + 100 \cdot (y - x^2)^2
+$$
```python
def rosenbrock(params):
@@ -351,9 +351,9 @@ The optimizer classes built here reappear in Phase 3 when we train a neural netw
2. **Momentum comparison.** Run SGD with momentum values [0.0, 0.5, 0.9, 0.99] on the Rosenbrock function. Track the loss at every step. Which momentum value converges fastest? Which overshoots?
-3. **Saddle point escape.** Define the function `f(x, y) = x^2 - y^2` (a saddle point at the origin). Start at (0.01, 0.01). Compare how vanilla GD, SGD with momentum, and Adam behave. Which escapes the saddle point?
+3. **Saddle point escape.** Define the function $f(x, y) = x^2 - y^2$ (a saddle point at the origin). Start at (0.01, 0.01). Compare how vanilla GD, SGD with momentum, and Adam behave. Which escapes the saddle point?
-4. **Implement learning rate decay.** Add an exponential decay schedule to the GradientDescent class: `lr = lr_0 * 0.999^step`. Compare convergence with and without decay on the Rosenbrock function.
+4. **Implement learning rate decay.** Add an exponential decay schedule to the GradientDescent class: $\text{lr} = \text{lr}_0 \cdot 0.999^{\text{step}}$. Compare convergence with and without decay on the Rosenbrock function.
## Key Terms
@@ -365,7 +365,7 @@ The optimizer classes built here reappear in Phase 3 when we train a neural netw
| SGD | "Random sampling" | Stochastic gradient descent. Compute gradient on a random subset instead of the full dataset. Almost always means mini-batch SGD in practice. |
| Mini-batch | "A chunk of data" | A small subset of training data (32-256 samples) used to estimate the gradient. Balances speed and gradient accuracy. |
| Adam | "The default optimizer" | Adaptive Moment Estimation. Tracks per-weight running averages of gradients and squared gradients to give each weight its own learning rate. |
-| Bias correction | "Fix the cold start" | Adam's first and second moments are initialized to zero. Bias correction divides by (1 - beta^t) to compensate during early steps. |
+| Bias correction | "Fix the cold start" | Adam's first and second moments are initialized to zero. Bias correction divides by $(1 - \beta^t)$ to compensate during early steps. |
| Learning rate schedule | "Change lr over time" | A function that adjusts the learning rate during training. Large steps early, small steps late. |
| Convex function | "One valley" | A function where any local minimum is the global minimum. Gradient descent always finds it. Neural network losses are not convex. |
| Saddle point | "Flat but not a minimum" | A point where the gradient is zero but it is a minimum in some directions and a maximum in others. Common in high dimensions. |
diff --git a/phases/01-math-foundations/09-information-theory/docs/en.md b/phases/01-math-foundations/09-information-theory/docs/en.md
index c7560ef449..ce819047cf 100644
--- a/phases/01-math-foundations/09-information-theory/docs/en.md
+++ b/phases/01-math-foundations/09-information-theory/docs/en.md
@@ -30,9 +30,9 @@ When something unlikely happens, it carries more information. A coin landing hea
The information content of an event with probability p is:
-```
-I(x) = -log(p(x))
-```
+$$
+I(x) = -\log(p(x))
+$$
Using log base 2 gives you bits. Using natural log gives you nats. Same idea, different units.
@@ -50,16 +50,18 @@ Certain events carry zero information. You already knew they would happen.
Entropy is the expected surprise across all possible outcomes of a distribution.
-```
-H(P) = -sum( p(x) * log(p(x)) ) for all x
-```
+$$
+H(P) = -\sum_{x} p(x) \log(p(x))
+$$
A fair coin has maximum entropy for a binary variable: 1 bit. A biased coin (99% heads) has low entropy: 0.08 bits. You already know what will happen, so each flip tells you almost nothing.
-```
-Fair coin: H = -(0.5 * log2(0.5) + 0.5 * log2(0.5)) = 1.0 bit
-Biased coin: H = -(0.99 * log2(0.99) + 0.01 * log2(0.01)) = 0.08 bits
-```
+$$
+\begin{aligned}
+\text{Fair coin:} \quad & H = -(0.5 \log_2(0.5) + 0.5 \log_2(0.5)) = 1.0 \text{ bit} \\
+\text{Biased coin:} \quad & H = -(0.99 \log_2(0.99) + 0.01 \log_2(0.01)) = 0.08 \text{ bits}
+\end{aligned}
+$$
Entropy measures the irreducible uncertainty in a distribution. You cannot compress below it.
@@ -67,17 +69,17 @@ Entropy measures the irreducible uncertainty in a distribution. You cannot compr
Cross-entropy measures the average surprise when you use distribution Q to encode events that actually come from distribution P.
-```
-H(P, Q) = -sum( p(x) * log(q(x)) ) for all x
-```
+$$
+H(P, Q) = -\sum_{x} p(x) \log(q(x))
+$$
P is the true distribution (the labels). Q is your model's predictions. If Q matches P perfectly, cross-entropy equals entropy. Any mismatch makes it larger.
In classification, P is a one-hot vector (the true class has probability 1, everything else 0). This simplifies cross-entropy to:
-```
-H(P, Q) = -log(q(true_class))
-```
+$$
+H(P, Q) = -\log(q(\text{true\_class}))
+$$
That is the entire cross-entropy loss formula for classification. Maximize the predicted probability of the correct class.
@@ -85,23 +87,27 @@ That is the entire cross-entropy loss formula for classification. Maximize the p
KL divergence measures how much extra surprise you get from using Q instead of P.
-```
-D_KL(P || Q) = sum( p(x) * log(p(x) / q(x)) ) for all x
- = H(P, Q) - H(P)
-```
+$$
+\begin{aligned}
+D_{KL}(P \parallel Q) &= \sum_{x} p(x) \log\left(\frac{p(x)}{q(x)}\right) \\
+&= H(P, Q) - H(P)
+\end{aligned}
+$$
Cross-entropy is entropy plus KL divergence. Since entropy of the true distribution is constant during training, minimizing cross-entropy is the same as minimizing KL divergence. You are pushing your model's distribution toward the true distribution.
-KL divergence is not symmetric: D_KL(P || Q) != D_KL(Q || P). It is not a true distance metric.
+KL divergence is not symmetric: $D_{KL}(P \parallel Q) \neq D_{KL}(Q \parallel P)$. It is not a true distance metric.
### Mutual Information
Mutual information measures how much knowing one variable tells you about another.
-```
-I(X; Y) = H(X) - H(X|Y)
- = H(X) + H(Y) - H(X, Y)
-```
+$$
+\begin{aligned}
+I(X; Y) &= H(X) - H(X \mid Y) \\
+&= H(X) + H(Y) - H(X, Y)
+\end{aligned}
+$$
If X and Y are independent, mutual information is zero. Knowing one tells you nothing about the other. If they are perfectly correlated, mutual information equals the entropy of either variable.
@@ -109,37 +115,37 @@ In feature selection, high mutual information between a feature and the target m
### Conditional Entropy
-H(Y|X) measures how much uncertainty remains about Y after you observe X.
+$H(Y \mid X)$ measures how much uncertainty remains about Y after you observe X.
-```
-H(Y|X) = H(X,Y) - H(X)
-```
+$$
+H(Y \mid X) = H(X,Y) - H(X)
+$$
Two extremes:
-- If X completely determines Y, then H(Y|X) = 0. Knowing X eliminates all uncertainty about Y. Example: X = temperature in Celsius, Y = temperature in Fahrenheit.
-- If X tells you nothing about Y, then H(Y|X) = H(Y). Knowing X does not reduce your uncertainty at all. Example: X = coin flip, Y = tomorrow's weather.
+- If X completely determines Y, then $H(Y \mid X) = 0$. Knowing X eliminates all uncertainty about Y. Example: X = temperature in Celsius, Y = temperature in Fahrenheit.
+- If X tells you nothing about Y, then $H(Y \mid X) = H(Y)$. Knowing X does not reduce your uncertainty at all. Example: X = coin flip, Y = tomorrow's weather.
-Conditional entropy is always non-negative and never exceeds H(Y):
+Conditional entropy is always non-negative and never exceeds $H(Y)$:
-```
-0 <= H(Y|X) <= H(Y)
-```
+$$
+0 \leq H(Y \mid X) \leq H(Y)
+$$
-In machine learning, conditional entropy appears in decision trees. At each split, the algorithm picks the feature X that minimizes H(Y|X) -- the feature that removes the most uncertainty about the label Y.
+In machine learning, conditional entropy appears in decision trees. At each split, the algorithm picks the feature X that minimizes $H(Y \mid X)$ -- the feature that removes the most uncertainty about the label Y.
### Joint Entropy
-H(X,Y) is the entropy of the joint distribution of X and Y together.
+$H(X,Y)$ is the entropy of the joint distribution of X and Y together.
-```
-H(X,Y) = -sum sum p(x,y) * log(p(x,y)) for all x, y
-```
+$$
+H(X,Y) = -\sum_{x} \sum_{y} p(x,y) \log(p(x,y))
+$$
Key property:
-```
-H(X,Y) <= H(X) + H(Y)
-```
+$$
+H(X,Y) \leq H(X) + H(Y)
+$$
Equality holds when X and Y are independent. If they share information, the joint entropy is less than the sum of individual entropies. The "missing" entropy is exactly the mutual information.
@@ -167,30 +173,32 @@ graph TD
```
The relationships:
-- H(X,Y) = H(X) + H(Y|X) = H(Y) + H(X|Y)
-- I(X;Y) = H(X) - H(X|Y) = H(Y) - H(Y|X)
-- H(X,Y) = H(X) + H(Y) - I(X;Y)
+- $H(X,Y) = H(X) + H(Y \mid X) = H(Y) + H(X \mid Y)$
+- $I(X;Y) = H(X) - H(X \mid Y) = H(Y) - H(Y \mid X)$
+- $H(X,Y) = H(X) + H(Y) - I(X;Y)$
### Mutual Information (Deep Dive)
-Mutual information I(X;Y) quantifies how much knowing one variable reduces uncertainty about the other.
+Mutual information $I(X;Y)$ quantifies how much knowing one variable reduces uncertainty about the other.
-```
-I(X;Y) = H(X) - H(X|Y)
- = H(Y) - H(Y|X)
- = H(X) + H(Y) - H(X,Y)
- = sum sum p(x,y) * log(p(x,y) / (p(x) * p(y)))
-```
+$$
+\begin{aligned}
+I(X;Y) &= H(X) - H(X \mid Y) \\
+&= H(Y) - H(Y \mid X) \\
+&= H(X) + H(Y) - H(X,Y) \\
+&= \sum_{x} \sum_{y} p(x,y) \log\left(\frac{p(x,y)}{p(x) \, p(y)}\right)
+\end{aligned}
+$$
Properties:
-- I(X;Y) >= 0 always. You never lose information by observing something.
-- I(X;Y) = 0 if and only if X and Y are independent.
-- I(X;Y) = I(Y;X). It is symmetric, unlike KL divergence.
-- I(X;X) = H(X). A variable shares all its information with itself.
+- $I(X;Y) \geq 0$ always. You never lose information by observing something.
+- $I(X;Y) = 0$ if and only if X and Y are independent.
+- $I(X;Y) = I(Y;X)$. It is symmetric, unlike KL divergence.
+- $I(X;X) = H(X)$. A variable shares all its information with itself.
**Mutual information for feature selection.** In ML, you want features that are informative about the target. Mutual information gives you a principled way to rank features:
-1. For each feature X_i, compute I(X_i; Y) where Y is the target variable.
+1. For each feature $X_i$, compute $I(X_i; Y)$ where Y is the target variable.
2. Rank features by MI score.
3. Keep the top k features.
@@ -198,19 +206,19 @@ This works for any relationship between feature and target -- linear, nonlinear,
| Method | Detects | Computational cost | Handles categorical? |
|--------|---------|-------------------|---------------------|
-| Pearson correlation | Linear relationships | O(n) | No |
-| Spearman correlation | Monotonic relationships | O(n log n) | No |
-| Mutual information | Any statistical dependency | O(n log n) with binning | Yes |
+| Pearson correlation | Linear relationships | $O(n)$ | No |
+| Spearman correlation | Monotonic relationships | $O(n \log n)$ | No |
+| Mutual information | Any statistical dependency | $O(n \log n)$ with binning | Yes |
### Label Smoothing and Cross-Entropy
Standard classification uses hard targets: [0, 0, 1, 0]. The true class gets probability 1, everything else gets 0. Label smoothing replaces these with soft targets:
-```
-soft_target = (1 - epsilon) * hard_target + epsilon / num_classes
-```
+$$
+\text{soft\_target} = (1 - \epsilon) \cdot \text{hard\_target} + \frac{\epsilon}{\text{num\_classes}}
+$$
-With epsilon = 0.1 and 4 classes:
+With $\epsilon = 0.1$ and 4 classes:
- Hard target: [0, 0, 1, 0]
- Soft target: [0.025, 0.025, 0.925, 0.025]
@@ -224,9 +232,9 @@ Why this helps:
The cross-entropy loss with label smoothing becomes:
-```
-L = (1 - epsilon) * CE(hard_target, prediction) + epsilon * H_uniform(prediction)
-```
+$$
+L = (1 - \epsilon) \cdot \text{CE}(\text{hard\_target}, \text{prediction}) + \epsilon \cdot H_{\text{uniform}}(\text{prediction})
+$$
The second term penalizes predictions that are far from uniform -- a direct regularization on confidence.
@@ -236,13 +244,15 @@ Three perspectives, same conclusion.
**Information theory view.** Cross-entropy measures how many bits you waste by using your model's distribution instead of the true distribution. Minimizing it makes your model the most efficient encoder of reality.
-**Maximum likelihood view.** For N training samples with true classes y_i:
+**Maximum likelihood view.** For N training samples with true classes $y_i$:
-```
-Likelihood = product( q(y_i) )
-Log-likelihood = sum( log(q(y_i)) )
-Negative log-likelihood = -sum( log(q(y_i)) )
-```
+$$
+\begin{aligned}
+\text{Likelihood} &= \prod_{i} q(y_i) \\
+\text{Log-likelihood} &= \sum_{i} \log(q(y_i)) \\
+\text{Negative log-likelihood} &= -\sum_{i} \log(q(y_i))
+\end{aligned}
+$$
That last line is cross-entropy loss. Minimizing cross-entropy = maximizing the likelihood of the training data under your model.
@@ -258,16 +268,18 @@ log base e -> nats (machine learning convention)
log base 10 -> hartleys (rarely used)
```
-1 nat = 1/ln(2) bits = 1.4427 bits. PyTorch and TensorFlow use natural log (nats) by default.
+$1 \text{ nat} = \frac{1}{\ln(2)} \text{ bits} = 1.4427 \text{ bits}$. PyTorch and TensorFlow use natural log (nats) by default.
### Perplexity
Perplexity is the exponential of cross-entropy. It tells you the effective number of equally likely choices the model is uncertain between.
-```
-Perplexity = 2^H(P,Q) (if using bits)
-Perplexity = e^H(P,Q) (if using nats)
-```
+$$
+\begin{aligned}
+\text{Perplexity} &= 2^{H(P,Q)} \quad \text{(if using bits)} \\
+\text{Perplexity} &= e^{H(P,Q)} \quad \text{(if using nats)}
+\end{aligned}
+$$
A language model with perplexity 50 is, on average, as confused as if it had to pick uniformly from 50 possible next tokens. Lower is better.
@@ -445,7 +457,7 @@ You built from scratch what `torch.nn.CrossEntropyLoss()` does internally. Now y
2. A model outputs logits [5.0, 2.0, 0.5] for a sample with true class 1. Compute the cross-entropy loss by hand, then verify with your `cross_entropy_loss` function. What logits would give zero loss?
-3. Show that KL divergence is not symmetric. Pick two distributions P and Q and compute D_KL(P || Q) and D_KL(Q || P). Explain why they differ.
+3. Show that KL divergence is not symmetric. Pick two distributions P and Q and compute $D_{KL}(P \parallel Q)$ and $D_{KL}(Q \parallel P)$. Explain why they differ.
4. Build a function that computes perplexity for a sequence of token predictions. Given a list of (true_token_index, predicted_logits) pairs, return the perplexity of the sequence.
@@ -453,7 +465,7 @@ You built from scratch what `torch.nn.CrossEntropyLoss()` does internally. Now y
| Term | What people say | What it actually means |
|------|----------------|----------------------|
-| Information content | "Surprise" | The number of bits (or nats) needed to encode an event: -log(p) |
+| Information content | "Surprise" | The number of bits (or nats) needed to encode an event: $-\log(p)$ |
| Entropy | "Randomness" | The average surprise across all outcomes of a distribution. Measures irreducible uncertainty. |
| Cross-entropy | "The loss function" | Average surprise when using model distribution Q to encode events from true distribution P. |
| KL divergence | "Distance between distributions" | Extra bits wasted by using Q instead of P. Equals cross-entropy minus entropy. Not symmetric. |
diff --git a/phases/01-math-foundations/11-singular-value-decomposition/docs/en.md b/phases/01-math-foundations/11-singular-value-decomposition/docs/en.md
index 79ea8819c7..2e7d98bd6f 100644
--- a/phases/01-math-foundations/11-singular-value-decomposition/docs/en.md
+++ b/phases/01-math-foundations/11-singular-value-decomposition/docs/en.md
@@ -68,20 +68,19 @@ The columns of U are called left singular vectors. The columns of V are called r
Each component of the SVD has a distinct geometric meaning.
-**Right singular vectors (columns of V):** These form an orthonormal basis for the input space (R^n). They are the directions in input space that the matrix maps to orthogonal directions in output space. Think of them as the natural coordinate system for the domain.
+**Right singular vectors (columns of V):** These form an orthonormal basis for the input space ($\mathbb{R}^n$). They are the directions in input space that the matrix maps to orthogonal directions in output space. Think of them as the natural coordinate system for the domain.
**Singular values (diagonal of Sigma):** These are the scaling factors. The i-th singular value tells you how much the matrix stretches vectors along the i-th right singular vector. A singular value of zero means the matrix crushes that direction entirely.
-**Left singular vectors (columns of U):** These form an orthonormal basis for the output space (R^m). The i-th left singular vector is the direction in output space where the i-th right singular vector lands (after scaling).
+**Left singular vectors (columns of U):** These form an orthonormal basis for the output space ($\mathbb{R}^m$). The i-th left singular vector is the direction in output space where the i-th right singular vector lands (after scaling).
The relationship between them:
-```
-A * v_i = sigma_i * u_i
+$$
+A v_i = \sigma_i u_i
+$$
-The matrix A takes the i-th right singular vector v_i,
-scales it by sigma_i, and maps it to the i-th left singular vector u_i.
-```
+The matrix $A$ takes the $i$-th right singular vector $v_i$, scales it by $\sigma_i$, and maps it to the $i$-th left singular vector $u_i$.
This gives you a coordinate-by-coordinate picture of what any matrix does.
@@ -89,85 +88,87 @@ This gives you a coordinate-by-coordinate picture of what any matrix does.
The SVD can be written as a sum of rank-1 matrices:
-```
-A = sigma_1 * u_1 * v_1^T + sigma_2 * u_2 * v_2^T + ... + sigma_r * u_r * v_r^T
+$$
+A = \sigma_1 u_1 v_1^T + \sigma_2 u_2 v_2^T + \cdots + \sigma_r u_r v_r^T
+$$
-Each term sigma_i * u_i * v_i^T is a rank-1 matrix (an outer product).
-The full matrix is the sum of r such matrices, where r is the rank.
-```
+Each term $\sigma_i u_i v_i^T$ is a rank-1 matrix (an outer product). The full matrix is the sum of $r$ such matrices, where $r$ is the rank.
This form is the foundation of low-rank approximation. Each term adds one layer of structure. The first term captures the single most important pattern. The second captures the next most important. And so on. Truncating this sum gives you the best possible approximation at any given rank.
-```
-Rank-1 approx: A_1 = sigma_1 * u_1 * v_1^T
- (captures the dominant pattern)
+Rank-1 approx: $A_1 = \sigma_1 u_1 v_1^T$ (captures the dominant pattern).
-Rank-2 approx: A_2 = sigma_1 * u_1 * v_1^T + sigma_2 * u_2 * v_2^T
- (captures the two most important patterns)
+Rank-2 approx: $A_2 = \sigma_1 u_1 v_1^T + \sigma_2 u_2 v_2^T$ (captures the two most important patterns).
-Rank-k approx: A_k = sum of top k terms
- (optimal by the Eckart-Young theorem)
-```
+Rank-k approx: $A_k =$ sum of top $k$ terms (optimal by the Eckart-Young theorem).
### Relationship to eigendecomposition
-SVD and eigendecomposition are deeply connected. The singular values and vectors of A come directly from the eigenvalues and eigenvectors of A^T A and A A^T.
+SVD and eigendecomposition are deeply connected. The singular values and vectors of $A$ come directly from the eigenvalues and eigenvectors of $A^T A$ and $A A^T$.
-```
-A^T A = V * Sigma^T * U^T * U * Sigma * V^T
- = V * Sigma^T * Sigma * V^T
- = V * D * V^T
+$$
+\begin{aligned}
+A^T A &= V \Sigma^T U^T U \Sigma V^T \\
+ &= V \Sigma^T \Sigma V^T \\
+ &= V D V^T
+\end{aligned}
+$$
-where D = Sigma^T * Sigma is a diagonal matrix with sigma_i^2 on the diagonal.
+where $D = \Sigma^T \Sigma$ is a diagonal matrix with $\sigma_i^2$ on the diagonal.
So:
-- The right singular vectors (V) are eigenvectors of A^T A
-- The singular values squared (sigma_i^2) are eigenvalues of A^T A
+- The right singular vectors ($V$) are eigenvectors of $A^T A$
+- The singular values squared ($\sigma_i^2$) are eigenvalues of $A^T A$
Similarly:
-A A^T = U * Sigma * V^T * V * Sigma^T * U^T
- = U * Sigma * Sigma^T * U^T
+
+$$
+\begin{aligned}
+A A^T &= U \Sigma V^T V \Sigma^T U^T \\
+ &= U \Sigma \Sigma^T U^T
+\end{aligned}
+$$
So:
-- The left singular vectors (U) are eigenvectors of A A^T
-- The eigenvalues of A A^T are also sigma_i^2
-```
+- The left singular vectors ($U$) are eigenvectors of $A A^T$
+- The eigenvalues of $A A^T$ are also $\sigma_i^2$
This connection tells you three things:
1. Singular values are always real and non-negative (they are square roots of eigenvalues of a positive semi-definite matrix).
-2. You could compute SVD via eigendecomposition of A^T A, but this squares the condition number and loses numerical precision. Dedicated SVD algorithms avoid this.
+2. You could compute SVD via eigendecomposition of $A^T A$, but this squares the condition number and loses numerical precision. Dedicated SVD algorithms avoid this.
3. When A is square and symmetric positive semi-definite, SVD and eigendecomposition are the same thing.
### Truncated SVD: low-rank approximation
The Eckart-Young-Mirsky theorem states that the best rank-k approximation to A (in both Frobenius and spectral norm) is obtained by keeping only the top k singular values and their corresponding vectors:
-```
-A_k = U_k * Sigma_k * V_k^T
+$$
+A_k = U_k \Sigma_k V_k^T
+$$
-where:
- U_k is m x k (first k columns of U)
- Sigma_k is k x k (top-left k x k block of Sigma)
- V_k is n x k (first k columns of V)
+where $U_k$ is $m \times k$ (first $k$ columns of $U$), $\Sigma_k$ is $k \times k$ (top-left $k \times k$ block of $\Sigma$), and $V_k$ is $n \times k$ (first $k$ columns of $V$).
-Approximation error = sigma_{k+1} (in spectral norm)
- = sqrt(sigma_{k+1}^2 + ... + sigma_r^2) (in Frobenius norm)
-```
+$$
+\begin{aligned}
+\text{Approximation error} &= \sigma_{k+1} \quad (\text{in spectral norm}) \\
+ &= \sqrt{\sigma_{k+1}^2 + \cdots + \sigma_r^2} \quad (\text{in Frobenius norm})
+\end{aligned}
+$$
This is not just "a good" approximation. It is provably the best possible approximation of rank k. No other rank-k matrix is closer to A.
| Component | Relative magnitude | Kept in rank-3 approx? |
|-----------|-------------------|------------------------|
-| sigma_1 | Largest | Yes |
-| sigma_2 | Large | Yes |
-| sigma_3 | Medium-large | Yes |
-| sigma_4 | Medium | No (error) |
-| sigma_5 | Medium-small | No (error) |
-| sigma_6 | Small | No (error) |
-| sigma_7 | Very small | No (error) |
-| sigma_8 | Tiny | No (error) |
+| $\sigma_1$ | Largest | Yes |
+| $\sigma_2$ | Large | Yes |
+| $\sigma_3$ | Medium-large | Yes |
+| $\sigma_4$ | Medium | No (error) |
+| $\sigma_5$ | Medium-small | No (error) |
+| $\sigma_6$ | Small | No (error) |
+| $\sigma_7$ | Very small | No (error) |
+| $\sigma_8$ | Tiny | No (error) |
-Keep top 3: A_3 captures the three largest singular values. Error = remaining values (sigma_4 through sigma_8).
+Keep top 3: $A_3$ captures the three largest singular values. Error = remaining values ($\sigma_4$ through $\sigma_8$).
If singular values decay fast, a small k captures most of the matrix. If they decay slowly, the matrix has no low-rank structure.
@@ -253,23 +254,23 @@ Noisy data has signal concentrated in the top singular values and noise spread a
| Component | Magnitude | Type |
|-----------|-----------|------|
-| sigma_1 | Very large | Signal |
-| sigma_2 | Large | Signal |
-| sigma_3 | Medium | Signal |
-| sigma_4 | Near zero | Negligible |
-| sigma_5 | Near zero | Negligible |
+| $\sigma_1$ | Very large | Signal |
+| $\sigma_2$ | Large | Signal |
+| $\sigma_3$ | Medium | Signal |
+| $\sigma_4$ | Near zero | Negligible |
+| $\sigma_5$ | Near zero | Negligible |
**Noisy signal singular values (noise adds to all):**
| Component | Magnitude | Type |
|-----------|-----------|------|
-| sigma_1 | Very large | Signal |
-| sigma_2 | Large | Signal |
-| sigma_3 | Medium | Signal |
-| sigma_4 | Small | Noise |
-| sigma_5 | Small | Noise |
-| sigma_6 | Small | Noise |
-| sigma_7 | Small | Noise |
+| $\sigma_1$ | Very large | Signal |
+| $\sigma_2$ | Large | Signal |
+| $\sigma_3$ | Medium | Signal |
+| $\sigma_4$ | Small | Noise |
+| $\sigma_5$ | Small | Noise |
+| $\sigma_6$ | Small | Noise |
+| $\sigma_7$ | Small | Noise |
```mermaid
graph TD
@@ -283,41 +284,40 @@ This is used in signal processing, scientific measurement, and data cleaning. An
### Pseudoinverse via SVD
-The Moore-Penrose pseudoinverse A+ generalizes matrix inversion to non-square and singular matrices. SVD makes computing it trivial.
+The Moore-Penrose pseudoinverse $A^+$ generalizes matrix inversion to non-square and singular matrices. SVD makes computing it trivial.
-```
-If A = U * Sigma * V^T, then:
+If $A = U \Sigma V^T$, then:
-A+ = V * Sigma+ * U^T
+$$
+A^+ = V \Sigma^+ U^T
+$$
-where Sigma+ is formed by:
- 1. Transpose Sigma (swap rows and columns)
- 2. Replace each non-zero diagonal entry sigma_i with 1/sigma_i
+where $\Sigma^+$ is formed by:
+ 1. Transpose $\Sigma$ (swap rows and columns)
+ 2. Replace each non-zero diagonal entry $\sigma_i$ with $1/\sigma_i$
3. Leave zeros as zeros
-For A (m x n): A+ is (n x m)
-For Sigma (m x n): Sigma+ is (n x m)
-```
+For $A$ ($m \times n$), $A^+$ is $n \times m$. For $\Sigma$ ($m \times n$), $\Sigma^+$ is $n \times m$.
-The pseudoinverse solves least-squares problems. If Ax = b has no exact solution (overdetermined system), then x = A+ b is the least-squares solution (minimizes ||Ax - b||).
+The pseudoinverse solves least-squares problems. If $Ax = b$ has no exact solution (overdetermined system), then $x = A^+ b$ is the least-squares solution (minimizes $\lVert Ax - b \rVert$).
-```
Overdetermined system (more equations than unknowns):
- [1 1] [3]
- [2 1] x = [5] No exact solution exists.
- [3 1] [6]
+$$
+\begin{bmatrix} 1 & 1 \\ 2 & 1 \\ 3 & 1 \end{bmatrix} x = \begin{bmatrix} 3 \\ 5 \\ 6 \end{bmatrix}
+$$
- x_ls = A+ b = V * Sigma+ * U^T * b
+No exact solution exists.
- This gives the x that minimizes the sum of squared residuals.
- Same result as the normal equations (A^T A)^(-1) A^T b,
- but numerically more stable.
-```
+$$
+x_{ls} = A^+ b = V \Sigma^+ U^T b
+$$
+
+This gives the $x$ that minimizes the sum of squared residuals. Same result as the normal equations $(A^T A)^{-1} A^T b$, but numerically more stable.
### Numerical stability advantages
-Computing eigendecomposition of A^T A squares the singular values (eigenvalues of A^T A are sigma_i^2). This squares the condition number, amplifying numerical errors.
+Computing eigendecomposition of $A^T A$ squares the singular values (eigenvalues of $A^T A$ are $\sigma_i^2$). This squares the condition number, amplifying numerical errors.
```
Example:
@@ -332,31 +332,31 @@ Example:
(6 extra digits of precision lost)
```
-Modern SVD algorithms (Golub-Kahan bidiagonalization) work directly on A, never forming A^T A. This is why you should always prefer `np.linalg.svd(A)` over `np.linalg.eig(A.T @ A)`.
+Modern SVD algorithms (Golub-Kahan bidiagonalization) work directly on $A$, never forming $A^T A$. This is why you should always prefer `np.linalg.svd(A)` over `np.linalg.eig(A.T @ A)`.
### Connection to PCA
PCA IS SVD on centered data. This is not an analogy. It is literally the same computation.
-```
-Given data matrix X (n_samples x n_features), centered (mean subtracted):
-
-Covariance matrix: C = (1/(n-1)) * X^T X
+Given data matrix $X$ ($n_\text{samples} \times n_\text{features}$), centered (mean subtracted), the covariance matrix is:
-PCA finds eigenvectors of C. But:
+$$
+C = \frac{1}{n-1} X^T X
+$$
- X = U * Sigma * V^T (SVD of X)
+PCA finds eigenvectors of $C$. But:
- X^T X = V * Sigma^2 * V^T
+$$
+\begin{aligned}
+X &= U \Sigma V^T \quad (\text{SVD of } X) \\
+X^T X &= V \Sigma^2 V^T \\
+C &= \frac{1}{n-1} V \Sigma^2 V^T
+\end{aligned}
+$$
- C = (1/(n-1)) * V * Sigma^2 * V^T
+So the principal components are exactly the right singular vectors $V$. The explained variance for each component is $\sigma_i^2 / (n-1)$.
-So the principal components are exactly the right singular vectors V.
-The explained variance for each component is sigma_i^2 / (n-1).
-
-In sklearn, PCA is implemented using SVD, not eigendecomposition.
-It is faster and more numerically stable.
-```
+In sklearn, PCA is implemented using SVD, not eigendecomposition. It is faster and more numerically stable.
This means everything you learned about dimensionality reduction in Lesson 10 is SVD under the hood. PCA is the most common application of SVD in machine learning.
@@ -368,7 +368,7 @@ svd-rank-reconstruction
### Step 1: SVD from scratch using power iteration
-The idea: to find the largest singular value and its vectors, use power iteration on A^T A (or A A^T). Then deflate the matrix and repeat for the next singular value.
+The idea: to find the largest singular value and its vectors, use power iteration on $A^T A$ (or $A A^T$). Then deflate the matrix and repeat for the next singular value.
```python
import numpy as np
@@ -443,7 +443,7 @@ def compress_image_svd(image_matrix, k):
compressed = U[:, :k] @ np.diag(S[:k]) @ Vt[:k, :]
return compressed
-image = np.random.seed(42)
+np.random.seed(42)
rows, cols = 200, 300
image = np.random.randn(rows, cols)
@@ -513,7 +513,7 @@ This lesson produces:
## Exercises
-1. Implement the full SVD from scratch without using power iteration. Instead, compute the eigendecomposition of A^T A to get V and the singular values, then compute U = A V Sigma^{-1}. Compare numerical accuracy with your power iteration version and with NumPy.
+1. Implement the full SVD from scratch without using power iteration. Instead, compute the eigendecomposition of $A^T A$ to get $V$ and the singular values, then compute $U = A V \Sigma^{-1}$. Compare numerical accuracy with your power iteration version and with NumPy.
2. Load a real grayscale image (or convert one to grayscale). Compress it at ranks 1, 5, 10, 25, 50, 100. For each rank, compute the compression ratio and the relative error. Find the rank where the image becomes visually acceptable.
@@ -521,20 +521,20 @@ This lesson produces:
4. Create a 100x50 document-term matrix with 3 synthetic topics. Each topic has 5 associated terms. Add noise. Apply SVD and verify that the top 3 singular values are much larger than the rest. Project documents into the 3D latent space and check that documents from the same topic cluster together.
-5. Generate a clean low-rank matrix (rank 3, size 50x40) and add Gaussian noise at different levels (sigma = 0.1, 0.5, 1.0, 2.0). For each noise level, find the optimal truncation rank by sweeping k from 1 to 40 and measuring reconstruction error against the clean matrix. Plot how the optimal k changes with noise level.
+5. Generate a clean low-rank matrix (rank 3, size 50x40) and add Gaussian noise at different levels ($\sigma$ = 0.1, 0.5, 1.0, 2.0). For each noise level, find the optimal truncation rank by sweeping k from 1 to 40 and measuring reconstruction error against the clean matrix. Plot how the optimal k changes with noise level.
## Key Terms
| Term | What people say | What it actually means |
|------|----------------|----------------------|
-| SVD | "Factor any matrix" | Decompose A into U Sigma V^T where U and V are orthogonal and Sigma is diagonal with non-negative entries. Works for any matrix of any shape. |
-| Singular value | "How important this component is" | The i-th diagonal entry of Sigma. Measures how much the matrix stretches along the i-th principal direction. Always non-negative, sorted in decreasing order. |
-| Left singular vector | "Output direction" | A column of U. The direction in output space that the i-th right singular vector maps to (after scaling by sigma_i). |
-| Right singular vector | "Input direction" | A column of V. The direction in input space that the matrix maps to the i-th left singular vector (after scaling by sigma_i). |
+| SVD | "Factor any matrix" | Decompose $A$ into $U \Sigma V^T$ where $U$ and $V$ are orthogonal and $\Sigma$ is diagonal with non-negative entries. Works for any matrix of any shape. |
+| Singular value | "How important this component is" | The $i$-th diagonal entry of $\Sigma$. Measures how much the matrix stretches along the $i$-th principal direction. Always non-negative, sorted in decreasing order. |
+| Left singular vector | "Output direction" | A column of $U$. The direction in output space that the $i$-th right singular vector maps to (after scaling by $\sigma_i$). |
+| Right singular vector | "Input direction" | A column of $V$. The direction in input space that the matrix maps to the $i$-th left singular vector (after scaling by $\sigma_i$). |
| Truncated SVD | "Low-rank approximation" | Keep only the top k singular values and their vectors. Produces the provably best rank-k approximation to the original matrix (Eckart-Young theorem). |
| Rank | "True dimensionality" | The number of non-zero singular values. Tells you how many independent directions the matrix actually uses. |
-| Pseudoinverse | "Generalized inverse" | V Sigma+ U^T. Inverts non-zero singular values, leaves zeros as zeros. Solves least-squares problems for non-square or singular matrices. |
-| Condition number | "How sensitive to errors" | sigma_max / sigma_min. A large condition number means small input changes cause large output changes. SVD reveals this directly. |
+| Pseudoinverse | "Generalized inverse" | $V \Sigma^+ U^T$. Inverts non-zero singular values, leaves zeros as zeros. Solves least-squares problems for non-square or singular matrices. |
+| Condition number | "How sensitive to errors" | $\sigma_{\max} / \sigma_{\min}$. A large condition number means small input changes cause large output changes. SVD reveals this directly. |
| Latent factor | "Hidden variable" | A dimension in the low-rank space discovered by SVD. In recommendations, a latent factor might correspond to genre preference. In NLP, it might correspond to a topic. |
| Frobenius norm | "Total matrix size" | Square root of the sum of squared entries. Equals the square root of the sum of squared singular values. Used to measure approximation error. |
| Eckart-Young theorem | "SVD gives the best compression" | For any target rank k, the truncated SVD minimizes the approximation error over all possible rank-k matrices. |
diff --git a/phases/01-math-foundations/13-numerical-stability/docs/en.md b/phases/01-math-foundations/13-numerical-stability/docs/en.md
index f59c8a0611..fcd5c785e4 100644
--- a/phases/01-math-foundations/13-numerical-stability/docs/en.md
+++ b/phases/01-math-foundations/13-numerical-stability/docs/en.md
@@ -96,7 +96,7 @@ Relative error: 19.2%
That is a 19% relative error from a single subtraction. In ML, this happens whenever you:
-- Compute variance of data with a large mean: `E[x^2] - E[x]^2` when E[x] is large
+- Compute variance of data with a large mean: $E[x^2] - E[x]^2$ when $E[x]$ is large
- Subtract nearly equal log-probabilities
- Compute finite-difference gradients with too-small epsilon
@@ -141,23 +141,25 @@ Computing `log(sum(exp(x_i)))` directly is numerically dangerous. If any `x_i` i
The trick: subtract the maximum value before exponentiating.
-```
-log(sum(exp(x_i))) = max(x) + log(sum(exp(x_i - max(x))))
-```
+$$
+\log\left(\sum_i \exp(x_i)\right) = \max(x) + \log\left(\sum_i \exp(x_i - \max(x))\right)
+$$
Why this works: after subtracting `max(x)`, the largest exponent is `exp(0) = 1`. No overflow is possible. At least one term in the sum is 1, so the sum is at least 1, and `log(1) = 0`. No underflow to `-inf` is possible.
Proof:
-```
-log(sum(exp(x_i)))
-= log(sum(exp(x_i - c + c))) (add and subtract c)
-= log(sum(exp(x_i - c) * exp(c))) (exp(a+b) = exp(a)*exp(b))
-= log(exp(c) * sum(exp(x_i - c))) (factor out exp(c))
-= c + log(sum(exp(x_i - c))) (log(a*b) = log(a) + log(b))
-```
+$$
+\begin{aligned}
+\log\left(\sum_i \exp(x_i)\right)
+&= \log\left(\sum_i \exp(x_i - c + c)\right) && \text{(add and subtract } c\text{)} \\
+&= \log\left(\sum_i \exp(x_i - c) \cdot \exp(c)\right) && (\exp(a+b) = \exp(a)\exp(b)) \\
+&= \log\left(\exp(c) \cdot \sum_i \exp(x_i - c)\right) && \text{(factor out } \exp(c)\text{)} \\
+&= c + \log\left(\sum_i \exp(x_i - c)\right) && (\log(ab) = \log(a) + \log(b))
+\end{aligned}
+$$
-Set `c = max(x)` and overflow is eliminated.
+Set $c = \max(x)$ and overflow is eliminated.
This trick appears everywhere in ML:
- Softmax normalization
@@ -170,9 +172,9 @@ This trick appears everywhere in ML:
Softmax converts logits to probabilities:
-```
-softmax(x_i) = exp(x_i) / sum(exp(x_j))
-```
+$$
+\text{softmax}(x_i) = \frac{\exp(x_i)}{\sum_j \exp(x_j)}
+$$
Without the trick, logits of [100, 101, 102] cause overflow:
@@ -242,19 +244,19 @@ Analytical gradients (from backpropagation) can have bugs. Numerical gradient ch
The centered difference formula:
-```
-df/dx ~= (f(x + h) - f(x - h)) / (2h)
-```
+$$
+\frac{df}{dx} \approx \frac{f(x + h) - f(x - h)}{2h}
+$$
-This is O(h^2) accurate, much better than the forward difference `(f(x+h) - f(x)) / h` which is only O(h).
+This is $O(h^2)$ accurate, much better than the forward difference $\frac{f(x+h) - f(x)}{h}$ which is only $O(h)$.
Choosing h: too large and the approximation is wrong. Too small and catastrophic cancellation destroys the answer. `h = 1e-5` to `1e-7` is typical.
The check: compute the relative difference between analytical and numerical gradients.
-```
-relative_error = |grad_analytical - grad_numerical| / max(|grad_analytical|, |grad_numerical|, 1e-8)
-```
+$$
+\text{relative\_error} = \frac{|\text{grad}_\text{analytical} - \text{grad}_\text{numerical}|}{\max(|\text{grad}_\text{analytical}|, |\text{grad}_\text{numerical}|, 10^{-8})}
+$$
Rules of thumb:
- relative_error < 1e-7: perfect, gradient is correct
@@ -350,9 +352,9 @@ Layer 50: values in [0, inf]
Normalization recenters and rescales activations at every layer:
-```
-LayerNorm(x) = (x - mean(x)) / (std(x) + epsilon) * gamma + beta
-```
+$$
+\text{LayerNorm}(x) = \frac{x - \text{mean}(x)}{\text{std}(x) + \epsilon} \cdot \gamma + \beta
+$$
The `epsilon` (typically 1e-5) prevents division by zero when all activations are identical. The learned parameters `gamma` and `beta` let the network restore any scale it needs.
@@ -364,7 +366,7 @@ This keeps values in a numerically safe range throughout the network, preventing
Cause: logits grew too large, softmax overflowed. Or learning rate is too high and weights diverged.
Fix: use stable softmax (max subtraction), reduce learning rate, add gradient clipping.
-**Bug: Loss is stuck at log(num_classes).**
+**Bug: Loss is stuck at $\log(\text{num\_classes})$.**
Cause: model outputs are near-uniform probabilities. Often means gradients are vanishing or the model is not learning at all.
Fix: check that data labels are correct, verify the loss function, check for dead ReLUs.
@@ -568,13 +570,13 @@ These stable implementations reappear in Phase 3 when building the training loop
## Exercises
-1. **Catastrophic cancellation.** Compute the variance of [1000000.0, 1000001.0, 1000002.0] using the naive formula `E[x^2] - E[x]^2` in float32. Then compute it using Welford's online algorithm. Compare the errors against the true variance (0.6667).
+1. **Catastrophic cancellation.** Compute the variance of [1000000.0, 1000001.0, 1000002.0] using the naive formula $E[x^2] - E[x]^2$ in float32. Then compute it using Welford's online algorithm. Compare the errors against the true variance (0.6667).
-2. **Precision hunt.** Find the smallest positive float32 value `x` such that `1.0 + x == 1.0` in Python. This is the machine epsilon. Verify it matches `numpy.finfo(numpy.float32).eps`.
+2. **Precision hunt.** Find the smallest positive float32 value `x` such that `1.0 + x > 1.0` in float32. This is the machine epsilon. Verify it matches `numpy.finfo(numpy.float32).eps`.
3. **Log-sum-exp edge cases.** Test your `logsumexp_stable` function with: (a) all values equal, (b) one value much larger than the rest, (c) all values very negative (-1000). Verify it gives correct results where the naive version fails.
-4. **Gradient checking a neural network layer.** Implement a single linear layer `y = Wx + b` and its analytical backward pass. Use `numerical_gradient` to verify correctness for a 3x2 weight matrix.
+4. **Gradient checking a neural network layer.** Implement a single linear layer $y = Wx + b$ and its analytical backward pass. Use `numerical_gradient` to verify correctness for a 3x2 weight matrix.
5. **Loss scaling experiment.** Simulate training with float16: create random gradients in the range [1e-9, 1e-3], convert to float16, and measure what fraction become zero. Then apply loss scaling (multiply by 1024), convert to float16, scale back, and measure the zero fraction again.
@@ -583,7 +585,7 @@ These stable implementations reappear in Phase 3 when building the training loop
| Term | What people say | What it actually means |
|------|----------------|----------------------|
| IEEE 754 | "The float standard" | International standard defining binary floating point formats, rounding rules, and special values (inf, nan). Every modern CPU and GPU implements it. |
-| Machine epsilon | "The precision limit" | The smallest value e such that 1.0 + e != 1.0 in a given float format. For float32, it is about 1.19e-7. |
+| Machine epsilon | "The precision limit" | The smallest value $e$ such that $1.0 + e \neq 1.0$ in a given float format. For float32, it is about 1.19e-7. |
| Catastrophic cancellation | "Precision loss from subtraction" | When subtracting nearly equal floating point numbers, significant digits cancel and rounding noise dominates the result. |
| Overflow | "Number too big" | A result exceeds the maximum representable value and becomes inf. exp(89) overflows float32. |
| Underflow | "Number too small" | A result is closer to zero than the smallest representable positive number and becomes 0.0. exp(-104) underflows float32. |
diff --git a/phases/01-math-foundations/14-norms-and-distances/docs/en.md b/phases/01-math-foundations/14-norms-and-distances/docs/en.md
index 6e60706291..227ad6b431 100644
--- a/phases/01-math-foundations/14-norms-and-distances/docs/en.md
+++ b/phases/01-math-foundations/14-norms-and-distances/docs/en.md
@@ -28,15 +28,15 @@ This lesson builds every major distance function from scratch, shows you when ea
### Norms: measuring vector magnitude
-A norm measures the "size" of a vector. Every distance function between two vectors can be written as the norm of their difference: d(a, b) = ||a - b||. So understanding norms is understanding distances.
+A norm measures the "size" of a vector. Many vector distances can be written as the norm of their difference: $d(a, b) = \|a - b\|$. So understanding norms is understanding many distances.
### L1 Norm (Manhattan distance)
The L1 norm sums the absolute values of all components.
-```
-||x||_1 = |x_1| + |x_2| + ... + |x_n|
-```
+$$
+\|x\|_1 = |x_1| + |x_2| + \dots + |x_n|
+$$
It is called Manhattan distance because it measures how far you walk on a city grid where you can only move along axes. No diagonals.
@@ -54,7 +54,7 @@ When to use L1:
- When you want robustness to outliers (a single huge difference does not dominate)
- Feature selection problems (L1 regularization promotes sparsity)
-Connection to L1 regularization (Lasso): adding ||w||_1 to your loss function penalizes the sum of absolute weight values. This pushes small weights to exactly zero, performing automatic feature selection. The L1 penalty creates diamond-shaped constraint regions in weight space, and the corners of diamonds lie on the axes where some weights are zero.
+Connection to L1 regularization (Lasso): adding $\|w\|_1$ to your loss function penalizes the sum of absolute weight values. This pushes small weights to exactly zero, performing automatic feature selection. The L1 penalty creates diamond-shaped constraint regions in weight space, and the corners of diamonds lie on the axes where some weights are zero.
Connection to loss functions: Mean Absolute Error (MAE) is the average L1 distance between predictions and targets. It penalizes all errors linearly, making it robust to outliers compared to MSE.
@@ -62,9 +62,9 @@ Connection to loss functions: Mean Absolute Error (MAE) is the average L1 distan
The L2 norm is the straight-line distance. Square root of the sum of squared components.
-```
-||x||_2 = sqrt(x_1^2 + x_2^2 + ... + x_n^2)
-```
+$$
+\|x\|_2 = \sqrt{x_1^2 + x_2^2 + \dots + x_n^2}
+$$
This is the distance you learned in geometry class. Pythagoras in n dimensions.
@@ -83,7 +83,7 @@ When to use L2:
- Physical distances (spatial data, sensor readings)
- Image similarity at the pixel level
-Connection to L2 regularization (Ridge): adding ||w||_2^2 to your loss function penalizes large weights. Unlike L1, it does not push weights to zero. It shrinks all weights toward zero proportionally. The L2 penalty creates circular constraint regions, so there are no corners on axes. Weights get small but rarely exactly zero.
+Connection to L2 regularization (Ridge): adding $\|w\|_2^2$ to your loss function penalizes large weights. Unlike L1, it does not push weights to zero. It shrinks all weights toward zero proportionally. The L2 penalty creates circular constraint regions, so there are no corners on axes. Weights get small but rarely exactly zero.
Connection to loss functions: Mean Squared Error (MSE) is the average of L2 distances squared. Squaring penalizes large errors more heavily than small ones.
@@ -96,9 +96,9 @@ MSE (L2 loss): (y - y_hat)^2 Quadratic penalty. Sensitive to outliers.
L1 and L2 are special cases of the Lp norm:
-```
-||x||_p = (|x_1|^p + |x_2|^p + ... + |x_n|^p)^(1/p)
-```
+$$
+\|x\|_p = \left( |x_1|^p + |x_2|^p + \dots + |x_n|^p \right)^{1/p}
+$$
Different values of p produce different shaped "unit balls" (the set of all points at distance 1 from the origin):
@@ -113,9 +113,9 @@ p=inf: Square/hypercube (flat sides along axes)
As p approaches infinity, the Lp norm converges to the maximum absolute component.
-```
-||x||_inf = max(|x_1|, |x_2|, ..., |x_n|)
-```
+$$
+\|x\|_\infty = \max(|x_1|, |x_2|, \dots, |x_n|)
+$$
The distance between two points is determined by the single dimension where they differ the most. All other dimensions are ignored.
@@ -135,13 +135,13 @@ When to use L-infinity:
Cosine similarity measures the angle between two vectors, ignoring their magnitudes.
-```
-cos_sim(a, b) = (a . b) / (||a||_2 * ||b||_2)
-```
+$$
+\text{cos\_sim}(a, b) = \frac{a \cdot b}{\|a\|_2 \cdot \|b\|_2}
+$$
It ranges from -1 (opposite directions) to +1 (same direction). Perpendicular vectors have cosine similarity 0.
-Cosine distance converts it to a distance: cosine_distance = 1 - cosine_similarity. This ranges from 0 (identical direction) to 2 (opposite direction).
+Cosine distance converts it to a distance: $\text{cosine\_distance} = 1 - \text{cosine\_similarity}$. This ranges from 0 (identical direction) to 2 (opposite direction).
```
a = (1, 0) b = (1, 1)
@@ -162,17 +162,18 @@ When to use cosine similarity:
The dot product of two vectors is:
-```
-a . b = a_1*b_1 + a_2*b_2 + ... + a_n*b_n
- = ||a|| * ||b|| * cos(angle)
-```
+$$
+\begin{aligned}
+a \cdot b &= a_1 b_1 + a_2 b_2 + \dots + a_n b_n \\
+&= \|a\| \cdot \|b\| \cdot \cos(\text{angle})
+\end{aligned}
+$$
Cosine similarity is the dot product normalized by both magnitudes. When both vectors are already unit-normalized (magnitude = 1), dot product and cosine similarity are identical.
-```
-If ||a|| = 1 and ||b|| = 1:
- a . b = cos(angle between a and b)
-```
+$$
+\text{If } \|a\| = 1 \text{ and } \|b\| = 1: \quad a \cdot b = \cos(\text{angle between } a \text{ and } b)
+$$
When they differ: dot product includes magnitude information. A vector with larger magnitude gets a higher dot product score. This matters in some retrieval systems where you want "popular" items to rank higher. The magnitude acts as an implicit quality or importance signal.
@@ -197,13 +198,13 @@ Euclidean distance treats all dimensions equally. But if your features are corre
Mahalanobis distance accounts for the covariance structure of the data.
-```
-d_M(x, y) = sqrt((x - y)^T * S^(-1) * (x - y))
-```
+$$
+d_M(x, y) = \sqrt{(x - y)^T \, S^{-1} \, (x - y)}
+$$
-where S is the covariance matrix of the data.
+where $S$ is the covariance matrix of the data.
-Intuitively: Mahalanobis distance first decorrelates and normalizes the data (whitening), then computes L2 distance in that transformed space. If S is the identity matrix (uncorrelated, unit variance features), Mahalanobis distance reduces to Euclidean distance.
+Intuitively: Mahalanobis distance first decorrelates and normalizes the data (whitening), then computes L2 distance in that transformed space. If $S$ is the identity matrix (uncorrelated, unit variance features), Mahalanobis distance reduces to Euclidean distance.
```
Example: height and weight are correlated.
@@ -225,9 +226,9 @@ When to use Mahalanobis distance:
Jaccard similarity measures overlap between two sets.
-```
-J(A, B) = |A intersect B| / |A union B|
-```
+$$
+J(A, B) = \frac{|A \cap B|}{|A \cup B|}
+$$
It ranges from 0 (no overlap) to 1 (identical sets). Jaccard distance = 1 - Jaccard similarity.
@@ -286,20 +287,20 @@ When to use edit distance:
KL divergence measures how one probability distribution differs from another. Covered in Lesson 09, but it belongs in this discussion because people use it as a "distance" despite it not being one.
-```
-D_KL(P || Q) = sum(p(x) * log(p(x) / q(x)))
-```
+$$
+D_{KL}(P \,\|\, Q) = \sum_x p(x) \log \frac{p(x)}{q(x)}
+$$
Critical property: KL divergence is NOT symmetric.
-```
-D_KL(P || Q) != D_KL(Q || P)
-```
+$$
+D_{KL}(P \,\|\, Q) \neq D_{KL}(Q \,\|\, P)
+$$
This means it fails the basic requirement of a distance metric. It also does not satisfy the triangle inequality. It is a divergence, not a distance.
-Forward KL (D_KL(P || Q)) is "mean-seeking": Q tries to cover all modes of P.
-Reverse KL (D_KL(Q || P)) is "mode-seeking": Q focuses on a single mode of P.
+Forward KL ($D_{KL}(P \mid\mid Q)$) is "mean-seeking": Q tries to cover all modes of P.
+Reverse KL ($D_{KL}(Q \mid\mid P)$) is "mode-seeking": Q focuses on a single mode of P.
When you see KL divergence:
- VAEs (the KL term in the ELBO pushes the latent distribution toward a prior)
@@ -311,15 +312,15 @@ When you see KL divergence:
Wasserstein distance measures the minimum "work" needed to transform one probability distribution into another. Think of it as: if one distribution is a pile of dirt and the other is a hole, how much dirt do you have to move and how far?
-```
-W(P, Q) = inf over all transport plans gamma of E[d(x, y)]
-```
+$$
+W(P, Q) = \inf_{\gamma} \; \mathbb{E}_{(x, y) \sim \gamma}[d(x, y)]
+$$
For 1D distributions, it simplifies to the integral of the absolute difference of the cumulative distribution functions:
-```
-W_1(P, Q) = integral |CDF_P(x) - CDF_Q(x)| dx
-```
+$$
+W_1(P, Q) = \int |\text{CDF}_P(x) - \text{CDF}_Q(x)| \, dx
+$$
Why Wasserstein matters:
- It is a true metric (symmetric, satisfies triangle inequality)
@@ -404,7 +405,7 @@ Why L1 produces sparsity but L2 does not: picture the constraint region in 2D we
Every distance function implies a nearest neighbor search problem: given a query point, find the closest points in a dataset.
-Exact nearest neighbor search is O(n * d) per query in a dataset of n points with d dimensions. For large datasets, this is too slow.
+Exact nearest neighbor search is $O(n \cdot d)$ per query in a dataset of n points with d dimensions. For large datasets, this is too slow.
Approximate Nearest Neighbor (ANN) algorithms trade a small amount of accuracy for massive speed gains:
@@ -469,7 +470,7 @@ When you call `model.encode(text)` and then search a vector database, this is wh
## Exercises
-1. Compute L1, L2, and L-infinity distances between (1, 2, 3) and (4, 0, 6). Verify that L-inf <= L2 <= L1 always holds for any pair of points. Prove why this ordering is guaranteed.
+1. Compute L1, L2, and L-infinity distances between (1, 2, 3) and (4, 0, 6). Verify that $L_\infty \leq L_2 \leq L_1$ always holds for any pair of points. Prove why this ordering is guaranteed.
2. Create two vectors where cosine similarity is high (> 0.9) but L2 distance is large (> 10). Explain geometrically what is happening. Then create two vectors where cosine similarity is low (< 0.3) but L2 distance is small (< 0.5).
diff --git a/phases/01-math-foundations/15-statistics-for-ml/docs/en.md b/phases/01-math-foundations/15-statistics-for-ml/docs/en.md
index 93ca067275..564472b7fc 100644
--- a/phases/01-math-foundations/15-statistics-for-ml/docs/en.md
+++ b/phases/01-math-foundations/15-statistics-for-ml/docs/en.md
@@ -32,37 +32,28 @@ Before you model anything, you need to know what your data looks like. Descripti
**Measures of central tendency** answer "where is the middle?"
-```
-Mean: sum of all values / count
- mu = (1/n) * sum(x_i)
+- **Mean:** sum of all values / count.
-Median: middle value when sorted
- Robust to outliers. If you have [1, 2, 3, 4, 1000], the mean is 202
- but the median is 3.
+$$
+\mu = \frac{1}{n} \sum x_i
+$$
-Mode: most frequent value
- Useful for categorical data. For continuous data, rarely informative.
-```
+- **Median:** middle value when sorted. Robust to outliers. If you have $[1, 2, 3, 4, 1000]$, the mean is 202 but the median is 3.
+- **Mode:** most frequent value. Useful for categorical data. For continuous data, rarely informative.
The mean is the balance point. The median is the halfway mark. When they diverge, your distribution is skewed. Income distributions have mean >> median (right skew from billionaires). Loss distributions during training often have mean << median (left skew from easy samples).
**Measures of spread** answer "how dispersed is the data?"
-```
-Variance: average squared deviation from the mean
- sigma^2 = (1/n) * sum((x_i - mu)^2)
-
-Standard deviation: square root of variance
- sigma = sqrt(sigma^2)
- Same units as the data, so more interpretable.
+- **Variance:** average squared deviation from the mean.
-Range: max - min
- Sensitive to outliers. Almost never useful alone.
+$$
+\sigma^2 = \frac{1}{n} \sum (x_i - \mu)^2
+$$
-IQR: Q3 - Q1 (interquartile range)
- The range of the middle 50% of the data.
- Robust to outliers. Used for box plots and outlier detection.
-```
+- **Standard deviation:** square root of variance, $\sigma = \sqrt{\sigma^2}$. Same units as the data, so more interpretable.
+- **Range:** $\max - \min$. Sensitive to outliers. Almost never useful alone.
+- **IQR:** $Q_3 - Q_1$ (interquartile range). The range of the middle 50% of the data. Robust to outliers. Used for box plots and outlier detection.
**Percentiles** divide sorted data into 100 equal parts. The 25th percentile (Q1) means 25% of values fall below this point. The 50th percentile is the median. The 75th percentile is Q3.
@@ -77,10 +68,12 @@ In ML, you care about percentiles for inference latency, prediction confidence d
**Sample vs population statistics.** When computing variance from a sample, divide by (n-1) instead of n. This is Bessel's correction. It compensates for the fact that your sample mean is not the true population mean. With n in the denominator, you systematically underestimate the true variance. With (n-1), the estimate is unbiased.
-```
-Population variance: sigma^2 = (1/N) * sum((x_i - mu)^2)
-Sample variance: s^2 = (1/(n-1)) * sum((x_i - x_bar)^2)
-```
+$$
+\begin{aligned}
+\text{Population variance:} \quad & \sigma^2 = \frac{1}{N} \sum (x_i - \mu)^2 \\
+\text{Sample variance:} \quad & s^2 = \frac{1}{n-1} \sum (x_i - \bar{x})^2
+\end{aligned}
+$$
In practice: if n is large (thousands of samples), the difference is negligible. If n is small (dozens of samples), it matters.
@@ -90,27 +83,24 @@ Correlation measures the strength and direction of a linear relationship between
**Pearson correlation coefficient** measures linear association:
-```
-r = sum((x_i - x_bar)(y_i - y_bar)) / (n * s_x * s_y)
+$$
+r = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{n \cdot s_x \cdot s_y}
+$$
-r = +1: perfect positive linear relationship
-r = -1: perfect negative linear relationship
-r = 0: no linear relationship (but there might be a nonlinear one!)
+- $r = +1$: perfect positive linear relationship
+- $r = -1$: perfect negative linear relationship
+- $r = 0$: no linear relationship (but there might be a nonlinear one!)
-Range: [-1, 1]
-```
+Range: $[-1, 1]$
Pearson assumes the relationship is linear and both variables are roughly normally distributed. It is sensitive to outliers. A single extreme point can drag r from 0.1 to 0.9.
**Spearman rank correlation** measures monotonic association:
-```
1. Replace each value with its rank (1, 2, 3, ...)
2. Compute Pearson correlation on the ranks
-Spearman catches any monotonic relationship, not just linear.
-If y = x^3, Pearson gives r < 1 but Spearman gives rho = 1.
-```
+Spearman catches any monotonic relationship, not just linear. If $y = x^3$, Pearson gives $r < 1$ but Spearman gives $\rho = 1$.
**When to use each:**
@@ -131,31 +121,34 @@ Spearman: Ordinal data (rankings, ratings).
The covariance between two variables measures how they vary together:
-```
-Cov(X, Y) = (1/n) * sum((x_i - x_bar)(y_i - y_bar))
+$$
+\text{Cov}(X, Y) = \frac{1}{n} \sum (x_i - \bar{x})(y_i - \bar{y})
+$$
-Cov(X, Y) > 0: X and Y tend to increase together
-Cov(X, Y) < 0: when X increases, Y tends to decrease
-Cov(X, Y) = 0: no linear co-movement
-```
+- $\text{Cov}(X, Y) > 0$: $X$ and $Y$ tend to increase together
+- $\text{Cov}(X, Y) < 0$: when $X$ increases, $Y$ tends to decrease
+- $\text{Cov}(X, Y) = 0$: no linear co-movement
-For d features, the covariance matrix C is a d x d matrix where C[i][j] = Cov(feature_i, feature_j). The diagonal entries C[i][i] are the variances of each feature.
+For $d$ features, the covariance matrix $C$ is a $d \times d$ matrix where $C[i][j] = \text{Cov}(\text{feature}_i, \text{feature}_j)$. The diagonal entries $C[i][i]$ are the variances of each feature.
-```
-C = | Var(x1) Cov(x1,x2) Cov(x1,x3) |
- | Cov(x2,x1) Var(x2) Cov(x2,x3) |
- | Cov(x3,x1) Cov(x3,x2) Var(x3) |
+$$
+C = \begin{bmatrix}
+\text{Var}(x_1) & \text{Cov}(x_1,x_2) & \text{Cov}(x_1,x_3) \\
+\text{Cov}(x_2,x_1) & \text{Var}(x_2) & \text{Cov}(x_2,x_3) \\
+\text{Cov}(x_3,x_1) & \text{Cov}(x_3,x_2) & \text{Var}(x_3)
+\end{bmatrix}
+$$
Properties:
- - Symmetric: C[i][j] = C[j][i]
- - Positive semi-definite: all eigenvalues >= 0
- - Diagonal = variances
- - Off-diagonal = covariances
-```
+
+- Symmetric: $C[i][j] = C[j][i]$
+- Positive semi-definite: all eigenvalues $\geq 0$
+- Diagonal = variances
+- Off-diagonal = covariances
**Connection to PCA.** PCA eigendecomposes the covariance matrix. The eigenvectors are the principal components (directions of maximum variance). The eigenvalues tell you how much variance each component captures. This is exactly what Lesson 10 covered, but now you see why the covariance matrix is the right thing to decompose: it encodes all pairwise linear relationships in your data.
-**Connection to correlation.** The correlation matrix is the covariance matrix of standardized variables (each divided by its standard deviation). Correlation normalizes covariance so all values fall in [-1, 1].
+**Connection to correlation.** The correlation matrix is the covariance matrix of standardized variables (each divided by its standard deviation). Correlation normalizes covariance so all values fall in $[-1, 1]$.
### Hypothesis Testing
@@ -174,28 +167,24 @@ Example:
**The p-value** is the probability of seeing data as extreme as what you observed, assuming H0 is true. It is NOT the probability that H0 is true. This is the single most common misunderstanding in statistics.
-```
-p-value = P(data this extreme | H0 is true)
+$$
+\text{p-value} = P(\text{data this extreme} \mid H_0 \text{ is true})
+$$
-If p-value < alpha (typically 0.05):
- Reject H0. The result is "statistically significant."
-If p-value >= alpha:
- Fail to reject H0. You do not have enough evidence.
- This does NOT mean H0 is true.
-```
+- If $\text{p-value} < \alpha$ (typically 0.05): Reject $H_0$. The result is "statistically significant."
+- If $\text{p-value} \geq \alpha$: Fail to reject $H_0$. You do not have enough evidence. This does NOT mean $H_0$ is true.
**Confidence intervals** give a range of plausible values for a parameter:
-```
95% confidence interval for the mean:
- x_bar +/- z * (s / sqrt(n))
-where z = 1.96 for 95% confidence
+$$
+\bar{x} \pm z \cdot \frac{s}{\sqrt{n}}
+$$
-Interpretation: if you repeated this experiment many times, 95% of the
-computed intervals would contain the true mean. It does NOT mean there
-is a 95% probability the true mean is in this specific interval.
-```
+where $z = 1.96$ for 95% confidence. This is the large-sample approximation; when $n$ is small and the population variance is unknown, use the $t$-distribution's critical value $t_{n-1,\,0.975}$ in place of $z$.
+
+Interpretation: if you repeated this experiment many times, 95% of the computed intervals would contain the true mean. It does NOT mean there is a 95% probability the true mean is in this specific interval.
The width of the confidence interval tells you about precision. Wide intervals mean high uncertainty. Narrow intervals mean your estimate is precise (but not necessarily accurate, if your data is biased).
@@ -205,27 +194,21 @@ The t-test compares means. There are several flavors.
**One-sample t-test:** is the population mean different from a hypothesized value?
-```
-t = (x_bar - mu_0) / (s / sqrt(n))
-
-degrees of freedom = n - 1
-```
+$$
+t = \frac{\bar{x} - \mu_0}{s / \sqrt{n}}, \qquad \text{degrees of freedom} = n - 1
+$$
**Two-sample t-test (independent):** are two group means different?
-```
-t = (x_bar_1 - x_bar_2) / sqrt(s1^2/n1 + s2^2/n2)
+$$
+t = \frac{\bar{x}_1 - \bar{x}_2}{\sqrt{s_1^2/n_1 + s_2^2/n_2}}
+$$
-This is Welch's t-test, which does not assume equal variances.
-Always use Welch's unless you have a specific reason for equal variances.
-```
+This is Welch's t-test, which does not assume equal variances. Always use Welch's unless you have a specific reason for equal variances.
**Paired t-test:** when measurements come in pairs (same model evaluated on same data splits):
-```
-Compute d_i = x_i - y_i for each pair
-Then run a one-sample t-test on the d_i values against mu_0 = 0
-```
+Compute $d_i = x_i - y_i$ for each pair. Then run a one-sample t-test on the $d_i$ values against $\mu_0 = 0$.
In ML, the paired t-test is common: you run both models on the same 10 cross-validation folds and compare their scores pairwise.
@@ -233,21 +216,24 @@ In ML, the paired t-test is common: you run both models on the same 10 cross-val
The chi-squared test checks if observed frequencies match expected frequencies. Useful for categorical data.
-```
-chi^2 = sum((observed - expected)^2 / expected)
+$$
+\chi^2 = \sum \frac{(\text{observed} - \text{expected})^2}{\text{expected}}
+$$
-Example: does a language model's output distribution match the
-training distribution across categories?
+Example: does a language model's output distribution match the training distribution across categories?
+```text
Category Observed Expected
Positive 120 100
Negative 80 100
-chi^2 = (120-100)^2/100 + (80-100)^2/100 = 4 + 4 = 8
-
-With 1 degree of freedom, chi^2 = 8 gives p < 0.005.
-The difference is significant.
```
+$$
+\chi^2 = \frac{(120-100)^2}{100} + \frac{(80-100)^2}{100} = 4 + 4 = 8
+$$
+
+With 1 degree of freedom, $\chi^2 = 8$ gives $p < 0.005$. The difference is significant.
+
### A/B Testing for ML Models
A/B testing in ML is not the same as web A/B testing. Model comparison has specific challenges:
@@ -297,13 +283,13 @@ engineering cost of deploying a new model.
**Effect size** quantifies how big the difference is, independent of sample size:
-```
-Cohen's d = (mean_1 - mean_2) / pooled_std
+$$
+\text{Cohen's } d = \frac{\text{mean}_1 - \text{mean}_2}{\text{pooled\_std}}
+$$
-d = 0.2: small effect
-d = 0.5: medium effect
-d = 0.8: large effect
-```
+- $d = 0.2$: small effect
+- $d = 0.5$: medium effect
+- $d = 0.8$: large effect
Always report both the p-value and the effect size. The p-value tells you if the difference is real. The effect size tells you if it matters.
@@ -311,23 +297,25 @@ Always report both the p-value and the effect size. The p-value tells you if the
When you test many hypotheses, some will be "significant" by chance. If you test 20 things at alpha = 0.05, you expect 1 false positive even when nothing is real.
-```
-P(at least one false positive) = 1 - (1 - alpha)^m
+$$
+P(\text{at least one false positive}) = 1 - (1 - \alpha)^m
+$$
+
+For $m = 20$ tests, $\alpha = 0.05$:
-m = 20 tests, alpha = 0.05:
-P(false positive) = 1 - 0.95^20 = 0.64
+$$
+P(\text{false positive}) = 1 - 0.95^{20} = 0.64
+$$
You have a 64% chance of at least one false positive.
-```
**Bonferroni correction:** divide alpha by the number of tests.
-```
-Adjusted alpha = alpha / m = 0.05 / 20 = 0.0025
+$$
+\text{Adjusted } \alpha = \frac{\alpha}{m} = \frac{0.05}{20} = 0.0025
+$$
-Only reject H0 if p-value < 0.0025.
-Conservative but simple. Works when tests are independent.
-```
+Only reject $H_0$ if $\text{p-value} < 0.0025$. Conservative but simple. Works when tests are independent.
In ML, this matters when you compare a model across multiple metrics, test many hyperparameter configurations, or evaluate on multiple datasets.
@@ -423,14 +411,13 @@ In ML experiments, you typically have small n (5 or 10 cross-validation folds),
The CLT says the distribution of sample means approaches a normal distribution as n grows, regardless of the underlying population distribution.
-```
-If X_1, X_2, ..., X_n are iid with mean mu and variance sigma^2:
+If $X_1, X_2, \ldots, X_n$ are iid with mean $\mu$ and variance $\sigma^2$:
- X_bar ~ Normal(mu, sigma^2 / n) as n -> infinity
+$$
+\bar{X} \sim \text{Normal}\left(\mu, \frac{\sigma^2}{n}\right) \quad \text{as } n \to \infty
+$$
-Works for n >= 30 in most cases.
-For highly skewed distributions, you might need n >= 100.
-```
+Works for $n \geq 30$ in most cases. For highly skewed distributions, you might need $n \geq 100$.
**Why this matters for ML:**
@@ -497,7 +484,7 @@ All from scratch, using only `math` and `random`. No numpy, no scipy.
| Standard deviation | Square root of variance. Measures spread in original units. |
| Percentile | Value below which a given percentage of data falls. |
| IQR | Interquartile range. Q3 minus Q1. The spread of the middle 50%. |
-| Pearson correlation | Measures linear association between two variables. Range [-1, 1]. |
+| Pearson correlation | Measures linear association between two variables. Range $[-1, 1]$. |
| Spearman correlation | Measures monotonic association using ranks. |
| Covariance matrix | Matrix of pairwise covariances between all features. |
| Null hypothesis | Default assumption of no effect or no difference. |
diff --git a/phases/01-math-foundations/16-sampling-methods/docs/en.md b/phases/01-math-foundations/16-sampling-methods/docs/en.md
index c39b35df97..88d51ac569 100644
--- a/phases/01-math-foundations/16-sampling-methods/docs/en.md
+++ b/phases/01-math-foundations/16-sampling-methods/docs/en.md
@@ -44,17 +44,17 @@ The core challenge: you can only sample directly from simple distributions (unif
Every sampling method starts here. A uniform random number generator produces values in [0, 1) where every sub-interval of equal length has equal probability.
-```
-U ~ Uniform(0, 1)
-
-P(a <= U <= b) = b - a for 0 <= a <= b <= 1
-
-Properties:
- E[U] = 0.5
- Var(U) = 1/12
-```
-
-To sample uniformly from a discrete set of n items, generate U and return floor(n * U). To sample from a continuous range [a, b], compute a + (b - a) * U.
+$$
+\begin{aligned}
+&U \sim \text{Uniform}(0, 1) \\
+&P(a \leq U \leq b) = b - a \quad \text{for } 0 \leq a \leq b \leq 1 \\
+&\text{Properties:} \\
+&\quad E[U] = 0.5 \\
+&\quad \text{Var}(U) = \tfrac{1}{12}
+\end{aligned}
+$$
+
+To sample uniformly from a discrete set of $n$ items, generate $U$ and return $\lfloor n U \rfloor$. To sample from a continuous range $[a, b]$, compute $a + (b - a) U$.
The key insight: a single uniform random number contains exactly the right amount of randomness to produce one sample from any distribution. The trick is finding the right transformation.
@@ -62,43 +62,45 @@ The key insight: a single uniform random number contains exactly the right amoun
The cumulative distribution function (CDF) maps values to probabilities:
-```
-F(x) = P(X <= x)
-
-Properties:
- F is non-decreasing
- F(-inf) = 0
- F(+inf) = 1
- F maps the real line to [0, 1]
-```
-
-The inverse CDF maps probabilities back to values. If U ~ Uniform(0, 1), then X = F_inverse(U) follows the target distribution.
-
-```
-Algorithm:
- 1. Generate u ~ Uniform(0, 1)
- 2. Return F_inverse(u)
-
-Why it works:
- P(X <= x) = P(F_inverse(U) <= x) = P(U <= F(x)) = F(x)
-```
+$$
+\begin{aligned}
+&F(x) = P(X \leq x) \\
+&\text{Properties:} \\
+&\quad F \text{ is non-decreasing} \\
+&\quad F(-\infty) = 0 \\
+&\quad F(+\infty) = 1 \\
+&\quad F \text{ maps the real line to } [0, 1]
+\end{aligned}
+$$
+
+The inverse CDF maps probabilities back to values. If $U \sim \text{Uniform}(0, 1)$, then $X = F^{-1}(U)$ follows the target distribution.
+
+$$
+\begin{aligned}
+&\text{Algorithm:} \\
+&\quad 1.\ \text{Generate } u \sim \text{Uniform}(0, 1) \\
+&\quad 2.\ \text{Return } F^{-1}(u) \\
+&\text{Why it works:} \\
+&\quad P(X \leq x) = P(F^{-1}(U) \leq x) = P(U \leq F(x)) = F(x)
+\end{aligned}
+$$
**Exponential distribution example:**
-```
-PDF: f(x) = lambda * exp(-lambda * x), x >= 0
-CDF: F(x) = 1 - exp(-lambda * x)
-
-Solve F(x) = u for x:
- u = 1 - exp(-lambda * x)
- exp(-lambda * x) = 1 - u
- x = -ln(1 - u) / lambda
-
-Since (1 - U) and U have the same distribution:
- x = -ln(u) / lambda
-```
-
-This works perfectly when you can write down F_inverse in closed form. For the normal distribution, there is no closed-form inverse CDF, so we use other methods (Box-Muller, or numerical approximation).
+$$
+\begin{aligned}
+&\text{PDF: } f(x) = \lambda \, e^{-\lambda x}, \quad x \geq 0 \\
+&\text{CDF: } F(x) = 1 - e^{-\lambda x} \\
+&\text{Solve } F(x) = u \text{ for } x: \\
+&\quad u = 1 - e^{-\lambda x} \\
+&\quad e^{-\lambda x} = 1 - u \\
+&\quad x = -\ln(1 - u) / \lambda \\
+&\text{Since } (1 - U) \text{ and } U \text{ have the same distribution:} \\
+&\quad x = -\ln(u) / \lambda
+\end{aligned}
+$$
+
+This works perfectly when you can write down $F^{-1}$ in closed form. For the normal distribution, there is no closed-form inverse CDF, so we use other methods (Box-Muller, or numerical approximation).
**Discrete version:** For discrete distributions, build the CDF as a cumulative sum, generate U, and find the first index where the cumulative sum exceeds U. This is how `sample_categorical` works in Lesson 06.
@@ -106,108 +108,111 @@ This works perfectly when you can write down F_inverse in closed form. For the n
When you cannot invert the CDF but can evaluate the target PDF up to a constant, rejection sampling works.
-```
-Target distribution: p(x) (can evaluate, possibly unnormalized)
-Proposal distribution: q(x) (can sample from)
-Bound: M such that p(x) <= M * q(x) for all x
-
-Algorithm:
- 1. Sample x ~ q(x)
- 2. Sample u ~ Uniform(0, 1)
- 3. If u < p(x) / (M * q(x)), accept x
- 4. Otherwise, reject and go to step 1
-
-Acceptance rate = 1/M
-```
+$$
+\begin{aligned}
+&\text{Target distribution: } p(x) \quad (\text{can evaluate, possibly unnormalized}) \\
+&\text{Proposal distribution: } q(x) \quad (\text{can sample from}) \\
+&\text{Bound: } M \text{ such that } p(x) \leq M \, q(x) \text{ for all } x \\
+&\text{Algorithm:} \\
+&\quad 1.\ \text{Sample } x \sim q(x) \\
+&\quad 2.\ \text{Sample } u \sim \text{Uniform}(0, 1) \\
+&\quad 3.\ \text{If } u < p(x) / (M \, q(x)), \text{ accept } x \\
+&\quad 4.\ \text{Otherwise, reject and go to step 1} \\
+&\text{Acceptance rate} = 1/M \quad \text{(when } p \text{ is normalized)}
+\end{aligned}
+$$
The tighter the bound M, the higher the acceptance rate. In low dimensions (1-3), rejection sampling works well. In high dimensions, the acceptance rate drops exponentially because most of the proposal volume gets rejected. This is the curse of dimensionality for rejection sampling.
**Example: sampling from a truncated normal.** Use a uniform proposal over the truncated range. The envelope M is the maximum of the normal PDF in that range.
-**Example: sampling from a semicircle.** Propose uniformly in the bounding rectangle. Accept if the point falls inside the semicircle. This is how Monte Carlo computes pi: the acceptance rate equals the area ratio pi/4.
+**Example: sampling from a semicircle.** Propose uniformly in the bounding rectangle. Accept if the point falls inside the semicircle. This is how Monte Carlo computes $\pi$: the acceptance rate equals the area ratio $\pi/4$.
### Importance Sampling
-Sometimes you do not need samples from the target distribution p(x). You need to estimate an expectation under p(x), and you have samples from a different distribution q(x).
-
-```
-Goal: estimate E_p[f(x)] = integral of f(x) * p(x) dx
-
-Rewrite:
- E_p[f(x)] = integral of f(x) * (p(x)/q(x)) * q(x) dx
- = E_q[f(x) * w(x)]
-
-where w(x) = p(x) / q(x) are the importance weights.
+Sometimes you do not need samples from the target distribution $p(x)$. You need to estimate an expectation under $p(x)$, and you have samples from a different distribution $q(x)$.
-Estimator:
- E_p[f(x)] ~ (1/N) * sum(f(x_i) * w(x_i)) where x_i ~ q(x)
-```
+$$
+\begin{aligned}
+&\text{Goal: estimate } E_p[f(x)] = \int f(x) \, p(x) \, dx \\
+&\text{Rewrite:} \\
+&\quad E_p[f(x)] = \int f(x) \, \frac{p(x)}{q(x)} \, q(x) \, dx \\
+&\quad\phantom{E_p[f(x)]} = E_q[f(x) \, w(x)] \\
+&\text{where } w(x) = p(x) / q(x) \text{ are the importance weights.} \\
+&\text{Estimator:} \\
+&\quad E_p[f(x)] \approx \frac{1}{N} \sum_i f(x_i) \, w(x_i) \quad \text{where } x_i \sim q(x)
+\end{aligned}
+$$
-This is critical in reinforcement learning. In PPO (Proximal Policy Optimization), you collect trajectories under an old policy pi_old but want to optimize a new policy pi_new. The importance weight is pi_new(a|s) / pi_old(a|s). PPO clips these weights to prevent the new policy from diverging too far from the old one.
+This is critical in reinforcement learning. In PPO (Proximal Policy Optimization), you collect trajectories under an old policy $\pi_\text{old}$ but want to optimize a new policy $\pi_\text{new}$. The importance weight is $\pi_\text{new}(a \mid s) / \pi_\text{old}(a \mid s)$. PPO clips these weights to prevent the new policy from diverging too far from the old one.
The variance of the importance sampling estimator depends on how similar q is to p. If q is very different from p, a few samples get enormous weights and dominate the estimate. Self-normalized importance sampling divides by the sum of weights to reduce this problem:
-```
-E_p[f(x)] ~ sum(w_i * f(x_i)) / sum(w_i)
-```
+$$
+E_p[f(x)] \approx \frac{\sum_i w_i \, f(x_i)}{\sum_i w_i}
+$$
### Monte Carlo Estimation
Monte Carlo estimation approximates integrals by averaging random samples. The law of large numbers guarantees convergence.
-```
-Goal: estimate I = integral of g(x) dx over domain D
-
-Method:
- 1. Sample x_1, ..., x_N uniformly from D
- 2. I ~ (Volume of D / N) * sum(g(x_i))
-
-Error: O(1 / sqrt(N)) regardless of dimension
-```
+$$
+\begin{aligned}
+&\text{Goal: estimate } I = \int_D g(x) \, dx \text{ over domain } D \\
+&\text{Method:} \\
+&\quad 1.\ \text{Sample } x_1, \ldots, x_N \text{ uniformly from } D \\
+&\quad 2.\ I \approx \frac{\text{Volume of } D}{N} \sum_i g(x_i) \\
+&\text{Error: } O(1 / \sqrt{N}) \quad \text{regardless of dimension}
+\end{aligned}
+$$
The error rate is dimension-independent. This is why Monte Carlo methods dominate in high dimensions where grid-based integration is impossible.
**Estimating pi:**
-```
-Sample (x, y) uniformly from [-1, 1] x [-1, 1]
-Count how many fall inside the unit circle: x^2 + y^2 <= 1
-pi ~ 4 * (count inside) / (total count)
-```
+$$
+\begin{aligned}
+&\text{Sample } (x, y) \text{ uniformly from } [-1, 1] \times [-1, 1] \\
+&\text{Count how many fall inside the unit circle: } x^2 + y^2 \leq 1 \\
+&\pi \approx 4 \cdot \frac{\text{count inside}}{\text{total count}}
+\end{aligned}
+$$
**Estimating expectations:**
-```
-E[f(X)] ~ (1/N) * sum(f(x_i)) where x_i ~ p(x)
-
-The sample mean converges to the true expectation.
-Variance of the estimator = Var(f(X)) / N
-```
+$$
+\begin{aligned}
+&E[f(X)] \approx \frac{1}{N} \sum_i f(x_i) \quad \text{where } x_i \sim p(x) \\
+&\text{The sample mean converges to the true expectation.} \\
+&\text{Variance of the estimator} = \text{Var}(f(X)) / N
+\end{aligned}
+$$
### Markov Chain Monte Carlo (MCMC): Metropolis-Hastings
-MCMC constructs a Markov chain whose stationary distribution is the target distribution p(x). After enough steps, samples from the chain are (approximately) samples from p(x).
-
-```
-Target: p(x) (known up to a normalizing constant)
-Proposal: q(x'|x) (how to propose the next state given the current state)
-
-Metropolis-Hastings algorithm:
- 1. Start at some x_0
- 2. For t = 1, 2, ..., T:
- a. Propose x' ~ q(x'|x_t)
- b. Compute acceptance ratio:
- alpha = [p(x') * q(x_t|x')] / [p(x_t) * q(x'|x_t)]
- c. Accept with probability min(1, alpha):
- - If u < alpha (u ~ Uniform(0,1)): x_{t+1} = x'
- - Otherwise: x_{t+1} = x_t
- 3. Discard first B samples (burn-in)
- 4. Return remaining samples
-```
-
-For symmetric proposals (q(x'|x) = q(x|x')), the ratio simplifies to p(x')/p(x). This is the original Metropolis algorithm.
-
-**Why it works.** The acceptance rule ensures detailed balance: the probability of being at x and moving to x' equals the probability of being at x' and moving to x. Detailed balance implies that p(x) is the stationary distribution of the chain.
+MCMC constructs a Markov chain whose stationary distribution is the target distribution $p(x)$. After enough steps, samples from the chain are (approximately) samples from $p(x)$.
+
+$$
+\begin{aligned}
+&\text{Target: } p(x) \quad (\text{known up to a normalizing constant}) \\
+&\text{Proposal: } q(x' \mid x) \quad (\text{how to propose the next state given the current state}) \\
+&\text{Metropolis-Hastings algorithm:} \\
+&\quad 1.\ \text{Start at some } x_0 \\
+&\quad 2.\ \text{For } t = 1, 2, \ldots, T: \\
+&\qquad \text{a. Propose } x' \sim q(x' \mid x_t) \\
+&\qquad \text{b. Compute acceptance ratio:} \\
+&\qquad\quad \alpha = \frac{p(x') \, q(x_t \mid x')}{p(x_t) \, q(x' \mid x_t)} \\
+&\qquad \text{c. Accept with probability } \min(1, \alpha): \\
+&\qquad\quad \text{- If } u < \alpha \ (u \sim \text{Uniform}(0,1)): x_{t+1} = x' \\
+&\qquad\quad \text{- Otherwise: } x_{t+1} = x_t \\
+&\quad 3.\ \text{Discard first } B \text{ samples (burn-in)} \\
+&\quad 4.\ \text{Return remaining samples}
+\end{aligned}
+$$
+
+For symmetric proposals ($q(x' \mid x) = q(x \mid x')$), the ratio simplifies to $p(x')/p(x)$. This is the original Metropolis algorithm.
+
+**Why it works.** The acceptance rule ensures detailed balance: the probability of being at $x$ and moving to $x'$ equals the probability of being at $x'$ and moving to $x$. Detailed balance implies that $p(x)$ is the stationary distribution of the chain.
**Practical considerations:**
- Burn-in: discard early samples before the chain reaches equilibrium
@@ -219,18 +224,19 @@ For symmetric proposals (q(x'|x) = q(x|x')), the ratio simplifies to p(x')/p(x).
Gibbs sampling is a special case of MCMC for multivariate distributions. Instead of proposing a move in all dimensions at once, it updates one variable at a time from its conditional distribution.
-```
-Target: p(x_1, x_2, ..., x_d)
-
-Algorithm:
- For each iteration t:
- Sample x_1^{t+1} ~ p(x_1 | x_2^t, x_3^t, ..., x_d^t)
- Sample x_2^{t+1} ~ p(x_2 | x_1^{t+1}, x_3^t, ..., x_d^t)
- ...
- Sample x_d^{t+1} ~ p(x_d | x_1^{t+1}, x_2^{t+1}, ..., x_{d-1}^{t+1})
-```
-
-Gibbs sampling requires that you can sample from each conditional distribution p(x_i | x_{-i}). This is straightforward for many models:
+$$
+\begin{aligned}
+&\text{Target: } p(x_1, x_2, \ldots, x_d) \\
+&\text{Algorithm:} \\
+&\quad \text{For each iteration } t: \\
+&\qquad \text{Sample } x_1^{t+1} \sim p(x_1 \mid x_2^t, x_3^t, \ldots, x_d^t) \\
+&\qquad \text{Sample } x_2^{t+1} \sim p(x_2 \mid x_1^{t+1}, x_3^t, \ldots, x_d^t) \\
+&\qquad \ldots \\
+&\qquad \text{Sample } x_d^{t+1} \sim p(x_d \mid x_1^{t+1}, x_2^{t+1}, \ldots, x_{d-1}^{t+1})
+\end{aligned}
+$$
+
+Gibbs sampling requires that you can sample from each conditional distribution $p(x_i \mid x_{-i})$. This is straightforward for many models:
- Bayesian networks: conditionals follow from the graph structure
- Gaussian mixtures: conditionals are Gaussian
- Ising models: each spin's conditional depends only on its neighbors
@@ -241,19 +247,20 @@ The acceptance rate is always 1 (every proposal is accepted) because sampling fr
### Temperature Sampling (Used in LLMs)
-Language models output logits z_1, ..., z_V for each token in the vocabulary. Softmax converts these to probabilities. Temperature rescales the logits before softmax:
+Language models output logits $z_1, \ldots, z_V$ for each token in the vocabulary. Softmax converts these to probabilities. Temperature rescales the logits before softmax:
-```
-p_i = exp(z_i / T) / sum(exp(z_j / T))
+$$
+\begin{aligned}
+&p_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)} \\
+&T = 1.0: \text{ standard softmax (original distribution)} \\
+&T \to 0: \text{ argmax (deterministic, always picks highest logit)} \\
+&T \to \infty: \text{ uniform (all tokens equally likely)} \\
+&T < 1.0: \text{ sharpens the distribution (more confident, less diverse)} \\
+&T > 1.0: \text{ flattens the distribution (less confident, more diverse)}
+\end{aligned}
+$$
-T = 1.0: standard softmax (original distribution)
-T -> 0: argmax (deterministic, always picks highest logit)
-T -> inf: uniform (all tokens equally likely)
-T < 1.0: sharpens the distribution (more confident, less diverse)
-T > 1.0: flattens the distribution (less confident, more diverse)
-```
-
-**Why it works.** Dividing logits by T < 1 amplifies differences between logits. If z_1 = 2 and z_2 = 1, dividing by T = 0.5 gives z_1/T = 4 and z_2/T = 2, making the gap larger. After softmax, the highest-logit token gets a much larger share.
+**Why it works.** Dividing logits by $T < 1$ amplifies differences between logits. If $z_1 = 2$ and $z_2 = 1$, dividing by $T = 0.5$ gives $z_1/T = 4$ and $z_2/T = 2$, making the gap larger. After softmax, the highest-logit token gets a much larger share.
**In practice:**
- T = 0.0: greedy decoding, best for factual Q&A
@@ -268,18 +275,19 @@ Temperature does not change which tokens are possible. It changes the probabilit
Top-k sampling restricts the candidate set to the k tokens with the highest probabilities, then renormalizes and samples from that restricted set.
-```
-Algorithm:
- 1. Compute softmax probabilities for all V tokens
- 2. Sort tokens by probability (descending)
- 3. Keep only the top k tokens
- 4. Renormalize: p_i' = p_i / sum(p_j for j in top-k)
- 5. Sample from the renormalized distribution
-
-k = 1: greedy decoding
-k = V: no filtering (standard sampling)
-k = 40: typical setting, removes long tail of unlikely tokens
-```
+$$
+\begin{aligned}
+&\text{Algorithm:} \\
+&\quad 1.\ \text{Compute softmax probabilities for all } V \text{ tokens} \\
+&\quad 2.\ \text{Sort tokens by probability (descending)} \\
+&\quad 3.\ \text{Keep only the top } k \text{ tokens} \\
+&\quad 4.\ \text{Renormalize: } p_i' = p_i / \textstyle\sum_{j \in \text{top-}k} p_j \\
+&\quad 5.\ \text{Sample from the renormalized distribution} \\
+&k = 1: \text{ greedy decoding} \\
+&k = V: \text{ no filtering (standard sampling)} \\
+&k = 40: \text{ typical setting, removes long tail of unlikely tokens}
+\end{aligned}
+$$
Top-k prevents the model from selecting extremely unlikely tokens (typos, nonsense) that exist in the long tail of the vocabulary distribution. The problem: k is fixed regardless of context. When the model is confident (one token has 95% probability), k = 40 still allows 39 alternatives. When the model is uncertain (probability is spread across 1000 tokens), k = 40 cuts off plausible options.
@@ -287,18 +295,19 @@ Top-k prevents the model from selecting extremely unlikely tokens (typos, nonsen
Top-p sampling dynamically adjusts the candidate set size. Instead of keeping a fixed number of tokens, it keeps the smallest set of tokens whose cumulative probability exceeds p.
-```
-Algorithm:
- 1. Compute softmax probabilities for all V tokens
- 2. Sort tokens by probability (descending)
- 3. Find smallest k such that sum of top-k probabilities >= p
- 4. Keep only those k tokens
- 5. Renormalize and sample
-
-p = 0.9: keeps tokens covering 90% of probability mass
-p = 1.0: no filtering
-p = 0.1: very restrictive, nearly greedy
-```
+$$
+\begin{aligned}
+&\text{Algorithm:} \\
+&\quad 1.\ \text{Compute softmax probabilities for all } V \text{ tokens} \\
+&\quad 2.\ \text{Sort tokens by probability (descending)} \\
+&\quad 3.\ \text{Find smallest } k \text{ such that sum of top-}k \text{ probabilities} \geq p \\
+&\quad 4.\ \text{Keep only those } k \text{ tokens} \\
+&\quad 5.\ \text{Renormalize and sample} \\
+&p = 0.9: \text{ keeps tokens covering 90\% of probability mass} \\
+&p = 1.0: \text{ no filtering} \\
+&p = 0.1: \text{ very restrictive, nearly greedy}
+\end{aligned}
+$$
When the model is confident, nucleus sampling keeps few tokens (maybe 2-3). When the model is uncertain, it keeps many (maybe 200). This adaptive behavior is why nucleus sampling generally produces better text than top-k.
@@ -313,35 +322,36 @@ Top-k and top-p can be combined. Apply top-k first, then top-p on the remaining
Variational autoencoders (VAEs) learn by encoding inputs into a distribution in latent space, sampling from that distribution, and decoding the sample back. The problem: you cannot backpropagate through a sampling operation.
-```
-Standard sampling (not differentiable):
- z ~ N(mu, sigma^2)
-
- The randomness blocks gradient flow.
- d/d_mu [sample from N(mu, sigma^2)] = ???
-```
+$$
+\begin{aligned}
+&\text{Standard sampling (not differentiable):} \\
+&\quad z \sim N(\mu, \sigma^2) \\
+&\quad \text{The randomness blocks gradient flow.} \\
+&\quad \frac{d}{d\mu}\left[\text{sample from } N(\mu, \sigma^2)\right] = \text{???}
+\end{aligned}
+$$
The reparameterization trick separates the randomness from the parameters:
-```
-Reparameterized sampling:
- epsilon ~ N(0, 1) (fixed random noise, no parameters)
- z = mu + sigma * epsilon (deterministic function of parameters)
-
- Now z is a deterministic, differentiable function of mu and sigma.
- d(z)/d(mu) = 1
- d(z)/d(sigma) = epsilon
-
- Gradients flow through mu and sigma.
-```
+$$
+\begin{aligned}
+&\text{Reparameterized sampling:} \\
+&\quad \epsilon \sim N(0, 1) \qquad (\text{fixed random noise, no parameters}) \\
+&\quad z = \mu + \sigma \epsilon \quad (\text{deterministic function of parameters}) \\
+&\quad \text{Now } z \text{ is a deterministic, differentiable function of } \mu \text{ and } \sigma. \\
+&\quad dz/d\mu = 1 \\
+&\quad dz/d\sigma = \epsilon \\
+&\quad \text{Gradients flow through } \mu \text{ and } \sigma.
+\end{aligned}
+$$
-This works because N(mu, sigma^2) has the same distribution as mu + sigma * N(0, 1). The key insight: move the randomness to a parameter-free source (epsilon), then express the sample as a differentiable transformation of the parameters.
+This works because $N(\mu, \sigma^2)$ has the same distribution as $\mu + \sigma N(0, 1)$. The key insight: move the randomness to a parameter-free source ($\epsilon$), then express the sample as a differentiable transformation of the parameters.
**In the VAE training loop:**
-1. Encoder outputs mu and log(sigma^2) for each input
-2. Sample epsilon ~ N(0, 1)
-3. Compute z = mu + sigma * epsilon
-4. Decode z to reconstruct the input
+1. Encoder outputs $\mu$ and $\log(\sigma^2)$ for each input
+2. Sample $\epsilon \sim N(0, 1)$
+3. Compute $z = \mu + \sigma \epsilon$
+4. Decode $z$ to reconstruct the input
5. Backpropagate through steps 4, 3, 2, 1 (possible because step 3 is differentiable)
Without the reparameterization trick, VAEs cannot be trained with standard backpropagation. This single insight made VAEs practical.
@@ -352,26 +362,28 @@ The reparameterization trick works for continuous distributions (Gaussian). For
**The Gumbel-Max trick (non-differentiable):**
-```
-To sample from a categorical distribution with log-probabilities log(p_1), ..., log(p_k):
- 1. Sample g_i ~ Gumbel(0, 1) for each category
- (g = -log(-log(u)), where u ~ Uniform(0, 1))
- 2. Return argmax(log(p_i) + g_i)
-
-This produces exact categorical samples.
-```
+$$
+\begin{aligned}
+&\text{To sample from a categorical distribution with log-probabilities } \log(p_1), \ldots, \log(p_k): \\
+&\quad 1.\ \text{Sample } g_i \sim \text{Gumbel}(0, 1) \text{ for each category} \\
+&\qquad (g = -\log(-\log(u)), \text{ where } u \sim \text{Uniform}(0, 1)) \\
+&\quad 2.\ \text{Return } \arg\max(\log(p_i) + g_i) \\
+&\text{This produces exact categorical samples.}
+\end{aligned}
+$$
**Gumbel-Softmax (differentiable approximation):**
-```
-Replace the hard argmax with a soft softmax:
- y_i = exp((log(p_i) + g_i) / tau) / sum(exp((log(p_j) + g_j) / tau))
-
-tau (temperature) controls the approximation:
- tau -> 0: approaches a one-hot vector (hard categorical)
- tau -> inf: approaches uniform (1/k, 1/k, ..., 1/k)
- tau = 1.0: soft approximation
-```
+$$
+\begin{aligned}
+&\text{Replace the hard argmax with a soft softmax:} \\
+&\quad y_i = \frac{\exp((\log(p_i) + g_i) / \tau)}{\sum_j \exp((\log(p_j) + g_j) / \tau)} \\
+&\tau \text{ (temperature) controls the approximation:} \\
+&\quad \tau \to 0: \text{ approaches a one-hot vector (hard categorical)} \\
+&\quad \tau \to \infty: \text{ approaches uniform } (1/k, 1/k, \ldots, 1/k) \\
+&\quad \tau = 1.0: \text{ soft approximation}
+\end{aligned}
+$$
Gumbel-Softmax produces a continuous relaxation of a discrete sample. The output is a probability vector (soft one-hot) instead of a hard one-hot. Gradients flow through the softmax. During the forward pass in training, you can use the "straight-through" estimator: use the hard argmax for the forward pass but the soft Gumbel-Softmax gradients for the backward pass.
@@ -385,25 +397,27 @@ Gumbel-Softmax produces a continuous relaxation of a discrete sample. The output
Standard Monte Carlo sampling can leave gaps in the sample space by chance. Stratified sampling forces even coverage by dividing the space into strata and sampling from each.
-```
-Standard Monte Carlo:
- Sample N points uniformly from [0, 1]
- Some regions may have clusters, others gaps
-
-Stratified sampling:
- Divide [0, 1] into N equal strata: [0, 1/N), [1/N, 2/N), ..., [(N-1)/N, 1)
- Sample one point uniformly within each stratum
- x_i = (i + u_i) / N where u_i ~ Uniform(0, 1), i = 0, ..., N-1
-```
+$$
+\begin{aligned}
+&\text{Standard Monte Carlo:} \\
+&\quad \text{Sample } N \text{ points uniformly from } [0, 1] \\
+&\quad \text{Some regions may have clusters, others gaps} \\
+&\text{Stratified sampling:} \\
+&\quad \text{Divide } [0, 1] \text{ into } N \text{ equal strata: } [0, 1/N), [1/N, 2/N), \ldots, [(N-1)/N, 1) \\
+&\quad \text{Sample one point uniformly within each stratum} \\
+&\quad x_i = (i + u_i) / N \quad \text{where } u_i \sim \text{Uniform}(0, 1), \ i = 0, \ldots, N-1
+\end{aligned}
+$$
Stratified sampling always has lower or equal variance compared to standard Monte Carlo:
-```
-Var(stratified) <= Var(standard Monte Carlo)
-
-The improvement is largest when f(x) varies smoothly.
-For piecewise-constant functions, stratified sampling is exact.
-```
+$$
+\begin{aligned}
+&\text{Var(stratified)} \leq \text{Var(standard Monte Carlo)} \\
+&\text{The improvement is largest when } f(x) \text{ varies smoothly.} \\
+&\text{For piecewise-constant functions, stratified sampling is exact.}
+\end{aligned}
+$$
**Applications:**
- Numerical integration (quasi-Monte Carlo)
@@ -415,19 +429,18 @@ For piecewise-constant functions, stratified sampling is exact.
Diffusion models generate images through a sampling process. The forward process adds Gaussian noise to an image over T steps until it becomes pure noise. The reverse process learns to denoise, recovering the original image step by step.
-```
-Forward process (known):
- x_t = sqrt(alpha_t) * x_{t-1} + sqrt(1 - alpha_t) * epsilon
- where epsilon ~ N(0, I)
-
- After T steps: x_T ~ N(0, I) (pure noise)
-
-Reverse process (learned):
- x_{t-1} = (1/sqrt(alpha_t)) * (x_t - (1 - alpha_t)/sqrt(1 - alpha_bar_t) * epsilon_theta(x_t, t)) + sigma_t * z
- where z ~ N(0, I)
-
- Each denoising step is a sampling step.
-```
+$$
+\begin{aligned}
+&\text{Forward process (known):} \\
+&\quad x_t = \sqrt{\alpha_t} \, x_{t-1} + \sqrt{1 - \alpha_t} \, \epsilon \\
+&\quad \text{where } \epsilon \sim N(0, I) \\
+&\quad \text{After } T \text{ steps: } x_T \sim N(0, I) \quad (\text{pure noise}) \\
+&\text{Reverse process (learned):} \\
+&\quad x_{t-1} = \frac{1}{\sqrt{\alpha_t}} \left( x_t - \frac{1 - \alpha_t}{\sqrt{1 - \bar{\alpha}_t}} \, \epsilon_\theta(x_t, t) \right) + \sigma_t \, z \\
+&\quad \text{where } z \sim N(0, I) \\
+&\quad \text{Each denoising step is a sampling step.}
+\end{aligned}
+$$
The connection to the methods in this lesson:
- Each denoising step uses the reparameterization trick (sample noise, apply deterministic transform)
@@ -454,10 +467,10 @@ def sample_uniform(a, b):
def sample_exponential_inverse_cdf(lam):
u = random.random()
- return -math.log(u) / lam
+ return -math.log1p(-u) / lam
```
-Generate 10,000 exponential samples and verify the mean is 1/lambda.
+Generate 10,000 exponential samples and verify the mean is $1/\lambda$.
### Step 2: Rejection sampling
@@ -484,7 +497,7 @@ def importance_sampling_estimate(f, target_pdf, proposal_pdf, proposal_sample, n
return total / n
```
-Estimate E[X^2] under a normal distribution using a uniform proposal. Compare to the known answer (mu^2 + sigma^2).
+Estimate $E[X^2]$ under a normal distribution using a uniform proposal. Compare to the known answer ($\mu^2 + \sigma^2$).
### Step 4: Monte Carlo estimation of pi
@@ -542,6 +555,8 @@ def softmax(logits):
return [e / total for e in exps]
def temperature_sample(logits, temperature):
+ if temperature == 0:
+ return max(range(len(logits)), key=lambda i: logits[i])
scaled = [z / temperature for z in logits]
probs = softmax(scaled)
return sample_from_probs(probs)
@@ -600,7 +615,7 @@ def gumbel_sample():
return -math.log(-math.log(u))
def gumbel_softmax(logits, temperature):
- gumbels = [math.log(p) + gumbel_sample() for p in logits]
+ gumbels = [z + gumbel_sample() for z in logits]
return softmax([g / temperature for g in gumbels])
```
@@ -642,13 +657,13 @@ You built these from scratch. Now you know what the library calls are doing.
## Exercises
-1. Implement inverse CDF sampling for the Cauchy distribution. The CDF is F(x) = 0.5 + arctan(x)/pi. Generate 10,000 samples and plot the histogram against the true PDF. Notice the heavy tails (extreme values far from center).
+1. Implement inverse CDF sampling for the Cauchy distribution. The CDF is $F(x) = 0.5 + \arctan(x)/\pi$. Generate 10,000 samples and plot the histogram against the true PDF. Notice the heavy tails (extreme values far from center).
2. Use rejection sampling to generate samples from a Beta(2, 5) distribution using a Uniform(0, 1) proposal. Plot the accepted samples against the true Beta PDF. What is the theoretical acceptance rate?
-3. Estimate the integral of sin(x) from 0 to pi using Monte Carlo with 1,000, 10,000, and 100,000 samples. Compare the error at each level. Verify that the error scales as O(1/sqrt(N)).
+3. Estimate the integral of $\sin(x)$ from $0$ to $\pi$ using Monte Carlo with 1,000, 10,000, and 100,000 samples. Compare the error at each level. Verify that the error scales as $O(1/\sqrt{N})$.
-4. Implement Metropolis-Hastings to sample from a 2D distribution p(x, y) proportional to exp(-(x^2 * y^2 + x^2 + y^2 - 8*x - 8*y) / 2). Plot the samples and the chain trajectory. Experiment with different proposal standard deviations.
+4. Implement Metropolis-Hastings to sample from a 2D distribution $p(x, y)$ proportional to $\exp(-(x^2 y^2 + x^2 + y^2 - 8x - 8y) / 2)$. Plot the samples and the chain trajectory. Experiment with different proposal standard deviations.
5. Build a complete text generation demo: given a vocabulary of 10 words with logits, generate sequences of 20 tokens using (a) greedy, (b) temperature=0.7, (c) top-k=3, (d) top-p=0.9. Compare the diversity of outputs across 5 runs.
@@ -657,22 +672,22 @@ You built these from scratch. Now you know what the library calls are doing.
| Term | What people say | What it actually means |
|------|----------------|----------------------|
| Sampling | "Drawing random values" | Generating values according to a probability distribution. The mechanism behind all generative AI |
-| Uniform distribution | "All equally likely" | Every value in [a, b] has equal probability density 1/(b-a). The starting point for all sampling methods |
-| Inverse CDF | "Probability transform" | F_inverse(U) converts a uniform sample into a sample from any distribution with known CDF. Exact and efficient |
+| Uniform distribution | "All equally likely" | Every value in $[a, b]$ has equal probability density $1/(b-a)$. The starting point for all sampling methods |
+| Inverse CDF | "Probability transform" | $F^{-1}(U)$ converts a uniform sample into a sample from any distribution with known CDF. Exact and efficient |
| Rejection sampling | "Propose and accept/reject" | Generate from a simple proposal, accept with probability proportional to target/proposal ratio. Exact but wastes samples |
-| Importance sampling | "Reweight samples" | Estimate expectations under p(x) using samples from q(x) by weighting each sample by p(x)/q(x). Core to PPO in RL |
-| Monte Carlo | "Average random samples" | Approximate integrals as sample averages. Error O(1/sqrt(N)) regardless of dimension |
+| Importance sampling | "Reweight samples" | Estimate expectations under $p(x)$ using samples from $q(x)$ by weighting each sample by $p(x)/q(x)$. Core to PPO in RL |
+| Monte Carlo | "Average random samples" | Approximate integrals as sample averages. Error $O(1/\sqrt{N})$ regardless of dimension |
| MCMC | "Random walk that converges" | Construct a Markov chain whose stationary distribution is the target. Metropolis-Hastings is the foundational algorithm |
| Metropolis-Hastings | "Accept uphill, sometimes downhill" | Propose moves, accept based on density ratio. Detailed balance ensures convergence to target distribution |
| Gibbs sampling | "One variable at a time" | Update each variable from its conditional distribution holding others fixed. 100% acceptance rate |
-| Temperature | "Confidence knob" | Divides logits by T before softmax. T<1 sharpens (more confident), T>1 flattens (more diverse) |
+| Temperature | "Confidence knob" | Divides logits by $T$ before softmax. $T<1$ sharpens (more confident), $T>1$ flattens (more diverse) |
| Top-k sampling | "Keep the k best" | Zero out all but the k highest-probability tokens, renormalize, sample. Fixed candidate set size |
| Nucleus sampling (top-p) | "Keep the probable ones" | Keep the smallest set of tokens whose cumulative probability exceeds p. Adaptive candidate set size |
-| Reparameterization trick | "Move randomness outside" | Write z = mu + sigma * epsilon where epsilon ~ N(0,1). Makes sampling differentiable. Essential for VAE training |
+| Reparameterization trick | "Move randomness outside" | Write $z = \mu + \sigma \epsilon$ where $\epsilon \sim N(0,1)$. Makes sampling differentiable. Essential for VAE training |
| Gumbel-Softmax | "Soft categorical sampling" | Differentiable approximation to categorical sampling using Gumbel noise + softmax with temperature |
| Stratified sampling | "Forced coverage" | Divide sample space into strata, sample from each. Always lower variance than naive Monte Carlo |
| Burn-in | "Warm-up period" | Initial MCMC samples discarded before the chain reaches its stationary distribution |
-| Detailed balance | "Reversibility condition" | p(x) * T(x->y) = p(y) * T(y->x). Sufficient condition for p to be the stationary distribution of a Markov chain |
+| Detailed balance | "Reversibility condition" | $p(x) \, T(x \to y) = p(y) \, T(y \to x)$. Sufficient condition for $p$ to be the stationary distribution of a Markov chain |
| Diffusion sampling | "Iterative denoising" | Generate data by starting from noise and applying learned denoising steps. Each step is a conditional sampling operation |
## Further Reading
diff --git a/phases/01-math-foundations/17-linear-systems/docs/en.md b/phases/01-math-foundations/17-linear-systems/docs/en.md
index a35f40c059..54cb33b9a0 100644
--- a/phases/01-math-foundations/17-linear-systems/docs/en.md
+++ b/phases/01-math-foundations/17-linear-systems/docs/en.md
@@ -111,7 +111,7 @@ Back substitute:
2*x1 + 2 + 2 = 8 --> x1 = 2
```
-Gaussian elimination costs O(n^3) operations. For a 1000x1000 system, that is about a billion floating-point operations. Fast, but you can do better if you need to solve multiple systems with the same A.
+Gaussian elimination costs $O(n^3)$ operations. For a 1000x1000 system, that is about a billion floating-point operations. Fast, but you can do better if you need to solve multiple systems with the same A.
### Partial pivoting: why it matters
@@ -146,17 +146,18 @@ A = L @ U
| 2 3 1 | | 1 2 1 | | 0 0 -2 |
```
-Why factor instead of just eliminating? Because once you have L and U, solving Ax = b for any new b costs only O(n^2):
+Why factor instead of just eliminating? Because once you have L and U, solving Ax = b for any new b costs only $O(n^2)$:
-```
-Ax = b
-LUx = b
-Let y = Ux:
- Ly = b (forward substitution, O(n^2))
- Ux = y (back substitution, O(n^2))
-```
+$$
+\begin{aligned}
+Ax &= b \\
+LUx &= b \\
+\text{let } y = Ux: \quad Ly &= b \quad (\text{forward substitution, } O(n^2)) \\
+Ux &= y \quad (\text{back substitution, } O(n^2))
+\end{aligned}
+$$
-The O(n^3) cost is paid once during factorization. Every subsequent solve is O(n^2). If you need to solve 1000 systems with the same A but different b vectors, LU saves a factor of 1000/3 in total work.
+The $O(n^3)$ cost is paid once during factorization. Every subsequent solve is $O(n^2)$. If you need to solve 1000 systems with the same A but different b vectors, LU saves a factor of 1000/3 in total work.
With partial pivoting, you get PA = LU where P is a permutation matrix recording the row swaps.
@@ -164,7 +165,7 @@ With partial pivoting, you get PA = LU where P is a permutation matrix recording
QR decomposition factors A into an orthogonal matrix Q and an upper triangular matrix R: A = QR.
-An orthogonal matrix has the property Q^T Q = I. Its columns are orthonormal vectors. Multiplying by Q preserves lengths and angles.
+An orthogonal matrix has the property $Q^T Q = I$. Its columns are orthonormal vectors. Multiplying by Q preserves lengths and angles.
```
A = Q @ R
@@ -178,27 +179,24 @@ To solve Ax = b:
Back substitute to get x.
```
-QR is numerically more stable than LU for solving least-squares problems. The Gram-Schmidt process builds Q column by column:
-
-```
-Given columns a1, a2, ... of A:
-
-q1 = a1 / ||a1||
-
-q2 = a2 - (a2 . q1) * q1 (subtract projection onto q1)
-q2 = q2 / ||q2|| (normalize)
+QR is numerically more stable than LU for solving least-squares problems. The Gram-Schmidt process builds Q column by column, given columns $a_1, a_2, \dots$ of A:
-q3 = a3 - (a3 . q1) * q1 - (a3 . q2) * q2
-q3 = q3 / ||q3||
-
-R[i][j] = qi . aj for i <= j
-```
+$$
+\begin{aligned}
+q_1 &= a_1 / \|a_1\| \\
+q_2 &= a_2 - (a_2 \cdot q_1)\, q_1 \quad (\text{subtract projection onto } q_1) \\
+q_2 &= q_2 / \|q_2\| \quad (\text{normalize}) \\
+q_3 &= a_3 - (a_3 \cdot q_1)\, q_1 - (a_3 \cdot q_2)\, q_2 \\
+q_3 &= q_3 / \|q_3\| \\
+R_{ij} &= q_i \cdot a_j \quad \text{for } i \leq j
+\end{aligned}
+$$
Each step removes the component along all previous q vectors, leaving only the new orthogonal direction.
### Cholesky decomposition
-When A is symmetric (A = A^T) and positive definite (all eigenvalues positive), you can factor it as A = L L^T where L is lower triangular. This is the Cholesky decomposition.
+When A is symmetric ($A = A^T$) and positive definite (all eigenvalues positive), you can factor it as $A = L L^T$ where L is lower triangular. This is the Cholesky decomposition.
```
A = L @ L^T
@@ -215,28 +213,27 @@ Cholesky is twice as fast as LU and requires half the storage. It only works for
- Covariance matrices are symmetric positive semi-definite (positive definite with regularization).
- The kernel matrix in Gaussian processes is symmetric positive definite.
- The Hessian of a convex function at a minimum is symmetric positive definite.
-- A^T A is always symmetric positive semi-definite.
+- $A^T A$ is always symmetric positive semi-definite.
-In Gaussian processes, you factor the kernel matrix K with Cholesky, then solve K alpha = y to get the predictive mean. The Cholesky factor also gives you the log-determinant for the marginal likelihood: log det(K) = 2 * sum(log(diag(L))).
+In Gaussian processes, you factor the kernel matrix K with Cholesky, then solve $K \alpha = y$ to get the predictive mean. The Cholesky factor also gives you the log-determinant for the marginal likelihood: $\log \det(K) = 2 \sum \log(\operatorname{diag}(L))$.
### Least squares: when Ax = b has no exact solution
-If A is m x n with m > n (more equations than unknowns), the system is overdetermined. There is no exact solution. Instead, you minimize the squared error:
+If A is $m \times n$ with $m > n$ (more equations than unknowns), the system is overdetermined. There is no exact solution. Instead, you minimize the squared error:
-```
-minimize ||Ax - b||^2
+$$
+\text{minimize } \|Ax - b\|^2
+$$
-This is the sum of squared residuals:
- sum((A[i,:] @ x - b[i])^2 for i in range(m))
-```
+This is the sum of squared residuals $\sum_{i} (A_{i,:} \cdot x - b_i)^2$.
The minimizer satisfies the normal equations:
-```
+$$
A^T A x = A^T b
-```
+$$
-Derivation: expand ||Ax - b||^2 = (Ax - b)^T (Ax - b) = x^T A^T A x - 2 x^T A^T b + b^T b. Take the gradient with respect to x, set it to zero: 2 A^T A x - 2 A^T b = 0.
+Derivation: expand $\|Ax - b\|^2 = (Ax - b)^T (Ax - b) = x^T A^T A x - 2 x^T A^T b + b^T b$. Take the gradient with respect to $x$, set it to zero: $2 A^T A x - 2 A^T b = 0$.
```
Original system (overdetermined, 4 equations, 2 unknowns):
@@ -249,7 +246,7 @@ Normal equations:
A^T A = | 4 10 | A^T b = | 22 |
| 10 30 | | 63 |
-Solve: x = [1.5, 1.7]
+Solve: x = [1.5, 1.6]
This is linear regression. x[0] is the intercept, x[1] is the slope.
```
@@ -258,33 +255,35 @@ This is linear regression. x[0] is the intercept, x[1] is the slope.
The connection is exact. In linear regression, your data matrix X has one row per sample and one column per feature. Your target vector y has one entry per sample. The weight vector w satisfies:
-```
-X^T X w = X^T y
-w = (X^T X)^(-1) X^T y
-```
+$$
+\begin{aligned}
+X^T X w &= X^T y \\
+w &= (X^T X)^{-1} X^T y
+\end{aligned}
+$$
This is the closed-form solution to linear regression. Every call to `sklearn.linear_model.LinearRegression.fit()` computes this (or an equivalent via QR or SVD).
-Add a regularization term lambda * I to the matrix and you get ridge regression:
+Add a regularization term $\lambda I$ to the matrix and you get ridge regression:
-```
-(X^T X + lambda * I) w = X^T y
-w = (X^T X + lambda * I)^(-1) X^T y
-```
+$$
+\begin{aligned}
+(X^T X + \lambda I) w &= X^T y \\
+w &= (X^T X + \lambda I)^{-1} X^T y
+\end{aligned}
+$$
-The regularization makes the matrix better conditioned (easier to invert accurately) and prevents overfitting by shrinking the weights toward zero. The matrix X^T X + lambda * I is always symmetric positive definite when lambda > 0, so you can use Cholesky to solve it.
+The regularization makes the matrix better conditioned (easier to invert accurately) and prevents overfitting by shrinking the weights toward zero. The matrix $X^T X + \lambda I$ is always symmetric positive definite when $\lambda > 0$, so you can use Cholesky to solve it.
### Pseudoinverse (Moore-Penrose)
-The pseudoinverse A+ generalizes matrix inversion to non-square and singular matrices. For any matrix A:
+The pseudoinverse $A^+$ generalizes matrix inversion to non-square and singular matrices. For any matrix A:
-```
-x = A+ b
-
-where A+ = V Sigma+ U^T (computed via SVD)
-```
+$$
+x = A^+ b \quad \text{where } A^+ = V \Sigma^+ U^T \quad (\text{computed via SVD})
+$$
-Sigma+ is formed by taking the reciprocal of each nonzero singular value and transposing the result. If A = U Sigma V^T, then A+ = V Sigma+ U^T.
+$\Sigma^+$ is formed by taking the reciprocal of each nonzero singular value and transposing the result. If $A = U \Sigma V^T$, then $A^+ = V \Sigma^+ U^T$.
```
A = U Sigma V^T (SVD)
@@ -297,9 +296,9 @@ A+ = V Sigma+ U^T
```
The pseudoinverse gives the minimum-norm least-squares solution. If the system has:
-- One solution: A+ b gives it.
-- No solution: A+ b gives the least-squares solution.
-- Infinite solutions: A+ b gives the one with the smallest ||x||.
+- One solution: $A^+ b$ gives it.
+- No solution: $A^+ b$ gives the least-squares solution.
+- Infinite solutions: $A^+ b$ gives the one with the smallest $\|x\|$.
NumPy's `np.linalg.lstsq` and `np.linalg.pinv` both use the SVD internally.
@@ -307,11 +306,11 @@ NumPy's `np.linalg.lstsq` and `np.linalg.pinv` both use the SVD internally.
The condition number measures how sensitive the solution is to small changes in the input. For a matrix A, the condition number is:
-```
-kappa(A) = ||A|| * ||A^(-1)|| = sigma_max / sigma_min
-```
+$$
+\kappa(A) = \|A\| \cdot \|A^{-1}\| = \frac{\sigma_{\max}}{\sigma_{\min}}
+$$
-where sigma_max and sigma_min are the largest and smallest singular values.
+where $\sigma_{\max}$ and $\sigma_{\min}$ are the largest and smallest singular values.
```
Well-conditioned (kappa ~ 1): Ill-conditioned (kappa ~ 10^15):
@@ -323,11 +322,11 @@ small change in x huge change in x
```
Rules of thumb:
-- kappa < 100: safe, solution is accurate.
-- kappa ~ 10^k: you lose about k digits of precision from your floating-point arithmetic.
-- kappa ~ 10^16 (for float64): the solution is meaningless. The matrix is effectively singular.
+- $\kappa < 100$: safe, solution is accurate.
+- $\kappa \sim 10^k$: you lose about $k$ digits of precision from your floating-point arithmetic.
+- $\kappa \sim 10^{16}$ (for float64): the solution is meaningless. The matrix is effectively singular.
-In ML, ill-conditioning happens when features are nearly collinear. Regularization (adding lambda * I) improves the condition number from sigma_max / sigma_min to (sigma_max + lambda) / (sigma_min + lambda).
+In ML, ill-conditioning happens when features are nearly collinear. For the regularized normal equations, adding $\lambda I$ changes the condition number from $\sigma_{\max}^2 / \sigma_{\min}^2$ to $(\sigma_{\max}^2 + \lambda) / (\sigma_{\min}^2 + \lambda)$.
### Iterative methods: conjugate gradient
@@ -362,29 +361,29 @@ The convergence rate depends on the condition number. Better conditioned systems
| Method | Requirements | Cost | Use case |
|--------|-------------|------|----------|
-| Gaussian elimination | Square, nonsingular A | O(n^3) | One-off solve of a square system |
-| LU decomposition | Square, nonsingular A | O(n^3) factor + O(n^2) solve | Multiple solves with the same A |
-| QR decomposition | Any A (m >= n) | O(mn^2) | Least squares, numerically stable |
-| Cholesky | Symmetric positive definite A | O(n^3/3) | Covariance matrices, Gaussian processes, ridge regression |
-| Normal equations | Overdetermined (m > n) | O(mn^2 + n^3) | Linear regression (small n) |
-| SVD / pseudoinverse | Any A | O(mn^2) | Rank-deficient systems, minimum-norm solutions |
-| Conjugate gradient | Symmetric positive definite, sparse A | O(n * k * nnz) | Large sparse systems, k = iterations |
+| Gaussian elimination | Square, nonsingular A | $O(n^3)$ | One-off solve of a square system |
+| LU decomposition | Square, nonsingular A | $O(n^3)$ factor + $O(n^2)$ solve | Multiple solves with the same A |
+| QR decomposition | Any A ($m \geq n$) | $O(mn^2)$ | Least squares, numerically stable |
+| Cholesky | Symmetric positive definite A | $O(n^3/3)$ | Covariance matrices, Gaussian processes, ridge regression |
+| Normal equations | Overdetermined ($m > n$) | $O(mn^2 + n^3)$ | Linear regression (small n) |
+| SVD / pseudoinverse | Any A | $O(mn^2)$ | Rank-deficient systems, minimum-norm solutions |
+| Conjugate gradient | Symmetric positive definite, sparse A | $O(n \cdot k \cdot nnz)$ | Large sparse systems, k = iterations |
### Connection to ML
Every method in this lesson appears in production ML:
-**Linear regression.** The closed-form solution solves the normal equations X^T X w = X^T y. This is done via Cholesky (if n is small) or QR (if numerical stability matters) or SVD (if the matrix might be rank-deficient).
+**Linear regression.** The closed-form solution solves the normal equations $X^T X w = X^T y$. This is done via Cholesky (if n is small) or QR (if numerical stability matters) or SVD (if the matrix might be rank-deficient).
-**Ridge regression.** Adds lambda * I to X^T X. The regularized system (X^T X + lambda * I) w = X^T y is always solvable via Cholesky because X^T X + lambda * I is symmetric positive definite for lambda > 0.
+**Ridge regression.** Adds $\lambda I$ to $X^T X$. The regularized system $(X^T X + \lambda I) w = X^T y$ is always solvable via Cholesky because $X^T X + \lambda I$ is symmetric positive definite for $\lambda > 0$.
-**Gaussian processes.** The predictive mean requires solving K alpha = y where K is the kernel matrix. Cholesky factorization of K is the standard approach. The log marginal likelihood uses log det(K) = 2 sum(log(diag(L))).
+**Gaussian processes.** The predictive mean requires solving $K \alpha = y$ where K is the kernel matrix. Cholesky factorization of K is the standard approach. The log marginal likelihood uses $\log \det(K) = 2 \sum \log(\operatorname{diag}(L))$.
**Neural network initialization.** Orthogonal initialization uses QR decomposition to create weight matrices whose columns are orthonormal. This prevents signal collapse in deep networks.
**Preconditioning.** Large-scale optimizers use incomplete Cholesky or incomplete LU as preconditioners for conjugate gradient solvers.
-**Feature engineering.** The condition number of X^T X tells you if your features are collinear. If kappa is large, drop features or add regularization.
+**Feature engineering.** The condition number of $X^T X$ tells you if your features are collinear. If $\kappa$ is large, drop features or add regularization.
```figure
linear-system-conditioning
@@ -545,7 +544,7 @@ This lesson produces:
1. Solve the system `[[1,2,3],[4,5,6],[7,8,10]] x = [6, 15, 27]` using your Gaussian elimination, your LU solver, and `np.linalg.solve`. Verify all three give the same answer within floating-point tolerance.
-2. Generate a 50x5 random matrix X and target y = X @ w_true + noise. Solve for w using normal equations, QR (via `np.linalg.qr`), SVD (via `np.linalg.svd`), and `np.linalg.lstsq`. Compare all four solutions. Measure the condition number of X^T X and explain how it affects which method you trust.
+2. Generate a 50x5 random matrix X and target y = X @ w_true + noise. Solve for w using normal equations, QR (via `np.linalg.qr`), SVD (via `np.linalg.svd`), and `np.linalg.lstsq`. Compare all four solutions. Measure the condition number of $X^T X$ and explain how it affects which method you trust.
3. Create a nearly singular matrix by making two columns almost identical (e.g., column 2 = column 1 + 1e-10 * noise). Compute its condition number. Solve Ax = b with and without regularization (add 0.01 * I). Compare the solutions and residuals. Explain why regularization helps.
@@ -558,20 +557,20 @@ This lesson produces:
| Term | What people say | What it actually means |
|------|----------------|----------------------|
| Linear system | "Solve for x" | A set of linear equations Ax = b. Finding x means finding the input that produces output b under transformation A. |
-| Gaussian elimination | "Row reduce" | Systematically zero out entries below the diagonal using row operations, producing an upper triangular system solvable by back substitution. O(n^3). |
+| Gaussian elimination | "Row reduce" | Systematically zero out entries below the diagonal using row operations, producing an upper triangular system solvable by back substitution. $O(n^3)$. |
| Partial pivoting | "Swap rows for stability" | Before eliminating in column k, swap the row with the largest absolute value in that column to the pivot position. Prevents division by small numbers. |
-| LU decomposition | "Factor into triangles" | Write A = LU where L is lower triangular (stores multipliers) and U is upper triangular (the eliminated matrix). Amortizes the O(n^3) cost over multiple solves. |
-| QR decomposition | "Orthogonal factorization" | Write A = QR where Q has orthonormal columns and R is upper triangular. More stable than LU for least squares. |
-| Cholesky decomposition | "Square root of a matrix" | For symmetric positive definite A, write A = LL^T. Half the cost of LU. Used for covariance matrices, kernel matrices, and ridge regression. |
-| Least squares | "Best fit when exact is impossible" | Minimize the sum of squared residuals ||Ax - b||^2 when the system is overdetermined (more equations than unknowns). |
-| Normal equations | "The calculus shortcut" | A^T A x = A^T b. Setting the gradient of ||Ax - b||^2 to zero. This IS the closed-form solution to linear regression. |
-| Pseudoinverse | "Inversion for non-square matrices" | A+ = V Sigma+ U^T via SVD. Gives the minimum-norm least-squares solution for any matrix, square or rectangular, singular or not. |
-| Condition number | "How trustworthy is this answer" | kappa = sigma_max / sigma_min. Measures sensitivity to input perturbations. Lose about log10(kappa) digits of precision. |
-| Ridge regression | "Regularized least squares" | Solve (X^T X + lambda I) w = X^T y. Adding lambda I improves conditioning and shrinks weights toward zero. Prevents overfitting. |
+| LU decomposition | "Factor into triangles" | Write $A = LU$ where L is lower triangular (stores multipliers) and U is upper triangular (the eliminated matrix). Amortizes the $O(n^3)$ cost over multiple solves. |
+| QR decomposition | "Orthogonal factorization" | Write $A = QR$ where Q has orthonormal columns and R is upper triangular. More stable than LU for least squares. |
+| Cholesky decomposition | "Square root of a matrix" | For symmetric positive definite A, write $A = LL^T$. Half the cost of LU. Used for covariance matrices, kernel matrices, and ridge regression. |
+| Least squares | "Best fit when exact is impossible" | Minimize the sum of squared residuals $\|Ax - b\|^2$ when the system is overdetermined (more equations than unknowns). |
+| Normal equations | "The calculus shortcut" | $A^T A x = A^T b$. Setting the gradient of $\|Ax - b\|^2$ to zero. This IS the closed-form solution to linear regression. |
+| Pseudoinverse | "Inversion for non-square matrices" | $A^+ = V \Sigma^+ U^T$ via SVD. Gives the minimum-norm least-squares solution for any matrix, square or rectangular, singular or not. |
+| Condition number | "How trustworthy is this answer" | $\kappa = \sigma_{\max} / \sigma_{\min}$. Measures sensitivity to input perturbations. Lose about $\log_{10}(\kappa)$ digits of precision. |
+| Ridge regression | "Regularized least squares" | Solve $(X^T X + \lambda I) w = X^T y$. Adding $\lambda I$ improves conditioning and shrinks weights toward zero. Prevents overfitting. |
| Conjugate gradient | "Iterative Ax=b for big matrices" | An iterative solver for symmetric positive definite systems. Converges in at most n steps. Practical for large sparse systems where factorization is too expensive. |
-| Overdetermined system | "More data than parameters" | m > n in an m-by-n system. No exact solution exists. Least squares finds the best approximation. This is every regression problem. |
-| Back substitution | "Solve from the bottom up" | Given an upper triangular system, solve the last equation first, then substitute backward. O(n^2). |
-| Forward substitution | "Solve from the top down" | Given a lower triangular system, solve the first equation first, then substitute forward. O(n^2). Used in the L step of LU solves. |
+| Overdetermined system | "More data than parameters" | $m > n$ in an m-by-n system. No exact solution exists. Least squares finds the best approximation. This is every regression problem. |
+| Back substitution | "Solve from the bottom up" | Given an upper triangular system, solve the last equation first, then substitute backward. $O(n^2)$. |
+| Forward substitution | "Solve from the top down" | Given a lower triangular system, solve the first equation first, then substitute forward. $O(n^2)$. Used in the L step of LU solves. |
## Further Reading
diff --git a/phases/01-math-foundations/18-convex-optimization/docs/en.md b/phases/01-math-foundations/18-convex-optimization/docs/en.md
index a8663e56a8..2fb207c25f 100644
--- a/phases/01-math-foundations/18-convex-optimization/docs/en.md
+++ b/phases/01-math-foundations/18-convex-optimization/docs/en.md
@@ -26,7 +26,7 @@ Understanding convexity does three things. First, it tells you when your problem
### Convex sets
-A set S is convex if for any two points in S, the line segment between them also lies entirely in S.
+A set $S$ is convex if for any two points in $S$, the line segment between them also lies entirely in $S$.
| Convex sets | Not convex |
|---|---|
@@ -34,12 +34,12 @@ A set S is convex if for any two points in S, the line segment between them also
| **Triangle**: same property holds for all interior points | **Donut/annulus**: the hole means some line segments leave the set |
| The line segment between any two points stays within the set | The line segment between some pairs of points exits the set |
-Formal test: for any points x, y in S and any t in [0, 1], the point tx + (1-t)y is also in S.
+Formal test: for any points $x, y$ in $S$ and any $t \in [0, 1]$, the point $tx + (1-t)y$ is also in $S$.
Examples of convex sets:
-- A line, a plane, all of R^n
+- A line, a plane, all of $\mathbb{R}^n$
- A ball (circle, sphere, hypersphere)
-- A halfspace: {x : a^T x <= b}
+- A halfspace: $\{x : a^T x \leq b\}$
- The intersection of any number of convex sets
Examples of non-convex sets:
@@ -49,11 +49,11 @@ Examples of non-convex sets:
### Convex functions
-A function f is convex if its domain is a convex set and for any two points x, y in its domain and any t in [0, 1]:
+A function $f$ is convex if its domain is a convex set and for any two points $x, y$ in its domain and any $t \in [0, 1]$:
-```
-f(tx + (1-t)y) <= t*f(x) + (1-t)*f(y)
-```
+$$
+f(tx + (1-t)y) \leq t \cdot f(x) + (1-t) \cdot f(y)
+$$
Geometrically: the line segment between any two points on the graph lies above or on the graph.
@@ -64,26 +64,26 @@ Geometrically: the line segment between any two points on the graph lies above o
| **Local minima** | Every local minimum is the global minimum | Multiple local minima may exist at different heights |
Common convex functions:
-- f(x) = x^2 (parabola)
-- f(x) = |x| (absolute value)
-- f(x) = e^x (exponential)
-- f(x) = max(0, x) (ReLU, though piecewise linear)
-- f(x) = -log(x) for x > 0 (negative log)
-- Any linear function f(x) = a^T x + b (both convex and concave)
+- $f(x) = x^2$ (parabola)
+- $f(x) = |x|$ (absolute value)
+- $f(x) = e^x$ (exponential)
+- $f(x) = \max(0, x)$ (ReLU, though piecewise linear)
+- $f(x) = -\log(x)$ for $x > 0$ (negative log)
+- Any linear function $f(x) = a^T x + b$ (both convex and concave)
### Testing for convexity
Three practical tests, from easiest to most rigorous.
-**Test 1: Second derivative test (1D).** If f''(x) >= 0 for all x, then f is convex.
+**Test 1: Second derivative test (1D).** If $f''(x) \geq 0$ for all $x$, then $f$ is convex.
-- f(x) = x^2: f''(x) = 2 >= 0. Convex.
-- f(x) = x^3: f''(x) = 6x. Negative for x < 0. Not convex.
-- f(x) = e^x: f''(x) = e^x > 0. Convex.
+- $f(x) = x^2$: $f''(x) = 2 \geq 0$. Convex.
+- $f(x) = x^3$: $f''(x) = 6x$. Negative for $x < 0$. Not convex.
+- $f(x) = e^x$: $f''(x) = e^x > 0$. Convex.
-**Test 2: Hessian test (multivariate).** If the Hessian matrix H(x) is positive semidefinite for all x, then f is convex. The Hessian is the matrix of second partial derivatives.
+**Test 2: Hessian test (multivariate).** If the Hessian matrix $H(x)$ is positive semidefinite for all $x$, then $f$ is convex. The Hessian is the matrix of second partial derivatives.
-**Test 3: Definition test.** Check the inequality f(tx + (1-t)y) <= t*f(x) + (1-t)*f(y) directly. Useful for functions where derivatives are hard to compute.
+**Test 3: Definition test.** Check the inequality $f(tx + (1-t)y) \leq t \cdot f(x) + (1-t) \cdot f(y)$ directly. Useful for functions where derivatives are hard to compute.
### Why convexity matters
@@ -129,21 +129,24 @@ Linear models with convex losses are convex. The moment you add hidden layers wi
### The Hessian matrix
-The Hessian H of a function f: R^n -> R is the n x n matrix of second partial derivatives.
+The Hessian $H$ of a function $f: \mathbb{R}^n \to \mathbb{R}$ is the $n \times n$ matrix of second partial derivatives.
-```
-H[i][j] = d^2 f / (dx_i dx_j)
-```
+$$
+H_{ij} = \frac{\partial^2 f}{\partial x_i \, \partial x_j}
+$$
-For f(x, y) = x^2 + 3xy + y^2:
+For $f(x, y) = x^2 + 3xy + y^2$:
-```
-df/dx = 2x + 3y d^2f/dx^2 = 2 d^2f/dxdy = 3
-df/dy = 3x + 2y d^2f/dydx = 3 d^2f/dy^2 = 2
+$$
+\begin{aligned}
+\frac{\partial f}{\partial x} = 2x + 3y \qquad & \frac{\partial^2 f}{\partial x^2} = 2 \qquad \frac{\partial^2 f}{\partial x \, \partial y} = 3 \\
+\frac{\partial f}{\partial y} = 3x + 2y \qquad & \frac{\partial^2 f}{\partial y \, \partial x} = 3 \qquad \frac{\partial^2 f}{\partial y^2} = 2
+\end{aligned}
+$$
-H = [ 2 3 ]
- [ 3 2 ]
-```
+$$
+H = \begin{bmatrix} 2 & 3 \\ 3 & 2 \end{bmatrix}
+$$
The Hessian tells you about curvature:
- Eigenvalues all positive: the function curves upward in every direction (convex at that point)
@@ -151,19 +154,23 @@ The Hessian tells you about curvature:
- Mixed signs: saddle point (curves up in some directions, down in others)
- Zero eigenvalue: flat in that direction (degenerate)
-For convexity, the Hessian must be positive semidefinite (all eigenvalues >= 0) everywhere, not just at one point.
+For convexity, the Hessian must be positive semidefinite (all eigenvalues $\geq 0$) everywhere, not just at one point.
### Newton's method
Gradient descent uses first-order information (the gradient). Newton's method uses second-order information (the Hessian). It fits a quadratic approximation at the current point and jumps directly to the minimum of that quadratic.
-```
Update rule:
- x_new = x - H^(-1) * gradient
+
+$$
+x_{\text{new}} = x - H^{-1} \cdot \text{gradient}
+$$
Compare to gradient descent:
- x_new = x - lr * gradient
-```
+
+$$
+x_{\text{new}} = x - \text{lr} \cdot \text{gradient}
+$$
Newton's method replaces the scalar learning rate with the inverse Hessian. This automatically adjusts the step size and direction based on local curvature.
@@ -190,14 +197,14 @@ Advantages:
- Scale-invariant (works regardless of how you parameterize the problem)
Disadvantages:
-- Computing the Hessian costs O(n^2) memory and O(n^3) to invert
-- For a neural network with 1 million weights, that is 10^12 entries and 10^18 operations
+- Computing the Hessian costs $O(n^2)$ memory and $O(n^3)$ to invert
+- For a neural network with 1 million weights, that is $10^{12}$ entries and $10^{18}$ operations
- Not practical for deep learning
### Constrained optimization
-Unconstrained optimization: minimize f(x) over all x.
-Constrained optimization: minimize f(x) subject to constraints.
+Unconstrained optimization: minimize $f(x)$ over all $x$.
+Constrained optimization: minimize $f(x)$ subject to constraints.
Real problems have constraints. You want to minimize cost but your budget is limited. You want to minimize error but your model complexity is bounded.
@@ -216,22 +223,24 @@ graph LR
The method of Lagrange multipliers converts a constrained problem into an unconstrained one.
-Problem: minimize f(x) subject to g(x) = 0.
+Problem: minimize $f(x)$ subject to $g(x) = 0$.
-Solution: introduce a new variable (the Lagrange multiplier lambda) and solve the unconstrained problem:
+Solution: introduce a new variable (the Lagrange multiplier $\lambda$) and solve the unconstrained problem:
-```
-L(x, lambda) = f(x) + lambda * g(x)
-```
+$$
+L(x, \lambda) = f(x) + \lambda \cdot g(x)
+$$
-At the solution, the gradient of L is zero:
+At the solution, the gradient of $L$ is zero:
-```
-dL/dx = df/dx + lambda * dg/dx = 0
-dL/dlambda = g(x) = 0
-```
+$$
+\begin{aligned}
+\frac{\partial L}{\partial x} &= \frac{\partial f}{\partial x} + \lambda \cdot \frac{\partial g}{\partial x} = 0 \\
+\frac{\partial L}{\partial \lambda} &= g(x) = 0
+\end{aligned}
+$$
-Geometric intuition: at the constrained minimum, the gradient of f must be parallel to the gradient of the constraint g. If they were not parallel, you could move along the constraint surface and reduce f further.
+Geometric intuition: at the constrained minimum, the gradient of $f$ must be parallel to the gradient of the constraint $g$. If they were not parallel, you could move along the constraint surface and reduce $f$ further.
```mermaid
graph LR
@@ -240,39 +249,41 @@ graph LR
S --- C["At the solution, gradient of f is parallel to gradient of g"]
```
-Example: minimize f(x,y) = x^2 + y^2 subject to x + y = 1.
-
-```
-L = x^2 + y^2 + lambda(x + y - 1)
+Example: minimize $f(x,y) = x^2 + y^2$ subject to $x + y = 1$.
-dL/dx = 2x + lambda = 0 => x = -lambda/2
-dL/dy = 2y + lambda = 0 => y = -lambda/2
-dL/dlambda = x + y - 1 = 0
+$$
+\begin{aligned}
+L &= x^2 + y^2 + \lambda(x + y - 1) \\[4pt]
+\frac{\partial L}{\partial x} &= 2x + \lambda = 0 \implies x = -\lambda/2 \\
+\frac{\partial L}{\partial y} &= 2y + \lambda = 0 \implies y = -\lambda/2 \\
+\frac{\partial L}{\partial \lambda} &= x + y - 1 = 0
+\end{aligned}
+$$
-From first two: x = y
-Substituting: 2x = 1, so x = y = 0.5, lambda = -1
-```
+From the first two: $x = y$. Substituting: $2x = 1$, so $x = y = 0.5$, $\lambda = -1$.
-The closest point on the line x + y = 1 to the origin is (0.5, 0.5).
+The closest point on the line $x + y = 1$ to the origin is $(0.5, 0.5)$.
### KKT conditions
The Karush-Kuhn-Tucker conditions extend Lagrange multipliers to inequality constraints.
-Problem: minimize f(x) subject to g_i(x) <= 0 for i = 1, ..., m.
+Problem: minimize $f(x)$ subject to $g_i(x) \leq 0$ for $i = 1, \ldots, m$.
The KKT conditions (necessary for optimality):
-```
-1. Stationarity: df/dx + sum(lambda_i * dg_i/dx) = 0
-2. Primal feasibility: g_i(x) <= 0 for all i
-3. Dual feasibility: lambda_i >= 0 for all i
-4. Complementary slackness: lambda_i * g_i(x) = 0 for all i
-```
+$$
+\begin{aligned}
+&\text{1. Stationarity:} && \frac{\partial f}{\partial x} + \sum_i \lambda_i \cdot \frac{\partial g_i}{\partial x} = 0 \\
+&\text{2. Primal feasibility:} && g_i(x) \leq 0 \quad \text{for all } i \\
+&\text{3. Dual feasibility:} && \lambda_i \geq 0 \quad \text{for all } i \\
+&\text{4. Complementary slackness:} && \lambda_i \cdot g_i(x) = 0 \quad \text{for all } i
+\end{aligned}
+$$
-Complementary slackness is the key insight: either the constraint is active (g_i = 0, the solution sits on the boundary) or the multiplier is zero (the constraint does not matter). A constraint that does not affect the solution has lambda = 0.
+Complementary slackness is the key insight: either the constraint is active ($g_i = 0$, the solution sits on the boundary) or the multiplier is zero (the constraint does not matter). A constraint that does not affect the solution has $\lambda = 0$.
-KKT conditions are central to SVMs. The support vectors are the data points where the constraint is active (lambda > 0). All other data points have lambda = 0 and do not affect the decision boundary.
+KKT conditions are central to SVMs. The support vectors are the data points where the constraint is active ($\lambda > 0$). All other data points have $\lambda = 0$ and do not affect the decision boundary.
### Regularization as constrained optimization
@@ -280,25 +291,31 @@ L1 and L2 regularization are not arbitrary tricks. They are constrained optimiza
**L2 regularization (Ridge):**
-```
-minimize Loss(w) subject to ||w||^2 <= t
+$$
+\text{minimize } \text{Loss}(w) \quad \text{subject to } \|w\|^2 \leq t
+$$
Equivalent unconstrained form:
-minimize Loss(w) + lambda * ||w||^2
-```
-The constraint ||w||^2 <= t defines a ball (circle in 2D, sphere in 3D). The solution is where the loss contours first touch this ball.
+$$
+\text{minimize } \text{Loss}(w) + \lambda \cdot \|w\|^2
+$$
+
+The constraint $\|w\|^2 \leq t$ defines a ball (circle in 2D, sphere in 3D). The solution is where the loss contours first touch this ball.
**L1 regularization (LASSO):**
-```
-minimize Loss(w) subject to ||w||_1 <= t
+$$
+\text{minimize } \text{Loss}(w) \quad \text{subject to } \|w\|_1 \leq t
+$$
Equivalent unconstrained form:
-minimize Loss(w) + lambda * ||w||_1
-```
-The constraint ||w||_1 <= t defines a diamond (rotated square in 2D).
+$$
+\text{minimize } \text{Loss}(w) + \lambda \cdot \|w\|_1
+$$
+
+The constraint $\|w\|_1 \leq t$ defines a diamond (rotated square in 2D).
| Property | L2 constraint (circle) | L1 constraint (diamond) |
|---|---|---|
@@ -315,12 +332,14 @@ Every constrained optimization problem (the primal) has a companion problem (the
The Lagrangian dual function:
-```
-Primal: minimize f(x) subject to g(x) <= 0
-Lagrangian: L(x, lambda) = f(x) + lambda * g(x)
-Dual function: d(lambda) = min_x L(x, lambda)
-Dual problem: maximize d(lambda) subject to lambda >= 0
-```
+$$
+\begin{aligned}
+\text{Primal:} \quad & \text{minimize } f(x) \text{ subject to } g(x) \leq 0 \\
+\text{Lagrangian:} \quad & L(x, \lambda) = f(x) + \lambda \cdot g(x) \\
+\text{Dual function:} \quad & d(\lambda) = \min_x L(x, \lambda) \\
+\text{Dual problem:} \quad & \text{maximize } d(\lambda) \text{ subject to } \lambda \geq 0
+\end{aligned}
+$$
Why duality matters:
- The dual problem is sometimes easier to solve than the primal
@@ -329,16 +348,16 @@ Why duality matters:
For SVMs specifically:
-```
-Primal: find w, b that maximize the margin 2/||w|| subject to
- y_i(w^T x_i + b) >= 1 for all i
-
-Dual: maximize sum(alpha_i) - 0.5 * sum_ij(alpha_i * alpha_j * y_i * y_j * x_i^T x_j)
- subject to alpha_i >= 0 and sum(alpha_i * y_i) = 0
+$$
+\begin{aligned}
+\text{Primal:} \quad & \text{find } w, b \text{ that maximize the margin } \frac{2}{\|w\|} \text{ subject to} \\
+& y_i(w^T x_i + b) \geq 1 \text{ for all } i \\[6pt]
+\text{Dual:} \quad & \text{maximize } \sum_i \alpha_i - \tfrac{1}{2} \sum_{ij} \alpha_i \alpha_j y_i y_j \, x_i^T x_j \\
+& \text{subject to } \alpha_i \geq 0 \text{ and } \sum_i \alpha_i y_i = 0
+\end{aligned}
+$$
-The dual only involves dot products x_i^T x_j.
-Replace x_i^T x_j with K(x_i, x_j) to get the kernel trick.
-```
+The dual only involves dot products $x_i^T x_j$. Replace $x_i^T x_j$ with $K(x_i, x_j)$ to get the kernel trick.
### Why deep learning works despite non-convexity
@@ -346,7 +365,7 @@ Neural network loss functions are wildly non-convex. By every classical measure,
**Most local minima are good enough.** In high-dimensional spaces, random critical points (where the gradient is zero) are overwhelmingly saddle points, not local minima. The few local minima that exist tend to have loss values close to the global minimum. Getting trapped in a terrible local minimum is extremely unlikely when the parameter space has millions of dimensions.
-**Saddle points, not local minima, are the real obstacle.** In a function with n parameters, a saddle point has a mix of positive and negative curvature directions. For a random critical point in high dimensions, the probability of all n eigenvalues being positive (local minimum) is roughly 2^(-n). Almost all critical points are saddle points. SGD's noise helps escape them.
+**Saddle points, not local minima, are the real obstacle.** In a function with $n$ parameters, a saddle point has a mix of positive and negative curvature directions. For a random critical point in high dimensions, the probability of all $n$ eigenvalues being positive (local minimum) is roughly $2^{-n}$. Almost all critical points are saddle points. SGD's noise helps escape them.
**Overparameterization smooths the landscape.** Networks with more parameters than training examples have smoother, more connected loss surfaces. Wider networks have fewer bad local minima. This is counterintuitive but empirically consistent.
@@ -365,21 +384,21 @@ Neural network loss functions are wildly non-convex. By every classical measure,
Pure Newton's method is impractical for large models. Several approximations make second-order information usable.
-**L-BFGS (Limited-memory BFGS):** Approximates the inverse Hessian using the last m gradient differences. Requires O(mn) memory instead of O(n^2). Works well for problems with up to ~10,000 parameters. Used in classical ML (logistic regression, CRFs) but not deep learning.
+**L-BFGS (Limited-memory BFGS):** Approximates the inverse Hessian using the last $m$ gradient differences. Requires $O(mn)$ memory instead of $O(n^2)$. Works well for problems with up to ~10,000 parameters. Used in classical ML (logistic regression, CRFs) but not deep learning.
**Natural gradient:** Uses the Fisher information matrix (expected Hessian of the log-likelihood) instead of the standard Hessian. This accounts for the geometry of probability distributions. K-FAC (Kronecker-Factored Approximate Curvature) approximates the Fisher matrix as a Kronecker product, making it practical for neural networks.
-**Hessian-free optimization:** Uses conjugate gradient to solve Hx = g without ever forming H. Only requires Hessian-vector products, which can be computed in O(n) time via automatic differentiation.
+**Hessian-free optimization:** Uses conjugate gradient to solve $Hx = g$ without ever forming $H$. Only requires Hessian-vector products, which can be computed in $O(n)$ time via automatic differentiation.
**Diagonal approximations:** Adam's second moment is a diagonal approximation of the Hessian's diagonal. AdaHessian extends this by using actual Hessian diagonal elements via Hutchinson's estimator.
| Method | Memory | Per-step cost | When to use |
|--------|--------|--------------|-------------|
-| Gradient descent | O(n) | O(n) | Baseline, large models |
-| Newton's method | O(n^2) | O(n^3) | Small convex problems |
-| L-BFGS | O(mn) | O(mn) | Medium convex problems |
-| Adam | O(n) | O(n) | Deep learning default |
-| K-FAC | O(n) | O(n) per layer | Research, large-batch training |
+| Gradient descent | $O(n)$ | $O(n)$ | Baseline, large models |
+| Newton's method | $O(n^2)$ | $O(n^3)$ | Small convex problems |
+| L-BFGS | $O(mn)$ | $O(mn)$ | Medium convex problems |
+| Adam | $O(n)$ | $O(n)$ | Deep learning default |
+| K-FAC | $O(n)$ | $O(n)$ per layer | Research, large-batch training |
```figure
convex-vs-nonconvex
@@ -516,13 +535,13 @@ print(f"Support vectors: {svm.n_support_}")
## Exercises
-1. **Convexity gallery.** Test these functions for convexity using the checker: f(x) = x^4, f(x) = sin(x), f(x,y) = x^2 + y^2, f(x,y) = x*y, f(x) = max(x, 0). Explain why each result makes sense.
+1. **Convexity gallery.** Test these functions for convexity using the checker: $f(x) = x^4$, $f(x) = \sin(x)$, $f(x,y) = x^2 + y^2$, $f(x,y) = xy$, $f(x) = \max(x, 0)$. Explain why each result makes sense.
-2. **Newton vs gradient descent race.** Run both methods on f(x,y) = 50*x^2 + y^2 from the starting point (10, 10). How many steps does each need to reach loss < 1e-10? What happens to gradient descent when the condition number (ratio of largest to smallest Hessian eigenvalue) increases?
+2. **Newton vs gradient descent race.** Run both methods on $f(x,y) = 50x^2 + y^2$ from the starting point $(10, 10)$. How many steps does each need to reach loss < 1e-10? What happens to gradient descent when the condition number (ratio of largest to smallest Hessian eigenvalue) increases?
-3. **Lagrange multiplier geometry.** Minimize f(x,y) = (x-3)^2 + (y-3)^2 subject to x + 2y = 4. Verify the solution by checking that the gradient of f is parallel to the gradient of g at the solution.
+3. **Lagrange multiplier geometry.** Minimize $f(x,y) = (x-3)^2 + (y-3)^2$ subject to $x + 2y = 4$. Verify the solution by checking that the gradient of $f$ is parallel to the gradient of $g$ at the solution.
-4. **Regularization constraint.** Implement L1-constrained optimization: minimize (x-3)^2 + (y-2)^2 subject to |x| + |y| <= 1. Show that the solution has one coordinate equal to zero (sparsity from the diamond constraint).
+4. **Regularization constraint.** Implement L1-constrained optimization: minimize $(x-3)^2 + (y-2)^2$ subject to $|x| + |y| \leq 1$. Show that the solution has one coordinate equal to zero (sparsity from the diamond constraint).
5. **Hessian eigenvalue analysis.** Compute the Hessian of the Rosenbrock function at (1,1) and at (-1,1). Compute eigenvalues at both points. What do the eigenvalues tell you about the curvature at the minimum versus far from it?
diff --git a/phases/01-math-foundations/19-complex-numbers/docs/en.md b/phases/01-math-foundations/19-complex-numbers/docs/en.md
index 02b85259cf..a743dee951 100644
--- a/phases/01-math-foundations/19-complex-numbers/docs/en.md
+++ b/phases/01-math-foundations/19-complex-numbers/docs/en.md
@@ -30,14 +30,14 @@ This lesson builds complex arithmetic from scratch, connects it to geometry, and
A complex number has two parts: a real part and an imaginary part.
-```
+$$
z = a + bi
+$$
where:
- a is the real part
- b is the imaginary part
- i is the imaginary unit, defined by i^2 = -1
-```
+- $a$ is the real part
+- $b$ is the imaginary part
+- $i$ is the imaginary unit, defined by $i^2 = -1$
That is it. You extend the number line into a plane. The real numbers sit on one axis. The imaginary numbers sit on the other. Every complex number is a point in this plane.
@@ -45,41 +45,48 @@ That is it. You extend the number line into a plane. The real numbers sit on one
**Addition.** Add the real parts together, add the imaginary parts together.
-```
-(a + bi) + (c + di) = (a + c) + (b + d)i
-
-Example: (3 + 2i) + (1 + 4i) = 4 + 6i
-```
-
-**Multiplication.** Use the distributive law and remember that i^2 = -1.
-
-```
-(a + bi)(c + di) = ac + adi + bci + bdi^2
- = ac + adi + bci - bd
- = (ac - bd) + (ad + bc)i
-
-Example: (3 + 2i)(1 + 4i) = 3 + 12i + 2i + 8i^2
- = 3 + 14i - 8
- = -5 + 14i
-```
+$$
+\begin{aligned}
+(a + bi) + (c + di) &= (a + c) + (b + d)i \\
+\text{Example: } (3 + 2i) + (1 + 4i) &= 4 + 6i
+\end{aligned}
+$$
+
+**Multiplication.** Use the distributive law and remember that $i^2 = -1$.
+
+$$
+\begin{aligned}
+(a + bi)(c + di) &= ac + adi + bci + bdi^2 \\
+ &= ac + adi + bci - bd \\
+ &= (ac - bd) + (ad + bc)i
+\end{aligned}
+$$
+
+$$
+\begin{aligned}
+\text{Example: } (3 + 2i)(1 + 4i) &= 3 + 12i + 2i + 8i^2 \\
+ &= 3 + 14i - 8 \\
+ &= -5 + 14i
+\end{aligned}
+$$
**Conjugate.** Flip the sign of the imaginary part.
-```
-conjugate of (a + bi) = a - bi
-```
+$$
+\text{conjugate of } (a + bi) = a - bi
+$$
The product of a complex number and its conjugate is always real:
-```
+$$
(a + bi)(a - bi) = a^2 + b^2
-```
+$$
**Division.** Multiply numerator and denominator by the conjugate of the denominator.
-```
-(a + bi) / (c + di) = (a + bi)(c - di) / (c^2 + d^2)
-```
+$$
+\frac{a + bi}{c + di} = \frac{(a + bi)(c - di)}{c^2 + d^2}
+$$
This eliminates the imaginary part from the denominator, giving you a clean complex number.
@@ -87,11 +94,9 @@ This eliminates the imaginary part from the denominator, giving you a clean comp
The complex plane maps every complex number to a 2D point. The horizontal axis is the real axis, the vertical axis is the imaginary axis.
-```
-z = 3 + 2i corresponds to the point (3, 2)
-z = -1 + 0i corresponds to the point (-1, 0) on the real axis
-z = 0 + 4i corresponds to the point (0, 4) on the imaginary axis
-```
+- $z = 3 + 2i$ corresponds to the point $(3, 2)$
+- $z = -1 + 0i$ corresponds to the point $(-1, 0)$ on the real axis
+- $z = 0 + 4i$ corresponds to the point $(0, 4)$ on the imaginary axis
A complex number is simultaneously a point and a vector from the origin. This dual interpretation is what makes complex numbers useful for geometry.
@@ -99,24 +104,25 @@ A complex number is simultaneously a point and a vector from the origin. This du
Any point in the plane can be described by its distance from the origin and its angle from the positive real axis.
-```
-z = r * (cos(theta) + i*sin(theta))
+$$
+z = r \cdot (\cos\theta + i\sin\theta)
+$$
where:
- r = |z| = sqrt(a^2 + b^2) (magnitude, or modulus)
- theta = atan2(b, a) (phase, or argument)
-```
+- $r = |z| = \sqrt{a^2 + b^2}$ (magnitude, or modulus)
+- $\theta = \operatorname{atan2}(b, a)$ (phase, or argument)
Rectangular form (a + bi) is good for addition. Polar form (r, theta) is good for multiplication.
**Multiplication in polar form.** Multiply the magnitudes, add the angles.
-```
-z1 = r1 * e^(i*theta1)
-z2 = r2 * e^(i*theta2)
-
-z1 * z2 = (r1 * r2) * e^(i*(theta1 + theta2))
-```
+$$
+\begin{aligned}
+z_1 &= r_1 \cdot e^{i\theta_1} \\
+z_2 &= r_2 \cdot e^{i\theta_2} \\
+z_1 \cdot z_2 &= (r_1 \cdot r_2) \cdot e^{i(\theta_1 + \theta_2)}
+\end{aligned}
+$$
This is why complex numbers are perfect for rotations. Multiplying by a complex number with magnitude 1 is a pure rotation.
@@ -124,39 +130,49 @@ This is why complex numbers are perfect for rotations. Multiplying by a complex
The bridge between complex exponentials and trigonometry:
-```
-e^(i*theta) = cos(theta) + i*sin(theta)
-```
+$$
+e^{i\theta} = \cos\theta + i\sin\theta
+$$
-This is the most important formula in this lesson. When theta = pi:
+This is the most important formula in this lesson. When $\theta = \pi$:
-```
-e^(i*pi) = cos(pi) + i*sin(pi) = -1 + 0i = -1
+$$
+e^{i\pi} = \cos\pi + i\sin\pi = -1 + 0i = -1
+$$
-Therefore: e^(i*pi) + 1 = 0
-```
+$$
+\text{Therefore: } e^{i\pi} + 1 = 0
+$$
-Five fundamental constants (e, i, pi, 1, 0) linked in one equation.
+Five fundamental constants ($e$, $i$, $\pi$, $1$, $0$) linked in one equation.
### Why Euler's formula matters for ML
-Euler's formula says that `e^(i*theta)` traces the unit circle as theta varies. At theta = 0, you are at (1, 0). At theta = pi/2, you are at (0, 1). At theta = pi, you are at (-1, 0). At theta = 3*pi/2, you are at (0, -1). A full rotation is theta = 2*pi.
+Euler's formula says that $e^{i\theta}$ traces the unit circle as $\theta$ varies. At $\theta = 0$, you are at $(1, 0)$. At $\theta = \pi/2$, you are at $(0, 1)$. At $\theta = \pi$, you are at $(-1, 0)$. At $\theta = 3\pi/2$, you are at $(0, -1)$. A full rotation is $\theta = 2\pi$.
This means complex exponentials ARE rotations. And rotations are everywhere in signal processing and ML.
### Connection to 2D rotations
-Multiplying the complex number (x + yi) by e^(i*theta) rotates the point (x, y) by angle theta around the origin.
+Multiplying the complex number $(x + yi)$ by $e^{i\theta}$ rotates the point $(x, y)$ by angle $\theta$ around the origin.
-```
Rotation via complex multiplication:
- (x + yi) * (cos(theta) + i*sin(theta))
- = (x*cos(theta) - y*sin(theta)) + (x*sin(theta) + y*cos(theta))i
+
+$$
+\begin{aligned}
+(x + yi) &\cdot (\cos\theta + i\sin\theta) \\
+&= (x\cos\theta - y\sin\theta) + (x\sin\theta + y\cos\theta)i
+\end{aligned}
+$$
Rotation via matrix multiplication:
- [cos(theta) -sin(theta)] [x] [x*cos(theta) - y*sin(theta)]
- [sin(theta) cos(theta)] [y] = [x*sin(theta) + y*cos(theta)]
-```
+
+$$
+\begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{bmatrix}
+\begin{bmatrix} x \\ y \end{bmatrix}
+=
+\begin{bmatrix} x\cos\theta - y\sin\theta \\ x\sin\theta + y\cos\theta \end{bmatrix}
+$$
They produce identical results. Complex multiplication IS 2D rotation. The rotation matrix is just complex multiplication written in matrix notation.
@@ -173,16 +189,16 @@ graph TD
### Phasors and rotating signals
-A complex exponential e^(i*omega*t) is a point rotating around the unit circle at angular frequency omega. As t increases, the point traces the circle.
+A complex exponential $e^{i\omega t}$ is a point rotating around the unit circle at angular frequency $\omega$. As $t$ increases, the point traces the circle.
-The real part of this rotating point is cos(omega*t). The imaginary part is sin(omega*t). A sinusoidal signal is the shadow of a rotating complex number.
+The real part of this rotating point is $\cos(\omega t)$. The imaginary part is $\sin(\omega t)$. A sinusoidal signal is the shadow of a rotating complex number.
-```
-e^(i*omega*t) = cos(omega*t) + i*sin(omega*t)
+$$
+e^{i\omega t} = \cos(\omega t) + i\sin(\omega t)
+$$
-Real part: cos(omega*t) -- a cosine wave
-Imaginary part: sin(omega*t) -- a sine wave
-```
+- Real part: $\cos(\omega t)$ -- a cosine wave
+- Imaginary part: $\sin(\omega t)$ -- a sine wave
This is the phasor representation. Instead of tracking a wiggly sine wave, you track a smoothly rotating arrow. Phase shifts become angle offsets. Amplitude changes become magnitude changes. Addition of signals becomes vector addition.
@@ -190,38 +206,38 @@ This is the phasor representation. Instead of tracking a wiggly sine wave, you t
The N-th roots of unity are N points equally spaced on the unit circle:
-```
-w_k = e^(2*pi*i*k/N) for k = 0, 1, 2, ..., N-1
-```
+$$
+w_k = e^{2\pi i k/N} \quad \text{for } k = 0, 1, 2, \ldots, N-1
+$$
-For N = 4, the roots are: 1, i, -1, -i (the four compass points).
-For N = 8, you get the four compass points plus the four diagonals.
+For $N = 4$, the roots are: $1, i, -1, -i$ (the four compass points).
+For $N = 8$, you get the four compass points plus the four diagonals.
Roots of unity are the foundation of the Discrete Fourier Transform. The DFT decomposes a signal into components at these N equally-spaced frequencies.
### Connection to the DFT
-The Discrete Fourier Transform of a signal x[0], x[1], ..., x[N-1] is:
+The Discrete Fourier Transform of a signal $x[0], x[1], \ldots, x[N-1]$ is:
-```
-X[k] = sum_{n=0}^{N-1} x[n] * e^(-2*pi*i*k*n/N)
-```
+$$
+X[k] = \sum_{n=0}^{N-1} x[n] \cdot e^{-2\pi i k n/N}
+$$
-Each X[k] measures how much the signal correlates with the k-th root of unity -- a complex sinusoid at frequency k. The DFT breaks a signal into N rotating phasors and tells you the amplitude and phase of each one.
+Each $X[k]$ measures how much the signal correlates with the $k$-th root of unity -- a complex sinusoid at frequency $k$. The DFT breaks a signal into $N$ rotating phasors and tells you the amplitude and phase of each one.
### Why i is not imaginary
The word "imaginary" is a historical accident. Descartes used it dismissively. But i is no more imaginary than negative numbers were when people first rejected them. Negative numbers answer "what do you subtract 5 from 3 to get?" The imaginary unit answers "what do you square to get -1?"
-More usefully: i is a 90-degree rotation operator. Multiply a real number by i once, you rotate 90 degrees to the imaginary axis. Multiply by i again (i^2), you rotate another 90 degrees -- now you are pointing in the negative real direction. That is why i^2 = -1. It is not mysterious. It is a half-turn built from two quarter-turns.
+More usefully: $i$ is a 90-degree rotation operator. Multiply a real number by $i$ once, you rotate 90 degrees to the imaginary axis. Multiply by $i$ again ($i^2$), you rotate another 90 degrees -- now you are pointing in the negative real direction. That is why $i^2 = -1$. It is not mysterious. It is a half-turn built from two quarter-turns.
This is why complex numbers are everywhere in engineering. Anything that rotates -- electromagnetic waves, quantum states, signal oscillations, positional encodings -- is naturally described by complex numbers.
### Complex exponentials vs trigonometric functions
-Before Euler's formula, engineers wrote signals as A*cos(omega*t + phi) -- amplitude A, frequency omega, phase phi. This works but makes arithmetic painful. Adding two cosines with different phases requires trigonometric identities.
+Before Euler's formula, engineers wrote signals as $A\cos(\omega t + \phi)$ -- amplitude $A$, frequency $\omega$, phase $\phi$. This works but makes arithmetic painful. Adding two cosines with different phases requires trigonometric identities.
-With complex exponentials, the same signal is A*e^(i*(omega*t + phi)). Adding two signals is just adding two complex numbers. Multiplying (modulating) is just multiplying magnitudes and adding angles. Phase shifts become angle additions. Frequency shifts become multiplications by phasors.
+With complex exponentials, the same signal is $A e^{i(\omega t + \phi)}$. Adding two signals is just adding two complex numbers. Multiplying (modulating) is just multiplying magnitudes and adding angles. Phase shifts become angle additions. Frequency shifts become multiplications by phasors.
The entire field of signal processing switched to complex exponential notation because the math is cleaner. The "real signal" is always just the real part of the complex representation. The imaginary part is carried along as bookkeeping, making all the algebra work out naturally.
@@ -229,10 +245,12 @@ The entire field of signal processing switched to complex exponential notation b
**Sinusoidal positional encodings** (original Transformer paper):
-```
-PE(pos, 2i) = sin(pos / 10000^(2i/d))
-PE(pos, 2i+1) = cos(pos / 10000^(2i/d))
-```
+$$
+\begin{aligned}
+PE(pos, 2i) &= \sin\left(\frac{pos}{10000^{2i/d}}\right) \\
+PE(pos, 2i+1) &= \cos\left(\frac{pos}{10000^{2i/d}}\right)
+\end{aligned}
+$$
The sin and cos pairs are the real and imaginary parts of complex exponentials at different frequencies. Each frequency provides a different "resolution" for encoding position. Low frequencies change slowly (coarse position). High frequencies change quickly (fine position). Together they give each position a unique frequency fingerprint.
@@ -240,13 +258,13 @@ The sin and cos pairs are the real and imaginary parts of complex exponentials a
| Operation | Algebraic Form | Geometric Meaning |
|-----------|---------------|-------------------|
-| Addition | (a+c) + (b+d)i | Vector addition in the plane |
-| Multiplication | (ac-bd) + (ad+bc)i | Rotate and scale |
-| Conjugate | a - bi | Reflect over real axis |
-| Magnitude | sqrt(a^2 + b^2) | Distance from origin |
-| Phase | atan2(b, a) | Angle from positive real axis |
+| Addition | $(a+c) + (b+d)i$ | Vector addition in the plane |
+| Multiplication | $(ac-bd) + (ad+bc)i$ | Rotate and scale |
+| Conjugate | $a - bi$ | Reflect over real axis |
+| Magnitude | $\sqrt{a^2 + b^2}$ | Distance from origin |
+| Phase | $\operatorname{atan2}(b, a)$ | Angle from positive real axis |
| Division | multiply by conjugate | Reverse rotation and rescale |
-| Power | r^n * e^(i*n*theta) | Rotate n times, scale by r^n |
+| Power | $r^n \cdot e^{in\theta}$ | Rotate $n$ times, scale by $r^n$ |
```mermaid
graph LR
@@ -326,7 +344,7 @@ Verify: `euler(theta).magnitude()` should always be 1.0. `euler(0)` should give
### Step 3: Rotation
-Rotating a point (x, y) by angle theta is one complex multiplication:
+Rotating a point $(x, y)$ by angle $\theta$ is one complex multiplication:
```python
point = Complex(3, 4)
@@ -350,7 +368,7 @@ def dft(signal):
return result
```
-This is the O(N^2) DFT. Each output X[k] is the sum of the signal samples multiplied by roots of unity.
+This is the $O(N^2)$ DFT. Each output $X[k]$ is the sum of the signal samples multiplied by roots of unity.
### Step 5: Inverse DFT
@@ -424,30 +442,30 @@ Run `code/complex_numbers.py` to generate `outputs/skill-complex-arithmetic.md`.
## Exercises
-1. **Complex arithmetic by hand.** Compute (2 + 3i) * (4 - i) and verify with the code. Then compute (5 + 2i) / (1 - 3i). Draw both results on the complex plane and check that multiplication rotated and scaled the first number.
+1. **Complex arithmetic by hand.** Compute $(2 + 3i)(4 - i)$ and verify with the code. Then compute $(5 + 2i) / (1 - 3i)$. Draw both results on the complex plane and check that multiplication rotated and scaled the first number.
-2. **Rotation sequence.** Start with the point (1, 0). Multiply by e^(i*pi/6) twelve times. Verify that you return to (1, 0) after 12 multiplications. Print the coordinates at each step and confirm they trace a regular 12-gon.
+2. **Rotation sequence.** Start with the point $(1, 0)$. Multiply by $e^{i\pi/6}$ twelve times. Verify that you return to $(1, 0)$ after 12 multiplications. Print the coordinates at each step and confirm they trace a regular 12-gon.
-3. **DFT of a known signal.** Create a signal that is the sum of sin(2*pi*3*t) and 0.5*sin(2*pi*7*t) sampled at 32 points. Run your DFT. Verify that the magnitude spectrum has peaks at frequencies 3 and 7, with the peak at 7 being half the height of the peak at 3.
+3. **DFT of a known signal.** Create a signal that is the sum of $\sin(2\pi \cdot 3t)$ and $0.5\sin(2\pi \cdot 7t)$ sampled at 32 points. Run your DFT. Verify that the magnitude spectrum has peaks at frequencies 3 and 7, with the peak at 7 being half the height of the peak at 3.
-4. **Roots of unity visualization.** Compute the 8th roots of unity. Verify that they sum to zero. Verify that multiplying any root by the primitive root e^(2*pi*i/8) gives the next root.
+4. **Roots of unity visualization.** Compute the 8th roots of unity. Verify that they sum to zero. Verify that multiplying any root by the primitive root $e^{2\pi i/8}$ gives the next root.
-5. **Rotation matrix equivalence.** For 10 random angles and 10 random points, verify that complex multiplication gives the same result as matrix-vector multiplication with the 2x2 rotation matrix. Print the maximum numerical difference.
+5. **Rotation matrix equivalence.** For 10 random angles and 10 random points, verify that complex multiplication gives the same result as matrix-vector multiplication with the $2\times2$ rotation matrix. Print the maximum numerical difference.
## Key Terms
| Term | What it means |
|------|---------------|
-| Complex number | A number a + bi where a is the real part, b is the imaginary part, and i^2 = -1 |
-| Imaginary unit | The number i, defined by i^2 = -1. Not imaginary in the philosophical sense -- it is a rotation operator |
+| Complex number | A number $a + bi$ where $a$ is the real part, $b$ is the imaginary part, and $i^2 = -1$ |
+| Imaginary unit | The number $i$, defined by $i^2 = -1$. Not imaginary in the philosophical sense -- it is a rotation operator |
| Complex plane | The 2D plane where the x-axis is real and the y-axis is imaginary. Also called the Argand plane |
-| Magnitude (modulus) | The distance from the origin: sqrt(a^2 + b^2). Written as \|z\| |
-| Phase (argument) | The angle from the positive real axis: atan2(b, a). Written as arg(z) |
-| Conjugate | The mirror image across the real axis: conjugate of a + bi is a - bi |
-| Polar form | Expressing z as r * e^(i*theta) instead of a + bi. Makes multiplication easy |
-| Euler's formula | e^(i*theta) = cos(theta) + i*sin(theta). Connects exponentials to trigonometry |
-| Phasor | A rotating complex number e^(i*omega*t) representing a sinusoidal signal |
-| Roots of unity | The N complex numbers e^(2*pi*i*k/N) for k = 0 to N-1. N equally spaced points on the unit circle |
+| Magnitude (modulus) | The distance from the origin: $\sqrt{a^2 + b^2}$. Written as $\vert z \vert$ |
+| Phase (argument) | The angle from the positive real axis: $\operatorname{atan2}(b, a)$. Written as $\arg(z)$ |
+| Conjugate | The mirror image across the real axis: conjugate of $a + bi$ is $a - bi$ |
+| Polar form | Expressing $z$ as $r \cdot e^{i\theta}$ instead of $a + bi$. Makes multiplication easy |
+| Euler's formula | $e^{i\theta} = \cos\theta + i\sin\theta$. Connects exponentials to trigonometry |
+| Phasor | A rotating complex number $e^{i\omega t}$ representing a sinusoidal signal |
+| Roots of unity | The $N$ complex numbers $e^{2\pi i k/N}$ for $k = 0$ to $N-1$. $N$ equally spaced points on the unit circle |
| DFT | Discrete Fourier Transform. Decomposes a signal into complex sinusoidal components using roots of unity |
| RoPE | Rotary Position Embedding. Uses complex multiplication to encode relative position in transformer attention |
diff --git a/phases/01-math-foundations/20-fourier-transform/docs/en.md b/phases/01-math-foundations/20-fourier-transform/docs/en.md
index 30177b2b7c..100079cc85 100644
--- a/phases/01-math-foundations/20-fourier-transform/docs/en.md
+++ b/phases/01-math-foundations/20-fourier-transform/docs/en.md
@@ -9,7 +9,7 @@
## Learning Objectives
-- Implement the DFT from scratch and verify it against the O(N log N) Cooley-Tukey FFT
+- Implement the DFT from scratch and verify it against the $O(N \log N)$ Cooley-Tukey FFT
- Interpret frequency coefficients: extract amplitude, phase, and power spectrum from a signal
- Apply the convolution theorem to perform convolution via FFT multiplication
- Connect Fourier frequency decomposition to transformer positional encodings and CNN convolution layers
@@ -30,65 +30,62 @@ This matters for ML because frequency-domain thinking appears everywhere. Convol
Given N samples x[0], x[1], ..., x[N-1], the Discrete Fourier Transform produces N frequency coefficients X[0], X[1], ..., X[N-1]:
-```
-X[k] = sum_{n=0}^{N-1} x[n] * e^(-2*pi*i*k*n/N)
-
-for k = 0, 1, ..., N-1
-```
+$$
+X[k] = \sum_{n=0}^{N-1} x[n] \cdot e^{-2\pi i k n / N} \quad \text{for } k = 0, 1, \ldots, N-1
+$$
-Each X[k] is a complex number. Its magnitude |X[k]| tells you the amplitude of frequency k. Its phase angle(X[k]) tells you the phase offset of that frequency.
+Each $X[k]$ is a complex number. Its magnitude $|X[k]|$ tells you the amplitude of frequency $k$. Its phase $\text{angle}(X[k])$ tells you the phase offset of that frequency.
-The key insight: `e^(-2*pi*i*k*n/N)` is a rotating phasor at frequency k. The DFT computes the correlation between the signal and each of N equally-spaced frequencies. If the signal contains energy at frequency k, the correlation is large. If not, it is near zero.
+The key insight: $e^{-2\pi i k n / N}$ is a rotating phasor at frequency $k$. The DFT computes the correlation between the signal and each of $N$ equally-spaced frequencies. If the signal contains energy at frequency $k$, the correlation is large. If not, it is near zero.
### What each coefficient means
-**X[0]: the DC component.** This is the sum of all samples -- proportional to the mean. It represents the constant (zero-frequency) offset of the signal.
+**$X[0]$: the DC component.** This is the sum of all samples -- proportional to the mean. It represents the constant (zero-frequency) offset of the signal.
-```
-X[0] = sum_{n=0}^{N-1} x[n] * e^0 = sum of all samples
-```
+$$
+X[0] = \sum_{n=0}^{N-1} x[n] \cdot e^0 = \text{sum of all samples}
+$$
-**X[k] for 1 <= k <= N/2: positive frequencies.** X[k] represents frequency k cycles per N samples. Higher k means higher frequency (faster oscillation).
+**$X[k]$ for $1 \leq k \leq N/2$: positive frequencies.** $X[k]$ represents frequency $k$ cycles per $N$ samples. Higher $k$ means higher frequency (faster oscillation).
-**X[N/2]: the Nyquist frequency.** The highest frequency you can represent with N samples. Above this, you get aliasing -- high frequencies masquerading as low ones.
+**$X[N/2]$: the Nyquist frequency.** The highest frequency you can represent with $N$ samples. Above this, you get aliasing -- high frequencies masquerading as low ones.
-**X[k] for N/2 < k < N: negative frequencies.** For real-valued signals, X[N-k] = conj(X[k]). The negative frequencies are mirror images of the positive ones. This is why the useful information is in the first N/2 + 1 coefficients.
+**$X[k]$ for $N/2 < k < N$: negative frequencies.** For real-valued signals, $X[N-k] = \text{conj}(X[k])$. The negative frequencies are mirror images of the positive ones. This is why the useful information is in the first $N/2 + 1$ coefficients.
### Inverse DFT
The inverse DFT reconstructs the original signal from its frequency coefficients:
-```
-x[n] = (1/N) * sum_{k=0}^{N-1} X[k] * e^(2*pi*i*k*n/N)
+$$
+x[n] = \frac{1}{N} \sum_{k=0}^{N-1} X[k] \cdot e^{2\pi i k n / N} \quad \text{for } n = 0, 1, \ldots, N-1
+$$
-for n = 0, 1, ..., N-1
-```
-
-The only differences from the forward DFT: the sign in the exponent is positive (not negative), and there is a 1/N normalization factor.
+The only differences from the forward DFT: the sign in the exponent is positive (not negative), and there is a $1/N$ normalization factor.
The inverse DFT is perfect reconstruction. No information is lost. You can go from time domain to frequency domain and back without any error. The DFT is a change of basis -- it re-expresses the same information in a different coordinate system.
### The FFT: making it fast
-The DFT as defined above is O(N^2): for each of N output coefficients, you sum over N input samples. For N = 1 million, that is 10^12 operations.
+The DFT as defined above is $O(N^2)$: for each of $N$ output coefficients, you sum over $N$ input samples. For $N$ = 1 million, that is $10^{12}$ operations.
-The Fast Fourier Transform (FFT) computes the same result in O(N log N). For N = 1 million, that is about 20 million operations instead of a trillion. This is what makes frequency analysis practical.
+The Fast Fourier Transform (FFT) computes the same result in $O(N \log N)$. For $N$ = 1 million, that is about 20 million operations instead of a trillion. This is what makes frequency analysis practical.
The Cooley-Tukey algorithm (the most common FFT) works by divide and conquer:
1. Split the signal into even-indexed and odd-indexed samples.
2. Compute the DFT of each half recursively.
-3. Combine the two half-size DFTs using "twiddle factors" e^(-2*pi*i*k/N).
+3. Combine the two half-size DFTs using "twiddle factors" $e^{-2\pi i k / N}$.
-```
-X[k] = E[k] + e^(-2*pi*i*k/N) * O[k] for k = 0, ..., N/2 - 1
-X[k + N/2] = E[k] - e^(-2*pi*i*k/N) * O[k] for k = 0, ..., N/2 - 1
+$$
+\begin{aligned}
+X[k] &= E[k] + e^{-2\pi i k / N} \cdot O[k] & \text{for } k = 0, \ldots, N/2 - 1 \\
+X[k + N/2] &= E[k] - e^{-2\pi i k / N} \cdot O[k] & \text{for } k = 0, \ldots, N/2 - 1
+\end{aligned}
+$$
-where E = DFT of even-indexed samples
- O = DFT of odd-indexed samples
-```
+where $E$ = DFT of even-indexed samples, $O$ = DFT of odd-indexed samples
-The symmetry means each level of recursion does O(N) work, and there are log2(N) levels. Total: O(N log N).
+The symmetry means each level of recursion does $O(N)$ work, and there are $\log_2(N)$ levels. Total: $O(N \log N)$.
```mermaid
graph TD
@@ -110,24 +107,28 @@ The FFT requires the signal length to be a power of 2. In practice, signals are
### Spectral analysis
-The **power spectrum** is |X[k]|^2 -- the squared magnitude of each frequency coefficient. It shows how much energy is at each frequency.
+The **power spectrum** is $|X[k]|^2$ -- the squared magnitude of each frequency coefficient. It shows how much energy is at each frequency.
-The **phase spectrum** is angle(X[k]) -- the phase offset of each frequency. For most analysis tasks, you care about the power spectrum and ignore the phase.
+The **phase spectrum** is $\text{angle}(X[k])$ -- the phase offset of each frequency. For most analysis tasks, you care about the power spectrum and ignore the phase.
-```
-Power at frequency k: P[k] = |X[k]|^2 = X[k].real^2 + X[k].imag^2
-Phase at frequency k: phi[k] = atan2(X[k].imag, X[k].real)
-```
+$$
+\begin{aligned}
+\text{Power at frequency } k: \quad & P[k] = |X[k]|^2 = X[k].\text{real}^2 + X[k].\text{imag}^2 \\
+\text{Phase at frequency } k: \quad & \phi[k] = \text{atan2}(X[k].\text{imag}, X[k].\text{real})
+\end{aligned}
+$$
### Frequency resolution
-The frequency resolution of the DFT depends on the number of samples N and the sampling rate fs.
+The frequency resolution of the DFT depends on the number of samples $N$ and the sampling rate $f_s$.
-```
-Frequency of bin k: f_k = k * fs / N
-Frequency resolution: delta_f = fs / N
-Maximum frequency: f_max = fs / 2 (Nyquist)
-```
+$$
+\begin{aligned}
+\text{Frequency of bin } k: \quad & f_k = k \cdot f_s / N \\
+\text{Frequency resolution}: \quad & \Delta f = f_s / N \\
+\text{Maximum frequency}: \quad & f_{\max} = f_s / 2 \quad \text{(Nyquist)}
+\end{aligned}
+$$
To resolve two frequencies that are close together, you need more samples. To capture high frequencies, you need a higher sampling rate.
@@ -137,20 +138,20 @@ This is one of the most important results in signal processing and directly rele
**Convolution in the time domain equals pointwise multiplication in the frequency domain.**
-```
-x * h = IFFT(FFT(x) . FFT(h))
+$$
+x * h = \text{IFFT}(\text{FFT}(x) \odot \text{FFT}(h))
+$$
-where * is convolution and . is element-wise multiplication
-```
+where $*$ is convolution and $\odot$ is element-wise multiplication
Why this matters:
-- Direct convolution of two signals of length N and M takes O(N*M) operations.
-- FFT-based convolution takes O(N log N): transform both, multiply, transform back.
+- Direct convolution of two signals of length $N$ and $M$ takes $O(N \cdot M)$ operations.
+- FFT-based convolution takes $O(N \log N)$: transform both, multiply, transform back.
- For large kernels, FFT convolution is dramatically faster.
- This is exactly what happens in convolutional layers with large receptive fields.
-Note: the DFT computes circular convolution (the signal wraps around). For linear convolution (no wraparound), zero-pad both signals to length N + M - 1 before computing.
+Note: the DFT computes circular convolution (the signal wraps around). For linear convolution (no wraparound), zero-pad both signals to length $N + M - 1$ before computing.
```mermaid
graph LR
@@ -183,24 +184,26 @@ Common windows:
| Hamming | Modified cosine | Moderate | Lower (-42 dB) | Audio processing, speech analysis |
| Blackman | Triple cosine | Wide | Very low (-58 dB) | When side lobe suppression is critical |
-```
-Hann window: w[n] = 0.5 * (1 - cos(2*pi*n / (N-1)))
-Hamming window: w[n] = 0.54 - 0.46 * cos(2*pi*n / (N-1))
-```
+$$
+\begin{aligned}
+\text{Hann window}: \quad & w[n] = 0.5 \cdot \left(1 - \cos\left(\frac{2\pi n}{N-1}\right)\right) \\
+\text{Hamming window}: \quad & w[n] = 0.54 - 0.46 \cdot \cos\left(\frac{2\pi n}{N-1}\right)
+\end{aligned}
+$$
-Apply the window by multiplying it element-wise with the signal before the DFT: `X = DFT(x * w)`.
+Apply the window by multiplying it element-wise with the signal before the DFT: $X = \text{DFT}(x \cdot w)$.
### DFT properties
| Property | Time Domain | Frequency Domain |
|----------|-------------|-----------------|
-| Linearity | a*x + b*y | a*X + b*Y |
-| Time shift | x[n - k] | X[f] * e^(-2*pi*i*f*k/N) |
-| Frequency shift | x[n] * e^(2*pi*i*f0*n/N) | X[f - f0] |
-| Convolution | x * h | X * H (pointwise) |
-| Multiplication | x * h (pointwise) | X * H (circular convolution, scaled by 1/N) |
-| Parseval's theorem | sum \|x[n]\|^2 | (1/N) * sum \|X[k]\|^2 |
-| Conjugate symmetry (real input) | x[n] real | X[k] = conj(X[N-k]) |
+| Linearity | $a \cdot x + b \cdot y$ | $a \cdot X + b \cdot Y$ |
+| Time shift | $x[n - k]$ | $X[f] \cdot e^{-2\pi i f k / N}$ |
+| Frequency shift | $x[n] \cdot e^{2\pi i f_0 n / N}$ | $X[f - f_0]$ |
+| Convolution | $x * h$ | $X \cdot H$ (pointwise) |
+| Multiplication | $x \cdot h$ (pointwise) | $X * H$ (circular convolution, scaled by $1/N$) |
+| Parseval's theorem | $\sum \vert x[n] \vert^2$ | $\frac{1}{N} \sum \vert X[k] \vert^2$ |
+| Conjugate symmetry (real input) | $x[n]$ real | $X[k] = \text{conj}(X[N-k])$ |
Parseval's theorem says the total energy is the same in both domains. Energy is conserved through the transform.
@@ -208,18 +211,20 @@ Parseval's theorem says the total energy is the same in both domains. Energy is
The original Transformer uses sinusoidal positional encodings:
-```
-PE(pos, 2i) = sin(pos / 10000^(2i/d_model))
-PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))
-```
+$$
+\begin{aligned}
+PE(pos, 2i) &= \sin\left(pos / 10000^{2i/d_{\text{model}}}\right) \\
+PE(pos, 2i+1) &= \cos\left(pos / 10000^{2i/d_{\text{model}}}\right)
+\end{aligned}
+$$
Each dimension pair (2i, 2i+1) oscillates at a different frequency. The frequencies are geometrically spaced from high (dimension 0,1) to low (last dimensions). This gives each position a unique pattern across all frequency bands -- similar to how Fourier coefficients uniquely identify a signal.
The key properties this provides:
- **Uniqueness:** No two positions have the same encoding.
-- **Bounded values:** sin and cos are always in [-1, 1].
-- **Relative position:** The encoding of position p+k can be expressed as a linear function of the encoding at position p. The model can learn to attend to relative positions.
+- **Bounded values:** sin and cos are always in $[-1, 1]$.
+- **Relative position:** The encoding of position $p+k$ can be expressed as a linear function of the encoding at position $p$. The model can learn to attend to relative positions.
### Connection to CNNs
@@ -231,7 +236,7 @@ By the convolution theorem, this is equivalent to:
3. Multiply in frequency domain
4. IFFT the result
-Standard CNN implementations use direct convolution (faster for small 3x3 kernels). But for large kernels or global convolution, FFT-based approaches are significantly faster. Some architectures (like FNet) replace attention entirely with FFT, achieving competitive accuracy with O(N log N) instead of O(N^2) complexity.
+Standard CNN implementations use direct convolution (faster for small $3 \times 3$ kernels). But for large kernels or global convolution, FFT-based approaches are significantly faster. Some architectures (like FNet) replace attention entirely with FFT, achieving competitive accuracy with $O(N \log N)$ instead of $O(N^2)$ complexity.
### Spectrograms and the Short-Time Fourier Transform
@@ -239,7 +244,7 @@ A single FFT gives you the frequency content of the entire signal, but tells you
The Short-Time Fourier Transform (STFT) solves this by computing FFTs on overlapping windows of the signal. The result is a spectrogram: a 2D representation with time on one axis and frequency on the other. The intensity at each point shows the energy at that frequency at that time.
-```
+```text
STFT procedure:
1. Choose a window size (e.g., 1024 samples)
2. Choose a hop size (e.g., 256 samples -- 75% overlap)
@@ -254,9 +259,9 @@ Spectrograms are the standard input representation for audio ML models. Speech r
### Aliasing
-If a signal contains frequencies above fs/2 (the Nyquist frequency), sampling at rate fs will create aliased copies. A 90 Hz signal sampled at 100 Hz looks identical to a 10 Hz signal. There is no way to distinguish them from the samples alone.
+If a signal contains frequencies above $f_s/2$ (the Nyquist frequency), sampling at rate $f_s$ will create aliased copies. A 90 Hz signal sampled at 100 Hz looks identical to a 10 Hz signal. There is no way to distinguish them from the samples alone.
-```
+```text
Example:
True signal: 90 Hz sine wave
Sampling rate: 100 Hz
@@ -273,7 +278,7 @@ This is why analog-to-digital converters include anti-aliasing filters that remo
A common misconception: zero-padding a signal before FFT improves frequency resolution. It does not. Zero-padding interpolates between existing frequency bins, giving you a smoother-looking spectrum. But it cannot reveal frequency detail that was not present in the original samples.
-True frequency resolution depends only on the observation time T = N / fs. To resolve two frequencies separated by delta_f, you need at least T = 1 / delta_f seconds of data. No amount of zero-padding changes this fundamental limit.
+True frequency resolution depends only on the observation time $T = N / f_s$. To resolve two frequencies separated by $\Delta f$, you need at least $T = 1 / \Delta f$ seconds of data. No amount of zero-padding changes this fundamental limit.
```figure
fourier-synthesis
@@ -283,7 +288,7 @@ fourier-synthesis
### Step 1: DFT from scratch
-The O(N^2) DFT follows directly from the definition.
+The $O(N^2)$ DFT follows directly from the definition.
```python
import math
@@ -426,33 +431,33 @@ Run `code/fourier.py` to generate `outputs/prompt-spectral-analyzer.md`.
1. **Pure tone identification.** Create a signal with a single sine wave at an unknown frequency (between 1 and 50 Hz), sampled at 128 Hz for 1 second. Use your DFT to identify the frequency. Verify the answer matches. Now add Gaussian noise with standard deviation 0.5 and repeat. How does noise affect the spectrum?
-2. **FFT vs DFT verification.** Generate a random signal of length 64. Compute both DFT (O(N^2)) and FFT. Verify that all coefficients match to within 1e-10. Time both functions on signals of length 256, 512, 1024, and 2048. Plot the ratio of DFT time to FFT time.
+2. **FFT vs DFT verification.** Generate a random signal of length 64. Compute both DFT ($O(N^2)$) and FFT. Verify that all coefficients match to within $10^{-10}$. Time both functions on signals of length 256, 512, 1024, and 2048. Plot the ratio of DFT time to FFT time.
3. **Convolution theorem proof by example.** Create signal x = [1, 2, 3, 4, 0, 0, 0, 0] and filter h = [1, 1, 1, 0, 0, 0, 0, 0]. Compute their circular convolution directly (nested loop). Then compute it via FFT (transform, multiply, inverse transform). Verify the results match. Now do linear convolution by zero-padding appropriately.
4. **Windowing effects.** Create a signal that is the sum of two sine waves at 10 Hz and 12 Hz (very close). Sample at 128 Hz for 1 second. Compute the power spectrum with no window, Hann window, and Hamming window. Which window makes it easiest to distinguish the two peaks? Why?
-5. **Positional encoding analysis.** Generate the sinusoidal positional encodings for d_model = 128 and max_pos = 512. For each pair of positions (p1, p2), compute the dot product of their encodings. Show that the dot product depends only on |p1 - p2|, not on the absolute positions. What happens to the dot product as the distance increases?
+5. **Positional encoding analysis.** Generate the sinusoidal positional encodings for $d_{\text{model}}$ = 128 and max_pos = 512. For each pair of positions $(p_1, p_2)$, compute the dot product of their encodings. Show that the dot product depends only on $|p_1 - p_2|$, not on the absolute positions. What happens to the dot product as the distance increases?
## Key Terms
| Term | What it means |
|------|---------------|
| DFT (Discrete Fourier Transform) | Converts N time-domain samples into N frequency-domain coefficients. Each coefficient is the correlation with a complex sinusoid at that frequency |
-| FFT (Fast Fourier Transform) | An O(N log N) algorithm to compute the DFT. The Cooley-Tukey algorithm splits even/odd indices recursively |
-| Inverse DFT | Reconstructs the time-domain signal from frequency coefficients. Same formula as DFT with flipped exponent sign and 1/N scaling |
-| Frequency bin | Each index k in the DFT output represents frequency k*fs/N Hz. The "bin" is the discrete frequency slot |
-| DC component | X[0], the zero-frequency coefficient. Proportional to the signal mean |
-| Nyquist frequency | fs/2, the maximum frequency representable at sampling rate fs. Frequencies above this alias |
-| Power spectrum | \|X[k]\|^2, the squared magnitude of each frequency coefficient. Shows energy distribution across frequencies |
-| Phase spectrum | angle(X[k]), the phase offset of each frequency component. Often ignored in analysis |
+| FFT (Fast Fourier Transform) | An $O(N \log N)$ algorithm to compute the DFT. The Cooley-Tukey algorithm splits even/odd indices recursively |
+| Inverse DFT | Reconstructs the time-domain signal from frequency coefficients. Same formula as DFT with flipped exponent sign and $1/N$ scaling |
+| Frequency bin | Each index $k$ in the DFT output represents frequency $k \cdot f_s / N$ Hz. The "bin" is the discrete frequency slot |
+| DC component | $X[0]$, the zero-frequency coefficient. Proportional to the signal mean |
+| Nyquist frequency | $f_s/2$, the maximum frequency representable at sampling rate $f_s$. Frequencies above this alias |
+| Power spectrum | $\vert X[k] \vert^2$, the squared magnitude of each frequency coefficient. Shows energy distribution across frequencies |
+| Phase spectrum | $\text{angle}(X[k])$, the phase offset of each frequency component. Often ignored in analysis |
| Spectral leakage | Spurious frequency content caused by treating a non-periodic signal as periodic. Reduced by windowing |
| Window function | A tapering function (Hann, Hamming, Blackman) applied before DFT to reduce spectral leakage |
-| Twiddle factor | The complex exponential e^(-2*pi*i*k/N) used to combine sub-DFTs in the FFT butterfly computation |
+| Twiddle factor | The complex exponential $e^{-2\pi i k / N}$ used to combine sub-DFTs in the FFT butterfly computation |
| Convolution theorem | Convolution in time domain equals pointwise multiplication in frequency domain. Fundamental to signal processing and CNNs |
| Circular convolution | Convolution where the signal wraps around. This is what the DFT naturally computes |
| Linear convolution | Standard convolution without wraparound. Achieved by zero-padding before DFT |
-| Parseval's theorem | Total energy is preserved through the Fourier transform. sum \|x[n]\|^2 = (1/N) sum \|X[k]\|^2 |
+| Parseval's theorem | Total energy is preserved through the Fourier transform. $\sum \vert x[n] \vert^2 = \frac{1}{N} \sum \vert X[k] \vert^2$ |
| Aliasing | When frequencies above Nyquist appear as lower frequencies due to insufficient sampling rate |
## Further Reading
diff --git a/phases/01-math-foundations/22-stochastic-processes/docs/en.md b/phases/01-math-foundations/22-stochastic-processes/docs/en.md
index 39f4507759..8eb1a44570 100644
--- a/phases/01-math-foundations/22-stochastic-processes/docs/en.md
+++ b/phases/01-math-foundations/22-stochastic-processes/docs/en.md
@@ -9,7 +9,7 @@
## Learning Objectives
-- Simulate 1D and 2D random walks and verify the sqrt(n) scaling of displacement
+- Simulate 1D and 2D random walks and verify the $\sqrt{n}$ scaling of displacement
- Build a Markov chain simulator and compute its stationary distribution via eigendecomposition
- Implement Metropolis-Hastings MCMC and Langevin dynamics for sampling from target distributions
- Connect the forward diffusion process to Brownian motion and explain how the reverse process generates data
@@ -38,9 +38,9 @@ All of these build on four foundational ideas:
Start at position 0. At each step, flip a fair coin. Heads: move right (+1). Tails: move left (-1).
-After n steps, your position is the sum of n random +/-1 values. The expected position is 0 (the walk is unbiased). But the expected distance from the origin grows as sqrt(n).
+After n steps, your position is the sum of n random $\pm 1$ values. The expected position is 0 (the walk is unbiased). But the expected distance from the origin grows as $\sqrt{n}$.
-This is counterintuitive. The walk is fair -- no drift in either direction. But over time, it wanders further and further from where it started. The standard deviation after n steps is sqrt(n).
+This is counterintuitive. The walk is fair -- no drift in either direction. But over time, it wanders further and further from where it started. The standard deviation after n steps is $\sqrt{n}$.
```
Step 0: Position = 0
@@ -51,31 +51,31 @@ Step 100: Expected distance from origin ~ 10 (sqrt(100))
Step 10000: Expected distance from origin ~ 100 (sqrt(10000))
```
-**In 2D**, the walk moves up, down, left, or right with equal probability. The same sqrt(n) scaling applies to the distance from the origin. The path traces a fractal-like pattern.
+**In 2D**, the walk moves up, down, left, or right with equal probability. The same $\sqrt{n}$ scaling applies to the distance from the origin. The path traces a fractal-like pattern.
-**Why sqrt(n)?** Each step is +1 or -1 with equal probability. After n steps, the position S_n = X_1 + X_2 + ... + X_n where each X_i is +/-1. The variance of each step is 1, and the steps are independent, so Var(S_n) = n. Standard deviation = sqrt(n). By the central limit theorem, S_n / sqrt(n) converges to a standard normal distribution.
+**Why $\sqrt{n}$?** Each step is $\pm 1$ with equal probability. After n steps, the position $S_n = X_1 + X_2 + \cdots + X_n$ where each $X_i$ is $\pm 1$. The variance of each step is 1, and the steps are independent, so $\text{Var}(S_n) = n$. Standard deviation $= \sqrt{n}$. By the central limit theorem, $S_n / \sqrt{n}$ converges to a standard normal distribution.
-This sqrt(n) scaling shows up everywhere in ML. SGD noise scales as 1/sqrt(batch_size). Embedding dimensions scale as sqrt(d). The square root is the signature of independent random additions.
+This $\sqrt{n}$ scaling shows up everywhere in ML. SGD noise scales as $1/\sqrt{\text{batch\_size}}$. Embedding dimensions scale as $\sqrt{d}$. The square root is the signature of independent random additions.
-**Connection to Brownian motion.** Take a random walk with step size 1/sqrt(n) and n steps per unit time. As n goes to infinity, the walk converges to Brownian motion B(t) -- a continuous-time process where B(t) is normally distributed with mean 0 and variance t.
+**Connection to Brownian motion.** Take a random walk with step size $1/\sqrt{n}$ and n steps per unit time. As n goes to infinity, the walk converges to Brownian motion $B(t)$ -- a continuous-time process where $B(t)$ is normally distributed with mean 0 and variance t.
Brownian motion is the mathematical foundation of diffusion. It models the random jiggling of particles in a fluid, the fluctuations of stock prices, and -- crucially -- the noise process in diffusion models.
-**Gambler's ruin.** A random walker starting at position k, with absorbing barriers at 0 and N. What is the probability of reaching N before 0? For a fair walk: P(reach N) = k/N. This is surprisingly simple and elegant. It connects to the theory of martingales -- the fair random walk is a martingale (expected future value = current value).
+**Gambler's ruin.** A random walker starting at position k, with absorbing barriers at 0 and N. What is the probability of reaching N before 0? For a fair walk: $P(\text{reach } N) = k/N$. This is surprisingly simple and elegant. It connects to the theory of martingales -- the fair random walk is a martingale (expected future value = current value).
### Markov Chains
A Markov chain is a system that transitions between states according to fixed probabilities. The key property: the next state depends only on the current state, not on the history.
-```
-P(X_{t+1} = j | X_t = i, X_{t-1} = ...) = P(X_{t+1} = j | X_t = i)
-```
+$$
+P(X_{t+1} = j \mid X_t = i, X_{t-1} = \ldots) = P(X_{t+1} = j \mid X_t = i)
+$$
This is the Markov property. It means you can describe the entire dynamics with a transition matrix P:
-```
-P[i][j] = probability of going from state i to state j
-```
+$$
+P[i][j] = \text{probability of going from state } i \text{ to state } j
+$$
Each row of P sums to 1 (you must go somewhere).
@@ -89,7 +89,7 @@ P = [[0.7, 0.1, 0.2], (if sunny: 70% sunny, 10% rainy, 20% cloudy)
[0.4, 0.2, 0.4]] (if cloudy: 40% sunny, 20% rainy, 40% cloudy)
```
-Start in any state. After many transitions, the distribution of states converges to the stationary distribution pi, where pi * P = pi. This is the left eigenvector of P with eigenvalue 1.
+Start in any state. After many transitions, the distribution of states converges to the stationary distribution $\pi$, where $\pi P = \pi$. This is the left eigenvector of P with eigenvalue 1.
For the weather chain, the stationary distribution might be [0.53, 0.18, 0.29] -- over the long run, it is sunny 53% of the time regardless of the starting state.
@@ -109,7 +109,7 @@ graph LR
**Computing the stationary distribution.** There are two approaches:
1. **Power method**: multiply any initial distribution by P repeatedly. After enough iterations, it converges.
-2. **Eigenvalue method**: find the left eigenvector of P with eigenvalue 1. This is the eigenvector of P^T with eigenvalue 1.
+2. **Eigenvalue method**: find the left eigenvector of P with eigenvalue 1. This is the eigenvector of $P^T$ with eigenvalue 1.
Both approaches require the chain to satisfy convergence conditions.
@@ -119,7 +119,7 @@ Both approaches require the chain to satisfy convergence conditions.
Most chains you encounter in ML satisfy both conditions.
-**Absorbing states.** A state is absorbing if once you enter it, you never leave (P[i][i] = 1). Absorbing Markov chains model processes with terminal states -- a game that ends, a customer who churns, a token sequence that hits the end-of-text token.
+**Absorbing states.** A state is absorbing if once you enter it, you never leave ($P[i][i] = 1$). Absorbing Markov chains model processes with terminal states -- a game that ends, a customer who churns, a token sequence that hits the end-of-text token.
**Mixing time.** How many steps until the chain is "close" to the stationary distribution? Formally, the number of steps until the total variation distance from stationarity drops below some threshold. Fast mixing = few steps needed. The spectral gap of P (1 minus the second-largest eigenvalue) controls the mixing time. Larger gap = faster mixing.
@@ -127,55 +127,55 @@ Most chains you encounter in ML satisfy both conditions.
Token generation in a language model is approximately a Markov process. Given the current context, the model outputs a distribution over the next token. Temperature controls the sharpness:
-```
-P(token_i) = exp(logit_i / temperature) / sum(exp(logit_j / temperature))
-```
+$$
+P(\text{token}_i) = \frac{\exp(\text{logit}_i / \text{temperature})}{\sum_j \exp(\text{logit}_j / \text{temperature})}
+$$
- Temperature = 1.0: standard distribution
- Temperature < 1.0: sharper (more deterministic)
- Temperature > 1.0: flatter (more random)
-- Temperature -> 0: argmax (greedy)
+- Temperature $\to 0$: argmax (greedy)
Top-k sampling truncates to the k highest-probability tokens. Top-p (nucleus) sampling truncates to the smallest set of tokens whose cumulative probability exceeds p. Both modify the Markov transition probabilities.
### Brownian Motion
-The continuous-time limit of the random walk. Position B(t) has three properties:
-1. B(0) = 0
-2. B(t) - B(s) is normally distributed with mean 0 and variance t - s (for t > s)
+The continuous-time limit of the random walk. Position $B(t)$ has three properties:
+1. $B(0) = 0$
+2. $B(t) - B(s)$ is normally distributed with mean 0 and variance $t - s$ (for $t > s$)
3. Increments on non-overlapping intervals are independent
Brownian motion is continuous but nowhere differentiable -- it jiggles at every scale. The path has fractal dimension 2 in the plane.
In discrete simulation, you approximate Brownian motion by:
-```
-B(t + dt) = B(t) + sqrt(dt) * z, where z ~ N(0, 1)
-```
+$$
+B(t + dt) = B(t) + \sqrt{dt} \cdot z, \quad \text{where } z \sim N(0, 1)
+$$
-The sqrt(dt) scaling is important. It comes from the central limit theorem applied to random walks.
+The $\sqrt{dt}$ scaling is important. It comes from the central limit theorem applied to random walks.
### Langevin Dynamics
-Gradient descent finds the minimum of a function. Langevin dynamics finds the probability distribution proportional to exp(-U(x)/T), where U is an energy function and T is temperature.
+Gradient descent finds the minimum of a function. Langevin dynamics finds the probability distribution proportional to $\exp(-U(x)/T)$, where U is an energy function and T is temperature.
-```
-x_{t+1} = x_t - dt * gradient(U(x_t)) + sqrt(2 * T * dt) * z_t
-```
+$$
+x_{t+1} = x_t - dt \cdot \nabla U(x_t) + \sqrt{2 \cdot T \cdot dt} \cdot z_t
+$$
Two forces act on the particle:
-1. **Gradient force** (-dt * gradient(U)): pushes toward low energy (like gradient descent)
-2. **Random force** (sqrt(2*T*dt) * z): pushes in random directions (exploration)
+1. **Gradient force** ($-dt \cdot \nabla U$): pushes toward low energy (like gradient descent)
+2. **Random force** ($\sqrt{2 T \, dt} \cdot z$): pushes in random directions (exploration)
-At temperature T = 0, this is pure gradient descent. At high temperature, it is nearly a random walk. At the right temperature, the particle explores the energy landscape and spends more time in low-energy regions.
+At temperature $T = 0$, this is pure gradient descent. Higher temperatures produce nearly random-walk behavior. The sweet spot lets the particle explore the energy landscape and spend more time in low-energy regions.
**Connection to diffusion models.** The forward process of a diffusion model is:
-```
-x_t = sqrt(alpha_t) * x_{t-1} + sqrt(1 - alpha_t) * noise
-```
+$$
+x_t = \sqrt{\alpha_t} \cdot x_{t-1} + \sqrt{1 - \alpha_t} \cdot \text{noise}
+$$
-This is a Markov chain that gradually mixes the data with noise. After enough steps, x_T is pure Gaussian noise.
+This is a Markov chain that gradually mixes the data with noise. After enough steps, $x_T$ is pure Gaussian noise.
The reverse process -- going from noise back to data -- is also a Markov chain, but its transition probabilities are learned by a neural network. The network learns to predict the noise that was added at each step, then subtracts it.
@@ -195,21 +195,21 @@ graph LR
### MCMC: Markov Chain Monte Carlo
-Sometimes you need to sample from a distribution p(x) that you can evaluate (up to a constant) but cannot sample from directly. Bayesian posteriors are the classic example -- you know the likelihood times the prior, but the normalizing constant is intractable.
+Sometimes you need to sample from a distribution $p(x)$ that you can evaluate (up to a constant) but cannot sample from directly. Bayesian posteriors are the classic example -- you know the likelihood times the prior, but the normalizing constant is intractable.
-**Metropolis-Hastings** constructs a Markov chain whose stationary distribution is p(x):
+**Metropolis-Hastings** constructs a Markov chain whose stationary distribution is $p(x)$:
1. Start at some position x
-2. Propose a new position x' from a proposal distribution Q(x'|x)
-3. Compute acceptance ratio: a = p(x') * Q(x|x') / (p(x) * Q(x'|x))
-4. Accept x' with probability min(1, a). Otherwise stay at x.
+2. Propose a new position $x'$ from a proposal distribution $Q(x' \mid x)$
+3. Compute acceptance ratio: $a = p(x') \cdot Q(x \mid x') / (p(x) \cdot Q(x' \mid x))$
+4. Accept $x'$ with probability $\min(1, a)$. Otherwise stay at x.
5. Repeat.
-If Q is symmetric (e.g., Q(x'|x) = Q(x|x') = N(x, sigma^2)), the ratio simplifies to a = p(x') / p(x). You only need the ratio of probabilities -- the normalizing constant cancels.
+If Q is symmetric (e.g., $Q(x' \mid x) = Q(x \mid x') = N(x, \sigma^2)$), the ratio simplifies to $a = p(x') / p(x)$. You only need the ratio of probabilities -- the normalizing constant cancels.
The chain is guaranteed to converge to p(x) under mild conditions. But convergence can be slow if the proposal is too small (random walk) or too large (high rejection). Tuning the proposal is the art of MCMC.
-**Why it works.** The acceptance ratio ensures detailed balance: the probability of being at x and moving to x' equals the probability of being at x' and moving to x. Detailed balance implies that p(x) is the stationary distribution of the chain. So after enough steps, the samples come from p(x).
+**Why it works.** The acceptance ratio ensures detailed balance: the probability of being at x and moving to $x'$ equals the probability of being at $x'$ and moving to x. Detailed balance implies that $p(x)$ is the stationary distribution of the chain. So after enough steps, the samples come from $p(x)$.
**Practical considerations:**
- **Burn-in**: discard the first N samples. The chain needs time to reach the stationary distribution from its starting point.
@@ -409,15 +409,15 @@ This lesson produces:
Diffusion models deserve special attention. DDPM (Ho et al., 2020) defines a forward Markov chain:
-```
-q(x_t | x_{t-1}) = N(x_t; sqrt(1-beta_t) * x_{t-1}, beta_t * I)
-```
+$$
+q(x_t \mid x_{t-1}) = N(x_t; \sqrt{1-\beta_t} \cdot x_{t-1}, \beta_t \cdot I)
+$$
-where beta_t is a noise schedule. After T steps, x_T is approximately N(0, I). The reverse process is parameterized by a neural network that predicts the noise:
+where $\beta_t$ is a noise schedule. After T steps, $x_T$ is approximately $N(0, I)$. The reverse process is parameterized by a neural network that predicts the noise:
-```
-p_theta(x_{t-1} | x_t) = N(x_{t-1}; mu_theta(x_t, t), sigma_t^2 * I)
-```
+$$
+p_\theta(x_{t-1} \mid x_t) = N(x_{t-1}; \mu_\theta(x_t, t), \sigma_t^2 \cdot I)
+$$
Every step of generation is a step in a learned Markov chain. Understanding Markov chains means understanding how and why diffusion models generate data.
@@ -427,13 +427,13 @@ The key insight across all these connections: stochastic processes are not just
## Exercises
-1. **Simulate 1000 random walks of 10000 steps.** Plot the distribution of final positions. Verify it is approximately Gaussian with mean 0 and standard deviation sqrt(10000) = 100.
+1. **Simulate 1000 random walks of 10000 steps.** Plot the distribution of final positions. Verify it is approximately Gaussian with mean 0 and standard deviation $\sqrt{10000} = 100$.
2. **Build a text generator using a Markov chain.** Train on a small corpus: for each word, count transitions to the next word. Build the transition matrix. Generate new sentences by sampling from the chain.
3. **Implement simulated annealing** using Metropolis-Hastings. Start at high temperature (accept almost everything) and gradually cool down (accept only improvements). Use it to find the minimum of a function with many local minima.
-4. **Compare Langevin dynamics at different temperatures.** Sample from a double-well potential U(x) = (x^2 - 1)^2. At low temperature, samples cluster in one well. At high temperature, they spread across both. Find the critical temperature where the chain mixes between wells.
+4. **Compare Langevin dynamics at different temperatures.** Sample from a double-well potential $U(x) = (x^2 - 1)^2$. At low temperature, samples cluster in one well. At high temperature, they spread across both. Find the critical temperature where the chain mixes between wells.
5. **Implement the forward diffusion process.** Start with a 1D signal (e.g., a sine wave). Add noise progressively over 100 steps with a linear noise schedule. Show how the signal degrades to pure noise. Then implement a simple denoiser that reverses the process (even a naive one that just subtracts the estimated noise).
@@ -443,9 +443,9 @@ The key insight across all these connections: stochastic processes are not just
|------|----------------|----------------------|
| Random walk | "Coin-flip movement" | A process where position changes by random increments at each step |
| Markov property | "Memoryless" | The future depends only on the present state, not on the history |
-| Transition matrix | "The probability table" | P[i][j] = probability of moving from state i to state j |
-| Stationary distribution | "The long-run average" | The distribution pi where pi*P = pi -- the chain's equilibrium |
-| Brownian motion | "Random jiggling" | The continuous-time limit of a random walk, B(t) ~ N(0, t) |
+| Transition matrix | "The probability table" | $P[i][j] =$ probability of moving from state i to state j |
+| Stationary distribution | "The long-run average" | The distribution $\pi$ where $\pi P = \pi$ -- the chain's equilibrium |
+| Brownian motion | "Random jiggling" | The continuous-time limit of a random walk, $B(t) \sim N(0, t)$ |
| Langevin dynamics | "Gradient descent with noise" | Update rule that combines deterministic gradient and random perturbation |
| MCMC | "Walking toward the target" | Constructing a Markov chain whose stationary distribution is the one you want |
| Metropolis-Hastings | "Propose and accept/reject" | MCMC algorithm that uses acceptance ratios to ensure convergence |
diff --git a/phases/02-ml-fundamentals/04-decision-trees/docs/en.md b/phases/02-ml-fundamentals/04-decision-trees/docs/en.md
index f27f7184c1..41c53c43f5 100644
--- a/phases/02-ml-fundamentals/04-decision-trees/docs/en.md
+++ b/phases/02-ml-fundamentals/04-decision-trees/docs/en.md
@@ -48,44 +48,46 @@ At each node, we have a set of samples. We want to split them so that the result
**Gini impurity** measures the probability that a randomly chosen sample would be misclassified if it were labeled according to the class distribution at that node.
-```
-Gini(S) = 1 - sum(p_k^2)
+$$
+\text{Gini}(S) = 1 - \sum_k p_k^2
+$$
-where p_k is the proportion of class k in set S.
-```
+where $p_k$ is the proportion of class $k$ in set $S$.
-For a pure node (all one class), Gini = 0. For a binary split with 50/50 classes, Gini = 0.5. Lower is better.
+For a pure node (all one class), $\text{Gini} = 0$. For a binary split with 50/50 classes, $\text{Gini} = 0.5$. Lower is better.
-```
Example: 6 cats, 4 dogs
-Gini = 1 - (0.6^2 + 0.4^2) = 1 - (0.36 + 0.16) = 0.48
-```
+$$
+\text{Gini} = 1 - (0.6^2 + 0.4^2) = 1 - (0.36 + 0.16) = 0.48
+$$
**Entropy** measures the information content (disorder) in a node. Covered in Phase 1 Lesson 09.
-```
-Entropy(S) = -sum(p_k * log2(p_k))
-```
+$$
+\text{Entropy}(S) = -\sum_k p_k \log_2(p_k)
+$$
-For a pure node, entropy = 0. For a 50/50 binary split, entropy = 1.0. Lower is better.
+For a pure node, $\text{entropy} = 0$. For a 50/50 binary split, $\text{entropy} = 1.0$. Lower is better.
-```
Example: 6 cats, 4 dogs
-Entropy = -(0.6 * log2(0.6) + 0.4 * log2(0.4))
- = -(0.6 * -0.737 + 0.4 * -1.322)
- = 0.442 + 0.529
- = 0.971 bits
-```
+$$
+\begin{aligned}
+\text{Entropy} &= -(0.6 \cdot \log_2(0.6) + 0.4 \cdot \log_2(0.4)) \\
+&= -(0.6 \cdot -0.737 + 0.4 \cdot -1.322) \\
+&= 0.442 + 0.529 \\
+&= 0.971 \text{ bits}
+\end{aligned}
+$$
**Information gain** is the reduction in impurity (entropy or Gini) after a split.
-```
-IG(S, feature, threshold) = Impurity(S) - weighted_avg(Impurity(S_left), Impurity(S_right))
+$$
+\text{IG}(S, \text{feature}, \text{threshold}) = \text{Impurity}(S) - \text{weighted\_avg}(\text{Impurity}(S_{\text{left}}), \text{Impurity}(S_{\text{right}}))
+$$
where the weights are the proportions of samples in each child.
-```
The greedy algorithm at each node: try every feature and every possible threshold. Pick the (feature, threshold) pair that maximizes information gain.
@@ -98,7 +100,7 @@ For a dataset with n features and m samples at the current node:
- Try every midpoint between consecutive distinct values as a threshold
- Compute the information gain for each threshold
2. Select the feature and threshold with the highest information gain
-3. Split the data into left (feature <= threshold) and right (feature > threshold)
+3. Split the data into left ($\text{feature} \leq \text{threshold}$) and right ($\text{feature} > \text{threshold}$)
4. Recurse on each child
This greedy approach does not guarantee the globally optimal tree. Finding the optimal tree is NP-hard. But greedy splitting works well in practice.
@@ -125,9 +127,9 @@ For regression, the leaf prediction is the mean of the target values in that lea
**Variance reduction** replaces information gain:
-```
-VR(S, feature, threshold) = Var(S) - weighted_avg(Var(S_left), Var(S_right))
-```
+$$
+\text{VR}(S, \text{feature}, \text{threshold}) = \text{Var}(S) - \text{weighted\_avg}(\text{Var}(S_{\text{left}}), \text{Var}(S_{\text{right}}))
+$$
Pick the split that reduces variance the most. The tree partitions the input space into regions, and predicts a constant (the mean) in each region.
@@ -155,7 +157,7 @@ Two sources of randomness make the trees diverse:
**Bagging (bootstrap aggregating):** Each tree is trained on a bootstrap sample, a random sample with replacement from the training data. About 63% of the original samples appear in each bootstrap (the rest are out-of-bag samples that can be used for validation).
-**Feature randomization:** At each split, only a random subset of features is considered. For classification, the default is sqrt(n_features). For regression, n_features/3. This prevents all trees from splitting on the same dominant feature.
+**Feature randomization:** At each split, only a random subset of features is considered. For classification, the default is $\sqrt{n_{\text{features}}}$. For regression, $n_{\text{features}}/3$. This prevents all trees from splitting on the same dominant feature.
The key insight: averaging many decorrelated trees reduces variance without increasing bias. Each individual tree may be mediocre. The ensemble is strong.
@@ -165,10 +167,9 @@ Random forests naturally provide feature importance scores. The most common meth
**Mean Decrease in Impurity (MDI):** For each feature, sum the total reduction in impurity across all trees and all nodes where that feature is used. Features that produce bigger impurity reductions at earlier splits are more important.
-```
-importance(feature_j) = sum over all nodes where feature_j is used:
- (n_samples_at_node / n_total_samples) * impurity_decrease
-```
+$$
+\text{importance}(\text{feature}_j) = \sum_{\text{nodes using feature}_j} \frac{n_{\text{samples at node}}}{n_{\text{total samples}}} \cdot \text{impurity\_decrease}
+$$
This is fast (computed during training) but biased toward high-cardinality features and features with many possible split points.
@@ -348,7 +349,7 @@ This lesson produces `outputs/prompt-tree-interpreter.md` -- a prompt that inter
1. Train a single decision tree on a 2D dataset with 3 classes. Manually trace the splits and draw the rectangular decision boundaries. Compare the boundaries at max_depth=2 vs max_depth=10.
-2. Implement variance reduction splitting for regression trees. Generate y = sin(x) + noise for 200 points and fit your regression tree. Plot the tree's piecewise-constant predictions against the true curve.
+2. Implement variance reduction splitting for regression trees. Generate $y = \sin(x) + \text{noise}$ for 200 points and fit your regression tree. Plot the tree's piecewise-constant predictions against the true curve.
3. Build a random forest with 1, 5, 10, 50, and 200 trees. Plot training accuracy and test accuracy vs number of trees. Observe that test accuracy plateaus but does not decrease (forests resist overfitting).
diff --git a/phases/02-ml-fundamentals/06-knn-and-distances/docs/en.md b/phases/02-ml-fundamentals/06-knn-and-distances/docs/en.md
index 7ecd324e9a..a520acdbfe 100644
--- a/phases/02-ml-fundamentals/06-knn-and-distances/docs/en.md
+++ b/phases/02-ml-fundamentals/06-knn-and-distances/docs/en.md
@@ -61,7 +61,7 @@ K is the single hyperparameter. It controls the bias-variance trade-off:
| Large K | Smoother boundaries. More robust to noise. May underfit |
| K = N | Predicts the majority class for every point. Maximum bias |
-A common starting point is K = sqrt(N) for a dataset of N points. Use odd K for binary classification to avoid ties.
+A common starting point is $K = \sqrt{N}$ for a dataset of $N$ points. Use odd K for binary classification to avoid ties.
```mermaid
graph LR
@@ -83,33 +83,31 @@ The distance function defines what "near" means. Different metrics produce diffe
**L2 (Euclidean)** is the default. Straight-line distance.
-```
-d(a, b) = sqrt(sum((a_i - b_i)^2))
-```
+$$
+d(a, b) = \sqrt{\sum_i (a_i - b_i)^2}
+$$
Sensitive to feature scale. Always standardize features before using L2 with KNN.
**L1 (Manhattan)** sums absolute differences. More robust to outliers than L2 because it does not square the differences.
-```
-d(a, b) = sum(|a_i - b_i|)
-```
+$$
+d(a, b) = \sum_i |a_i - b_i|
+$$
**Cosine distance** measures the angle between vectors, ignoring magnitude. Essential for text and embedding data.
-```
-d(a, b) = 1 - (a . b) / (||a|| * ||b||)
-```
+$$
+d(a, b) = 1 - \frac{a \cdot b}{\|a\| \, \|b\|}
+$$
-**Minkowski** generalizes L1 and L2 with parameter p.
+**Minkowski** generalizes L1 and L2 with parameter $p$.
-```
-d(a, b) = (sum(|a_i - b_i|^p))^(1/p)
+$$
+d(a, b) = \left( \sum_i |a_i - b_i|^p \right)^{1/p}
+$$
-p=1: Manhattan
-p=2: Euclidean
-p->inf: Chebyshev (max absolute difference)
-```
+$p=1$: Manhattan. $p=2$: Euclidean. $p \to \infty$: Chebyshev (max absolute difference).
Which metric to use depends on the data:
@@ -127,14 +125,13 @@ Standard KNN gives equal weight to all K neighbors. But a neighbor at distance 0
**Distance-weighted KNN** weights each neighbor inversely by distance:
-```
-weight_i = 1 / (distance_i + epsilon)
+$$
+\text{weight}_i = \frac{1}{\text{distance}_i + \epsilon}
+$$
-For classification: weighted vote
-For regression: weighted average = sum(w_i * y_i) / sum(w_i)
-```
+For classification: weighted vote. For regression: weighted average $= \dfrac{\sum_i w_i \, y_i}{\sum_i w_i}$.
-The epsilon prevents division by zero when a query point exactly matches a training point.
+The $\epsilon$ prevents division by zero when a query point exactly matches a training point.
Weighted KNN is less sensitive to the choice of K because distant neighbors contribute very little regardless.
@@ -162,7 +159,7 @@ Practical consequence: KNN works well up to about 20-50 features. Beyond that, y
### KD-trees: fast nearest neighbor search
-Brute-force KNN computes the distance from the query to every training point. That is O(n * d) per query. For large datasets, this is too slow.
+Brute-force KNN computes the distance from the query to every training point. That is $O(n \cdot d)$ per query. For large datasets, this is too slow.
A KD-tree recursively partitions the space along feature axes. At each level, it splits along one dimension at the median value.
@@ -178,7 +175,7 @@ graph TD
To find the nearest neighbor, traverse the tree to the leaf containing the query, then backtrack and check neighboring partitions only if they could contain closer points.
-Average query time: O(log n) for low dimensions. But KD-trees degrade to O(n) in high dimensions (d > 20) because the backtracking eliminates fewer and fewer branches.
+Average query time: $O(\log n)$ for low dimensions. But KD-trees degrade to $O(n)$ in high dimensions ($d > 20$) because the backtracking eliminates fewer and fewer branches.
### Ball trees: better for moderate dimensions
@@ -197,8 +194,8 @@ KNN is a lazy learner: it does no work at training time and all work at predicti
| Aspect | Lazy (KNN) | Eager (SVM, neural net) |
|--------|------------|------------------------|
-| Training time | O(1) just store data | O(n * epochs) |
-| Prediction time | O(n * d) per query | O(d) or O(parameters) |
+| Training time | $O(1)$ just store data | $O(n \cdot \text{epochs})$ |
+| Prediction time | $O(n \cdot d)$ per query | $O(d)$ or $O(\text{parameters})$ |
| Memory at prediction | Store entire training set | Store model parameters only |
| Adapts to new data | Add points instantly | Retrain the model |
| Decision boundary | Implicit, computed on the fly | Explicit, fixed after training |
@@ -213,13 +210,15 @@ Lazy learning is ideal when:
Instead of majority voting, KNN for regression averages the target values of the K neighbors.
-```
-prediction = (1/K) * sum(y_i for i in K nearest neighbors)
+$$
+\text{prediction} = \frac{1}{K} \sum_{i \in \text{K nearest neighbors}} y_i
+$$
Or with distance weighting:
-prediction = sum(w_i * y_i) / sum(w_i)
-where w_i = 1 / distance_i
-```
+
+$$
+\text{prediction} = \frac{\sum_i w_i \, y_i}{\sum_i w_i}, \quad \text{where } w_i = \frac{1}{\text{distance}_i}
+$$
KNN regression produces piecewise-constant (or piecewise-smooth with weighting) predictions. It cannot extrapolate beyond the range of the training data. If the training targets are all between 0 and 100, KNN will never predict 200.
@@ -363,12 +362,12 @@ distances, indices = index.search(query_vectors, k=5)
| Lazy learning | No computation at training time. All work happens at prediction time. KNN is the canonical example |
| Eager learning | Heavy computation at training time to build a compact model. Most ML algorithms are eager |
| Curse of dimensionality | In high dimensions, distances converge and neighborhoods expand to cover most of the space, making KNN ineffective |
-| KD-tree | Binary tree that recursively partitions space along feature axes. O(log n) queries in low dimensions |
+| KD-tree | Binary tree that recursively partitions space along feature axes. $O(\log n)$ queries in low dimensions |
| Ball tree | Tree of nested hyperspheres. Works better than KD-trees in moderate dimensions (up to ~50) |
| Weighted KNN | Neighbors weighted inversely by distance. Closer neighbors have more influence on the prediction |
| Feature scaling | Normalizing features to comparable ranges. Required for distance-based methods like KNN |
| Majority vote | Classification by counting which class is most common among K neighbors |
-| Brute force search | Computing distance to every training point. O(n*d) per query. Exact but slow for large n |
+| Brute force search | Computing distance to every training point. $O(n \cdot d)$ per query. Exact but slow for large n |
| Approximate nearest neighbor | Algorithms (HNSW, LSH, IVF) that find approximately nearest points much faster than exact search |
| Voronoi diagram | The partition of space where each region contains all points closer to one training point than any other. K=1 KNN produces Voronoi boundaries |
diff --git a/phases/02-ml-fundamentals/10-bias-variance/docs/en.md b/phases/02-ml-fundamentals/10-bias-variance/docs/en.md
index 8836068ef5..101c5202a1 100644
--- a/phases/02-ml-fundamentals/10-bias-variance/docs/en.md
+++ b/phases/02-ml-fundamentals/10-bias-variance/docs/en.md
@@ -54,23 +54,23 @@ High variance (overfitting):
### The Decomposition
-For any point x, the expected prediction error under squared loss decomposes exactly:
+For any point $x$, the expected prediction error under squared loss decomposes exactly:
-```
-Expected Error = Bias^2 + Variance + Irreducible Noise
-
-where:
- Bias^2 = (E[f_hat(x)] - f(x))^2
- Variance = E[(f_hat(x) - E[f_hat(x)])^2]
- Noise = E[(y - f(x))^2] (sigma^2)
-```
+$$
+\begin{aligned}
+\text{Expected Error} &= \text{Bias}^2 + \text{Variance} + \text{Irreducible Noise} \\
+\text{Bias}^2 &= (E[\hat{f}(x)] - f(x))^2 \\
+\text{Variance} &= E[(\hat{f}(x) - E[\hat{f}(x)])^2] \\
+\text{Noise} &= E[(y - f(x))^2] = \sigma^2
+\end{aligned}
+$$
-- `f(x)` is the true function
-- `f_hat(x)` is your model's prediction
-- `E[...]` is the expectation over different training sets
-- `y` is the observed label (true function plus noise)
+- $f(x)$ is the true function
+- $\hat{f}(x)$ is your model's prediction
+- $E[\ldots]$ is the expectation over different training sets
+- $y$ is the observed label (true function plus noise)
-The noise term is irreducible. No model can do better than sigma^2 on noisy data. Your job is to find the right balance between bias^2 and variance.
+The noise term is irreducible. No model can do better than $\sigma^2$ on noisy data. Your job is to find the right balance between $\text{bias}^2$ and variance.
### Model Complexity vs Error
@@ -101,7 +101,7 @@ Regularization deliberately increases bias to reduce variance. It constrains the
- **Dropout:** Randomly disables neurons during training. Forces redundant representations.
- **Early stopping:** Stops training before the model fully fits the training data.
-The regularization strength (lambda, dropout rate, number of epochs) directly controls where you sit on the bias-variance curve. More regularization means more bias, less variance.
+The regularization strength ($\lambda$, dropout rate, number of epochs) directly controls where you sit on the bias-variance curve. More regularization means more bias, less variance.
### Double Descent: The Modern Perspective
@@ -131,9 +131,9 @@ Why does this happen? At the interpolation threshold, the model has just enough
| Regime | Parameters vs Samples | Behavior |
|--------|----------------------|----------|
-| Underparameterized | p << n | Classical tradeoff applies |
-| Interpolation threshold | p ~ n | Variance peaks, test error spikes |
-| Overparameterized | p >> n | Implicit regularization kicks in, test error drops |
+| Underparameterized | $p \ll n$ | Classical tradeoff applies |
+| Interpolation threshold | $p \sim n$ | Variance peaks, test error spikes |
+| Overparameterized | $p \gg n$ | Implicit regularization kicks in, test error drops |
For practical purposes: if you are using neural networks or large tree ensembles, do not stop at the interpolation threshold. Either stay well below it (with explicit regularization) or go well past it. The worst place to be is right at the threshold.
@@ -170,7 +170,7 @@ flowchart TD
**When variance is the problem:**
- Get more training data
- Use bagging (random forests)
-- Increase regularization (higher lambda, more dropout)
+- Increase regularization (higher $\lambda$, more dropout)
- Feature selection (remove noisy features)
- Use cross-validation to detect it early
@@ -180,7 +180,7 @@ Ensemble methods are the most practical tool for fighting variance.
**Bagging (Bootstrap Aggregating)** trains multiple models on different bootstrap samples of the training data, then averages their predictions. Each individual model has high variance, but the average has much lower variance. Random forests are bagging applied to decision trees.
-Why it works mathematically: if you average N independent predictions, each with variance sigma^2, the variance of the average is sigma^2 / N. The models are not truly independent (they all see similar data), so the reduction is less than 1/N, but it is still substantial.
+Why it works mathematically: if you average $N$ independent predictions, each with variance $\sigma^2$, the variance of the average is $\sigma^2 / N$. The models are not truly independent (they all see similar data), so the reduction is less than $1/N$, but it is still substantial.
**Boosting** reduces bias by building models sequentially, where each new model focuses on the errors of the ensemble so far. Gradient boosting and AdaBoost are the main examples. Boosting can overfit if you add too many models, so you need early stopping or regularization.
@@ -264,7 +264,7 @@ The code in `code/bias_variance.py` runs the full bias-variance decomposition ex
### Step 1: Generate Synthetic Data from a Known Function
-We use `f(x) = sin(1.5x) + 0.5x` with Gaussian noise. Knowing the true function lets us compute exact bias and variance.
+We use $f(x) = \sin(1.5x) + 0.5x$ with Gaussian noise. Knowing the true function lets us compute exact bias and variance.
```python
def true_function(x):
@@ -306,10 +306,10 @@ variance = np.mean(predictions.var(axis=0))
total_error = np.mean(np.mean((predictions - y_true) ** 2, axis=1))
```
-- `mean_pred` is E[f_hat(x)] estimated from bootstrap samples
+- `mean_pred` is $E[\hat{f}(x)]$ estimated from bootstrap samples
- `bias_sq` is the squared gap between average prediction and truth
- `variance` is the average spread of predictions across bootstrap samples
-- `total_error` should approximately equal bias^2 + variance + noise
+- `total_error` should approximately equal $\text{bias}^2 + \text{variance} + \text{noise}$
### Step 4: Learning Curves
@@ -440,9 +440,9 @@ This lesson produces: `outputs/prompt-model-diagnostics.md`
2. Increase the training set size from 30 to 300. How does this affect the variance component? Does the optimal polynomial degree shift?
-3. Add L2 regularization (Ridge regression) to the experiment. For a fixed high-degree polynomial (degree 15), sweep lambda from 0 to 100. Plot bias^2 and variance as functions of lambda.
+3. Add L2 regularization (Ridge regression) to the experiment. For a fixed high-degree polynomial (degree 15), sweep $\lambda$ from 0 to 100. Plot $\text{bias}^2$ and variance as functions of $\lambda$.
-4. Modify the true function from a polynomial to `sin(x)`. How does the bias-variance decomposition change? Is there still a clear optimal degree?
+4. Modify the true function from a polynomial to $\sin(x)$. How does the bias-variance decomposition change? Is there still a clear optimal degree?
5. Implement a simple bootstrap aggregating (bagging) wrapper: train 10 models on bootstrap samples and average predictions. Show that this reduces variance without increasing bias much.
diff --git a/phases/02-ml-fundamentals/18-feature-selection/docs/en.md b/phases/02-ml-fundamentals/18-feature-selection/docs/en.md
index d4c75d5817..b7f5241023 100644
--- a/phases/02-ml-fundamentals/18-feature-selection/docs/en.md
+++ b/phases/02-ml-fundamentals/18-feature-selection/docs/en.md
@@ -62,9 +62,9 @@ The simplest filter. If a feature barely varies across samples, it carries almos
Consider a feature that is 0.0 for 999 out of 1000 samples. Its variance is near zero. No model can use it to distinguish between classes. Remove it.
-```
-variance(x) = mean((x - mean(x))^2)
-```
+$$
+\text{variance}(x) = \text{mean}((x - \text{mean}(x))^2)
+$$
Set a threshold (e.g., 0.01). Drop every feature with variance below it. This removes constant or near-constant features without looking at the target variable at all.
@@ -76,15 +76,15 @@ Limitation: a feature can have high variance and still be pure noise. Variance t
Mutual information measures how much knowing the value of feature X reduces uncertainty about target Y.
-```
-I(X; Y) = sum_x sum_y p(x, y) * log(p(x, y) / (p(x) * p(y)))
-```
+$$
+I(X; Y) = \sum_x \sum_y p(x, y) \cdot \log\left(\frac{p(x, y)}{p(x) \cdot p(y)}\right)
+$$
-If X and Y are independent, p(x, y) = p(x) * p(y), so the log term is zero and I(X; Y) = 0. The more X tells you about Y, the higher the mutual information.
+If X and Y are independent, $p(x, y) = p(x) \cdot p(y)$, so the log term is zero and $I(X; Y) = 0$. The more X tells you about Y, the higher the mutual information.
Key advantage over correlation: mutual information captures nonlinear relationships. A feature might have zero correlation with the target but high mutual information because the relationship is quadratic or periodic.
-For continuous features, discretize into bins first (histogram-based estimation). The number of bins affects the estimate -- too few bins lose information, too many bins add noise. A common choice: sqrt(n) bins or Sturges' rule (1 + log2(n)).
+For continuous features, discretize into bins first (histogram-based estimation). The number of bins affects the estimate -- too few bins lose information, too many bins add noise. A common choice: $\sqrt{n}$ bins or Sturges' rule ($1 + \log_2(n)$).
```mermaid
flowchart LR
@@ -122,9 +122,9 @@ The cost: you train the model N - target times. With 500 features and a target o
L1 regularization adds the absolute value of weights to the loss function:
-```
-loss = prediction_error + alpha * sum(|w_i|)
-```
+$$
+\text{loss} = \text{prediction\_error} + \alpha \cdot \sum |w_i|
+$$
The alpha parameter controls how aggressively features are pruned. Higher alpha means more weights go to exactly zero.
@@ -142,11 +142,9 @@ Decision trees and their ensembles (random forests, gradient boosting) naturally
For a random forest with T trees:
-```
-importance(feature_j) = (1/T) * sum over all trees of
- sum over all nodes splitting on feature_j of
- (n_samples * impurity_decrease)
-```
+$$
+\text{importance}(\text{feature}_j) = \frac{1}{T} \sum_{\text{trees}} \; \sum_{\substack{\text{nodes splitting} \\ \text{on feature}_j}} (n_\text{samples} \cdot \text{impurity\_decrease})
+$$
This gives a normalized importance score for each feature. It handles nonlinear relationships and feature interactions automatically.
diff --git a/phases/03-deep-learning-core/01-the-perceptron/docs/en.md b/phases/03-deep-learning-core/01-the-perceptron/docs/en.md
index 9f9e5ab436..02a7c5a6e2 100644
--- a/phases/03-deep-learning-core/01-the-perceptron/docs/en.md
+++ b/phases/03-deep-learning-core/01-the-perceptron/docs/en.md
@@ -40,10 +40,9 @@ graph LR
The step function is brutal: if the weighted sum plus bias is >= 0, output 1. Otherwise, output 0.
-```
-step(z) = 1 if z >= 0
- 0 if z < 0
-```
+$$
+\text{step}(z) = \begin{cases} 1 & \text{if } z \geq 0 \\ 0 & \text{if } z < 0 \end{cases}
+$$
This is a linear classifier. The weights and bias define a line (or hyperplane in higher dimensions) that splits the input space into two regions.
@@ -359,7 +358,7 @@ This lesson produces:
## Exercises
1. Train a perceptron on a NAND gate (the universal gate - any logic circuit can be built from NAND). Verify its weights and bias form a valid decision boundary.
-2. Modify the Perceptron class to track the decision boundary (w1*x1 + w2*x2 + b = 0) at each epoch. Print how the line shifts during training on the AND gate.
+2. Modify the Perceptron class to track the decision boundary ($w_1 x_1 + w_2 x_2 + b = 0$) at each epoch. Print how the line shifts during training on the AND gate.
3. Build a 3-input perceptron that outputs 1 only when at least 2 of the 3 inputs are 1 (a majority vote function). Is this linearly separable? Why?
## Key Terms
@@ -372,7 +371,7 @@ This lesson produces:
| Activation function | "The thing that squishes values" | A function applied after the weighted sum - step function for perceptrons, sigmoid/ReLU for modern networks |
| Linearly separable | "You can draw a line between them" | A dataset where a single hyperplane can perfectly separate the classes |
| XOR problem | "The thing perceptrons can't do" | Proof that single-layer networks cannot learn non-linearly-separable functions |
-| Decision boundary | "Where the classifier switches" | The hyperplane w*x + b = 0 that divides input space into two classes |
+| Decision boundary | "Where the classifier switches" | The hyperplane $w \cdot x + b = 0$ that divides input space into two classes |
| Multi-layer perceptron | "A real neural network" | Perceptrons stacked in layers, where each layer's output feeds the next layer's input |
## Further Reading
diff --git a/phases/03-deep-learning-core/06-optimizers/docs/en.md b/phases/03-deep-learning-core/06-optimizers/docs/en.md
index b8896c2e3d..32b1ca1be4 100644
--- a/phases/03-deep-learning-core/06-optimizers/docs/en.md
+++ b/phases/03-deep-learning-core/06-optimizers/docs/en.md
@@ -18,7 +18,7 @@
You computed the gradients. You know that weight #4,721 should decrease by 0.003 to reduce the loss. But 0.003 in what units? Scaled by what? And should you move the same amount on step 1 as on step 1,000?
-Vanilla gradient descent applies the same learning rate to every parameter on every step: w = w - lr * gradient. This creates three problems that make training neural networks painful in practice.
+Vanilla gradient descent applies the same learning rate to every parameter on every step: $w = w - \text{lr} \cdot \text{gradient}$. This creates three problems that make training neural networks painful in practice.
First, oscillation. The loss landscape is rarely shaped like a smooth bowl. It's more like a long, narrow valley. The gradient points across the valley (steep direction), not along it (shallow direction). Gradient descent bounces back and forth across the narrow dimension while making tiny progress along the useful one. You've seen this: loss drops fast then plateaus, not because the model converged but because it's oscillating.
@@ -34,9 +34,9 @@ Adam solves all three. It maintains two running averages per parameter -- the me
The simplest optimizer. Compute the gradient on a mini-batch and step in the opposite direction.
-```
-w = w - lr * gradient
-```
+$$
+w = w - \text{lr} \cdot \text{gradient}
+$$
The "stochastic" means you use a random subset (mini-batch) of data to estimate the gradient, rather than the full dataset. This noise is actually useful -- it helps escape sharp local minima. But the noise also causes oscillation.
@@ -46,12 +46,14 @@ Learning rate is the only knob. Too high: the loss diverges. Too low: training t
The ball-rolling-downhill analogy is overused but accurate. Instead of stepping by the gradient alone, you maintain a velocity that accumulates past gradients.
-```
-m_t = beta * m_{t-1} + gradient
-w = w - lr * m_t
-```
+$$
+\begin{aligned}
+m_t &= \beta \cdot m_{t-1} + \text{gradient} \\
+w &= w - \text{lr} \cdot m_t
+\end{aligned}
+$$
-Beta (typically 0.9) controls how much history to keep. With beta = 0.9, the momentum is roughly the average of the last 10 gradients (1 / (1 - 0.9) = 10).
+Beta (typically 0.9) controls how much history to keep. With $\beta = 0.9$, the momentum is roughly the average of the last 10 gradients ($1 / (1 - 0.9) = 10$).
Why this fixes oscillation: gradients that point in the same direction accumulate. Gradients that flip direction cancel out. In that narrow valley, the "across" component flips sign each step and gets dampened. The "along" component stays consistent and gets amplified. The result is smooth acceleration in the useful direction.
@@ -61,56 +63,62 @@ Real numbers: SGD alone on a badly conditioned loss landscape might take 10,000
The first per-parameter adaptive learning rate method that actually worked. Proposed by Hinton in a Coursera lecture (never formally published).
-```
-s_t = beta * s_{t-1} + (1 - beta) * gradient^2
-w = w - lr * gradient / (sqrt(s_t) + epsilon)
-```
+$$
+\begin{aligned}
+s_t &= \beta \cdot s_{t-1} + (1 - \beta) \cdot \text{gradient}^2 \\
+w &= w - \text{lr} \cdot \frac{\text{gradient}}{\sqrt{s_t} + \epsilon}
+\end{aligned}
+$$
-s_t tracks the running average of squared gradients. Parameters with consistently large gradients get divided by a large number (smaller effective learning rate). Parameters with small gradients get divided by a small number (larger effective learning rate).
+$s_t$ tracks the running average of squared gradients. Parameters with consistently large gradients get divided by a large number (smaller effective learning rate). Parameters with small gradients get divided by a small number (larger effective learning rate).
This solves the "one learning rate for all parameters" problem. A weight that's already been getting large updates is probably near its target -- slow it down. A weight that's been getting tiny updates might be undertrained -- speed it up.
-Epsilon (typically 1e-8) prevents division by zero when a parameter hasn't been updated.
+Epsilon (typically $1\text{e-}8$) prevents division by zero when a parameter hasn't been updated.
### Adam: Momentum + RMSProp
Adam combines both ideas. It maintains two exponential moving averages per parameter:
-```
-m_t = beta1 * m_{t-1} + (1 - beta1) * gradient (first moment: mean)
-v_t = beta2 * v_{t-1} + (1 - beta2) * gradient^2 (second moment: variance)
-```
+$$
+\begin{aligned}
+m_t &= \beta_1 \cdot m_{t-1} + (1 - \beta_1) \cdot \text{gradient} \quad &&\text{(first moment: mean)} \\
+v_t &= \beta_2 \cdot v_{t-1} + (1 - \beta_2) \cdot \text{gradient}^2 \quad &&\text{(second moment: variance)}
+\end{aligned}
+$$
-**Bias correction** is the key detail most explanations skip. At step 1, m_1 = (1 - beta1) * gradient. With beta1 = 0.9, that's 0.1 * gradient -- ten times too small. The moving average hasn't warmed up yet. Bias correction compensates:
+**Bias correction** is the key detail most explanations skip. At step 1, $m_1 = (1 - \beta_1) \cdot \text{gradient}$. With $\beta_1 = 0.9$, that's $0.1 \cdot \text{gradient}$ -- ten times too small. The moving average hasn't warmed up yet. Bias correction compensates:
-```
-m_hat = m_t / (1 - beta1^t)
-v_hat = v_t / (1 - beta2^t)
-```
+$$
+\begin{aligned}
+\hat{m} &= \frac{m_t}{1 - \beta_1^t} \\
+\hat{v} &= \frac{v_t}{1 - \beta_2^t}
+\end{aligned}
+$$
-At step 1 with beta1 = 0.9: m_hat = m_1 / (1 - 0.9) = m_1 / 0.1 = the actual gradient. At step 100: (1 - 0.9^100) is approximately 1.0, so the correction vanishes. Bias correction matters for the first ~10 steps and is irrelevant after ~50.
+At step 1 with $\beta_1 = 0.9$: $\hat{m} = m_1 / (1 - 0.9) = m_1 / 0.1 =$ the actual gradient. At step 100: $(1 - 0.9^{100})$ is approximately 1.0, so the correction vanishes. Bias correction matters for the first ~10 steps and is irrelevant after ~50.
The update:
-```
-w = w - lr * m_hat / (sqrt(v_hat) + epsilon)
-```
+$$
+w = w - \text{lr} \cdot \frac{\hat{m}}{\sqrt{\hat{v}} + \epsilon}
+$$
-Adam defaults: lr = 0.001, beta1 = 0.9, beta2 = 0.999, epsilon = 1e-8. These defaults work for 80% of problems. When they don't, change lr first. Then beta2. Almost never change beta1 or epsilon.
+Adam defaults: $\text{lr} = 0.001$, $\beta_1 = 0.9$, $\beta_2 = 0.999$, $\epsilon = 1\text{e-}8$. These defaults work for 80% of problems. When they don't, change lr first. Then $\beta_2$. Almost never change $\beta_1$ or $\epsilon$.
### AdamW: Weight Decay Done Right
-L2 regularization adds lambda * w^2 to the loss. In vanilla SGD, this is equivalent to weight decay (subtracting lambda * w from the weight at each step). In Adam, this equivalence breaks.
+L2 regularization adds $\lambda \cdot w^2$ to the loss. In vanilla SGD, this is equivalent to weight decay (subtracting $\lambda \cdot w$ from the weight at each step). In Adam, this equivalence breaks.
The Loshchilov & Hutter insight: when you add L2 to the loss and then Adam processes the gradient, the adaptive learning rate scales the regularization term too. Parameters with large gradient variance get less regularization. Parameters with small variance get more. This is not what you want -- you want uniform regularization regardless of the gradient statistics.
AdamW fixes this by applying weight decay directly to the weights, after the Adam update:
-```
-w = w - lr * m_hat / (sqrt(v_hat) + epsilon) - lr * lambda * w
-```
+$$
+w = w - \text{lr} \cdot \frac{\hat{m}}{\sqrt{\hat{v}} + \epsilon} - \text{lr} \cdot \lambda \cdot w
+$$
-The weight decay term (lr * lambda * w) is not scaled by Adam's adaptive factor. Every parameter gets the same proportional shrinkage.
+The weight decay term ($\text{lr} \cdot \lambda \cdot w$) is not scaled by Adam's adaptive factor. Every parameter gets the same proportional shrinkage.
This seems like a minor detail. It's not. AdamW converges to better solutions than Adam + L2 regularization on virtually every task. It's the default optimizer in PyTorch for training transformers, diffusion models, and most modern architectures. BERT, GPT, LLaMA, Stable Diffusion -- all trained with AdamW.
@@ -424,11 +432,11 @@ This lesson produces:
## Exercises
-1. Implement Nesterov momentum, where you compute the gradient at the "lookahead" position (w - lr * beta * v) instead of the current position. Compare convergence to standard momentum on the circle dataset.
+1. Implement Nesterov momentum, where you compute the gradient at the "lookahead" position ($w - \text{lr} \cdot \beta \cdot v$) instead of the current position. Compare convergence to standard momentum on the circle dataset.
2. Implement a learning rate warmup schedule: linear ramp from 0 to max_lr over the first 10% of training steps, then cosine decay to 0. Train with Adam + warmup vs Adam without warmup. Measure how many epochs it takes to reach 90% accuracy on the circle dataset.
-3. Track the effective learning rate for each parameter during Adam training. The effective rate is lr * m_hat / (sqrt(v_hat) + eps). Plot the distribution of effective rates after 10, 50, and 200 steps. Are all parameters being updated at the same speed?
+3. Track the effective learning rate for each parameter during Adam training. The effective rate is $\text{lr} \cdot \hat{m} / (\sqrt{\hat{v}} + \epsilon)$. Plot the distribution of effective rates after 10, 50, and 200 steps. Are all parameters being updated at the same speed?
4. Implement gradient clipping (clip by global norm). Set the max gradient norm to 1.0. Train with and without clipping using a high learning rate (lr=0.01 for Adam). Count how many runs diverge (loss goes to NaN) with and without clipping over 10 random seeds.
@@ -439,12 +447,12 @@ This lesson produces:
| Term | What people say | What it actually means |
|------|----------------|----------------------|
| Learning rate | "Step size" | The scalar multiplier on the gradient update; the single most impactful hyperparameter in training |
-| SGD | "Basic gradient descent" | Stochastic gradient descent: update weights by subtracting lr * gradient, computed on a mini-batch |
+| SGD | "Basic gradient descent" | Stochastic gradient descent: update weights by subtracting $\text{lr} \cdot \text{gradient}$, computed on a mini-batch |
| Momentum | "Rolling ball analogy" | Exponential moving average of past gradients; dampens oscillation and accelerates consistent directions |
| RMSProp | "Adaptive learning rate" | Divides each parameter's gradient by the running RMS of its recent gradients; equalizes learning rates |
| Adam | "The default optimizer" | Combines momentum (first moment) and RMSProp (second moment) with bias correction for the initial steps |
| AdamW | "Adam done right" | Adam with decoupled weight decay; applies regularization directly to weights rather than through the gradient |
-| Bias correction | "Warmup for running averages" | Dividing by (1 - beta^t) to compensate for the zero-initialization of Adam's moment estimates |
+| Bias correction | "Warmup for running averages" | Dividing by $(1 - \beta^t)$ to compensate for the zero-initialization of Adam's moment estimates |
| Weight decay | "Shrink the weights" | Subtracting a fraction of the weight value at each step; a regularizer that penalizes large weights |
| Learning rate schedule | "Changing lr over time" | A function that adjusts the learning rate during training; warmup + cosine decay is the modern default |
| Gradient clipping | "Capping the gradient norm" | Scaling down the gradient vector when its norm exceeds a threshold; prevents exploding gradient updates |
diff --git a/phases/03-deep-learning-core/07-regularization/docs/en.md b/phases/03-deep-learning-core/07-regularization/docs/en.md
index f2857b50cf..64171c561d 100644
--- a/phases/03-deep-learning-core/07-regularization/docs/en.md
+++ b/phases/03-deep-learning-core/07-regularization/docs/en.md
@@ -43,20 +43,22 @@ graph LR
The simplest regularization technique with the most elegant interpretation. During training, randomly set each neuron's output to zero with probability p.
-```
-output = activation(z) * mask where mask[i] ~ Bernoulli(1 - p)
-```
+$$
+\text{output} = \text{activation}(z) \cdot \text{mask}, \quad \text{where } \text{mask}[i] \sim \text{Bernoulli}(1 - p)
+$$
With p = 0.5, half the neurons are zeroed on every forward pass. The network must learn redundant representations because it can't predict which neurons will be available. This prevents co-adaptation -- neurons learning to rely on specific other neurons being present.
-The ensemble interpretation: a network with N neurons and dropout creates 2^N possible subnetworks (every combination of which neurons are on or off). Training with dropout approximately trains all 2^N subnetworks simultaneously, each on different mini-batches. At test time, you use all neurons (no dropout) and scale outputs by (1 - p) to match the expected value during training. This is equivalent to averaging the predictions of 2^N subnetworks -- a massive ensemble from a single model.
+The ensemble interpretation: a network with $N$ neurons and dropout creates $2^N$ possible subnetworks (every combination of which neurons are on or off). Training with dropout approximately trains all $2^N$ subnetworks simultaneously, each on different mini-batches. At test time, you use all neurons (no dropout) and scale outputs by $(1 - p)$ to match the expected value during training. This is equivalent to averaging the predictions of $2^N$ subnetworks -- a massive ensemble from a single model.
In practice, the scaling is applied during training instead of testing (inverted dropout):
-```
-During training: output = activation(z) * mask / (1 - p)
-During testing: output = activation(z) (no change needed)
-```
+$$
+\begin{aligned}
+\text{During training:} \quad &\text{output} = \text{activation}(z) \cdot \text{mask} / (1 - p) \\
+\text{During testing:} \quad &\text{output} = \text{activation}(z) \quad (\text{no change needed})
+\end{aligned}
+$$
This is cleaner because test code doesn't need to know about dropout at all.
@@ -66,11 +68,11 @@ Default rates: p = 0.1 for transformers, p = 0.5 for MLPs, p = 0.2-0.3 for CNNs.
Add the squared magnitude of all weights to the loss:
-```
-total_loss = task_loss + (lambda / 2) * sum(w_i^2)
-```
+$$
+\text{total\_loss} = \text{task\_loss} + \frac{\lambda}{2} \sum_i w_i^2
+$$
-The gradient of the regularization term is lambda * w. This means at every step, each weight is shrunk toward zero by a fraction proportional to its magnitude. Large weights get penalized more. The model is pushed toward solutions where no single weight dominates.
+The gradient of the regularization term is $\lambda w$. This means at every step, each weight is shrunk toward zero by a fraction proportional to its magnitude. Large weights get penalized more. The model is pushed toward solutions where no single weight dominates.
Why this helps generalization: overfit models tend to have large weights that amplify noise in the training data. Weight decay keeps weights small, which limits the model's effective capacity and forces it to rely on robust, generalizable features rather than memorized quirks.
@@ -88,12 +90,14 @@ Normalize the output of each layer across the mini-batch before passing it to th
For a mini-batch of activations at some layer:
-```
-mu = (1/B) * sum(x_i) (batch mean)
-sigma^2 = (1/B) * sum((x_i - mu)^2) (batch variance)
-x_hat = (x_i - mu) / sqrt(sigma^2 + eps) (normalize)
-y = gamma * x_hat + beta (scale and shift)
-```
+$$
+\begin{aligned}
+\mu &= \frac{1}{B} \sum_i x_i &&\text{(batch mean)} \\
+\sigma^2 &= \frac{1}{B} \sum_i (x_i - \mu)^2 &&\text{(batch variance)} \\
+\hat{x} &= \frac{x_i - \mu}{\sqrt{\sigma^2 + \epsilon}} &&\text{(normalize)} \\
+y &= \gamma \cdot \hat{x} + \beta &&\text{(scale and shift)}
+\end{aligned}
+$$
Gamma and beta are learnable parameters that let the network undo the normalization if that's optimal. Without them, you'd be forcing every layer's output to be zero-mean unit-variance, which might not be what the network wants.
@@ -107,14 +111,16 @@ BatchNorm has a fundamental limitation: it depends on batch statistics. With bat
Normalize across features instead of across the batch. For a single sample:
-```
-mu = (1/D) * sum(x_j) (feature mean)
-sigma^2 = (1/D) * sum((x_j - mu)^2) (feature variance)
-x_hat = (x_j - mu) / sqrt(sigma^2 + eps)
-y = gamma * x_hat + beta
-```
+$$
+\begin{aligned}
+\mu &= \frac{1}{D} \sum_j x_j &&\text{(feature mean)} \\
+\sigma^2 &= \frac{1}{D} \sum_j (x_j - \mu)^2 &&\text{(feature variance)} \\
+\hat{x} &= \frac{x_j - \mu}{\sqrt{\sigma^2 + \epsilon}} \\
+y &= \gamma \cdot \hat{x} + \beta
+\end{aligned}
+$$
-D is the feature dimension. Each sample is normalized independently -- no dependence on batch size. This is why transformers use LayerNorm instead of BatchNorm. Sequences have variable lengths, batch sizes are often small (or 1 during generation), and the computation is identical between training and inference.
+$D$ is the feature dimension. Each sample is normalized independently -- no dependence on batch size. This is why transformers use LayerNorm instead of BatchNorm. Sequences have variable lengths, batch sizes are often small (or 1 during generation), and the computation is identical between training and inference.
LayerNorm in transformers is applied after each self-attention block and each feed-forward block (Post-LN), or before them (Pre-LN, which is more stable for training).
@@ -122,10 +128,12 @@ LayerNorm in transformers is applied after each self-attention block and each fe
LayerNorm without the mean subtraction. Proposed by Zhang & Sennrich (2019).
-```
-rms = sqrt((1/D) * sum(x_j^2))
-y = gamma * x / rms
-```
+$$
+\begin{aligned}
+\text{rms} &= \sqrt{\frac{1}{D} \sum_j x_j^2} \\
+y &= \gamma \cdot \frac{x}{\text{rms}}
+\end{aligned}
+$$
That's it. No mean computation, no beta parameter. The observation: the re-centering (mean subtraction) in LayerNorm contributes very little to the model's performance, but costs computation. Removing it gives the same accuracy with about 10% less overhead.
@@ -516,7 +524,7 @@ This lesson produces:
| Overfitting | "Model memorized the data" | When a model's training performance significantly exceeds its test performance, indicating it learned noise rather than signal |
| Regularization | "Preventing overfitting" | Any technique that constrains model complexity to improve generalization: dropout, weight decay, normalization, augmentation |
| Dropout | "Random neuron deletion" | Zeroing random neurons during training with probability p, forcing redundant representations; equivalent to training an ensemble |
-| Weight decay | "L2 penalty" | Shrinking all weights toward zero by subtracting lambda * w at each step; penalizes complexity through weight magnitude |
+| Weight decay | "L2 penalty" | Shrinking all weights toward zero by subtracting $\lambda w$ at each step; penalizes complexity through weight magnitude |
| Batch normalization | "Normalize per batch" | Normalizing layer outputs across the batch dimension using batch statistics during training and running averages during inference |
| Layer normalization | "Normalize per sample" | Normalizing across features within each sample; batch-independent, used in transformers where batch size varies |
| RMSNorm | "LayerNorm without the mean" | Root mean square normalization; drops the mean subtraction from LayerNorm for 10% speedup with equal accuracy |
diff --git a/phases/04-computer-vision/01-image-fundamentals/docs/en.md b/phases/04-computer-vision/01-image-fundamentals/docs/en.md
index 7801d53453..26ec9acea7 100644
--- a/phases/04-computer-vision/01-image-fundamentals/docs/en.md
+++ b/phases/04-computer-vision/01-image-fundamentals/docs/en.md
@@ -173,9 +173,9 @@ For most modern CNNs you feed RGB. You meet other spaces when:
Grayscale from RGB is a weighted sum, not an average, because the human eye is more sensitive to green than to red or blue:
-```
-Y = 0.299 R + 0.587 G + 0.114 B (ITU-R BT.601, the classic weights)
-```
+$$
+Y = 0.299\,R + 0.587\,G + 0.114\,B \qquad \text{(ITU-R BT.601, the classic weights)}
+$$
### Aspect ratio, resizing, and interpolation
diff --git a/phases/04-computer-vision/02-convolutions-from-scratch/docs/en.md b/phases/04-computer-vision/02-convolutions-from-scratch/docs/en.md
index 5f40ade8e8..e014023533 100644
--- a/phases/04-computer-vision/02-convolutions-from-scratch/docs/en.md
+++ b/phases/04-computer-vision/02-convolutions-from-scratch/docs/en.md
@@ -10,7 +10,7 @@
## Learning Objectives
- Implement 2D convolution from scratch using only NumPy, including the nested-loop version and a vectorised `im2col` version
-- Compute output spatial size for any combination of input size, kernel size, padding, and stride, and justify the `(H - K + 2P) / S + 1` formula
+- Compute output spatial size for any combination of input size, kernel size, padding, and stride, and justify the $(H - K + 2P) / S + 1$ formula
- Hand-design kernels (edge, blur, sharpen, Sobel) and explain why each one produces the pattern of activations it does
- Stack convolutions into a feature extractor and connect the depth-of-the-stack to the size of the receptive field
@@ -74,9 +74,9 @@ That one formula — **shared weights, locality, sliding window** — is the ent
Given input spatial size `H`, kernel size `K`, padding `P`, stride `S`:
-```
-H_out = floor( (H - K + 2P) / S ) + 1
-```
+$$
+H_{out} = \left\lfloor \frac{H - K + 2P}{S} \right\rfloor + 1
+$$
Memorise this. You will compute it dozens of times per architecture.
@@ -88,7 +88,7 @@ Memorise this. You will compute it dozens of times per architecture.
| Pool 2x2 | 32 | 2 | 0 | 2 | 16 |
| Large receptive field | 32 | 7 | 3 | 2 | 16 |
-"Same padding" means pick P so that H_out == H when S == 1. For odd K, that is P = (K - 1) / 2. That is why 3x3 kernels dominate — they are the smallest odd kernel that still has a centre.
+"Same padding" means pick P so that H_out == H when S == 1. For odd K, that is $P = (K - 1) / 2$. That is why 3x3 kernels dominate — they are the smallest odd kernel that still has a centre.
### Padding
@@ -146,7 +146,7 @@ Output: (C_out, H', W') 64 x 3 x 3
Parameter count: C_out * C_in * K * K + C_out (the + C_out is biases)
```
-That last line is the one you will calculate when planning a model. A 64-channel 3x3 conv on a 3-channel input has `64 * 3 * 3 * 3 + 64 = 1,792` parameters. Cheap.
+That last line is the one you will calculate when planning a model. A 64-channel 3x3 conv on a 3-channel input has $64 \times 3 \times 3 \times 3 + 64 = 1{,}792$ parameters. Cheap.
### The im2col trick
@@ -172,11 +172,11 @@ Every production conv implementation is some variant of this plus cache-tiling t
A single 3x3 conv looks at 9 input pixels. Stack two 3x3 convs and a neuron in the second layer looks at 5x5 input pixels. Three 3x3 convs give 7x7. In general:
-```
-RF after L stacked K x K convs (stride 1) = 1 + L * (K - 1)
+$$
+\text{RF after } L \text{ stacked } K \times K \text{ convs (stride 1)} = 1 + L \cdot (K - 1)
+$$
-With strides: RF grows multiplicatively with stride along each layer.
-```
+With strides: RF grows multiplicatively with stride along each layer.
The entire reason "3x3 all the way down" works (VGG, ResNet, ConvNeXt) is that two 3x3 convs see the same input area as one 5x5 conv but with fewer parameters and an extra non-linearity in between.
diff --git a/phases/04-computer-vision/03-cnns-lenet-to-resnet/docs/en.md b/phases/04-computer-vision/03-cnns-lenet-to-resnet/docs/en.md
index 59e2ec3b32..652f6fda14 100644
--- a/phases/04-computer-vision/03-cnns-lenet-to-resnet/docs/en.md
+++ b/phases/04-computer-vision/03-cnns-lenet-to-resnet/docs/en.md
@@ -72,7 +72,7 @@ stack: conv 3x3 -> conv 3x3 -> pool 2x2
repeat: 16 or 19 conv layers
```
-Two 3x3 convs see the same 5x5 input area as one 5x5 conv but with fewer parameters (2*9*C^2 = 18C^2 vs 25*C^2) and an extra ReLU in between. VGG turned this observation into an entire architecture. The simplicity — one block type, repeated — made it the reference point for everything that came after.
+Two 3x3 convs see the same 5x5 input area as one 5x5 conv but with fewer parameters ($2 \cdot 9 \cdot C^2 = 18C^2$ vs $25 \cdot C^2$) and an extra ReLU in between. VGG turned this observation into an entire architecture. The simplicity — one block type, repeated — made it the reference point for everything that came after.
Cost: 138M parameters, slow to train, expensive at inference.
@@ -103,16 +103,19 @@ Each branch specialises — 1x1 for channel mixing, 3x3 for local texture, 5x5 f
By 2015, VGG-19 worked and VGG-32 did not. Depth was supposed to help, but past ~20 layers both training and test loss got worse. That is not overfitting. That is the optimiser failing to find useful weights because gradients shrink multiplicatively through every layer.
-```
Plain deep network:
- y = f_L( f_{L-1}( ... f_1(x) ... ) )
+
+$$
+y = f_L( f_{L-1}( \dots f_1(x) \dots ) )
+$$
Gradient wrt early layer:
- dL/dW_1 = dL/dy * df_L/df_{L-1} * ... * df_2/df_1 * df_1/dW_1
-Each multiplicative term has magnitude roughly (weight magnitude) * (activation gain).
-Stack 100 of them with gains < 1 and the gradient is effectively zero.
-```
+$$
+\frac{\partial L}{\partial W_1} = \frac{\partial L}{\partial y} \cdot \frac{\partial f_L}{\partial f_{L-1}} \cdot \dots \cdot \frac{\partial f_2}{\partial f_1} \cdot \frac{\partial f_1}{\partial W_1}
+$$
+
+Each multiplicative term has magnitude roughly (weight magnitude) $\times$ (activation gain). Stack 100 of them with gains $< 1$ and the gradient is effectively zero.
VGG worked at 19 layers because batch norm (published simultaneously) kept activations well-scaled. But even batch norm could not rescue depth beyond 30-ish layers.
@@ -120,12 +123,14 @@ VGG worked at 19 layers because batch norm (published simultaneously) kept activ
He, Zhang, Ren, Sun proposed one change that fixed everything:
-```
-standard block: y = F(x)
-residual block: y = F(x) + x
-```
+$$
+\begin{aligned}
+\text{standard block:} \quad & y = F(x) \\
+\text{residual block:} \quad & y = F(x) + x
+\end{aligned}
+$$
-The `+ x` means the layer can always choose to do nothing by driving `F(x)` to zero. A 1,000-layer ResNet is now at most as bad as a 1-layer network, because every extra block has a trivial escape hatch. With that guarantee, the optimiser is willing to make every block *slightly* useful — and slightly useful, stacked 100 times, is state-of-the-art.
+The `+ x` means the layer can always choose to do nothing by driving $F(x)$ to zero. A 1,000-layer ResNet is now at most as bad as a 1-layer network, because every extra block has a trivial escape hatch. With that guarantee, the optimiser is willing to make every block *slightly* useful — and slightly useful, stacked 100 times, is state-of-the-art.
```mermaid
flowchart LR
@@ -377,7 +382,7 @@ This lesson produces:
| Term | What people say | What it actually means |
|------|----------------|----------------------|
| Backbone | "The model" | The stack of convolutional blocks that produces the feature map fed to the task head |
-| Residual connection | "Skip connection" | `y = F(x) + x`; lets the optimiser learn identity by setting F to zero, which makes arbitrary depth trainable |
+| Residual connection | "Skip connection" | $y = F(x) + x$; lets the optimiser learn identity by setting $F$ to zero, which makes arbitrary depth trainable |
| BasicBlock | "Two 3x3 convs with a skip" | The ResNet-18/34 building block: conv-BN-ReLU-conv-BN-add-ReLU |
| Bottleneck | "1x1 down, 3x3, 1x1 up" | The ResNet-50/101/152 block; cheap at high channel counts because the 3x3 runs on a reduced width |
| Degradation problem | "Deeper is worse" | Past ~20 plain conv layers, both training and test error increase; solved by residual connections, not by more data |
diff --git a/phases/04-computer-vision/08-instance-segmentation-mask-rcnn/docs/en.md b/phases/04-computer-vision/08-instance-segmentation-mask-rcnn/docs/en.md
index bf4a68feab..d039011c6c 100644
--- a/phases/04-computer-vision/08-instance-segmentation-mask-rcnn/docs/en.md
+++ b/phases/04-computer-vision/08-instance-segmentation-mask-rcnn/docs/en.md
@@ -86,14 +86,14 @@ Upsample the 28x28 mask to the proposal's original pixel size to produce the fin
Mask R-CNN has four losses added together:
-```
-L = L_rpn_cls + L_rpn_box + L_box_cls + L_box_reg + L_mask
-```
-
-- `L_rpn_cls`, `L_rpn_box` — objectness + box regression for the RPN proposals.
-- `L_box_cls` — cross-entropy over (C+1) classes (including background) on the head's classifier.
-- `L_box_reg` — smooth L1 on the head's box refinement.
-- `L_mask` — per-pixel binary cross-entropy on the 28x28 mask output.
+$$
+L = L_{\text{rpn\_cls}} + L_{\text{rpn\_box}} + L_{\text{box\_cls}} + L_{\text{box\_reg}} + L_{\text{mask}}
+$$
+
+- $L_{\text{rpn\_cls}}$, $L_{\text{rpn\_box}}$ — objectness + box regression for the RPN proposals.
+- $L_{\text{box\_cls}}$ — cross-entropy over $(C+1)$ classes (including background) on the head's classifier.
+- $L_{\text{box\_reg}}$ — smooth L1 on the head's box refinement.
+- $L_{\text{mask}}$ — per-pixel binary cross-entropy on the 28x28 mask output.
Each loss has its own default weight; the torchvision implementation exposes them as constructor arguments.
diff --git a/phases/04-computer-vision/10-image-generation-diffusion/docs/en.md b/phases/04-computer-vision/10-image-generation-diffusion/docs/en.md
index 5b9fffb404..7207615cd7 100644
--- a/phases/04-computer-vision/10-image-generation-diffusion/docs/en.md
+++ b/phases/04-computer-vision/10-image-generation-diffusion/docs/en.md
@@ -9,7 +9,7 @@
## Learning Objectives
-- Derive the forward noising process `x_0 -> x_1 -> ... -> x_T` and explain why the closed-form `q(x_t | x_0)` holds for any t
+- Derive the forward noising process $x_0 \to x_1 \to \dots \to x_T$ and explain why the closed-form $q(x_t \mid x_0)$ holds for any $t$
- Implement a DDPM-style training objective that regresses the noise added at each step, and a sampler that walks back from pure noise to an image
- Build a time-conditioned U-Net (small enough to train on CPU) that predicts the noise for any timestep
- Explain the difference between DDPM and DDIM sampling, and when each is appropriate (Lesson 23 covers flow matching and rectified flow in depth)
@@ -26,35 +26,33 @@ This lesson builds the minimal DDPM: forward noising, backward denoising, traini
### The forward process
-Take an image `x_0`. Add a tiny amount of Gaussian noise to get `x_1`. Add a tiny amount more to get `x_2`. Keep going for T steps until `x_T` is nearly indistinguishable from pure Gaussian noise.
+Take an image $x_0$. Add a tiny amount of Gaussian noise to get $x_1$. Add a tiny amount more to get $x_2$. Keep going for T steps until $x_T$ is nearly indistinguishable from pure Gaussian noise.
-```
-q(x_t | x_{t-1}) = N(x_t; sqrt(1 - beta_t) * x_{t-1}, beta_t * I)
-```
+$$
+q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{1 - \beta_t} \cdot x_{t-1},\; \beta_t I)
+$$
-`beta_t` is a small variance schedule, typically linear from 0.0001 to 0.02 over T=1000 steps. Each step slightly shrinks the signal and injects fresh noise.
+$\beta_t$ is a small variance schedule, typically linear from 0.0001 to 0.02 over T=1000 steps. Each step slightly shrinks the signal and injects fresh noise.
### The closed-form jump
-Adding noise one step at a time is a Markov chain, but the math folds: you can sample `x_t` directly from `x_0` in one step.
-
-```
-Define alpha_t = 1 - beta_t
-Define alpha_bar_t = prod_{s=1..t} alpha_s
-
-Then:
- q(x_t | x_0) = N(x_t; sqrt(alpha_bar_t) * x_0, (1 - alpha_bar_t) * I)
+Adding noise one step at a time is a Markov chain, but the math folds: you can sample $x_t$ directly from $x_0$ in one step.
-Equivalently:
- x_t = sqrt(alpha_bar_t) * x_0 + sqrt(1 - alpha_bar_t) * epsilon
- where epsilon ~ N(0, I)
-```
+$$
+\begin{aligned}
+&\text{Define } \alpha_t = 1 - \beta_t \\
+&\text{Define } \bar{\alpha}_t = \prod_{s=1}^{t} \alpha_s \\[4pt]
+\text{Then:}\quad & q(x_t \mid x_0) = \mathcal{N}(x_t; \sqrt{\bar{\alpha}_t} \cdot x_0,\; (1 - \bar{\alpha}_t) I) \\[4pt]
+\text{Equivalently:}\quad & x_t = \sqrt{\bar{\alpha}_t} \cdot x_0 + \sqrt{1 - \bar{\alpha}_t} \cdot \epsilon \\
+&\text{where } \epsilon \sim \mathcal{N}(0, I)
+\end{aligned}
+$$
-This single equation is the whole reason diffusion is practical. During training you pick a random `t`, sample `x_t` directly from `x_0`, and train in one step — no simulation of the full Markov chain needed.
+This single equation is the whole reason diffusion is practical. During training you pick a random $t$, sample $x_t$ directly from $x_0$, and train in one step — no simulation of the full Markov chain needed.
### The reverse process
-The forward process is fixed. The reverse process `p(x_{t-1} | x_t)` is what the neural network learns. Diffusion models do not predict `x_{t-1}` directly; they predict the noise `epsilon` added at step t, and the math derives `x_{t-1}` from it.
+The forward process is fixed. The reverse process $p(x_{t-1} \mid x_t)$ is what the neural network learns. Diffusion models do not predict $x_{t-1}$ directly; they predict the noise $\epsilon$ added at step t, and the math derives $x_{t-1}$ from it.
```mermaid
flowchart LR
@@ -78,18 +76,18 @@ flowchart LR
For every training step:
-1. Sample a real image `x_0`.
-2. Sample a timestep `t` uniformly from [1, T].
-3. Sample noise `epsilon ~ N(0, I)`.
-4. Compute `x_t = sqrt(alpha_bar_t) * x_0 + sqrt(1 - alpha_bar_t) * epsilon`.
-5. Predict `epsilon_theta(x_t, t)` with the network.
-6. Minimise `|| epsilon - epsilon_theta(x_t, t) ||^2`.
+1. Sample a real image $x_0$.
+2. Sample a timestep $t$ uniformly from $[1, T]$.
+3. Sample noise $\epsilon \sim \mathcal{N}(0, I)$.
+4. Compute $x_t = \sqrt{\bar{\alpha}_t} \cdot x_0 + \sqrt{1 - \bar{\alpha}_t} \cdot \epsilon$.
+5. Predict $\epsilon_\theta(x_t, t)$ with the network.
+6. Minimise $\lVert \epsilon - \epsilon_\theta(x_t, t) \rVert^2$.
That is it. The neural network learns to predict the noise at any timestep. The loss is MSE. There is no adversarial game, no collapse, no oscillation.
### The sampler (DDPM)
-To generate: start from `x_T ~ N(0, I)` and walk backwards one step at a time.
+To generate: start from $x_T \sim \mathcal{N}(0, I)$ and walk backwards one step at a time.
```
for t = T, T-1, ..., 1:
@@ -111,7 +109,7 @@ Training is the same. Sampling changes. DDIM (Song et al., 2020) defines a deter
### Time conditioning
-The network `epsilon_theta(x_t, t)` needs to know which timestep it is denoising. Modern diffusion models inject `t` via sinusoidal time embeddings (same idea as positional encoding in transformers) that get added to feature maps at every U-Net level.
+The network $\epsilon_\theta(x_t, t)$ needs to know which timestep it is denoising. Modern diffusion models inject $t$ via sinusoidal time embeddings (same idea as positional encoding in transformers) that get added to feature maps at every U-Net level.
```
t_embedding = sinusoidal(t)
@@ -301,9 +299,9 @@ This lesson produces:
## Exercises
-1. **(Easy)** Visualise the forward process: take one image and plot `x_t` at `t in [0, 100, 250, 500, 750, 1000]`. Verify that `x_1000` looks like pure Gaussian noise.
+1. **(Easy)** Visualise the forward process: take one image and plot $x_t$ at $t \in \{0, 100, 250, 500, 750, 1000\}$. Verify that $x_{1000}$ looks like pure Gaussian noise.
2. **(Medium)** Train the TinyUNet on the synthetic-circles dataset for 20 epochs and sample 16 circles. Compare DDPM (1000 steps) and DDIM (50 steps) sampling — do they produce similar images from the same noise seed?
-3. **(Hard)** Implement a cosine noise schedule (Nichol & Dhariwal, 2021): `alpha_bar_t = cos^2((t/T + s) / (1 + s) * pi / 2)`. Train the same model with linear and cosine schedules and show that cosine gives better samples at low step counts.
+3. **(Hard)** Implement a cosine noise schedule (Nichol & Dhariwal, 2021): $\bar{\alpha}_t = \cos^2\left(\frac{t/T + s}{1 + s} \cdot \frac{\pi}{2}\right)$. Train the same model with linear and cosine schedules and show that cosine gives better samples at low step counts.
## Key Terms
@@ -311,10 +309,10 @@ This lesson produces:
|------|----------------|----------------------|
| Forward process | "Add noise over time" | Fixed Markov chain that corrupts an image into Gaussian noise over T steps |
| Reverse process | "Denoise step by step" | Learned distribution that walks back from noise to image |
-| Epsilon prediction | "Predict the noise" | The training target: `epsilon_theta(x_t, t)` predicts the noise added at step t |
+| Epsilon prediction | "Predict the noise" | The training target: $\epsilon_\theta(x_t, t)$ predicts the noise added at step t |
| Beta schedule | "Noise amounts" | Sequence of T small variances that define how much noise enters per step |
-| alpha_bar_t | "Cumulative retain factor" | Product of (1 - beta_s) up to time t; bigger t means less signal left |
-| DDPM sampler | "Ancestral, stochastic" | Samples each x_{t-1} from its conditional Gaussian; 1000 steps |
+| $\bar{\alpha}_t$ | "Cumulative retain factor" | Product of $(1 - \beta_s)$ up to time t; bigger t means less signal left |
+| DDPM sampler | "Ancestral, stochastic" | Samples each $x_{t-1}$ from its conditional Gaussian; 1000 steps |
| DDIM sampler | "Deterministic, fast" | Rewrites sampling as a deterministic ODE; 20-100 steps with similar quality |
| Time conditioning | "Tell the model which t" | Sinusoidal embedding of t injected into the U-Net so it knows the noise level |
diff --git a/phases/04-computer-vision/13-3d-vision-nerf/docs/en.md b/phases/04-computer-vision/13-3d-vision-nerf/docs/en.md
index 4a195b160b..e6d5b764a6 100644
--- a/phases/04-computer-vision/13-3d-vision-nerf/docs/en.md
+++ b/phases/04-computer-vision/13-3d-vision-nerf/docs/en.md
@@ -26,7 +26,7 @@ The two representations dominate for different reasons. Point clouds are what se
### Point clouds
-A point cloud is an unordered set of N points in R^3, optionally each with features (colour, intensity, normal).
+A point cloud is an unordered set of $N$ points in $\mathbb{R}^3$, optionally each with features (colour, intensity, normal).
```
cloud = [
@@ -44,9 +44,9 @@ No grid, no connectivity. Two properties make this hard for neural networks:
PointNet (Qi et al., 2017) solved both with one idea: apply a shared MLP to every point, then aggregate with a symmetric function (max pool). The result is a fixed-size vector that does not depend on order.
-```
-f(P) = max_{p in P} MLP(p)
-```
+$$
+f(P) = \max_{p \in P} \text{MLP}(p)
+$$
This is the entire core of PointNet. Deeper variants (PointNet++, Point Transformer) add hierarchical sampling and local aggregation but the symmetric-function trick is unchanged.
@@ -89,22 +89,23 @@ A loss compares the rendered pixel to the ground-truth pixel in the training pho
A vanilla MLP on `(x, y, z)` cannot represent high-frequency details because MLPs are spectrally biased toward low frequencies. NeRF fixes this by encoding each coordinate into a Fourier feature vector before the MLP:
-```
-gamma(p) = (sin(2^0 pi p), cos(2^0 pi p), sin(2^1 pi p), cos(2^1 pi p), ...)
-```
+$$
+\gamma(p) = (\sin(2^0 \pi p), \cos(2^0 \pi p), \sin(2^1 \pi p), \cos(2^1 \pi p), \ldots)
+$$
-Up to L=10 frequency levels. This is the same trick transformers use for positions, and it appears again in diffusion time conditioning (Lesson 10). Without it, NeRFs look blurry.
+Up to $L=10$ frequency levels. This is the same trick transformers use for positions, and it appears again in diffusion time conditioning (Lesson 10). Without it, NeRFs look blurry.
### Volumetric rendering
-```
-C(r) = sum_i T_i * (1 - exp(-sigma_i * delta_i)) * c_i
-
-T_i = exp(- sum_{j 4d) -> GELU -> Linear(4d -> d)
-```
+MLP is two-layer with GELU: $\text{Linear}(d \to 4d) \to \text{GELU} \to \text{Linear}(4d \to d)$
ViT-B/16 stacks 12 of these blocks, each with 12 attention heads, totalling 86M parameters.
### Why pre-LN
-Early transformers used post-LN (`x = LN(x + sublayer(x))`) and struggled to train past 6-8 layers without warmup. Pre-LN (`x = x + sublayer(LN(x))`) trains deeper networks stably without warmup. Every ViT and every modern LLM uses pre-LN.
+Early transformers used post-LN ($x = \text{LN}(x + \text{sublayer}(x))$) and struggled to train past 6-8 layers without warmup. Pre-LN ($x = x + \text{sublayer}(\text{LN}(x))$) trains deeper networks stably without warmup. Every ViT and every modern LLM uses pre-LN.
### Patch size trade-off
- 16x16 patches -> 196 tokens, standard.
- 32x32 patches -> 49 tokens, faster but lower resolution.
-- 8x8 patches -> 784 tokens, finer but O(n^2) attention cost scales badly.
+- 8x8 patches -> 784 tokens, finer but $O(n^2)$ attention cost scales badly.
Bigger patches = fewer tokens = faster but less spatial detail. SwinV2 uses 4x4 patches in hierarchical windows.
@@ -257,7 +259,7 @@ This lesson produces:
| Patch embedding | "The first conv" | A conv with kernel size = stride = patch size; turns the image into a grid of token embeddings |
| Class token | "[CLS]" | A learned vector prepended to the token sequence; its final output is the global image representation |
| Positional embedding | "Learned pos" | A learned vector added to every token so the transformer knows where each patch came from |
-| Pre-LN | "LayerNorm before sublayer" | The stable transformer variant: `x + sublayer(LN(x))` instead of `LN(x + sublayer(x))` |
+| Pre-LN | "LayerNorm before sublayer" | The stable transformer variant: $x + \text{sublayer}(\text{LN}(x))$ instead of $\text{LN}(x + \text{sublayer}(x))$ |
| Multi-head attention | "Parallel attention" | Standard transformer attention split into num_heads independent subspaces, concatenated afterwards |
| ViT-B/16 | "Base, patch 16" | The canonical size: dim=768, depth=12, heads=12, patch_size=16, image=224; ~86M params |
| DeiT | "Data-efficient ViT" | ViT trained on ImageNet-1k alone with strong augmentation; proved large pretraining datasets are not strictly required |
diff --git a/phases/04-computer-vision/17-self-supervised-vision/docs/en.md b/phases/04-computer-vision/17-self-supervised-vision/docs/en.md
index ea115ef5b1..17d2174597 100644
--- a/phases/04-computer-vision/17-self-supervised-vision/docs/en.md
+++ b/phases/04-computer-vision/17-self-supervised-vision/docs/en.md
@@ -41,14 +41,13 @@ flowchart LR
Take one image, apply two random augmentations, get two views. Feed both through the same encoder plus a projection head. Minimise a loss that says "these two embeddings should be close" and "this embedding should be far from every other image's embeddings in the batch."
-```
-Loss for positive pair (z_i, z_j) among 2N views per batch:
+Loss for positive pair $(z_i, z_j)$ among $2N$ views per batch:
- L_ij = -log( exp(sim(z_i, z_j) / tau) / sum_k in batch \ {i} exp(sim(z_i, z_k) / tau) )
+$$
+L_{ij} = -\log\left( \frac{\exp(\text{sim}(z_i, z_j) / \tau)}{\sum_{k \in \text{batch} \setminus \{i\}} \exp(\text{sim}(z_i, z_k) / \tau)} \right)
+$$
-sim = cosine similarity
-tau = temperature (0.1 standard)
-```
+where $\text{sim}$ is cosine similarity and $\tau$ is the temperature (0.1 standard).
This is the InfoNCE loss. It requires many negatives per positive, so batch size matters — SimCLR needs 512-8192. MoCo introduced a momentum queue of past batches to decouple negative count from batch size.
@@ -56,12 +55,13 @@ This is the InfoNCE loss. It requires many negatives per positive, so batch size
Two networks with the same architecture: student and teacher. The teacher is an exponential moving average (EMA) of the student's weights. Both see augmented views of the image. The student's output is trained to match the teacher's — no explicit negatives.
-```
-loss = CE( student_output(view_1), teacher_output(view_2) )
- + CE( student_output(view_2), teacher_output(view_1) )
-
-teacher_weights = m * teacher_weights + (1 - m) * student_weights (m ≈ 0.996)
-```
+$$
+\begin{aligned}
+\text{loss} &= \text{CE}\big( \text{student\_output}(\text{view}_1),\ \text{teacher\_output}(\text{view}_2) \big) \\
+&\quad + \text{CE}\big( \text{student\_output}(\text{view}_2),\ \text{teacher\_output}(\text{view}_1) \big) \\
+\text{teacher\_weights} &= m \cdot \text{teacher\_weights} + (1 - m) \cdot \text{student\_weights} \quad (m \approx 0.996)
+\end{aligned}
+$$
Why it does not collapse to "predict a constant": the teacher's output is centred (subtract per-dimension mean) and sharpened (divide by small temperature). Centering prevents one dimension from dominating; sharpening prevents output collapse to uniform.
@@ -173,7 +173,7 @@ print(f"InfoNCE with identical pairs: {loss_same:.3f}")
print(f"InfoNCE with random pairs: {loss_random:.3f}")
```
-Identical pairs should give a low loss (close to 0 for a large batch and cold temperature). Random pairs should give log(2N-1) = ~log(31) = ~3.4 with a 16-pair batch.
+Identical pairs should give a low loss (close to 0 for a large batch and cold temperature). Random pairs should give $\log(2N-1) \approx \log(31) \approx 3.4$ with a 16-pair batch.
### Step 4: MAE-style masking
@@ -227,7 +227,7 @@ This lesson produces:
## Exercises
-1. **(Easy)** Verify that InfoNCE loss drops when you decrease temperature for well-aligned embeddings and rises when you decrease temperature for random embeddings. Produce a plot `tau in [0.05, 0.1, 0.2, 0.5]` vs loss.
+1. **(Easy)** Verify that InfoNCE loss drops when you decrease temperature for well-aligned embeddings and rises when you decrease temperature for random embeddings. Produce a plot $\tau \in [0.05, 0.1, 0.2, 0.5]$ vs loss.
2. **(Medium)** Implement a DINO-style centre buffer. Show that without the centring, the student collapses to a constant vector within a few epochs.
3. **(Hard)** Train MAE on CIFAR-100 using the TinyUNet from Lesson 10 as the backbone. Report linear-probe accuracy at 10, 50, and 200 epochs. Show that a MAE-pretrained linear probe beats a from-scratch supervised linear probe on the same 1,000-image subset.
diff --git a/phases/04-computer-vision/20-image-retrieval-metric/docs/en.md b/phases/04-computer-vision/20-image-retrieval-metric/docs/en.md
index fca0c3ffce..bcff72ac97 100644
--- a/phases/04-computer-vision/20-image-retrieval-metric/docs/en.md
+++ b/phases/04-computer-vision/20-image-retrieval-metric/docs/en.md
@@ -54,9 +54,9 @@ For most production use cases, start with a pretrained backbone and only add a m
### Triplet loss formally
-```
-L = max(0, ||f(a) - f(p)||^2 - ||f(a) - f(n)||^2 + margin)
-```
+$$
+L = \max(0, \|f(a) - f(p)\|^2 - \|f(a) - f(n)\|^2 + \text{margin})
+$$
Pull anchor `a` close to positive `p`, push it away from negative `n`, with a `margin` that ensures a gap. The three-image structure generalises to any similarity ordering.
@@ -69,7 +69,7 @@ Two metrics, two conventions:
- **Cosine**: angle between vectors. Requires L2-normalised embeddings.
- **L2**: Euclidean distance. Works on raw or normalised embeddings, but is usually paired with L2-normalised + squared L2.
-For most modern nets the two are equivalent: `||a - b||^2 = 2 - 2 cos(a, b)` when `||a|| = ||b|| = 1`. Pick the convention that matches your embedding training; mixing them silently changes what "nearest" means.
+For most modern nets the two are equivalent: $\|a - b\|^2 = 2 - 2\cos(a, b)$ when $\|a\| = \|b\| = 1$. Pick the convention that matches your embedding training; mixing them silently changes what "nearest" means.
### Recall@K
@@ -231,7 +231,7 @@ This lesson produces:
| Term | What people say | What it actually means |
|------|----------------|----------------------|
| Metric learning | "Shape the space" | Training an encoder so distances in its output space reflect a target similarity |
-| Triplet loss | "Pull and push" | L = max(0, d(a, p) - d(a, n) + margin); the canonical metric-learning loss |
+| Triplet loss | "Pull and push" | $L = \max(0, d(a, p) - d(a, n) + \text{margin})$; the canonical metric-learning loss |
| Semi-hard mining | "Useful negatives" | Negatives further from the anchor than the positive but within margin; empirically the most informative |
| Proxy-based loss | "Class prototypes" | One learned proxy per class; cross-entropy over similarity-to-proxies; no pair mining |
| Recall@K | "Top-K hit rate" | Fraction of queries with at least one correct result in the top K |
diff --git a/phases/04-computer-vision/21-keypoint-pose/docs/en.md b/phases/04-computer-vision/21-keypoint-pose/docs/en.md
index c9331228d9..fc725e9686 100644
--- a/phases/04-computer-vision/21-keypoint-pose/docs/en.md
+++ b/phases/04-computer-vision/21-keypoint-pose/docs/en.md
@@ -48,11 +48,11 @@ Top-down (HRNet, ViTPose) is the accuracy leader; bottom-up (OpenPose, HigherHRN
### Heatmap regression
-Instead of regressing `(x, y)` directly, predict an `H x W` heatmap per keypoint with a Gaussian blob centred at the true location.
+Instead of regressing `(x, y)` directly, predict an $H \times W$ heatmap per keypoint with a Gaussian blob centred at the true location.
-```
-target[k, y, x] = exp(-((x - cx_k)^2 + (y - cy_k)^2) / (2 sigma^2))
-```
+$$
+\text{target}[k, y, x] = \exp\left(-\frac{(x - cx_k)^2 + (y - cy_k)^2}{2\sigma^2}\right)
+$$
At inference, the argmax of each heatmap is the predicted keypoint location.
@@ -60,7 +60,7 @@ Why heatmaps work better than direct regression: the network's spatial structure
### Sub-pixel localisation
-Argmax gives integer coordinates. For sub-pixel precision, refine by fitting a parabola to the argmax and its neighbours, or use the well-known offset `(dx, dy) = 0.25 * (heatmap[y, x+1] - heatmap[y, x-1], ...)` direction.
+Argmax gives integer coordinates. For sub-pixel precision, refine by fitting a parabola to the argmax and its neighbours, or use the well-known offset $(dx, dy) = 0.25 \cdot (\text{heatmap}[y, x+1] - \text{heatmap}[y, x-1], \ldots)$ direction.
### Part Affinity Fields (PAFs)
@@ -215,7 +215,7 @@ This lesson produces:
| Pose | "The skeleton" | An ordered set of keypoints belonging to one instance |
| Top-down | "Detect then pose" | Two-stage pipeline: person detector + per-crop keypoint model; highest accuracy |
| Bottom-up | "Pose first, group later" | Single-pass all-keypoint prediction + grouping; constant time in crowd size |
-| Heatmap | "Gaussian target" | H x W tensor per keypoint with peak at the true location; the preferred regression target |
+| Heatmap | "Gaussian target" | $H \times W$ tensor per keypoint with peak at the true location; the preferred regression target |
| PAF | "Part Affinity Field" | 2-channel unit vector field encoding limb directions; used to group keypoints into instances |
| OKS | "Keypoint IoU" | Object Keypoint Similarity; the COCO metric for pose |
| HRNet | "High-Resolution Net" | The dominant top-down keypoint architecture; preserves high-res features throughout |
diff --git a/phases/04-computer-vision/23-diffusion-transformers-rectified-flow/docs/en.md b/phases/04-computer-vision/23-diffusion-transformers-rectified-flow/docs/en.md
index 8e5c6a4be6..be72a87f9b 100644
--- a/phases/04-computer-vision/23-diffusion-transformers-rectified-flow/docs/en.md
+++ b/phases/04-computer-vision/23-diffusion-transformers-rectified-flow/docs/en.md
@@ -55,15 +55,15 @@ flowchart LR
### Rectified flow in one paragraph
-DDPM defines the forward process as a noisy SDE where `x_t` is increasingly corrupted. The learned reverse is a second SDE, solved by 1000 small steps.
+DDPM defines the forward process as a noisy SDE where $x_t$ is increasingly corrupted. The learned reverse is a second SDE, solved by 1000 small steps.
Rectified flow defines a **straight-line** interpolation between clean data and pure noise:
-```
-x_t = (1 - t) * x_0 + t * epsilon, t in [0, 1]
-```
+$$
+x_t = (1 - t) \cdot x_0 + t \cdot \epsilon, \quad t \in [0, 1]
+$$
-Train a network to predict the velocity `v_theta(x_t, t) = epsilon - x_0` — the forward direction along the straight-line path from clean data to noise (`dx_t/dt`). During sampling, you integrate this velocity backward to step from noise toward data. The resulting ODE is much closer to a straight line, so far fewer integration steps are needed to sample.
+Train a network to predict the velocity $v_\theta(x_t, t) = \epsilon - x_0$ — the forward direction along the straight-line path from clean data to noise ($dx_t/dt$). During sampling, you integrate this velocity backward to step from noise toward data. The resulting ODE is much closer to a straight line, so far fewer integration steps are needed to sample.
SD3 calls this **Rectified Flow Matching**. FLUX, Z-Image, and most 2026 models use the same objective. Typical inference: 20-30 Euler steps (deterministic) vs 50+ DDIM steps in the old DDPM regime. Distilled / turbo / schnell / LCM variants take it down to 1-4 steps.
@@ -92,7 +92,7 @@ Rectified flow changes the sampler, not the conditioning. Classifier-free guidan
Four names for the same idea: distil a slow many-step model into a fast few-step model.
-- **LCM (Latent Consistency Model)** — train a student that predicts the final `x_0` from any intermediate `x_t` in one step.
+- **LCM (Latent Consistency Model)** — train a student that predicts the final $x_0$ from any intermediate $x_t$ in one step.
- **SDXL Turbo / FLUX schnell** — 1-4 step models trained with adversarial diffusion distillation.
- **SD Turbo** — OpenAI-style Consistency Models adapted to latent diffusion.
@@ -237,7 +237,7 @@ def rectified_flow_train_step(model, x0, optimizer, device):
return loss.item()
```
-Compare with DDPM's noise-prediction loss (Lesson 10): same structure, different target. Instead of predicting the noise `epsilon`, we predict the **velocity** `epsilon - x_0`, which points from data to noise along the straight-line interpolation.
+Compare with DDPM's noise-prediction loss (Lesson 10): same structure, different target. Instead of predicting the noise $\epsilon$, we predict the **velocity** $\epsilon - x_0$, which points from data to noise along the straight-line interpolation.
### Step 4: Euler sampler
@@ -336,7 +336,7 @@ This lesson produces:
| MMDiT | "Multi-modal DiT (SD3)" | Separate weight streams for text and image tokens that share a joint self-attention |
| Single-stream / double-stream | "FLUX trick" | First N blocks double-stream (separate weights per modality), later blocks single-stream (concat + shared weights) for efficiency |
| Rectified flow | "Straight-line noise-to-data" | Linear interpolation between data and noise; network predicts velocity; fewer ODE steps needed at inference |
-| Velocity target | "epsilon - x_0" | The regression target in rectified flow; points from clean data to noise |
+| Velocity target | "$\epsilon - x_0$" | The regression target in rectified flow; points from clean data to noise |
| CFG guidance | "classifier-free guidance" | Mix conditional and unconditional predictions; still used in rectified-flow models |
| Schnell / turbo / LCM | "1-4 step distillation" | Small-step variants distilled from full-quality models; production real-time |
diff --git a/phases/07-transformers-deep-dive/02-self-attention-from-scratch/docs/en.md b/phases/07-transformers-deep-dive/02-self-attention-from-scratch/docs/en.md
index fdd602ab55..c833e951a3 100644
--- a/phases/07-transformers-deep-dive/02-self-attention-from-scratch/docs/en.md
+++ b/phases/07-transformers-deep-dive/02-self-attention-from-scratch/docs/en.md
@@ -11,7 +11,7 @@
- Implement scaled dot-product self-attention from scratch using only NumPy, including query/key/value projections and the softmax-weighted sum
- Build a multi-head attention layer that splits heads, computes parallel attention, and concatenates results
-- Trace how the attention matrix captures token relationships and explain why scaling by sqrt(d_k) prevents softmax saturation
+- Trace how the attention matrix captures token relationships and explain why scaling by $\sqrt{d_k}$ prevents softmax saturation
- Apply causal masking to convert bidirectional attention into autoregressive (decoder-style) attention
## The Problem
@@ -109,11 +109,11 @@ attention-matrix
### Why Scale?
-The dot products grow with dimension dk. If dk = 64, dot products can be in the range of tens, pushing softmax into regions where gradients vanish. The fix: divide by sqrt(dk).
+The dot products grow with dimension $d_k$. If $d_k = 64$, dot products can be in the range of tens, pushing softmax into regions where gradients vanish. The fix: divide by $\sqrt{d_k}$.
-```
-Scaled scores = (Q @ K^T) / sqrt(dk)
-```
+$$
+\text{Scaled scores} = \frac{Q K^T}{\sqrt{d_k}}
+$$
This keeps values in a range where softmax produces useful gradients.
@@ -135,12 +135,15 @@ Now each token has a set of weights saying how much to attend to every other tok
The final output for each token is a weighted sum of all value vectors:
-```
-output_i = sum( attention_weight[i][j] * v_j for all j )
+$$
+\text{output}_i = \sum_j \text{attention\_weight}[i][j] \cdot v_j
+$$
For token 1:
- output_1 = 0.52 * v1 + 0.09 * v2 + 0.07 * v3 + 0.14 * v4 + 0.08 * v5
-```
+
+$$
+\text{output}_1 = 0.52 \cdot v_1 + 0.09 \cdot v_2 + 0.07 \cdot v_3 + 0.14 \cdot v_4 + 0.08 \cdot v_5
+$$
### Full Pipeline
@@ -159,9 +162,9 @@ flowchart LR
Formula in one line:
-```
-Attention(Q, K, V) = softmax( Q @ K^T / sqrt(dk) ) @ V
-```
+$$
+\text{Attention}(Q, K, V) = \text{softmax}\!\left( \frac{Q K^T}{\sqrt{d_k}} \right) V
+$$
```figure
softmax-attention-scaling
@@ -302,7 +305,7 @@ print(f"\nAttn weights (averaged over heads):")
print(attn_weights[0].detach().numpy().round(3))
```
-The key difference: multi-head attention runs multiple attention functions in parallel, each with its own Q, K, V projections of size dk = d_model / n_heads, then concatenates results. This lets the model attend to different relationship types simultaneously.
+The key difference: multi-head attention runs multiple attention functions in parallel, each with its own Q, K, V projections of size $d_k = d_{\text{model}} / n_{\text{heads}}$, then concatenates results. This lets the model attend to different relationship types simultaneously.
## Ship It
@@ -322,7 +325,7 @@ This lesson produces:
| Query (Q) | "The question vector" | A learned projection of the input that represents what information this token is looking for |
| Key (K) | "The label vector" | A learned projection that represents what information this token contains, matched against queries |
| Value (V) | "The content vector" | A learned projection carrying the actual information that gets aggregated based on attention scores |
-| Scaled dot-product attention | "The attention formula" | softmax(QK^T / sqrt(dk)) @ V - scaling prevents softmax saturation in high dimensions |
+| Scaled dot-product attention | "The attention formula" | $\text{softmax}(QK^T / \sqrt{d_k})\,V$ - scaling prevents softmax saturation in high dimensions |
| Self-attention | "The token looks at itself and others" | Attention where Q, K, V all come from the same sequence, letting every position attend to every other position |
| Attention weights | "How much focus" | A probability distribution over positions, produced by softmax over scaled dot products |
| Multi-head attention | "Parallel attention" | Running multiple attention functions with different projections, then concatenating results for richer representations |
diff --git a/phases/07-transformers-deep-dive/04-positional-encoding/docs/en.md b/phases/07-transformers-deep-dive/04-positional-encoding/docs/en.md
index d6dc3744d2..5b14496de5 100644
--- a/phases/07-transformers-deep-dive/04-positional-encoding/docs/en.md
+++ b/phases/07-transformers-deep-dive/04-positional-encoding/docs/en.md
@@ -9,7 +9,7 @@
## The Problem
-Scaled dot-product attention is order-blind. The attention matrix `softmax(Q K^T / √d) V` is computed from pairwise similarities. Shuffle the rows of `X`, get the rows of the output shuffled the same way. Nothing inside attention cares about position.
+Scaled dot-product attention is order-blind. The attention matrix $\text{softmax}(Q K^T / \sqrt{d}) V$ is computed from pairwise similarities. Shuffle the rows of `X`, get the rows of the output shuffled the same way. Nothing inside attention cares about position.
That is not a bug in a bag-of-words model. For language, code, audio, video — anything where order carries meaning — it is fatal.
@@ -29,10 +29,12 @@ As of 2026, essentially every frontier open model uses RoPE: Llama 2/3/4, Qwen 2
Pre-compute a fixed matrix `PE` of shape `(max_len, d_model)`:
-```
-PE[pos, 2i] = sin(pos / 10000^(2i / d_model))
-PE[pos, 2i+1] = cos(pos / 10000^(2i / d_model))
-```
+$$
+\begin{aligned}
+PE[pos, 2i] &= \sin(pos / 10000^{2i / d_{model}}) \\
+PE[pos, 2i+1] &= \cos(pos / 10000^{2i / d_{model}})
+\end{aligned}
+$$
Then `X' = X + PE[:N]` before attention. Each dimension is a sinusoid at a different frequency. The model learns to read position from the phase pattern. Fails beyond `max_len`: nothing told the model what happens at position 2048 when it only saw positions 0–2047.
@@ -40,14 +42,15 @@ Then `X' = X + PE[:N]` before attention. Each dimension is a sinusoid at a diffe
Rotate the Q and K vectors (not embeddings). For a pair of dimensions `(2i, 2i+1)`:
-```
-[q'_2i ] [ cos(pos·θ_i) -sin(pos·θ_i) ] [q_2i ]
-[q'_2i+1 ] = [ sin(pos·θ_i) cos(pos·θ_i) ] [q_2i+1 ]
+$$
+\begin{bmatrix} q'_{2i} \\ q'_{2i+1} \end{bmatrix} = \begin{bmatrix} \cos(pos \cdot \theta_i) & -\sin(pos \cdot \theta_i) \\ \sin(pos \cdot \theta_i) & \cos(pos \cdot \theta_i) \end{bmatrix} \begin{bmatrix} q_{2i} \\ q_{2i+1} \end{bmatrix}
+$$
-θ_i = base^(-2i / d_head), base = 10000 by default
-```
+$$
+\theta_i = base^{-2i / d_{head}}, \quad base = 10000 \text{ by default}
+$$
-Apply the same rotation to keys with position `pos_k`. The dot product `q'_m · k'_n` becomes a function of `(m - n)` alone. That is: **the attention score depends only on the relative distance**, even though the rotation was keyed off absolute positions. Beautiful trick.
+Apply the same rotation to keys with position $pos_k$. The dot product $q'_m \cdot k'_n$ becomes a function of $(m - n)$ alone. That is: **the attention score depends only on the relative distance**, even though the rotation was keyed off absolute positions. Beautiful trick.
Extending RoPE: `base` can be scaled (NTK-aware, YaRN, LongRoPE) to extrapolate to longer contexts without retraining. Llama 3 extended from 8K to 128K context this way.
@@ -55,11 +58,11 @@ Extending RoPE: `base` can be scaled (NTK-aware, YaRN, LongRoPE) to extrapolate
Skip the embedding trick. Bias the attention scores directly:
-```
-attn_score[i, j] = (q_i · k_j) / √d - m_h · |i - j|
-```
+$$
+\text{attn\_score}[i, j] = (q_i \cdot k_j) / \sqrt{d} - m_h \cdot |i - j|
+$$
-Where `m_h` is a head-specific slope (e.g. `1 / 2^(8·h/H)`). Closer tokens get boosted; far tokens get penalized. No training-time cost. The paper shows length extrapolation beats sinusoidal and matches RoPE on its original trained length.
+Where $m_h$ is a head-specific slope (e.g. $1 / 2^{8 \cdot h/H}$). Closer tokens get boosted; far tokens get penalized. No training-time cost. The paper shows length extrapolation beats sinusoidal and matches RoPE on its original trained length.
### What to pick in 2026
@@ -113,7 +116,7 @@ def apply_rope(x, pos, base=10000):
return out
```
-Crucial: apply the same function to Q at position `m` and K at position `n`. Their dot product picks up a `cos((m-n)·θ_i)` factor on every coordinate pair. Attention learns relative position for free.
+Crucial: apply the same function to Q at position `m` and K at position `n`. Their dot product picks up a $\cos((m-n) \cdot \theta_i)$ factor on every coordinate pair. Attention learns relative position for free.
### Step 3: ALiBi slopes and bias
@@ -146,7 +149,7 @@ model = AutoModel.from_pretrained("meta-llama/Llama-3.2-3B")
**Long-context tricks in 2026:**
-- **NTK-aware interpolation.** Rescale `base` to `base * (scale_factor)^(d/(d-2))` when extending from 4K to 16K+.
+- **NTK-aware interpolation.** Rescale `base` to $base \cdot (scale\_factor)^{d/(d-2)}$ when extending from 4K to 16K+.
- **YaRN.** Smarter interpolation that preserves attention entropy on long contexts. Llama 3.1 128K uses it.
- **LongRoPE.** Microsoft's 2024 method that uses evolutionary search to pick per-dimension scale factors. Phi-3-Long uses it.
- **Position interpolation + fine-tuning.** Just shrink positions by the extension factor and fine-tune for 1–5B tokens. Surprisingly effective.
@@ -168,7 +171,7 @@ See `outputs/skill-positional-encoding-picker.md`. The skill picks an encoding s
| Positional encoding | "Tells attention about order" | Any signal added to embeddings or attention that encodes position. |
| Sinusoidal | "The original one" | `sin/cos` at geometric frequencies added to embeddings; doesn't extrapolate. |
| RoPE | "Rotary embeddings" | Rotate Q, K by position-dependent angle; dot product encodes relative distance. |
-| ALiBi | "Linear bias trick" | Add `-m·\|i-j\|` to attention scores; no embedding needed, great extrapolation. |
+| ALiBi | "Linear bias trick" | Add $-m \cdot \vert i-j \vert$ to attention scores; no embedding needed, great extrapolation. |
| base | "RoPE's knob" | The frequency scaler in RoPE; increase to extend context at inference. |
| NTK-aware | "A RoPE scaling trick" | Rescale `base` so high-frequency dims aren't squeezed when context expands. |
| YaRN | "The fancy one" | Per-dimension interpolation+extrapolation that preserves attention entropy. |
diff --git a/phases/07-transformers-deep-dive/13-scaling-laws/docs/en.md b/phases/07-transformers-deep-dive/13-scaling-laws/docs/en.md
index 1555d67f0e..9e97e712ba 100644
--- a/phases/07-transformers-deep-dive/13-scaling-laws/docs/en.md
+++ b/phases/07-transformers-deep-dive/13-scaling-laws/docs/en.md
@@ -14,7 +14,7 @@ When you have C FLOPs of training compute and want the best model, you face two
1. **How many parameters (N)?** Bigger model, higher capacity.
2. **How many training tokens (D)?** More data, better use of capacity.
-FLOPs scale approximately as `6 × N × D`. You can push N up and D down, or D up and N down. Which is better?
+FLOPs scale approximately as $6 \times N \times D$. You can push N up and D down, or D up and N down. Which is better?
Before 2022, the answer was "push N hard." GPT-3 (2020) was 175B parameters trained on ~300B tokens. A ratio of about 1.7 tokens per parameter. The Kaplan scaling laws backed this up.
@@ -30,23 +30,25 @@ Hoffmann et al. (2022), training a small family of models called Chinchilla, fou
From the Chinchilla paper, loss follows:
-```
-L(N, D) = A / N^α + B / D^β + E
-```
+$$
+L(N, D) = \frac{A}{N^\alpha} + \frac{B}{D^\beta} + E
+$$
-- `N` = parameters (non-embedding).
-- `D` = training tokens.
-- `α ≈ 0.34`, `β ≈ 0.28` (roughly symmetric).
-- `E ≈ 1.69`, the irreducible loss ceiling.
-- `A ≈ 406`, `B ≈ 411`.
+- $N$ = parameters (non-embedding).
+- $D$ = training tokens.
+- $\alpha \approx 0.34$, $\beta \approx 0.28$ (roughly symmetric).
+- $E \approx 1.69$, the irreducible loss ceiling.
+- $A \approx 406$, $B \approx 411$.
-Two terms trade against each other as you scale. Take the derivative w.r.t. `N` at fixed compute (C = 6ND) and solve:
+Two terms trade against each other as you scale. Take the derivative w.r.t. $N$ at fixed compute ($C = 6ND$) and solve:
-```
-N_opt ≈ 0.6 × (C/6)^0.5
-D_opt ≈ 0.6 × (C/6)^0.5
-D_opt / N_opt ≈ 20
-```
+$$
+\begin{aligned}
+N_{opt} &\approx 0.6 \times (C/6)^{0.5} \\
+D_{opt} &\approx 0.6 \times (C/6)^{0.5} \\
+D_{opt} / N_{opt} &\approx 20
+\end{aligned}
+$$
Compute-optimal: 20 tokens per parameter.
@@ -134,8 +136,8 @@ See `outputs/skill-training-budget-estimator.md`. The skill picks `(N, D, hours,
## Exercises
1. **Easy.** Run `code/main.py`. Print Chinchilla-optimal `(N, D)` for compute budgets `1e20`, `1e22`, `1e24`. Compare to the real model table.
-2. **Medium.** Implement the Hoffmann loss-as-function-of-compute curve. Plot loss vs `log10(C)` for the compute-optimal frontier. Identify when the law predicts we'd need `>10^28` FLOPs for the next 0.1 reduction in cross-entropy.
-3. **Hard.** Fit your own scaling law on 5 tiny models (100K to 10M params) trained on the same dataset. Estimate `α` and `E`. How well do your exponents match published ones?
+2. **Medium.** Implement the Hoffmann loss-as-function-of-compute curve. Plot loss vs $\log_{10}(C)$ for the compute-optimal frontier. Identify when the law predicts we'd need $>10^{28}$ FLOPs for the next 0.1 reduction in cross-entropy.
+3. **Hard.** Fit your own scaling law on 5 tiny models (100K to 10M params) trained on the same dataset. Estimate $\alpha$ and $E$. How well do your exponents match published ones?
## Key Terms
@@ -143,10 +145,10 @@ See `outputs/skill-training-budget-estimator.md`. The skill picks `(N, D, hours,
|------|-----------------|-----------------------|
| Parameters (N) | "Model size" | Non-embedding weight count; determines capacity. |
| Tokens (D) | "Training data" | Number of training tokens seen; determines how well the parameters get used. |
-| Compute (C) | "FLOPs spent" | Approximately `6 × N × D` for a standard transformer. |
-| Chinchilla-optimal | "D/N ≈ 20" | Ratio that minimizes loss per FLOP of pretraining. |
-| Over-training | "Past Chinchilla" | Spend extra training FLOPs to save inference FLOPs; D/N >> 20. |
-| Irreducible loss | "The floor" | The `E` term in the scaling law; the entropy of the data itself. |
+| Compute (C) | "FLOPs spent" | Approximately $6 \times N \times D$ for a standard transformer. |
+| Chinchilla-optimal | "$D/N \approx 20$" | Ratio that minimizes loss per FLOP of pretraining. |
+| Over-training | "Past Chinchilla" | Spend extra training FLOPs to save inference FLOPs; $D/N \gg 20$. |
+| Irreducible loss | "The floor" | The $E$ term in the scaling law; the entropy of the data itself. |
| Emergent capability | "Sudden jumps at scale" | Often a scorer artifact; continuous loss is smooth. |
| Effective compute | "Training-efficiency multiplier" | Better data / optimizer / architecture multiplies how far a FLOP goes. |
diff --git a/phases/07-transformers-deep-dive/15-attention-variants/docs/en.md b/phases/07-transformers-deep-dive/15-attention-variants/docs/en.md
index 3748c7b996..5089c9cc5a 100644
--- a/phases/07-transformers-deep-dive/15-attention-variants/docs/en.md
+++ b/phases/07-transformers-deep-dive/15-attention-variants/docs/en.md
@@ -9,12 +9,12 @@
## The Problem
-Full attention costs `O(N²)` memory and `O(N²)` compute in sequence length. For a 128K-context Llama 3 70B that is 16 billion attention entries per layer, times 80 layers. Flash Attention (Lesson 12) hides the `O(N²)` activation memory but does not change the arithmetic cost — every token still attends to every other token.
+Full attention costs $O(N^2)$ memory and $O(N^2)$ compute in sequence length. For a 128K-context Llama 3 70B that is 16 billion attention entries per layer, times 80 layers. Flash Attention (Lesson 12) hides the $O(N^2)$ activation memory but does not change the arithmetic cost — every token still attends to every other token.
Three classes of variants change the topology of the attention matrix itself:
-1. **Sliding window attention (SWA).** Each token attends to a fixed window of neighbors, not the full prefix. Memory and compute drop to `O(N · W)` where `W` is the window. Gemma 2/3, Mistral 7B's first layers, Phi-3-Long.
-2. **Sparse / block attention.** Only selected pairs `(i, j)` get scored; the rest are forced to zero weight. Longformer, BigBird, OpenAI sparse transformer.
+1. **Sliding window attention (SWA).** Each token attends to a fixed window of neighbors, not the full prefix. Memory and compute drop to $O(N \cdot W)$ where $W$ is the window. Gemma 2/3, Mistral 7B's first layers, Phi-3-Long.
+2. **Sparse / block attention.** Only selected pairs $(i, j)$ get scored; the rest are forced to zero weight. Longformer, BigBird, OpenAI sparse transformer.
3. **Differential attention.** Compute two attention maps with separate Q/K projections, subtract one from the other. Kills the "attention sink" that bleeds weight into the first few tokens. Microsoft's DIFF Transformer (2024).
These coexist. A 2026 frontier model often mixes them: most layers are SWA-1024, every fifth is global full attention, and a handful are differential heads that clean up retrieval. Gemma 3's 5:1 SWA-to-global ratio is the current textbook default.
@@ -23,7 +23,7 @@ These coexist. A 2026 frontier model often mixes them: most layers are SWA-1024,
### Sliding Window Attention (SWA)
-Each query at position `i` attends only to positions in `[i - W, i]` (causal SWA) or `[i - W/2, i + W/2]` (bidirectional). Tokens outside the window get `-inf` in the score matrix.
+Each query at position $i$ attends only to positions in $[i - W, i]$ (causal SWA) or $[i - W/2, i + W/2]$ (bidirectional). Tokens outside the window get $-\infty$ in the score matrix.
```
full causal: sliding window (W=4):
@@ -39,19 +39,19 @@ positions 0-7 positions 0-7, W=4
7 | x x x x x x x x 7 | x x x x
```
-For `N = 8192` and `W = 1024`, the score matrix has 1024 × 8192 non-zero rows in expectation — an 8× reduction.
+For $N = 8192$ and $W = 1024$, the score matrix has $1024 \times 8192$ non-zero rows in expectation — an 8× reduction.
-**KV cache shrinks with SWA.** Only the last `W` tokens of K and V need to be kept per layer. For a Gemma-3-ish config (1024 window, 128K context), KV cache drops 128×.
+**KV cache shrinks with SWA.** Only the last $W$ tokens of K and V need to be kept per layer. For a Gemma-3-ish config (1024 window, 128K context), KV cache drops 128×.
-**Quality cost.** SWA-only transformers struggle with long-range retrieval. The fix: interleave SWA layers with full-attention layers. Gemma 3 uses 5:1 SWA:global. Mistral 7B used a causal-SWA stack where information "flows forward" through overlapping windows — each layer extends effective receptive field by `W`, and after `L` layers the model can attend `L × W` tokens back.
+**Quality cost.** SWA-only transformers struggle with long-range retrieval. The fix: interleave SWA layers with full-attention layers. Gemma 3 uses 5:1 SWA:global. Mistral 7B used a causal-SWA stack where information "flows forward" through overlapping windows — each layer extends effective receptive field by $W$, and after $L$ layers the model can attend $L \times W$ tokens back.
### Sparse / Block Attention
-Pick an `N × N` sparsity pattern ahead of time. Three canonical shapes:
+Pick an $N \times N$ sparsity pattern ahead of time. Three canonical shapes:
-- **Local + strided (OpenAI sparse transformer).** Attend to the last `W` tokens plus every `stride`-th token before that. Captures both local and long-range at `O(N · sqrt(N))` compute.
+- **Local + strided (OpenAI sparse transformer).** Attend to the last $W$ tokens plus every `stride`-th token before that. Captures both local and long-range at $O(N \cdot \sqrt{N})$ compute.
- **Longformer / BigBird.** Local window + a small set of global tokens (e.g. `[CLS]`) that attend to everyone and are attended by everyone + random-sparse links. Empirical 2× context at matched quality.
-- **Native Sparse Attention (DeepSeek, 2025).** Learn which blocks of `(Q, K)` matter; skip the zero blocks at kernel level. FlashAttention-compatible.
+- **Native Sparse Attention (DeepSeek, 2025).** Learn which blocks of $(Q, K)$ matter; skip the zero blocks at kernel level. FlashAttention-compatible.
Sparse attention is a kernel-engineering story. The math is simple (mask the score matrix); the win comes from never loading the zero entries into SRAM. FlashAttention-3 and the 2026 FlexAttention API make custom sparse patterns first-class in PyTorch.
@@ -61,13 +61,15 @@ Regular attention has an "attention sink" problem: softmax forces every row to s
Differential attention fixes this by computing **two** attention maps and subtracting:
-```
-A1 = softmax(Q1 K1^T / √d)
-A2 = softmax(Q2 K2^T / √d)
-DiffAttn = (A1 - λ · A2) V
-```
+$$
+\begin{aligned}
+A_1 &= \text{softmax}(Q_1 K_1^T / \sqrt{d}) \\
+A_2 &= \text{softmax}(Q_2 K_2^T / \sqrt{d}) \\
+\text{DiffAttn} &= (A_1 - \lambda \cdot A_2) V
+\end{aligned}
+$$
-where `λ` is a learned scalar (typically 0.5–0.8). A1 captures real content weights; A2 captures the sink. Subtraction cancels the sink, reallocates weight to relevant tokens.
+where $\lambda$ is a learned scalar (typically 0.5–0.8). $A_1$ captures real content weights; $A_2$ captures the sink. Subtraction cancels the sink, reallocates weight to relevant tokens.
Reported results (Microsoft 2024): 5–10% lower perplexity, 1.5–2× longer effective context at same trained length, sharper needle-in-haystack retrieval.
@@ -75,12 +77,12 @@ Reported results (Microsoft 2024): 5–10% lower perplexity, 1.5–2× longer ef
| Variant | Compute | KV cache | Quality vs full | Production use |
|---------|---------|----------|-----------------|----------------|
-| Full attention | O(N²) | O(N) per layer | baseline | every model's default layer |
-| SWA (window 1024) | O(N·W) | O(W) per layer | -0.1 ppl, good with global layers | Gemma 2/3, Phi-3-Long |
-| Local + strided sparse | O(N·√N) | mixed | similar to SWA | OpenAI sparse transformer, Longformer |
-| BigBird (local + global + random) | O(N) approx | mixed | matches full at 2× context | early long-context BERT |
-| Native Sparse (DeepSeek-V3.2) | O(N · active fraction) | O(N) | within 0.05 ppl | DeepSeek-V3.2, 2025 |
-| Differential | O(2·N²) | O(2N) | -5 to -10% ppl | DIFF Transformer, early 2026 models |
+| Full attention | $O(N^2)$ | $O(N)$ per layer | baseline | every model's default layer |
+| SWA (window 1024) | $O(N \cdot W)$ | $O(W)$ per layer | -0.1 ppl, good with global layers | Gemma 2/3, Phi-3-Long |
+| Local + strided sparse | $O(N \cdot \sqrt{N})$ | mixed | similar to SWA | OpenAI sparse transformer, Longformer |
+| BigBird (local + global + random) | $O(N)$ approx | mixed | matches full at 2× context | early long-context BERT |
+| Native Sparse (DeepSeek-V3.2) | $O(N \cdot \text{active fraction})$ | $O(N)$ | within 0.05 ppl | DeepSeek-V3.2, 2025 |
+| Differential | $O(2 \cdot N^2)$ | $O(2N)$ | -5 to -10% ppl | DIFF Transformer, early 2026 models |
```figure
gqa-kv-sharing
@@ -185,17 +187,17 @@ See `outputs/skill-attention-variant-picker.md`. The skill picks an attention to
1. **Easy.** Run `code/main.py`. Verify SWA at `window=4` zeroes everything outside the last 4 tokens per row. Verify `window=n` reproduces full causal attention bit-identically.
2. **Medium.** Implement causal SWA with `window=1024` on top of the Lesson 07 capstone. Train for 1,000 steps on tinyshakespeare. How much does val loss regress vs full attention? How much does peak memory drop?
3. **Hard.** Implement a Gemma-3-style 5:1 layer mix (5 SWA, 1 global) in the capstone model. Compare loss, memory, and generation quality against pure-SWA and pure-global baselines at matched parameters.
-4. **Hard.** Implement differential attention with a learned `λ` per head. Train on a synthetic retrieval task (one needle, 2,000 distractors). Measure retrieval accuracy vs a single-attention baseline at matched parameters.
+4. **Hard.** Implement differential attention with a learned $\lambda$ per head. Train on a synthetic retrieval task (one needle, 2,000 distractors). Measure retrieval accuracy vs a single-attention baseline at matched parameters.
## Key Terms
| Term | What people say | What it actually means |
|------|-----------------|-----------------------|
-| Sliding window attention (SWA) | "Local attention" | Each query attends to its last `W` tokens; KV cache shrinks to `O(W)`. |
-| Effective receptive field | "How far back the model sees" | In an `L`-layer SWA stack with window `W`, up to `L × W` tokens. |
+| Sliding window attention (SWA) | "Local attention" | Each query attends to its last $W$ tokens; KV cache shrinks to $O(W)$. |
+| Effective receptive field | "How far back the model sees" | In an $L$-layer SWA stack with window $W$, up to $L \times W$ tokens. |
| Longformer / BigBird | "Local + global + random" | Sparse patterns with a few always-attending global tokens; early long-context approach. |
| Native Sparse Attention | "DeepSeek's kernel trick" | Learn block-level sparsity; skip zero blocks at the kernel level while keeping quality. |
-| Differential attention | "Two maps, one subtracts" | DIFF Transformer: subtract a learned `λ` times a second attention map from the first to cancel attention sinks. |
+| Differential attention | "Two maps, one subtracts" | DIFF Transformer: subtract a learned $\lambda$ times a second attention map from the first to cancel attention sinks. |
| Attention sink | "Weight bleeds to token 0" | Softmax normalization forces rows to sum to 1; uninformative queries dump weight on position 0. |
| FlexAttention | "Mask-as-Python" | PyTorch 2.5+ API that compiles arbitrary mask functions into FlashAttention-shape kernels. |
| Layer type mix | "5:1 SWA-to-global" | Interleave sparse and full attention layers in a stack to keep quality at lower memory. |
diff --git a/phases/08-generative-ai/01-generative-models-taxonomy-history/docs/en.md b/phases/08-generative-ai/01-generative-models-taxonomy-history/docs/en.md
index 2171edd65f..8b65f03ae4 100644
--- a/phases/08-generative-ai/01-generative-models-taxonomy-history/docs/en.md
+++ b/phases/08-generative-ai/01-generative-models-taxonomy-history/docs/en.md
@@ -9,9 +9,9 @@
## The Problem
-A generative model does one job: given training samples drawn from some unknown distribution `p_data(x)`, output new samples that look like they came from the same distribution. Faces, sentences, MIDI files, protein structures — all the same problem if you squint.
+A generative model does one job: given training samples drawn from some unknown distribution $p_{\text{data}}(x)$, output new samples that look like they came from the same distribution. Faces, sentences, MIDI files, protein structures — all the same problem if you squint.
-The rub is that `p_data` lives in a space with millions of dimensions (a 512x512 RGB image is ~786k dimensions), the samples sit on a thin manifold inside that space, and you only have maybe 10M examples. Brute-forcing the density is hopeless. Every generative model is a compromise that trades one hard problem for a slightly less hard one.
+The rub is that $p_{\text{data}}$ lives in a space with millions of dimensions (a 512x512 RGB image is ~786k dimensions), the samples sit on a thin manifold inside that space, and you only have maybe 10M examples. Brute-forcing the density is hopeless. Every generative model is a compromise that trades one hard problem for a slightly less hard one.
Five families have survived the last twelve years. Knowing which compromise each family makes tells you why it wins on some tasks and collapses on others.
@@ -19,13 +19,13 @@ Five families have survived the last twelve years. Knowing which compromise each
-**1. Explicit density, tractable.** Write `log p(x)` as a sum you can actually evaluate. Autoregressive models (PixelCNN, WaveNet, GPT) factorize `p(x) = ∏ p(x_i | x_ A plain autoencoder compresses then reconstructs. It memorizes. It does not generate. Add one trick — force the code to look Gaussian — and you get a sampler. That single trick, the reparameterization of `z = μ + σ·ε`, is why every latent-diffusion and flow-matching image model you use in 2026 has a VAE at the input.
+> A plain autoencoder compresses then reconstructs. It memorizes. It does not generate. Add one trick — force the code to look Gaussian — and you get a sampler. That single trick, the reparameterization of $z = \mu + \sigma \cdot \epsilon$, is why every latent-diffusion and flow-matching image model you use in 2026 has a VAE at the input.
**Type:** Build
**Languages:** Python
@@ -11,9 +11,9 @@
Compress a 784-pixel MNIST digit to a 16-number code, then reconstruct. A plain autoencoder will ace reconstruction MSE but the code space is a lumpy mess. Pick a random point in the code space, decode it, and you get noise. It has no sampler. It is a compression model dressed up.
-What you actually want is: (a) the code space is a clean, smooth distribution you can sample from — say an isotropic Gaussian `N(0, I)`, (b) decoding any sample produces a plausible digit, and (c) the encoder and decoder still compress well. Three goals, one architecture, one loss.
+What you actually want is: (a) the code space is a clean, smooth distribution you can sample from — say an isotropic Gaussian $N(0, I)$, (b) decoding any sample produces a plausible digit, and (c) the encoder and decoder still compress well. Three goals, one architecture, one loss.
-Kingma's 2013 VAE solves this by training the encoder to output a *distribution* `q(z|x) = N(μ(x), σ(x)²)`, pulling that distribution toward the prior `N(0, I)` via a KL penalty, and then sampling `z` from `q(z|x)` before decoding. At inference time, drop the encoder, sample `z ~ N(0, I)`, decode. The KL penalty is what forces the code space to be structured.
+Kingma's 2013 VAE solves this by training the encoder to output a *distribution* $q(z \mid x) = N(\mu(x), \sigma(x)^2)$, pulling that distribution toward the prior $N(0, I)$ via a KL penalty, and then sampling $z$ from $q(z \mid x)$ before decoding. At inference time, drop the encoder, sample $z \sim N(0, I)$, decode. The KL penalty is what forces the code space to be structured.
In 2026 VAEs rarely ship standalone — they have been outclassed by diffusion for raw image quality — but they are the encoder of choice for every latent-diffusion model (SD 1/2/XL/3, Flux, AudioCraft). Learn the VAE and you learn the invisible first layer of every image pipeline you use.
@@ -21,22 +21,24 @@ In 2026 VAEs rarely ship standalone — they have been outclassed by diffusion f
-**Autoencoder.** `z = encoder(x)`, `x̂ = decoder(z)`, loss = `||x - x̂||²`. Code space unstructured.
+**Autoencoder.** $z = \text{encoder}(x)$, $\hat{x} = \text{decoder}(z)$, loss = $\|x - \hat{x}\|^2$. Code space unstructured.
-**VAE encoder.** Outputs two vectors: `μ(x)` and `log σ²(x)`. These define `q(z|x) = N(μ, diag(σ²))`.
+**VAE encoder.** Outputs two vectors: $\mu(x)$ and $\log \sigma^2(x)$. These define $q(z \mid x) = N(\mu, \mathrm{diag}(\sigma^2))$.
-**Reparameterization trick.** Sampling from `q(z|x)` is not differentiable. Rewrite the sample as `z = μ + σ·ε` where `ε ~ N(0, I)`. Now `z` is a deterministic function of `(μ, σ)` plus a non-parameter noise — gradients flow through `μ` and `σ`.
+**Reparameterization trick.** Sampling from $q(z \mid x)$ is not differentiable. Rewrite the sample as $z = \mu + \sigma \cdot \epsilon$ where $\epsilon \sim N(0, I)$. Now $z$ is a deterministic function of $(\mu, \sigma)$ plus a non-parameter noise — gradients flow through $\mu$ and $\sigma$.
**Loss.** Evidence Lower BOund (ELBO), two terms:
-```
-loss = reconstruction + β · KL[q(z|x) || N(0, I)]
- = ||x - x̂||² + β · Σ_i ( σ_i² + μ_i² - log σ_i² - 1 ) / 2
-```
+$$
+\begin{aligned}
+\text{loss} &= \text{reconstruction} + \beta \cdot \mathrm{KL}[q(z \mid x) \,\|\, N(0, I)] \\
+&= \|x - \hat{x}\|^2 + \beta \cdot \sum_i ( \sigma_i^2 + \mu_i^2 - \log \sigma_i^2 - 1 ) / 2
+\end{aligned}
+$$
-Reconstruction pushes `x̂` toward `x`. KL pushes `q(z|x)` toward the prior. They trade off. Small β (<1) = sharper samples, code space less Gaussian. Large β (>1) = cleaner code space, blurrier samples. β-VAE (Higgins 2017) made this knob famous and kicked off disentanglement research.
+Reconstruction pushes $\hat{x}$ toward $x$. KL pushes $q(z \mid x)$ toward the prior. They trade off. Small $\beta < 1$ = sharper samples, code space less Gaussian. Large $\beta > 1$ = cleaner code space, blurrier samples. β-VAE (Higgins 2017) made this knob famous and kicked off disentanglement research.
-**Sampling.** At inference: draw `z ~ N(0, I)`, forward through decoder. One forward pass — no iterative sampling like diffusion.
+**Sampling.** At inference: draw $z \sim N(0, I)$, forward through decoder. One forward pass — no iterative sampling like diffusion.
```figure
vae-latent-grid
@@ -56,7 +58,7 @@ def encode(x, enc):
return mu, log_sigma2
```
-`log σ²` instead of `σ` so the network output is unconstrained (softplus of σ is a trap — gradients die at σ ≈ 0).
+$\log \sigma^2$ instead of $\sigma$ so the network output is unconstrained (softplus of $\sigma$ is a trap — gradients die at $\sigma \approx 0$).
### Step 2: reparameterize and decode
@@ -94,10 +96,10 @@ That is the generative model. Five lines.
## Pitfalls
-- **Posterior collapse.** KL term drives `q(z|x) → N(0, I)` so aggressively that `z` carries no info about `x`. Fix: β-annealing (start β=0, ramp to 1), free bits, or skip the KL on inactive dimensions.
+- **Posterior collapse.** KL term drives $q(z \mid x) \to N(0, I)$ so aggressively that $z$ carries no info about $x$. Fix: β-annealing (start β=0, ramp to 1), free bits, or skip the KL on inactive dimensions.
- **Blurry samples.** The Gaussian decoder likelihood implies MSE reconstruction, which is Bayes-optimal for L2 (the mean) — the mean of a set of plausible digits is a fuzzy digit. Fix: discrete decoder (VQ-VAE, NVAE), or use the VAE only as an encoder and stack diffusion on the latents (this is what Stable Diffusion does).
- **β too large, too early.** See posterior collapse. Start at β≈0.01 and ramp.
-- **Latent dim too small.** 16-D works for MNIST, 256-D for ImageNet 256², 2048-D for ImageNet 1024². Stable Diffusion's VAE compresses 512×512×3 → 64×64×4 (32x downsample factor in spatial area, 32x in channels).
+- **Latent dim too small.** 16-D works for MNIST, 256-D for ImageNet $256^2$, 2048-D for ImageNet $1024^2$. Stable Diffusion's VAE compresses $512 \times 512 \times 3 \to 64 \times 64 \times 4$ (32x downsample factor in spatial area, 32x in channels).
## Use It
@@ -118,7 +120,7 @@ A latent-diffusion model is a VAE with a diffusion model living between encoder
Save `outputs/skill-vae-trainer.md`.
-Skill takes: dataset profile + latent-dim target + downstream use (reconstruction, sampling, or latent-diffusion input) and outputs: architecture choice (plain/β/VQ/RVQ), β schedule, latent dim, decoder likelihood (Gaussian vs categorical), and evaluation plan (recon MSE, KL per dim, Fréchet distance between `q(z|x)` and `N(0, I)`).
+Skill takes: dataset profile + latent-dim target + downstream use (reconstruction, sampling, or latent-diffusion input) and outputs: architecture choice (plain/β/VQ/RVQ), β schedule, latent dim, decoder likelihood (Gaussian vs categorical), and evaluation plan (recon MSE, KL per dim, Fréchet distance between $q(z \mid x)$ and $N(0, I)$).
## Exercises
@@ -130,21 +132,21 @@ Skill takes: dataset profile + latent-dim target + downstream use (reconstructio
| Term | What people say | What it actually means |
|------|-----------------|-----------------------|
-| Autoencoder | Encode-decode network | `x → z → x̂`, learn MSE. Not generative. |
+| Autoencoder | Encode-decode network | $x \to z \to \hat{x}$, learn MSE. Not generative. |
| VAE | AE with a sampler | Encoder outputs a distribution, KL penalty shapes code space. |
-| ELBO | Evidence lower bound | `log p(x) ≥ recon - KL[q(z\|x) \|\| p(z)]`; tight when `q = p(z\|x)`. |
-| Reparameterization | `z = μ + σ·ε` | Rewrites stochastic node as deterministic + pure noise. Enables backprop through sampling. |
-| Prior | `p(z)` | Target distribution for the latent, typically `N(0, I)`. |
-| Posterior collapse | "KL term wins" | Encoder ignores `x`, outputs the prior; decoder must hallucinate. |
-| β-VAE | Tunable KL weight | `loss = recon + β·KL`. Higher β = more disentangled but blurrier. |
-| VQ-VAE | Discrete latent | Replace continuous `z` with nearest codebook vector; enables transformer modelling. |
+| ELBO | Evidence lower bound | $\log p(x) \geq \text{recon} - \mathrm{KL}[q(z \mid x) \,\vert\vert\, p(z)]$; tight when $q = p(z \mid x)$. |
+| Reparameterization | $z = \mu + \sigma \cdot \epsilon$ | Rewrites stochastic node as deterministic + pure noise. Enables backprop through sampling. |
+| Prior | $p(z)$ | Target distribution for the latent, typically $N(0, I)$. |
+| Posterior collapse | "KL term wins" | Encoder ignores $x$, outputs the prior; decoder must hallucinate. |
+| β-VAE | Tunable KL weight | $\text{loss} = \text{recon} + \beta \cdot \mathrm{KL}$. Higher β = more disentangled but blurrier. |
+| VQ-VAE | Discrete latent | Replace continuous $z$ with nearest codebook vector; enables transformer modelling. |
## Production note: the VAE is the hottest path in a diffusion server
-In a Stable Diffusion / Flux / SD3 pipeline the VAE is called twice per request — once to encode (if doing img2img / inpainting) and once to decode. At 1024² the decoder pass is often the single largest activation-memory peak in the whole pipeline because it upsamples `128×128×16` latents back to `1024×1024×3`. Two practical consequences:
+In a Stable Diffusion / Flux / SD3 pipeline the VAE is called twice per request — once to encode (if doing img2img / inpainting) and once to decode. At $1024^2$ the decoder pass is often the single largest activation-memory peak in the whole pipeline because it upsamples $128 \times 128 \times 16$ latents back to $1024 \times 1024 \times 3$. Two practical consequences:
-- **Slice or tile the decode.** `diffusers` exposes `pipe.vae.enable_slicing()` and `pipe.vae.enable_tiling()`. Tiling trades a small seam artifact for `O(tile²)` memory instead of `O(H·W)`. Essential for 1024²+ on consumer GPUs.
-- **bf16 decoder, fp32 numerics for the final resize.** The SD 1.x VAE was released in fp32 and *silently produces NaNs* when cast to fp16 at 1024²+. SDXL ships `madebyollin/sdxl-vae-fp16-fix` — always prefer the fp16-fix variant or use bf16.
+- **Slice or tile the decode.** `diffusers` exposes `pipe.vae.enable_slicing()` and `pipe.vae.enable_tiling()`. Tiling trades a small seam artifact for $O(\text{tile}^2)$ memory instead of $O(H \cdot W)$. Essential for $1024^2$+ on consumer GPUs.
+- **bf16 decoder, fp32 numerics for the final resize.** The SD 1.x VAE was released in fp32 and *silently produces NaNs* when cast to fp16 at $1024^2$+. SDXL ships `madebyollin/sdxl-vae-fp16-fix` — always prefer the fp16-fix variant or use bf16.
## Further Reading
diff --git a/phases/08-generative-ai/06-diffusion-ddpm-from-scratch/docs/en.md b/phases/08-generative-ai/06-diffusion-ddpm-from-scratch/docs/en.md
index d22b0bb943..628510c210 100644
--- a/phases/08-generative-ai/06-diffusion-ddpm-from-scratch/docs/en.md
+++ b/phases/08-generative-ai/06-diffusion-ddpm-from-scratch/docs/en.md
@@ -9,47 +9,47 @@
## The Problem
-You want a sampler for `p_data(x)`. GANs play a minimax game that often diverges. VAEs produce blurry samples from a Gaussian decoder. What you really want is a training objective that is (a) a single stable loss (no saddle point, no minimax), (b) a lower bound on `log p(x)` (so you have likelihoods), and (c) samples that match SOTA quality.
+You want a sampler for $p_{\text{data}}(x)$. GANs play a minimax game that often diverges. VAEs produce blurry samples from a Gaussian decoder. What you really want is a training objective that is (a) a single stable loss (no saddle point, no minimax), (b) a lower bound on $\log p(x)$ (so you have likelihoods), and (c) samples that match SOTA quality.
-Sohl-Dickstein et al. (2015) had a theoretical answer: define a Markov chain `q(x_t | x_{t-1})` that gradually adds Gaussian noise, and train a reverse chain `p_θ(x_{t-1} | x_t)` to denoise. Ho, Jain, Abbeel (2020) showed the loss could be simplified to one line — predict the noise — and cleaned up the math. In 2020 this was a curiosity. In 2021 it produced state-of-the-art samples. In 2022 it became Stable Diffusion. In 2026 it is the substrate.
+Sohl-Dickstein et al. (2015) had a theoretical answer: define a Markov chain $q(x_t \mid x_{t-1})$ that gradually adds Gaussian noise, and train a reverse chain $p_\theta(x_{t-1} \mid x_t)$ to denoise. Ho, Jain, Abbeel (2020) showed the loss could be simplified to one line — predict the noise — and cleaned up the math. In 2020 this was a curiosity. In 2021 it produced state-of-the-art samples. In 2022 it became Stable Diffusion. In 2026 it is the substrate.
## The Concept
-**Forward process `q`.** Add Gaussian noise in `T` small steps. The closed form — the reason the math is tractable — is that the cumulative step is also Gaussian:
+**Forward process $q$.** Add Gaussian noise in $T$ small steps. The closed form — the reason the math is tractable — is that the cumulative step is also Gaussian:
-```
-q(x_t | x_0) = N( sqrt(α̅_t) · x_0, (1 - α̅_t) · I )
-```
+$$
+q(x_t \mid x_0) = \mathcal{N}\left( \sqrt{\bar{\alpha}_t} \cdot x_0,\ (1 - \bar{\alpha}_t) \cdot I \right)
+$$
-where `α̅_t = ∏_{s=1..t} (1 - β_s)` for a schedule of `β_t`. Pick `β_t` from 1e-4 to 0.02 linearly over T=1000 steps and `x_T` is approximately `N(0, I)`.
+where $\bar{\alpha}_t = \prod_{s=1}^{t} (1 - \beta_s)$ for a schedule of $\beta_t$. Pick $\beta_t$ from 1e-4 to 0.02 linearly over T=1000 steps and $x_T$ is approximately $\mathcal{N}(0, I)$.
-**Reverse process `p_θ`.** Learn a neural net `ε_θ(x_t, t)` that predicts the noise that was added. Given `x_t`, denoise by:
+**Reverse process $p_\theta$.** Learn a neural net $\varepsilon_\theta(x_t, t)$ that predicts the noise that was added. Given $x_t$, denoise by:
-```
-x_{t-1} = (1 / sqrt(α_t)) · ( x_t - (β_t / sqrt(1 - α̅_t)) · ε_θ(x_t, t) ) + σ_t · z
-```
+$$
+x_{t-1} = \frac{1}{\sqrt{\alpha_t}} \cdot \left( x_t - \frac{\beta_t}{\sqrt{1 - \bar{\alpha}_t}} \cdot \varepsilon_\theta(x_t, t) \right) + \sigma_t \cdot z
+$$
-where `σ_t` is either `sqrt(β_t)` or a learned variance. The expression is ugly but it is just algebra — solving for `x_{t-1}` given the posterior `q(x_{t-1} | x_t, x_0)` and substituting `x_0` with its noise-predicted estimate.
+where $\sigma_t$ is either $\sqrt{\beta_t}$ or a learned variance. The expression is ugly but it is just algebra — solving for $x_{t-1}$ given the posterior $q(x_{t-1} \mid x_t, x_0)$ and substituting $x_0$ with its noise-predicted estimate.
**Training loss.**
-```
-L_simple = E_{x_0, t, ε} [ || ε - ε_θ( sqrt(α̅_t) · x_0 + sqrt(1 - α̅_t) · ε, t ) ||² ]
-```
+$$
+L_{\text{simple}} = \mathbb{E}_{x_0, t, \varepsilon} \left[ \left\| \varepsilon - \varepsilon_\theta\left( \sqrt{\bar{\alpha}_t} \cdot x_0 + \sqrt{1 - \bar{\alpha}_t} \cdot \varepsilon,\ t \right) \right\|^2 \right]
+$$
-Sample `x_0` from data, pick a random `t`, sample `ε ~ N(0, I)`, compute the noisy `x_t` in one shot via the closed form, and regress on the noise. One loss, no minimax, no KL, no reparameterization tricks.
+Sample $x_0$ from data, pick a random $t$, sample $\varepsilon \sim \mathcal{N}(0, I)$, compute the noisy $x_t$ in one shot via the closed form, and regress on the noise. One loss, no minimax, no KL, no reparameterization tricks.
-**Sampling.** Start `x_T ~ N(0, I)`. Iterate the reverse step from `t = T` to `1`. Done.
+**Sampling.** Start $x_T \sim \mathcal{N}(0, I)$. Iterate the reverse step from $t = T$ to $1$. Done.
## Why it works
Three intuitions:
-1. **Denoising is easy; generating is hard.** At `t=T`, the data is pure noise — the net has to solve a trivial problem. At `t=0`, the net only has to clean up a few pixels. At intermediate `t`, the problem is hard but the net has many gradients flowing through the same weights from every noise level.
+1. **Denoising is easy; generating is hard.** At $t=T$, the data is pure noise — the net has to solve a trivial problem. At $t=0$, the net only has to clean up a few pixels. At intermediate $t$, the problem is hard but the net has many gradients flowing through the same weights from every noise level.
-2. **Score matching in disguise.** Vincent (2011) proved that predicting the noise is equivalent to estimating `∇_x log q(x_t | x_0)`, the *score*. The reverse SDE uses this score to walk up the density gradient — a guided random walk toward high-probability regions.
+2. **Score matching in disguise.** Vincent (2011) proved that predicting the noise is equivalent to estimating $\nabla_x \log q(x_t \mid x_0)$, the *score*. The reverse SDE uses this score to walk up the density gradient — a guided random walk toward high-probability regions.
3. **The ELBO reduces to simple MSE.** The full variational lower bound has a KL term per timestep. With DDPM's parameterization those KL terms simplify to MSE on noise prediction with specific coefficients; Ho dropped the coefficients (calling it "simple" loss) and quality *improved*.
@@ -114,17 +114,17 @@ For a 1-D problem with 40 timesteps and a 24-unit MLP, this learns the two-mode
The net needs to know which timestep it is denoising. Two standard options:
-- **Sinusoidal embedding.** Like Transformer positional encoding. `embed(t) = [sin(t/ω_0), cos(t/ω_0), sin(t/ω_1), ...]`. Pass through an MLP, broadcast into the net.
+- **Sinusoidal embedding.** Like Transformer positional encoding. $\text{embed}(t) = [\sin(t/\omega_0), \cos(t/\omega_0), \sin(t/\omega_1), \ldots]$. Pass through an MLP, broadcast into the net.
- **Film / group-norm conditioning.** Project embedding to per-channel scale/bias (FiLM) at each block.
Our toy code uses sinusoidal → concat. Production U-Nets use FiLM.
## Pitfalls
-- **Schedule matters a lot.** Linear `β` is the DDPM default but cosine schedule (Nichol & Dhariwal, 2021) gives better FID for the same compute. Switch schedules if quality plateaus.
-- **Timestep embedding is fragile.** Passing raw `t` as a float works for toy 1-D but fails for images; always use a proper embedding.
-- **V-prediction vs ε-prediction.** For narrow regimes (very small or very large t), `ε` has poor signal-to-noise. V-prediction (`v = α·ε - σ·x`) is more stable; SDXL, SD3, and Flux use it.
-- **Classifier-free guidance.** At inference, compute both conditional and unconditional `ε`, then `ε_cfg = (1 + w) · ε_cond - w · ε_uncond` with `w ≈ 3-7`. Covered in Lesson 08.
+- **Schedule matters a lot.** Linear $\beta$ is the DDPM default but cosine schedule (Nichol & Dhariwal, 2021) gives better FID for the same compute. Switch schedules if quality plateaus.
+- **Timestep embedding is fragile.** Passing raw $t$ as a float works for toy 1-D but fails for images; always use a proper embedding.
+- **V-prediction vs ε-prediction.** For narrow regimes (very small or very large t), $\varepsilon$ has poor signal-to-noise. V-prediction ($v = \alpha \cdot \varepsilon - \sigma \cdot x$) is more stable; SDXL, SD3, and Flux use it.
+- **Classifier-free guidance.** At inference, compute both conditional and unconditional $\varepsilon$, then $\varepsilon_{\text{cfg}} = (1 + w) \cdot \varepsilon_{\text{cond}} - w \cdot \varepsilon_{\text{uncond}}$ with $w \approx 3\text{-}7$. Covered in Lesson 08.
- **1000 steps is a lot.** Production uses DDIM (20-50 steps), DPM-Solver (10-20 steps), or distillation (1-4 steps). See Lesson 12.
## Use It
@@ -147,19 +147,19 @@ Save `outputs/skill-diffusion-trainer.md`. Skill takes a dataset + compute budge
1. **Easy.** Change T from 40 to 10 in `code/main.py`. How does sample quality (visual histogram of outputs) degrade? At what T does the two-mode structure collapse?
2. **Medium.** Switch from ε-prediction to v-prediction. Re-derive the reverse step. Compare final sample quality.
-3. **Hard.** Add classifier-free guidance. Condition on a class label `c ∈ {0, 1}`, drop it 10% of the time during training, and at sampling time use `ε = (1+w)·ε_cond - w·ε_uncond`. Measure the conditional-mode-hit rate at `w = 0, 1, 3, 7`.
+3. **Hard.** Add classifier-free guidance. Condition on a class label $c \in \{0, 1\}$, drop it 10% of the time during training, and at sampling time use $\varepsilon = (1+w) \cdot \varepsilon_{\text{cond}} - w \cdot \varepsilon_{\text{uncond}}$. Measure the conditional-mode-hit rate at $w = 0, 1, 3, 7$.
## Key Terms
| Term | What people say | What it actually means |
|------|-----------------|-----------------------|
-| Forward process | "Adding noise" | Fixed Markov chain `q(x_t \| x_{t-1})` that destroys the data. |
-| Reverse process | "Denoising" | Learned chain `p_θ(x_{t-1} \| x_t)` that reconstructs the data. |
+| Forward process | "Adding noise" | Fixed Markov chain $q(x_t \mid x_{t-1})$ that destroys the data. |
+| Reverse process | "Denoising" | Learned chain $p_\theta(x_{t-1} \mid x_t)$ that reconstructs the data. |
| β schedule | "The noise ladder" | Per-step variance; linear, cosine, or sigmoid. |
-| α̅ | "Alpha bar" | Cumulative product `∏(1 - β)`; gives closed-form `x_t` from `x_0`. |
-| Simple loss | "MSE on noise" | `\|\|ε - ε_θ(x_t, t)\|\|²`; all variational derivations collapse to this. |
+| α̅ | "Alpha bar" | Cumulative product $\prod(1 - \beta)$; gives closed-form $x_t$ from $x_0$. |
+| Simple loss | "MSE on noise" | $\lVert \varepsilon - \varepsilon_\theta(x_t, t) \rVert^2$; all variational derivations collapse to this. |
| ε-prediction | "Predict noise" | Output is the noise added; standard DDPM. |
-| V-prediction | "Predict velocity" | Output is `α·ε - σ·x`; better conditioning across t. |
+| V-prediction | "Predict velocity" | Output is $\alpha \cdot \varepsilon - \sigma \cdot x$; better conditioning across t. |
| DDPM | "The paper" | Ho et al. 2020; linear β, 1000 steps, U-Net. |
| DDIM | "Deterministic sampler" | Non-Markov sampler, 20-50 steps, same training objective. |
| Classifier-free guidance | "CFG" | Mix conditional and unconditional noise predictions to amplify conditioning. |
@@ -168,11 +168,11 @@ Save `outputs/skill-diffusion-trainer.md`. Skill takes a dataset + compute budge
The DDPM paper runs T=1000 reverse steps. Nobody ships that in production. Every real inference stack picks one of three strategies — and each maps cleanly to production framing of "where is the latency coming from":
-1. **Faster sampler, same model.** DDIM (20-50 steps), DPM-Solver++ (10-20), UniPC (8-16). Drop-in replacement of the reverse loop; the trained `ε_θ` weights are untouched. Cuts latency 20-50×.
+1. **Faster sampler, same model.** DDIM (20-50 steps), DPM-Solver++ (10-20), UniPC (8-16). Drop-in replacement of the reverse loop; the trained $\varepsilon_\theta$ weights are untouched. Cuts latency 20-50×.
2. **Distillation.** Train a student to match the teacher in fewer steps: Progressive Distillation (2 → 1), Consistency Models (arbitrary → 1-4), LCM, SDXL-Turbo, SD3-Turbo. Cuts latency another 5-10×, requires retraining.
3. **Caching and compilation.** `torch.compile(unet, mode="reduce-overhead")`, TensorRT-LLM's diffusion backends, `xformers`/SDPA attention, bf16 weights. Cuts per-step latency ~2×. Stacks with (1) and (2).
-For a production diffusion server the budget conversation is the same as production literature describes for LLMs: latency is `num_steps × step_cost + VAE_decode`, throughput is `batch_size × (num_steps × step_cost)^-1`. TTFT is small (one step); TPOT-equivalent is the full response time because image generation is "all-at-once" from the user's perspective.
+For a production diffusion server the budget conversation is the same as production literature describes for LLMs: latency is $\text{num\_steps} \times \text{step\_cost} + \text{VAE\_decode}$, throughput is $\text{batch\_size} \times (\text{num\_steps} \times \text{step\_cost})^{-1}$. TTFT is small (one step); TPOT-equivalent is the full response time because image generation is "all-at-once" from the user's perspective.
## Further Reading
diff --git a/phases/08-generative-ai/13-flow-matching-rectified-flows/docs/en.md b/phases/08-generative-ai/13-flow-matching-rectified-flows/docs/en.md
index 6c0e31f594..9f687bed40 100644
--- a/phases/08-generative-ai/13-flow-matching-rectified-flows/docs/en.md
+++ b/phases/08-generative-ai/13-flow-matching-rectified-flows/docs/en.md
@@ -9,9 +9,9 @@
## The Problem
-DDPM's reverse process is a 1000-step stochastic walk from `N(0, I)` back to the data distribution. DDIM collapsed it to 20-50 deterministic steps. You want fewer steps — ideally one. The blocker is that the ODE solving the reverse process is stiff; the path is curved.
+DDPM's reverse process is a 1000-step stochastic walk from $N(0, I)$ back to the data distribution. DDIM collapsed it to 20-50 deterministic steps. You want fewer steps — ideally one. The blocker is that the ODE solving the reverse process is stiff; the path is curved.
-If you could train the model such that the path from noise to data was a *straight line*, a single Euler step from `t=1` to `t=0` would work. Flow matching builds this directly: define a straight-line interpolation from `x_1 ∼ N(0, I)` to `x_0 ∼ data`, train a vector field `v_θ(x, t)` to match its time derivative, integrate at inference.
+If you could train the model such that the path from noise to data was a *straight line*, a single Euler step from $t=1$ to $t=0$ would work. Flow matching builds this directly: define a straight-line interpolation from $x_1 \sim N(0, I)$ to $x_0 \sim \text{data}$, train a vector field $v_\theta(x, t)$ to match its time derivative, integrate at inference.
Rectified flow (Liu 2022) goes further: iteratively straighten the paths with a reflow procedure that produces a progressively closer-to-linear ODE. After two reflow iterations, a 2-step sampler matches 50-step DDPM quality.
@@ -23,41 +23,41 @@ Rectified flow (Liu 2022) goes further: iteratively straighten the paths with a
Define:
-```
-x_t = t · x_1 + (1 - t) · x_0, t ∈ [0, 1]
-```
+$$
+x_t = t \cdot x_1 + (1 - t) \cdot x_0, \quad t \in [0, 1]
+$$
-where `x_0 ~ data` and `x_1 ~ N(0, I)`. The time derivative along this straight line is constant:
+where $x_0 \sim \text{data}$ and $x_1 \sim N(0, I)$. The time derivative along this straight line is constant:
-```
-dx_t / dt = x_1 - x_0
-```
+$$
+\frac{dx_t}{dt} = x_1 - x_0
+$$
-Define a neural vector field `v_θ(x_t, t)` and train it to match this derivative:
+Define a neural vector field $v_\theta(x_t, t)$ and train it to match this derivative:
-```
-L = E_{x_0, x_1, t} || v_θ(x_t, t) - (x_1 - x_0) ||²
-```
+$$
+L = \mathbb{E}_{x_0, x_1, t} \, \| v_\theta(x_t, t) - (x_1 - x_0) \|^2
+$$
-This is the **conditional flow matching** loss (Lipman 2023). Training is simulation-free: you never unroll the ODE. Just sample `(x_0, x_1, t)` and regress.
+This is the **conditional flow matching** loss (Lipman 2023). Training is simulation-free: you never unroll the ODE. Just sample $(x_0, x_1, t)$ and regress.
### Sampling
At inference, integrate the learned vector field *backwards* in time:
-```
-x_{t-Δt} = x_t - Δt · v_θ(x_t, t)
-```
+$$
+x_{t-\Delta t} = x_t - \Delta t \cdot v_\theta(x_t, t)
+$$
-Start at `x_1 ~ N(0, I)`, Euler-step down to `t=0`.
+Start at $x_1 \sim N(0, I)$, Euler-step down to $t=0$.
### Rectified flow (Liu 2022)
-Straight-line flow works but the learned paths are *not actually straight* — they curve because many `x_0`s can map to the same `x_1`. Rectified flow's reflow step:
+Straight-line flow works but the learned paths are *not actually straight* — they curve because many $x_0$s can map to the same $x_1$. Rectified flow's reflow step:
-1. Train flow model v_1 with random pairings.
-2. Sample N pairs `(x_1, x_0)` by integrating v_1 from `x_1` to its landing `x_0`.
-3. Train v_2 on those paired examples. Because the pairs are now "ODE-matched", the straight-line interpolant between them is genuinely flatter.
+1. Train flow model $v_1$ with random pairings.
+2. Sample $N$ pairs $(x_1, x_0)$ by integrating $v_1$ from $x_1$ to its landing $x_0$.
+3. Train $v_2$ on those paired examples. Because the pairs are now "ODE-matched", the straight-line interpolant between them is genuinely flatter.
4. Repeat.
In practice 2 reflow iterations get you to near-linear, enabling 2-4 step inference. SDXL-Turbo, SD3-Turbo, LCM are all distilled-from-flow-matching models.
@@ -67,18 +67,18 @@ In practice 2 reflow iterations get you to near-linear, enabling 2-4 step infere
Three reasons:
1. **Simulation-free training** — no ODE unrolling during training, trivial to implement.
-2. **Better loss geometry** — straight paths have consistent signal-to-noise, whereas DDPM ε-loss has bad SNR at edges of the schedule.
+2. **Better loss geometry** — straight paths have consistent signal-to-noise, whereas DDPM $\epsilon$-loss has bad SNR at edges of the schedule.
3. **Faster inference** — 4-8 steps at SDXL-Turbo quality; 1 step with consistency distillation.
## Flow matching vs DDPM — the exact connection
-Flow matching with a Gaussian-conditional path is diffusion *with a specific noise schedule*. Pick the `x_t = α(t) x_0 + σ(t) x_1` schedule and flow matching recovers Stratonovich-reformulated diffusion with `v = α'·x_0 - σ'·x_1`. The two are algebraically equivalent for Gaussian paths.
+Flow matching with a Gaussian-conditional path is diffusion *with a specific noise schedule*. Pick the $x_t = \alpha(t) x_0 + \sigma(t) x_1$ schedule and flow matching recovers Stratonovich-reformulated diffusion with $v = \alpha' \cdot x_0 - \sigma' \cdot x_1$. The two are algebraically equivalent for Gaussian paths.
What flow matching added: the *clarity* of the target (a plain velocity), a cleaner loss, and the license to experiment with non-Gaussian interpolants.
## Build It
-`code/main.py` implements 1-D flow matching on a two-mode Gaussian mixture. The vector field `v_θ(x, t)` is a tiny MLP trained with the straight-line target. At inference, integrate 1, 2, 4, and 20 Euler steps and compare sample quality.
+`code/main.py` implements 1-D flow matching on a two-mode Gaussian mixture. The vector field $v_\theta(x, t)$ is a tiny MLP trained with the straight-line target. At inference, integrate 1, 2, 4, and 20 Euler steps and compare sample quality.
### Step 1: training loss
@@ -111,10 +111,10 @@ Expect the 4-step sampler to already match the 20-step quality — a big deal fo
## Pitfalls
-- **Time parameterization.** Flow matching uses `t ∈ [0, 1]` with `t=0` at data, `t=1` at noise. DDPM uses `t ∈ [0, T]` with `t=0` at data, `t=T` at noise. Same direction, different scale. Papers get this wrong constantly.
+- **Time parameterization.** Flow matching uses $t \in [0, 1]$ with $t=0$ at data, $t=1$ at noise. DDPM uses $t \in [0, T]$ with $t=0$ at data, $t=T$ at noise. Same direction, different scale. Papers get this wrong constantly.
- **Schedule choice.** Rectified flow's straight line is "the" flow-matching schedule, but you can use cosine or logit-normal t-sampling (SD3 does this) for better scale coverage.
- **Reflow cost.** Generating the paired dataset for reflow is a full inference pass per sample. Only do reflow when you really need 1-2 step inference.
-- **Classifier-free guidance still applies.** Just swap ε for v in the linear combination: `v_cfg = (1+w) v_cond - w v_uncond`.
+- **Classifier-free guidance still applies.** Just swap $\epsilon$ for $v$ in the linear combination: $v_\text{cfg} = (1+w) v_\text{cond} - w \, v_\text{uncond}$.
## Use It
@@ -136,21 +136,21 @@ Save `outputs/skill-fm-tuner.md`. Skill takes a diffusion-style model spec and c
## Exercises
1. **Easy.** Run `code/main.py` and compare 1-step vs 20-step MSE vs the true data distribution.
-2. **Medium.** Switch from uniform `t` sampling to logit-normal (concentrates sampling at mid-t). Does the model quality improve?
-3. **Hard.** Implement one reflow iteration: generate paired (x_0, x_1) by integrating the first model, train a second model on the pairs, and compare 1-step sample quality.
+2. **Medium.** Switch from uniform $t$ sampling to logit-normal (concentrates sampling at mid-t). Does the model quality improve?
+3. **Hard.** Implement one reflow iteration: generate paired $(x_0, x_1)$ by integrating the first model, train a second model on the pairs, and compare 1-step sample quality.
## Key Terms
| Term | What people say | What it actually means |
|------|-----------------|-----------------------|
-| Flow matching | "Straight-line diffusion" | Train `v_θ(x, t)` to match `x_1 - x_0` along an interpolant. |
+| Flow matching | "Straight-line diffusion" | Train $v_\theta(x, t)$ to match $x_1 - x_0$ along an interpolant. |
| Rectified flow | "Reflow" | Iterative procedure that straightens learned flows. |
-| Velocity field | "v_θ" | Output of the model — the direction to move `x_t`. |
-| Straight-line interpolant | "The path" | `x_t = (1-t)·x_0 + t·x_1`; trivial target derivative. |
+| Velocity field | "v_θ" | Output of the model — the direction to move $x_t$. |
+| Straight-line interpolant | "The path" | $x_t = (1-t) \cdot x_0 + t \cdot x_1$; trivial target derivative. |
| Euler sampler | "1st order ODE solver" | Simplest integrator; works well when paths are straight. |
-| Logit-normal t | "SD3 sampling" | Concentrate `t` sampling toward mid-values where gradients are strongest. |
-| Consistency distillation | "1-step sampler" | Train a student to map any `x_t` directly to `x_0`. |
-| CFG with velocity | "v-CFG" | `v_cfg = (1+w) v_cond - w v_uncond`; same trick, new variable. |
+| Logit-normal t | "SD3 sampling" | Concentrate $t$ sampling toward mid-values where gradients are strongest. |
+| Consistency distillation | "1-step sampler" | Train a student to map any $x_t$ directly to $x_0$. |
+| CFG with velocity | "v-CFG" | $v_\text{cfg} = (1+w) v_\text{cond} - w \, v_\text{uncond}$; same trick, new variable. |
## Production note: Flux.1-schnell is flow matching at its fastest
diff --git a/phases/08-generative-ai/19-visual-autoregressive-var/docs/en.md b/phases/08-generative-ai/19-visual-autoregressive-var/docs/en.md
index 60d7242106..c00329144f 100644
--- a/phases/08-generative-ai/19-visual-autoregressive-var/docs/en.md
+++ b/phases/08-generative-ai/19-visual-autoregressive-var/docs/en.md
@@ -31,13 +31,13 @@ f -> tokenize at 2x2: token grid z_2 of shape (2, 2)
f -> tokenize at (H/p)x(W/p): token grid z_K of shape (H/p, W/p)
```
-Each z_k uses the same codebook (typical size 4096-16384). The tokenization at each scale is not independent — it is trained so that summing the residuals at each scale reconstructs f:
+Each $z_k$ uses the same codebook (typical size 4096-16384). The tokenization at each scale is not independent — it is trained so that summing the residuals at each scale reconstructs $f$:
-```
-f ≈ upsample(embed(z_1), target_size) + ... + upsample(embed(z_K), target_size)
-```
+$$
+f \approx \text{upsample}(\text{embed}(z_1), \text{target\_size}) + \cdots + \text{upsample}(\text{embed}(z_K), \text{target\_size})
+$$
-This is a **residual VQ** variant. Scale k captures what scales 1..k-1 missed. Decoder takes the sum of all scale embeddings and produces the image.
+This is a **residual VQ** variant. Scale $k$ captures what scales $1..k-1$ missed. Decoder takes the sum of all scale embeddings and produces the image.
The multi-scale VQ tokenizer is trained once (like VQGAN) and then frozen. All the generative work is done by the autoregressive model on top.
@@ -50,9 +50,9 @@ Input sequence structure:
[START, z_1 tokens, z_2 tokens, z_3 tokens, ..., z_K tokens]
```
-Position embeddings encode both scale index and spatial position within the scale. Attention is causal in scale order: token at scale k, position (i, j) can attend to all tokens at scales 1..k and to tokens at scale k itself that come earlier in whatever intra-scale order is used (VAR uses fixed positional attention with no intra-scale causality — all positions within a scale are predicted in parallel).
+Position embeddings encode both scale index and spatial position within the scale. Attention is causal in scale order: token at scale $k$, position $(i, j)$ can attend to all tokens at scales $1..k$ and to tokens at scale $k$ itself that come earlier in whatever intra-scale order is used (VAR uses fixed positional attention with no intra-scale causality — all positions within a scale are predicted in parallel).
-Training loss: at each scale k, predict the tokens z_k given all prior-scale tokens. Cross-entropy loss on the discrete VQ codes. Same structure as GPT except the "sequence" is now scale-structured.
+Training loss: at each scale $k$, predict the tokens $z_k$ given all prior-scale tokens. Cross-entropy loss on the discrete VQ codes. Same structure as GPT except the "sequence" is now scale-structured.
### Generation
@@ -73,7 +73,7 @@ For K = 10 scales, generation is 10 transformer forward passes. Each pass produc
Three structural wins:
1. **Coarse-to-fine aligns with natural image statistics.** Human visual perception and image datasets both exhibit scale-dependent regularities: low-frequency structure is stable and predictable; high-frequency detail is conditional on low-frequency content. Next-scale prediction exploits this.
2. **Parallel generation within scale.** Unlike GPT-style token AR, VAR produces all tokens at a scale in one step. Effective generation length is log-scale instead of linear.
-3. **No generation order bias.** Tokens at scale k see all of scale k-1; there is no "left-of" or "above" bias that forces early tokens to commit before late context is available.
+3. **No generation order bias.** Tokens at scale $k$ see all of scale $k-1$; there is no "left-of" or "above" bias that forces early tokens to commit before late context is available.
### Scaling Law
@@ -121,9 +121,9 @@ This lesson produces `outputs/skill-var-tokenizer-designer.md` — a skill for d
| VAR | "Visual AutoRegressive" | Image generation by next-scale prediction over a pyramid of VQ token grids |
| Next-scale prediction | "Predict coarser, then finer" | The model predicts tokens at increasing resolution scales, conditioning on all previous scales |
| Multi-scale VQ tokenizer | "Residual VQ" | VQ-VAE that produces K token grids of increasing resolution, with decoder summing all scales |
-| Scale k | "Pyramid level k" | One of K resolution levels, from 1x1 at k=1 up to (H/p)x(W/p) at k=K |
+| Scale k | "Pyramid level k" | One of K resolution levels, from 1x1 at $k=1$ up to (H/p)x(W/p) at $k=K$ |
| Parallel-within-scale | "One forward per scale" | All tokens at scale k are predicted in one transformer pass, not autoregressively |
-| Causal-across-scales | "Scale-ordered attention" | Token at scale k can attend to all of scales 1..k but not scales k+1..K |
+| Causal-across-scales | "Scale-ordered attention" | Token at scale $k$ can attend to all of scales $1..k$ but not scales $k+1..K$ |
| Residual VQ | "Additive tokenization" | Each scale's tokens encode the residual left by lower scales; decoder sums all scale embeddings |
| VAR scaling law | "Image GPT scaling" | FID follows a predictable power law in compute, like language models' perplexity |
| HART | "Hybrid VAR + text" | Text-conditional VAR variant combining MaskGIT-style iterative decoding with VAR's scale structure |
diff --git a/phases/10-llms-from-scratch/01-tokenizers/docs/en.md b/phases/10-llms-from-scratch/01-tokenizers/docs/en.md
index f8969be6c0..d71ccb6349 100644
--- a/phases/10-llms-from-scratch/01-tokenizers/docs/en.md
+++ b/phases/10-llms-from-scratch/01-tokenizers/docs/en.md
@@ -135,10 +135,12 @@ GPT-2 introduced this approach. The base vocabulary covers every possible byte.
WordPiece looks similar to BPE but picks merges differently. Instead of raw frequency, it maximizes the likelihood of the training data:
-```
-BPE merge criterion: count(A, B)
-WordPiece merge criterion: count(AB) / (count(A) * count(B))
-```
+$$
+\begin{aligned}
+\text{BPE merge criterion:} \quad & \text{count}(A, B) \\
+\text{WordPiece merge criterion:} \quad & \frac{\text{count}(AB)}{\text{count}(A) \cdot \text{count}(B)}
+\end{aligned}
+$$
BPE asks: "Which pair appears most often?" WordPiece asks: "Which pair appears together more often than you would expect by chance?" This subtle difference produces different vocabularies. WordPiece favors merges where co-occurrence is surprising, not just frequent.
@@ -461,7 +463,7 @@ This lesson produces `outputs/prompt-tokenizer-analyzer.md` -- a reusable prompt
|------|----------------|----------------------|
| Token | "A word" | A unit in the model's vocabulary -- could be a character, subword, word, or multi-word chunk |
| BPE | "Some compression thing" | Byte Pair Encoding -- iteratively merge the most frequent adjacent pair of tokens until the target vocabulary size is reached |
-| WordPiece | "BERT's tokenizer" | Like BPE but merges maximize the likelihood ratio count(AB)/(count(A)*count(B)) instead of raw frequency |
+| WordPiece | "BERT's tokenizer" | Like BPE but merges maximize the likelihood ratio $\text{count}(AB)/(\text{count}(A) \cdot \text{count}(B))$ instead of raw frequency |
| SentencePiece | "A tokenizer library" | A language-agnostic tokenizer that operates on raw Unicode without pre-tokenization, supporting BPE and Unigram algorithms |
| Vocabulary size | "How many words it knows" | The total number of unique tokens: GPT-2 has 50,257, BERT has 30,522, Llama 3 has 128,256 |
| Fertility | "Not a tokenizer term" | Average number of tokens per word -- measures tokenizer efficiency across languages (1.0 is perfect, 3.0 means the model works three times harder) |
diff --git a/phases/10-llms-from-scratch/16-differential-attention-v2/docs/en.md b/phases/10-llms-from-scratch/16-differential-attention-v2/docs/en.md
index eca0a58a9d..f4e7c90c55 100644
--- a/phases/10-llms-from-scratch/16-differential-attention-v2/docs/en.md
+++ b/phases/10-llms-from-scratch/16-differential-attention-v2/docs/en.md
@@ -16,7 +16,7 @@
## The Problem
-Standard softmax attention has a mathematical property that turns into an operational headache at scale. For a query `q`, the attention weights are `softmax(qK^T / sqrt(d))`. Softmax can never produce exact zeros — every non-matching token gets some positive mass. That residual mass is noise, and it scales with context length. At 128k tokens, even if each non-matching token gets only 0.001% of the probability, 127,999 of them combined contribute about 12% of the total. The model has to learn to route around a noise floor that grows with context.
+Standard softmax attention has a mathematical property that turns into an operational headache at scale. For a query $q$, the attention weights are $\text{softmax}(qK^T / \sqrt{d})$. Softmax can never produce exact zeros — every non-matching token gets some positive mass. That residual mass is noise, and it scales with context length. At 128k tokens, even if each non-matching token gets only 0.001% of the probability, 127,999 of them combined contribute about 12% of the total. The model has to learn to route around a noise floor that grows with context.
Empirically this shows up as attention-head interference: hallucinated citations in long-context RAG, lost-in-the-middle failures on 100k-token retrieval tasks, and subtle accuracy degradation on needle-in-haystack benchmarks past 32k. The Differential Transformer paper (arXiv:2410.05258, ICLR 2025) measured the gap: DIFF Transformers hit lower perplexity, higher long-context accuracy, and fewer hallucinations than same-size baselines.
@@ -26,38 +26,40 @@ DIFF V1 had three problems that kept it out of frontier pre-training pipelines.
### The noise floor of softmax
-For a query `q` and keys `K = [k_1, ..., k_N]`, attention weights are:
+For a query $q$ and keys $K = [k_1, \ldots, k_N]$, attention weights are:
-```
-w_i = exp(q . k_i / sqrt(d)) / sum_j exp(q . k_j / sqrt(d))
-```
+$$
+w_i = \frac{\exp(q \cdot k_i / \sqrt{d})}{\sum_j \exp(q \cdot k_j / \sqrt{d})}
+$$
-No `w_i` is ever zero. If `k_i` is completely unrelated to `q`, the score `q . k_i` is not 0 — it fluctuates around zero with variance `||q||^2 / d`. After softmax normalization, each unrelated token still contributes `O(1/N)` to the weighted sum. The total contribution of unrelated tokens is `O((N-1)/N) = O(1)` — not a small quantity.
+No $w_i$ is ever zero. If $k_i$ is completely unrelated to $q$, the score $q \cdot k_i$ is not 0 — it fluctuates around zero with variance $\|q\|^2 / d$. After softmax normalization, each unrelated token still contributes $O(1/N)$ to the weighted sum. The total contribution of unrelated tokens is $O((N-1)/N) = O(1)$ — not a small quantity.
What the model wants is something like a hard top-k: high weight on matching tokens, near-zero weight everywhere else. Softmax is too smooth to do that directly.
### The differential idea
-Split each head's Q and K projections into two: Q = (Q_1, Q_2) and K = (K_1, K_2). Compute two attention maps:
+Split each head's Q and K projections into two: $Q = (Q_1, Q_2)$ and $K = (K_1, K_2)$. Compute two attention maps:
-```
-A_1 = softmax(Q_1 K_1^T / sqrt(d))
-A_2 = softmax(Q_2 K_2^T / sqrt(d))
-```
+$$
+\begin{aligned}
+A_1 &= \text{softmax}(Q_1 K_1^T / \sqrt{d}) \\
+A_2 &= \text{softmax}(Q_2 K_2^T / \sqrt{d})
+\end{aligned}
+$$
Output:
-```
-DiffAttn = (A_1 - lambda * A_2) V
-```
+$$
+\text{DiffAttn} = (A_1 - \lambda A_2) V
+$$
The subtraction cancels whatever noise distribution the two maps share. If both maps have roughly uniform weight on the 127k unrelated tokens (which they will, at random initialization), those cancel. The signal — peaked weight on the few actually relevant tokens — only cancels if it appears in both maps at the same magnitude, which it will not once the model trains.
-`lambda` is a learnable scalar per head, parameterized as `lambda = exp(lambda_q1 dot lambda_k1) - exp(lambda_q2 dot lambda_k2) + lambda_init`. It can be negative. `lambda_init` defaults to a small positive number like 0.8.
+$\lambda$ is a learnable scalar per head, parameterized as $\lambda = \exp(\lambda_{q1} \cdot \lambda_{k1}) - \exp(\lambda_{q2} \cdot \lambda_{k2}) + \lambda_{\text{init}}$. It can be negative. $\lambda_{\text{init}}$ defaults to a small positive number like 0.8.
### Why this matches headed noise-canceling
-Think of two noisy microphones recording the same voice. Both pick up the speaker plus correlated background noise. Subtract one from the other and the shared noise drops out. The voice survives because the two signals differ in phase or amplitude by enough to prevent full cancellation. The per-head `lambda` learns exactly this balance.
+Think of two noisy microphones recording the same voice. Both pick up the speaker plus correlated background noise. Subtract one from the other and the shared noise drops out. The voice survives because the two signals differ in phase or amplitude by enough to prevent full cancellation. The per-head $\lambda$ learns exactly this balance.
### V1 vs V2: the diff
@@ -182,9 +184,9 @@ This lesson produces `outputs/skill-diff-attention-integrator.md`. Given a model
| Term | What people say | What it actually means |
|------|----------------|------------------------|
-| Differential attention | "Two softmaxes minus each other" | Split Q, K into two halves, compute two softmax maps, subtract the second (scaled by lambda) from the first, then multiply by V |
-| Noise floor | "The non-zero tail of softmax" | The O(1/N) weight softmax puts on every unrelated token, which sums to O(1) across long contexts |
-| lambda | "The subtraction scale" | Per-head learnable scalar parameterized as `exp(lq1.lk1) - exp(lq2.lk2) + lambda_init`; can be negative |
+| Differential attention | "Two softmaxes minus each other" | Split Q, K into two halves, compute two softmax maps, subtract the second (scaled by $\lambda$) from the first, then multiply by V |
+| Noise floor | "The non-zero tail of softmax" | The $O(1/N)$ weight softmax puts on every unrelated token, which sums to $O(1)$ across long contexts |
+| lambda | "The subtraction scale" | Per-head learnable scalar parameterized as $\exp(l_{q1} \cdot l_{k1}) - \exp(l_{q2} \cdot l_{k2}) + \lambda_{\text{init}}$; can be negative |
| DIFF V1 | "The ICLR 2025 version" | Original Differential Transformer; halves head dim to preserve parameter count, needs custom kernel, slower decode |
| DIFF V2 | "The January 2026 fix" | Doubles Q heads keeping KV heads; matches baseline decode speed and works with FlashAttention |
| Per-head RMSNorm | "The V1 stabilizer" | Extra norm V1 applied after the difference; V2 removed it to prevent late-training instability |
diff --git a/phases/11-llm-engineering/04-embeddings/docs/en.md b/phases/11-llm-engineering/04-embeddings/docs/en.md
index 495c638c66..f0e755b15e 100644
--- a/phases/11-llm-engineering/04-embeddings/docs/en.md
+++ b/phases/11-llm-engineering/04-embeddings/docs/en.md
@@ -39,9 +39,9 @@ In 2013, Tomas Mikolov and colleagues at Google published Word2Vec. The core ins
The famous result:
-```
-king - man + woman = queen
-```
+$$
+\text{king} - \text{man} + \text{woman} = \text{queen}
+$$
Vector arithmetic on word embeddings captures semantic relationships. The direction from "man" to "woman" is roughly the same as the direction from "king" to "queen." This was the moment the field realized that geometry could encode meaning.
@@ -100,21 +100,21 @@ Given two embedding vectors, three ways to measure how similar they are:
**Cosine similarity**: the cosine of the angle between two vectors. Ranges from -1 (opposite) to 1 (identical direction). Ignores magnitude -- a 10-word sentence and a 500-word document can score 1.0 if they point the same direction. This is the default for 90% of use cases.
-```
-cosine_sim(a, b) = dot(a, b) / (||a|| * ||b||)
-```
+$$
+\text{cosine\_sim}(a, b) = \frac{a \cdot b}{\lVert a \rVert \, \lVert b \rVert}
+$$
**Dot product**: the raw inner product of two vectors. Identical to cosine similarity when vectors are normalized (unit length). Faster to compute. OpenAI's embeddings are normalized, so dot product and cosine give the same ranking.
-```
-dot(a, b) = sum(a_i * b_i)
-```
+$$
+a \cdot b = \sum_i a_i \, b_i
+$$
**Euclidean (L2) distance**: straight-line distance in the vector space. Smaller = more similar. Sensitive to magnitude differences. Use when the absolute position in space matters, not just the direction.
-```
-L2(a, b) = sqrt(sum((a_i - b_i)^2))
-```
+$$
+L2(a, b) = \sqrt{\sum_i (a_i - b_i)^2}
+$$
When to use which:
@@ -134,7 +134,7 @@ Vector databases solve this with Approximate Nearest Neighbor (ANN) algorithms.
2. Top layers are sparse -- long-range connections between distant clusters
3. Bottom layers are dense -- fine-grained connections between nearby vectors
4. Search starts at the top layer, greedily descending to refine
-5. Returns approximate top-k results in O(log n) time instead of O(n)
+5. Returns approximate top-k results in $O(\log n)$ time instead of $O(n)$
HNSW trades a small accuracy loss (typically 95-99% recall) for massive speed gains. At 10 million vectors, brute force takes seconds. HNSW takes milliseconds.
@@ -491,7 +491,7 @@ This lesson produces:
| Embedding | "Text to numbers" | A dense vector where geometric proximity encodes semantic similarity |
| Word2Vec | "The OG embedding" | 2013 model that learned word vectors by predicting context words; proved vector arithmetic encodes meaning |
| Cosine similarity | "How similar are two vectors" | Cosine of the angle between vectors; 1 = identical direction, 0 = orthogonal, -1 = opposite |
-| HNSW | "Fast vector search" | Hierarchical Navigable Small World graph -- multi-layer structure enabling O(log n) approximate nearest neighbor search |
+| HNSW | "Fast vector search" | Hierarchical Navigable Small World graph -- multi-layer structure enabling $O(\log n)$ approximate nearest neighbor search |
| Bi-encoder | "Embed separately, compare fast" | Encodes query and document independently into vectors; enables pre-computation and fast retrieval |
| Cross-encoder | "Slow but accurate reranker" | Processes query-document pair jointly through the full model; higher accuracy, no pre-computation |
| Matryoshka embeddings | "Truncatable vectors" | Embeddings trained so the first N dimensions capture the most important information, enabling variable-size storage |
diff --git a/phases/11-llm-engineering/06-rag/docs/en.md b/phases/11-llm-engineering/06-rag/docs/en.md
index 94ae30ae82..e7571bfbc0 100644
--- a/phases/11-llm-engineering/06-rag/docs/en.md
+++ b/phases/11-llm-engineering/06-rag/docs/en.md
@@ -94,21 +94,21 @@ Given two vectors, how do you measure similarity? Three options:
**Cosine similarity**: the cosine of the angle between two vectors. Ranges from -1 (opposite) to 1 (identical). Ignores magnitude, only cares about direction. This is the default for RAG.
-```
-cosine_sim(a, b) = dot(a, b) / (||a|| * ||b||)
-```
+$$
+\text{cosine\_sim}(a, b) = \frac{a \cdot b}{\lVert a \rVert \, \lVert b \rVert}
+$$
**Dot product**: the raw inner product. Larger vectors get higher scores. Useful when magnitude carries information (longer documents might be more relevant).
-```
-dot(a, b) = sum(a_i * b_i)
-```
+$$
+a \cdot b = \sum_i a_i \, b_i
+$$
**L2 (Euclidean) distance**: straight-line distance in the vector space. Smaller distance = more similar. Sensitive to magnitude differences.
-```
-L2(a, b) = sqrt(sum((a_i - b_i)^2))
-```
+$$
+L2(a, b) = \sqrt{\sum_i (a_i - b_i)^2}
+$$
Cosine similarity is the standard. It handles documents of different lengths gracefully because it normalizes by magnitude. When someone says "vector search," they almost always mean cosine similarity.
diff --git a/phases/11-llm-engineering/13-production-app/docs/en.md b/phases/11-llm-engineering/13-production-app/docs/en.md
index 98ef6ffdb2..95c8155e5f 100644
--- a/phases/11-llm-engineering/13-production-app/docs/en.md
+++ b/phases/11-llm-engineering/13-production-app/docs/en.md
@@ -228,7 +228,7 @@ Before you ship, estimate your monthly cost. This spreadsheet decides if your bu
| Input price per 1M tokens | $5.00 | OpenAI GPT-5 pricing |
| Output price per 1M tokens | $15.00 | OpenAI GPT-5 pricing |
| Cache hit rate | 35% | Measured from cache metrics |
-| Effective daily queries | 32,500 | 50,000 * (1 - 0.35) |
+| Effective daily queries | 32,500 | $50{,}000 \times (1 - 0.35)$ |
**Monthly LLM cost:**
- Input: 32,500 queries/day x 1,500 tokens x 30 days / 1M x $2.50 = **$3,656**
diff --git a/phases/12-multimodal-ai/01-vision-transformer-patch-tokens/docs/en.md b/phases/12-multimodal-ai/01-vision-transformer-patch-tokens/docs/en.md
index 6cb5f97188..700d5b5590 100644
--- a/phases/12-multimodal-ai/01-vision-transformer-patch-tokens/docs/en.md
+++ b/phases/12-multimodal-ai/01-vision-transformer-patch-tokens/docs/en.md
@@ -28,11 +28,11 @@ By 2026 the ViT primitive is the unquestioned foundation. Every open-weights VLM
### Patches as tokens
-Given an image `x` of shape `(H, W, 3)` and a patch size `P`, you carve the image into a grid of `(H/P) x (W/P)` non-overlapping patches. Each patch is a `P x P x 3` cube of pixels. Flatten each cube to a `3 P^2` vector. Apply a shared linear projection `W_E` of shape `(3 P^2, D)` to map each patch into the model's hidden dimension `D`.
+Given an image `x` of shape $(H, W, 3)$ and a patch size $P$, you carve the image into a grid of $(H/P) \times (W/P)$ non-overlapping patches. Each patch is a $P \times P \times 3$ cube of pixels. Flatten each cube to a $3P^2$ vector. Apply a shared linear projection $W_E$ of shape $(3P^2, D)$ to map each patch into the model's hidden dimension $D$.
For the ViT-B/16 canonical config:
-- Resolution 224, patch size 16 → grid 14x14 → 196 patch tokens.
-- Each patch is `16 x 16 x 3 = 768` pixel values, projected to `D = 768`.
+- Resolution 224, patch size 16 → grid $14 \times 14$ → 196 patch tokens.
+- Each patch is $16 \times 16 \times 3 = 768$ pixel values, projected to $D = 768$.
- Add a learnable `[CLS]` token → sequence length 197.
The patch projection is mathematically identical to a 2D convolution with kernel size `P`, stride `P`, and `D` output channels. That is how production code actually implements it — `nn.Conv2d(3, D, kernel_size=P, stride=P)`. The "linear projection" framing is conceptual; the kernel framing is efficient.
@@ -77,15 +77,17 @@ ViT-g/14 (1B params, patch 14, resolution 224 → 256 tokens) and SigLIP SO400m/
The full calculation lives in `code/main.py`. For ViT-B/16 at 224:
-```
-patch_embed = 3 * 16 * 16 * 768 + 768 = 591k
-cls + pos = 768 + 197 * 768 = 152k
-block = 4 * 768^2 (QKVO) + 2 * 4 * 768^2 (MLP) + 2 * 2*768 (LN)
- = 12 * 768^2 + 3k = 7.1M
-12 blocks = 85M
-final LN = 1.5k
-total ≈ 86M
-```
+$$
+\begin{aligned}
+\text{patch\_embed} &= 3 \cdot 16 \cdot 16 \cdot 768 + 768 = 591\text{k} \\
+\text{cls} + \text{pos} &= 768 + 197 \cdot 768 = 152\text{k} \\
+\text{block} &= 4 \cdot 768^2 \,(\text{QKVO}) + 2 \cdot 4 \cdot 768^2 \,(\text{MLP}) + 2 \cdot 2 \cdot 768 \,(\text{LN}) \\
+&= 12 \cdot 768^2 + 3\text{k} = 7.1\text{M} \\
+12 \text{ blocks} &= 85\text{M} \\
+\text{final LN} &= 1.5\text{k} \\
+\text{total} &\approx 86\text{M}
+\end{aligned}
+$$
Ball-park every ViT this way before you load the checkpoint. The backbone size sets your VRAM floor in any downstream VLM.
diff --git a/phases/13-tools-and-protocols/03-parallel-and-streaming-tool-calls/docs/en.md b/phases/13-tools-and-protocols/03-parallel-and-streaming-tool-calls/docs/en.md
index d9043759d0..4d00954899 100644
--- a/phases/13-tools-and-protocols/03-parallel-and-streaming-tool-calls/docs/en.md
+++ b/phases/13-tools-and-protocols/03-parallel-and-streaming-tool-calls/docs/en.md
@@ -102,7 +102,7 @@ Order in the reply does not matter for correctness on OpenAI or Anthropic. Gemin
### Benchmark: sequential vs parallel
-The harness in `code/main.py` simulates three executors with 400, 600, and 800 ms latency. Sequential runs it in 1800 ms total. Parallel runs it in max(400, 600, 800) = 800 ms. The difference is constant, not proportional, so the savings grow with tool count.
+The harness in `code/main.py` simulates three executors with 400, 600, and 800 ms latency. Sequential runs it in 1800 ms total. Parallel runs it in $\max(400, 600, 800) = 800$ ms. The difference is constant, not proportional, so the savings grow with tool count.
Real-world caveat: parallel calls stress downstream APIs. A 10-way fan-out to a rate-limited service will fail. Phase 13 · 17 covers gateway-level backpressure; retry semantics are planned for a future phase.
@@ -126,7 +126,7 @@ This lesson produces `outputs/skill-parallel-call-safety-check.md`. Given a tool
## Exercises
-1. Run `code/main.py` and vary the simulated latencies. Confirm that the parallel-to-sequential ratio is approximately `max/sum` (real runs deviate slightly from the ideal because of thread scheduling, serialization, and harness overhead). At what latency distribution does parallel stop mattering?
+1. Run `code/main.py` and vary the simulated latencies. Confirm that the parallel-to-sequential ratio is approximately $\max/\text{sum}$ (real runs deviate slightly from the ideal because of thread scheduling, serialization, and harness overhead). At what latency distribution does parallel stop mattering?
2. Extend the accumulator to handle a "call was cancelled mid-stream" case by dropping its buffer and emitting a `cancelled` event. What provider documents this case explicitly? Check Anthropic's `content_block_stop` semantics and OpenAI's `finish_reason: "length"` behavior.
diff --git a/phases/16-multi-agent-and-swarms/19-swarm-optimization-pso-aco/docs/en.md b/phases/16-multi-agent-and-swarms/19-swarm-optimization-pso-aco/docs/en.md
index 9fd4d8deed..e4a8879ad6 100644
--- a/phases/16-multi-agent-and-swarms/19-swarm-optimization-pso-aco/docs/en.md
+++ b/phases/16-multi-agent-and-swarms/19-swarm-optimization-pso-aco/docs/en.md
@@ -19,17 +19,19 @@ The same patterns apply to agent *routing* in multi-agent systems. An ACO-style
### PSO refresher (Kennedy & Eberhart 1995)
-Particle Swarm Optimization: population of particles in a continuous search space. Each particle has position `x_i` and velocity `v_i`. Each iteration:
-
-```
-v_i <- w * v_i + c1 * r1 * (p_best_i - x_i) + c2 * r2 * (g_best - x_i)
-x_i <- x_i + v_i
-evaluate fitness(x_i)
-update p_best_i if improved
-update g_best if global best
-```
-
-Where `p_best` is particle's own best, `g_best` is swarm's best, `w, c1, c2` are inertia + cognitive + social weights, `r1, r2` are random factors.
+Particle Swarm Optimization: population of particles in a continuous search space. Each particle has position $x_i$ and velocity $v_i$. Each iteration:
+
+$$
+\begin{aligned}
+v_i &\leftarrow w \cdot v_i + c_1 \cdot r_1 \cdot (p_{\text{best},i} - x_i) + c_2 \cdot r_2 \cdot (g_{\text{best}} - x_i) \\
+x_i &\leftarrow x_i + v_i \\
+&\text{evaluate } \mathrm{fitness}(x_i) \\
+&\text{update } p_{\text{best},i} \text{ if improved} \\
+&\text{update } g_{\text{best}} \text{ if global best}
+\end{aligned}
+$$
+
+Where $p_{\text{best}}$ is particle's own best, $g_{\text{best}}$ is swarm's best, $w, c_1, c_2$ are inertia + cognitive + social weights, $r_1, r_2$ are random factors.
### PSO on LLM outputs — LMPSO
diff --git a/phases/18-ethics-safety-alignment/02-reward-hacking-goodhart/docs/en.md b/phases/18-ethics-safety-alignment/02-reward-hacking-goodhart/docs/en.md
index 7b57e07874..0441e93957 100644
--- a/phases/18-ethics-safety-alignment/02-reward-hacking-goodhart/docs/en.md
+++ b/phases/18-ethics-safety-alignment/02-reward-hacking-goodhart/docs/en.md
@@ -26,14 +26,16 @@ Gao, Schulman, Hilton (2023) measured this directly. Train a "gold" reward model
Goodhart's original formulation: "When a measure becomes a target, it ceases to be a good measure." Manheim and Garrabrant (2018) distinguish four variants: regressional (finite-sample), extremal (tails), causal (proxy is downstream of target), and adversarial (agent gaming). For RLHF, extremal + adversarial are the dominant modes.
-Gao et al. give a functional form. Let `d = sqrt(KL(pi || pi_init))`. Let `R_proxy(d)` be mean proxy reward and `R_gold(d)` mean gold reward. Empirically:
+Gao et al. give a functional form. Let $d = \sqrt{KL(\pi \mid\mid \pi_{init})}$. Let $R_{proxy}(d)$ be mean proxy reward and $R_{gold}(d)$ mean gold reward. Empirically:
-```
-R_proxy(d) = alpha * d - beta_proxy * d^2
-R_gold(d) = alpha * d - beta_gold * d^2
-```
+$$
+\begin{aligned}
+R_{proxy}(d) &= \alpha \cdot d - \beta_{proxy} \cdot d^2 \\
+R_{gold}(d) &= \alpha \cdot d - \beta_{gold} \cdot d^2
+\end{aligned}
+$$
-with `beta_gold > beta_proxy`. Both rise from zero KL, both peak, the gold peak is closer to the origin. At large `d`, gold falls below baseline even while proxy keeps climbing. The proxy-gold gap has the same signature across BoN sampling, PPO, and SFT-to-best.
+with $\beta_{gold} > \beta_{proxy}$. Both rise from zero KL, both peak, the gold peak is closer to the origin. At large $d$, gold falls below baseline even while proxy keeps climbing. The proxy-gold gap has the same signature across BoN sampling, PPO, and SFT-to-best.
This is the "over-optimization curve." It is not a bug in a specific reward model. It is the shape of the problem.
@@ -101,7 +103,7 @@ This lesson produces `outputs/skill-reward-hack-auditor.md`. Given a trained RLH
| Gold reward | "what we actually want" | The target the proxy is a noisy measurement of; in practice, a larger-sample RM or human eval |
| Proxy reward | "the RM" | The scalar used during training; by construction, it is what the optimizer sees |
| Over-optimization curve | "the reward-hacking U-curve" | Proxy climbs, gold peaks then falls as KL from initial policy grows |
-| KL budget | "how far we can drift" | `sqrt(KL(pi \|\| pi_init))`; Gao et al. plot reward against this |
+| KL budget | "how far we can drift" | $\sqrt{KL(\pi \mid\mid \pi_{init})}$; Gao et al. plot reward against this |
| Catastrophic Goodhart | "KL does not save you" | Under heavy-tailed reward error, KL-constrained optimal policy can maximize proxy while providing no gold utility |
| Unfaithful reasoning | "wrong CoT, right answer" | Chain-of-thought that does not causally drive the final prediction |
| Evaluator tampering | "gaming the scorer" | Agent modifies its environment, scratchpad, or the RM's inputs to register success |
diff --git a/phases/19-capstone-projects/10-multi-agent-software-team/docs/en.md b/phases/19-capstone-projects/10-multi-agent-software-team/docs/en.md
index 18b8249adf..27045a91f6 100644
--- a/phases/19-capstone-projects/10-multi-agent-software-team/docs/en.md
+++ b/phases/19-capstone-projects/10-multi-agent-software-team/docs/en.md
@@ -78,7 +78,7 @@ Coder A Coder B Coder C Coder D (4 parallel)
6. **Tester.** Gemini 2.5 Pro runs the test suite in a clean sandbox. Captures artifacts. Emits `test_passed` or `test_failed` with stacktraces. Failed tests loop back to the coder owning the failing subtask.
-7. **Handoff accounting.** Every message crossing a role boundary gets a span in Langfuse with payload size and model used. Compute per-subtask token amplification (coder_tokens + reviewer_tokens + tester_tokens + architect_share / coder_tokens).
+7. **Handoff accounting.** Every message crossing a role boundary gets a span in Langfuse with payload size and model used. Compute per-subtask token amplification $(\text{coder\_tokens} + \text{reviewer\_tokens} + \text{tester\_tokens} + \text{architect\_share} / \text{coder\_tokens})$.
8. **Eval.** Run on 50 SWE-bench Pro issues. Compare pass@1 and $-per-solved-issue against a single-agent baseline (one Sonnet 4.7 in a single worktree).
diff --git a/site/lesson.html b/site/lesson.html
index b43f0b759f..c615fc8467 100644
--- a/site/lesson.html
+++ b/site/lesson.html
@@ -18,7 +18,10 @@
+
+
+
@@ -1626,6 +1649,7 @@
+
@@ -2192,6 +2216,8 @@
var blockquoteLines = [];
var mermaidBlocks = 0;
var inLabChallenge = false;
+ var inMathBlock = false;
+ var mathBlockLines = [];
function flushLabChallenge() {
if (!inLabChallenge) return '';
@@ -2261,6 +2287,17 @@
for (var i = 0; i < lines.length; i++) {
var line = lines[i];
+ if (inMathBlock) {
+ if (line.match(/^\s*\$\$\s*$/)) {
+ inMathBlock = false;
+ out += renderMathBlock(mathBlockLines.join('\n'));
+ mathBlockLines = [];
+ } else {
+ mathBlockLines.push(line);
+ }
+ continue;
+ }
+
if (inCodeBlock) {
if (line.match(/^```\s*$/)) {
inCodeBlock = false;
@@ -2300,6 +2337,25 @@
continue;
}
+ // Block math: $$ ... $$ on one line, or a lone $$ opening a multi-line
+ // display block closed by a matching lone $$.
+ var mathOneLine = line.match(/^\s*\$\$(.+?)\$\$\s*$/);
+ if (mathOneLine) {
+ out += flushList();
+ out += flushTable();
+ out += flushBlockquote();
+ out += renderMathBlock(mathOneLine[1].trim());
+ continue;
+ }
+ if (line.match(/^\s*\$\$\s*$/)) {
+ out += flushList();
+ out += flushTable();
+ out += flushBlockquote();
+ inMathBlock = true;
+ mathBlockLines = [];
+ continue;
+ }
+
if (line.match(/^\s*\|.*\|\s*$/)) {
out += flushList();
out += flushBlockquote();
@@ -2440,6 +2496,9 @@
out += flushTable();
out += flushBlockquote();
out += flushLabChallenge();
+ if (inMathBlock && mathBlockLines.length) {
+ out += renderMathBlock(mathBlockLines.join('\n'));
+ }
return out;
}
@@ -2475,18 +2534,64 @@
return cells.map(function (c) { return c.trim(); });
}
+ // KaTeX renders math to HTML strings synchronously. These helpers degrade
+ // gracefully to the raw TeX if KaTeX failed to load or the source is invalid,
+ // so a bad formula never blanks the lesson.
+ function renderMathInline(tex) {
+ if (typeof katex === 'undefined') {
+ return '' + escapeHtml(tex) + '';
+ }
+ try {
+ return katex.renderToString(tex, { displayMode: false, throwOnError: false });
+ } catch (e) {
+ return '' + escapeHtml(tex) + '';
+ }
+ }
+
+ function renderMathBlock(tex) {
+ if (typeof katex === 'undefined') {
+ return '
';
+ }
+ }
+
function inlineFormat(text) {
+ // Protect inline-code and inline-math spans with private-use sentinels
+ // BEFORE escaping/markdown so their contents survive verbatim. Code is
+ // stashed first so a $ inside `code` is never mistaken for math.
+ var stash = [];
+ function keep(html) { stash.push(html); return '' + (stash.length - 1) + ''; }
+
+ text = text.replace(/`([^`]+)`/g, function (m, code) {
+ return keep('' + escapeHtml(code) + '');
+ });
+
+ // Inline math: $...$ using the Pandoc rule — opener not followed by a
+ // space, closer not preceded by a space and not followed by a digit.
+ // This leaves prose dollar amounts ($50,000, $0.25/$2.50) untouched.
+ text = text.replace(/\$(?=\S)([^$\n]+?)\$(?!\d)/g, function (whole, tex) {
+ if (/\s$/.test(tex)) return whole;
+ return keep(renderMathInline(tex));
+ });
+
text = text.replace(/&/g, '&').replace(//g, '>').replace(/"/g, '"').replace(/'/g, ''');
text = text.replace(/\*\*\*(.+?)\*\*\*/g, '$1');
text = text.replace(/\*\*(.+?)\*\*/g, '$1');
text = text.replace(/\*(.+?)\*/g, '$1');
- text = text.replace(/`([^`]+)`/g, '$1');
text = text.replace(/\[([^\]]+)\]\(([^)]+)\)/g, function (m, label, href) {
if (/^https?:\/\/|^mailto:/i.test(href)) {
return '' + label + '';
}
return label;
});
+
+ text = text.replace(/(\d+)/g, function (m, i) { return stash[+i]; });
return text;
}