Technical — the stages

1. Collect & split — set aside evaluation data before making any modeling choices (§3).
2. Preprocess & engineer features — scaling, encoding, missing values. Every transformation’s parameters must be learned on training data only.
3. Train — the learning algorithm adjusts the model’s parameters (weights) to minimize a loss.
4. Validate — compare candidate models and hyperparameters (values you choose, not learn — §7).
5. Test — one final unbiased estimate on data that influenced zero decisions.
6. Deploy & monitor — real data drifts; performance decays without retraining (Chapter 5).

Technical

The paradigms differ only in where the error signal comes from. Supervised learning compares the prediction to a human-provided target. Self-supervised supervision is automatic: hide part of the input, make the model reconstruct it — the data is its own teacher at massive scale, which is why LLMs can be trained on trillions of tokens without human labeling.

Technical — vocabulary

• Example (row): one observation. Feature (x): an input attribute. Label (y): the value to predict.
• Training set (~70%) — the algorithm adjusts parameters here.
• Validation set (~15%) — used to compare models, pick hyperparameters, choose the epoch to stop at.
• Test set (~15%) — touched once, at the end. Proportions are conventions, not laws; with little data, use cross-validation instead (§7).

Technical — bias & variance

• Bias = systematic error from a too-rigid hypothesis class (the straight line can’t bend). High bias → underfitting.
• Variance = sensitivity to the particular training sample — retrain on a different sample and the wiggly line changes wildly. High variance → overfitting.
• Expected test error ≈ bias² + variance + irreducible noise. Decreasing one typically raises the other — the bias–variance tradeoff, controlled by model capacity and regularization (§5).

Simplification: the classic tradeoff story is a useful mental model, but modern heavily over-parameterized networks can have low test error despite near-zero training error (“double descent”). The diagnosis via train vs validation curves below stays valid regardless.

Technical

Generalization = performance on unseen data; the train–validation gap measures it. Regularized objective (add a penalty to the loss):

J_reg(w) = Loss(w) + λ Σ w_i²   (L2 / ridge / weight decay)

J_reg(w) = Loss(w) + λ Σ |w_i|   (L1 / lasso)

• L2 shrinks all weights toward zero — smoother functions, no exact zeros. Default in deep learning (“weight decay”).
• L1 drives some weights to exactly zero → feature selection / sparse models.
• Early stopping — stop at the validation-loss minimum (the sweet spot in §4’s plot).
• Dropout (deep nets) — randomly disable neurons during training so no single path is relied upon.
• Data augmentation — enlarges effective data; the most “physical” regularizer of all.

λ is a hyperparameter: too small → overfit; too large → underfit. Choose it on validation data only.

Technical — the confusion matrix

Precision = TP / (TP + FP)  ·  “when I alarm, how often am I right?”

Recall = TP / (TP + FN)  ·  “of all real cases, how many did I catch?”

F1 = 2·P·R / (P + R)  ·  harmonic mean — punishes imbalance between P and R

Choosing: cancer screening / fraud → favor recall (missed cases are costly); spam filter / auto-moderation → favor precision (false alarms erode trust); balanced costs → accuracy or F1. ROC-AUC summarizes ranking quality across thresholds — useful, but misleading on extreme imbalance.

Technical — regression metrics

MAE = (1/n) Σ |y_i − ŷ_i|  ·  robust to outliers, same units as y

MSE = (1/n) Σ (y_i − ŷ_i)²  ·  punishes large errors hard; RMSE = √MSE restores units

R² = 1 − SS_res / SS_tot  ·  variance explained vs. always predicting the mean (can be negative!)

Rule of thumb: outliers matter → MAE; large errors are disproportionately costly → RMSE; always report at least two metrics and the residual distribution.

Technical — k-fold CV (k = 5)

Parameter vs hyperparameter: parameters (weights) are learned by the optimizer inside a fold; hyperparameters (λ, tree depth, learning rate, k in k-NN) are chosen by you across folds.

Tuning strategies: grid search (try the Cartesian product — explodes fast) vs random search (sample configurations — usually finds good regions with far fewer runs). Nested CV is the rigorous version where tuning happens strictly inside training folds. Whatever the strategy: any feature selection or preprocessing must live inside each fold, or you leak (§3).

Technical

• Weights (w) — learned multipliers; sign = excite/inhibit, magnitude = importance. This is what training updates.
• Bias (b) — learned offset; lets the neuron fire even with all-zero input. Shifts the decision threshold.
• Activation f — the nonlinearity. Without it, any stack of layers collapses algebraically into a single linear map: f(W₂·(W₁x)) = (W₂W₁)x — depth would buy nothing. Common choices: ReLU max(0, z) (default in CNNs), GELU (transformers), sigmoid/softmax (output layers).
• Layer — a column of neurons that each see the same inputs. A dense layer with n inputs and m neurons = an n × m weight matrix + m biases.

Technical — one layer

z = Wx + b   ·   a = f(z)

Stack L layers and feed each layer’s activations as the next layer’s inputs. Shapes: a batch of inputs (B, n₀) flows through (B, n₁) → … → (B, n_L).

Technical

Regression — Mean Squared Error:

MSE = (1/n) Σ (y_i − ŷ_i)²

Classification — cross-entropy. If the model assigns probability p to the correct class:

L = −ln(p)

Confidently correct → p ≈ 1 → loss ≈ 0. Confidently wrong → p → 0 → loss → ∞. The logarithm punishes confident mistakes disproportionately, which keeps gradients informative.

Connect to our GPT: next-token prediction is classification over the vocabulary (65 classes for TinyShakespeare). That is why the untrained model’s loss is ln(65) ≈ 4.17 — the log of uniform probability 1/65 (see the Transformers sheet below, §9).

Technical — the chain rule, in one worked example

Model: ŷ = w·x. Truth: y = 8. Input: x = 2. Current weight: w = 3. Loss: L = (ŷ − y)².

1. Forward: ŷ = 3 × 2 = 6 → L = (6 − 8)² = 4.
2. dL/dŷ = 2(ŷ − y) = 2(6 − 8) = −4.
3. dŷ/dw = x = 2.
4. Chain rule: dL/dw = (−4) × 2 = −8. Interpretation: increasing w decreases the loss strongly (negative gradient points downhill).
5. Update with η = 0.1: w ← 3 − 0.1×(−8) = 3.8.
6. Re-forward: ŷ = 7.6, L = 0.16. Loss fell 4 → 0.16 in one step.

Backpropagation is exactly this, applied to every weight of every layer simultaneously — the chain rule threaded through the whole computational graph, sharing intermediate results so the total cost is only ~2× the forward pass. No weight is “told” anything manually; every gradient is computed mechanically from the loss.

Technical

Learning rate — the hyperparameter that matters most:

• Too large → loss oscillates or explodes (NaNs).
• Too small → glacial progress, risk of getting stuck.
• Schedules — warmup (ramp from ~0 over the first ~100 steps: early gradients are wild) then decay (cosine or step-down: fine adjustments need fine steps).
• Connect to our GPT: train.py uses AdamW, betas (0.9, 0.95), grad clipping at 1.0, and a 100-step linear warmup into cosine decay — every element of this section, live.

Technical — the three words

• Batch (iteration) — one forward + backward + update pass on B examples.
• Epoch — one full pass over the training set: iterations/epoch = ⌈N / batch_size⌉.
• Step — usually synonymous with iteration (some papers count optimizer steps under gradient accumulation differently).

Example: N = 10,000, batch 64 → 157 iterations per epoch; 20 epochs = 3,140 iterations.

Technical — the operation

Place the K×K kernel over a patch of the input; multiply element-wise; sum; write one output number. Slide by stride S. Padding P zeroes a border to control output size. One kernel produces one feature map (channel); a conv layer applies F independent kernels, giving F feature maps.

output size = ⌊(N − K + 2P) / S⌋ + 1

Parameters per layer: F × (K × K × C_in) + F — independent of image size!

Stride & padding effects

Technical

• Max pool 2×2, stride 2 — halves H and W; no learnable parameters; per-channel.
• Average pooling — smoother; global average pooling (GAP) collapses each final map to a single number, replacing big flatten+FC heads.
• Receptive field — the input region influencing one output neuron. With two stacked 3×3 convs: the second layer sees 5×5 of the input; add a 2×2 pool and the next conv spans more. Deep stacks trade resolution for semantic reach — exactly what classification needs.

Why the shapes work out

• Pad-1 convs preserve 28→28 and 14→14; pools do all the shrinking (÷2 each time).
• Channel count doubles (1→16→32) as spatial size halves — the standard budget shift.
• Flatten turns the final (32, 7, 7) volume into a 1,568-vector; the linear head maps it to 10 class logits; cross-entropy + argmax finish the job.
• Modern refinements: BatchNorm after each conv (stabilizes activations), ResNet skip connections out = f(x) + x (lets 50–150-layer nets train — He et al., 2015), GAP heads instead of big FC layers.

Technical — the two modes

Why a low LR when fine-tuning? Pretrained weights sit in a good basin; big steps destroy them. Typical recipe: head at 1e-3, backbone at 1e-5 (discriminative LRs).

Connect forward: this is precisely the bridge to LLM fine-tuning — LoRA/QLoRA (your GPT Phase 2) is parameter-efficient fine-tuning applied to language models instead of CNN backbones.

Technical

• Augmentation — random flips, crops, rotations, color jitter, Cutout/MixUp: label-preserving distortions that multiply effective data. Rule: the transform must not change the label (a flipped “6” is not a “9”; digit recognition must skip horizontal flips).
• Dropout — randomly zero activations in FC heads (typically p = 0.2–0.5).
• Weight decay — L2 penalty via AdamW.
• Early stopping — keep the checkpoint at the validation minimum.
• BatchNorm — normalizes layer inputs over the batch; stabilizes and mildly regularizes. (Contrast with the LayerNorm used in transformers — per-token, batch-independent — the Transformers sheet §6 below.)

Technical

• Character-level — vocab = every distinct character (TinyShakespeare: 65). Simple, but sequences get long and every token carries little meaning.
• BPE (byte-pair encoding) — start from 256 bytes; repeatedly merge the most frequent adjacent pair into a new token. GPT-family models use BPE variants; vocabularies run ~50K–200K. Our repo ships both: tokenizer.py (char) and the BPE you built against a pytest spec.
• The vocabulary is fixed before training — the model never learns new tokens, only what the existing ones mean.

Connect to model.py: tok_emb = nn.Embedding(vocab_size, n_embd) — row id of that matrix is the token’s learned vector.

Technical

x₀ = Embedding_tok(id) + Embedding_pos(position)

• Learned absolute positions (GPT-2, our model.py) — a block_size × n_embd table; position 300 simply doesn’t exist if block_size = 256. This is why generate() crops the context.
• Sinusoidal (original Transformer) — fixed geometric pattern; extrapolates a bit beyond trained lengths.
• RoPE (modern LLMs — Llama, Qwen) — rotates Q/K pairs by position-dependent angles, so relative distance appears in the attention score; the basis of modern long-context tricks. Simplification: RoPE is applied inside attention, not added at input — details in the RoPE paper.

Technical

All three are learned linear projections of the same input x — three weight matrices updated by backprop:

Q = x·W_Q    K = x·W_K    V = x·W_V

In model.py these are one fused nn.Linear(n_embd, 3·n_embd) split into three — mathematically identical, faster on GPU. The retrieval analogy is soft: Q·K is content-based addressing with graded, differentiable matches — not exact key lookup.

The formula

Attention(Q, K, V) = softmax( Q·K^T / √d_k ) · V

Technical

Split d_model across h heads: each head works in d_head = d_model/h dimensions, runs the §4 pipeline independently, then all head outputs are concatenated and mixed by a final projection W_O:

MHA(x) = Concat(head₁,…,head_h)·W_O

Our GPT: n_embd = 384, n_head = 6 → head_dim = 64. Same total computation as one 384-dim attention, but six specialized subspaces instead of one. Simplification: heads are not guaranteed to be cleanly interpretable; “each head learns one linguistic rule” is an idealized story that only sometimes holds in practice.

Technical — the three remaining parts

• FFN / MLP — per-token, position-wise: expand 4× (384 → 1536 in our GPT), GELU, project back. This is where most parameters live (~2/3 of the block) and where much “knowledge” is stored. Attention = communicate; MLP = compute.
• Residual connections — x = x + attn(ln x), x = x + mlp(ln x) (pre-norm, as in GPT-2 and our model.py). The addition creates a gradient highway: during backprop, gradients flow through the + unchanged, letting deep stacks train. Also: each block only has to learn an increment on the representation.
• LayerNorm — normalizes each token’s vector across its features (mean ≈ 0, unit-ish variance, learned scale/shift). Keeps activation scales stable across depth. Placed before each sublayer (pre-norm). Contrast: BatchNorm normalizes across the batch — awkward for variable-length sequences, which is why transformers use LN.

Technical — one axis: the attention pattern

• Encoder-only — bidirectional attention; trained with masked-token / contrastive objectives; outputs representations, not text. (BERT, embedding models — including RAG retrievers!)
• Decoder-only — causal attention; trained on next-token prediction; generates. (GPT, Llama, Qwen — and everything in your roadmap.) Dominant because one objective scales endlessly.
• Encoder–decoder — encoder understands, decoder generates with cross-attention into encoder states. (T5, BART; original Transformer was this form for translation.)

Technical

A lower-triangular matrix of allowed positions is added to the attention scores before softmax: future positions get −∞, so softmax assigns them exactly 0. In model.py: torch.tril(...), registered as a buffer, sliced to the current T each forward.

Technical — targets are the input, shifted

In data.py: x = stream[i:i+T], y = stream[i+1:i+T+1]. Position t of x must predict the token at position t of y. The model’s head produces logits (one raw score per vocab entry); cross-entropy with the true next token gives the loss; −ln p is the same formula as the CNN sheet §3, just over 65 (or 50K) classes.

The famous sanity check: an untrained model outputs ≈uniform logits → p(correct) ≈ 1/65 → loss = ln(65) ≈ 4.17. Our train.py prints this before training begins. Watch loss go 4.17 → ~1.5: that’s the model moving from uniform guessing to confidently right — in a loss sense; decoding behavior is §11.

Technical — a precise diff

The activations inside the network change with every input in both modes; what never changes at inference is the parameter set. That distinction — activations vs parameters — resolves the classic beginner confusion “but the model computes different things, so isn’t it learning?”

Technical — the loop in generate()

• Temperature divides logits before softmax: <1 sharpens toward greedy, >1 flattens toward uniform; 0 (=argmax) is deterministic but repetitive.
• Top-k keeps only the k highest-probability tokens before sampling — kills the long low-quality tail. (Top-p/nucleus: keep smallest set with cumulative mass ≥ p.)
• Each iteration runs a full forward pass over the cropped context — correct, simple, O(T²) total for T tokens. (Production serving adds a KV cache to reuse past computations; that’s an engineering optimization, not a change to the math — see LLM Fundamentals, Stage 3, §10.)

Technical — from file to tensor

1. Decode — JPEG/PNG → pixel array. Each value originally 0–255 (uint8).
2. Resize / crop — networks need fixed input size (e.g., 224×224). Resize distorts; resize-then-center-crop preserves aspect ratio.
3. Scale — divide by 255 → [0, 1]. Networks train badly on large raw magnitudes.
4. Normalize — subtract dataset mean, divide by std (per channel), e.g. ImageNet’s mean=[0.485, 0.456, 0.406]. Matches the distribution the backbone was pretrained on.
5. Batch & reorder — PyTorch conv expects (B, C, H, W) — channels first. Getting (B, H, W, C) instead is the classic silent bug.

Augmentation lives here too — random flips/crops/jitter (Stage 2, §11) are preprocessing steps applied on the fly each epoch.

Technical — outputs and losses

• Classification — output: one probability vector over classes. Loss: cross-entropy.
• Detection — output: N boxes, each [x, y, w, h, class scores, objectness]. Loss: box regression (e.g., smooth-L1) + classification, plus the matching problem: which prediction counts for which object? (Solved by IoU matching → §6.)
• Segmentation — output: per-pixel class map (semantic) or per-object masks (instance). Loss: pixel-wise cross-entropy / Dice. Architectures: encoder–decoder (U-Net style) that upsamples back to full resolution.

Technical

The backbone is the convolutional trunk (ResNet, CSPDarknet, EfficientNet…) that maps the input image to feature maps. It is exactly the Stage 2 §9 architecture minus the classifier head. Key idea: as depth grows, spatial resolution shrinks while each location summarizes a larger region — features become semantic (“this area contains fur and whiskers”) rather than geometric (“a diagonal edge here”).

• Feature pyramid — detectors usually consume several scales of the backbone (e.g., strides 8, 16, 32): fine maps for small objects, coarse maps for large ones (FPN, Lin et al. 2017).
• Receptive field — a detection head cell must “see” enough image to recognize its object; deep backbones provide this for free (Stage 2 §8).
• The detector head sits on top; the backbone is usually initialized from ImageNet pretraining (§9).

Technical — the mechanism

• Backbone produces the grid; each cell predicts B boxes + objectness (probability that a box contains any object) + class scores.
• Box coordinates are predicted relative to the cell (offsets + sizes) — easier to learn than absolute pixels, and each cell only proposes objects centered on it.
• One combined loss: localization (box regression) + objectness (binary) + classification — trained end-to-end.
• Consequences: extremely fast (single pass), global context (the whole image is seen when predicting each box) — but historically weaker on small/clumped objects than two-stage methods.
• Modern variants (v3–v8+, anchor-free designs, better heads) refine this recipe; the grid-in-one-pass idea is constant.

Technical — RoI pooling in one sentence

Proposed boxes have arbitrary sizes, but stage 2 needs fixed-length vectors: RoI pooling crops each region out of the shared feature map and re-pools it to a fixed grid (e.g., 7×7) — so one classifier head handles all regions. Anchors: predefined box shapes/scales at each location that the RPN refines rather than predicting raw boxes from scratch. Simplification: modern variants replace anchors with learned predictions and improve the pooling (RoIAlign); the two-stage principle stands.

Technical — the definition

A box is [x, y, w, h] or corner form [x1, y1, x2, y2]. Two boxes match by Intersection over Union:

IoU(A, B) = area(A ∩ B) / area(A ∪ B)   ∈  [0, 1]

1.0 = identical boxes; 0 = disjoint. Convention: a detection is a true positive at threshold t (commonly 0.5) if it overlaps its ground-truth object with IoU ≥ t, correct class, and no other prediction claimed that object first.

Technical — the algorithm

1. Discard boxes with confidence below a threshold.
2. Sort remaining by score, descending.
3. Take the top box → keep it.
4. Remove every remaining box with IoU > threshold (typically 0.5) against it.
5. Repeat with the next-highest survivor until done.

Technical — the ladder from AP to mAP

1. Match predictions to ground truth: sort by confidence; a prediction is TP if IoU ≥ t with an unclaimed GT of the same class; otherwise FP. (Each GT can be claimed once — duplicates are FPs.)
2. AP at threshold t, per class = area under the precision–recall curve for that class (interpolated so the curve is monotonically non-increasing).
3. mAP@t = mean of AP over all classes at threshold t. (PASCAL VOC used t = 0.5.)
4. COCO mAP@[.5:.95] = mean of mAP over t = 0.50, 0.55, …, 0.95 — rewards precise localization, not just “found something.”

Related COCO metrics: AP_small / AP_medium / AP_large (object-size slices) and AR (average recall at fixed detections per image). Report the exact metric name — “mAP 0.4” means nothing without the protocol.

Technical

• Initialize the backbone from image-classification pretraining (COCO/ImageNet); replace the head with your class count.
• Fine-tune: heads first (higher LR), backbone at lower LR; unfreeze more if data allows. Detection needs more data than classification to fine-tune well — boxes are information-richer but scarcer annotations.
• Domain gap matters more than raw size: aerial/medical/industrial imagery may need full fine-tuning or domain-specific pretraining (Stage 2 §10 decision rule applies).
• Established toolchains: torchvision.models.detection (Faster R-CNN, RetinaNet), MMDetection, Ultralytics YOLO — all ship pretrained checkpoints.

Technical — the recipe

1. Forward pass; capture activations A^k of the last conv layer and the score y_c for class c.
2. Backward from y_c; the gradient of y_c w.r.t. each map is averaged spatially → weight α^k.
3. Class heatmap: ReLU( Σ_k α^k A^k ), resize to image size.

L^c_Grad-CAM = ReLU( Σ_k α^k_c A^k ),   α^k_c = (1/Z) Σ_i Σ_j ∂y_c/∂A^k_ij

Uses: debugging (model looking at watermark → dataset bias), failure analysis (§11), sanity-checking transfer learning. Limits: coarse (last-conv resolution), shows correlation with the prediction, not causal correctness; a good heatmap does not prove the model generalizes.

Technical — a failure-analysis loop

1. Slice the metrics: mAP per class, per object size, per lighting/scene condition. Aggregate mAP hides everything.
2. Read the confusion: which classes merge? (motorbike ↔ bicycle → fine-grained features missing or annotation inconsistency.)
3. Inspect error types: localization-right/class-wrong; class-right/box-loose (fails at high IoU); duplicates (NMS mis-tuned); missed small objects.
4. Grad-CAM a sample of errors — is the model looking at the object, the context, or a watermark?
5. Fix at the dataset level first: more/better annotations of the failing slice usually beats architecture tweaks.

Technical — the three claims in the name

• Language — models text sequences; everything else is representation learned en route.
• Model — a transformer (decoder-only, mostly) with billions of parameters — the numbers training adjusts (Stage 2: training changes weights; inference uses them).
• Large — scale across three axes at once: parameters, data (tokens), compute. GPT-3 (2020): 175B parameters, ~300B training tokens (published figures). Scaling laws (Kaplan 2020; Chinchilla/Hoffmann 2022) quantify how loss falls predictably as all three grow — Chinchilla’s rule of thumb: ~20 training tokens per parameter for compute-optimal pretraining.

What it is not: a database (facts live lossily in weights), a search engine (no lookup without retrieval — see Chapter 4), or a logic engine (probabilistic, sampled outputs).

Two details the diagram hides

• Streaming: the loop emits each token as soon as it’s sampled — that’s why chat UIs type word-by-word; the pipeline runs once per token, not once per answer.
• Stop conditions: generation ends on a stop token (e.g., EOS) or a max-token budget. The model has no built-in notion of “finished my point” beyond learned EOS behavior.

Technical — the product view

• Cost & limits are token-denominated: API pricing, context limits, and rate limits all count tokens, not words. English averages roughly ~4 characters/token (varies by domain and tokenizer).
• Tokenization is per-model: the same text has different token counts under GPT-2 vs Llama-3 tokenizers. Always measure with the actual tokenizer — never guess from words.
• Failure modes follow the vocabulary: rare languages fragment into many tokens (longer, costlier, sometimes weaker performance), arithmetic and letter-counting suffer because numbers and words aren’t digit-letter sequences to the model (Stage 2 §1).
• Round-trip is guaranteed — decode(encode(x)) == x — but word boundaries are not: “chatGPT” might be two tokens, “GPT” another single one.

Technical — two different vectors called “embedding”

• Token embedding (static, input): a fixed learned vector per vocabulary entry — row lookup (Stage 2 §2). The same vector for “bank” everywhere.
• Contextual representation (per position, per layer): after attention mixes information across tokens, each position’s vector is context-specific: “bank” near “river” vs “money” becomes different vectors inside the network. This is the real mechanism behind “understanding”: geometry shaped by context.

Consequences: (1) meaning is compositional — sentence representations differ from any bag of word vectors; (2) this is why the same model can do sentiment, translation, and code — contexts induce the needed geometry; (3) embedding models for retrieval (Chapter 4) are encoder-only systems trained so that semantically similar texts land close together — a different training objective on the same broad idea.

Technical — the math

p_i = exp(z_i / T) / Σ_j exp(z_j / T)

• Logits z — raw, unnormalized scores, one per vocabulary entry.
• Temperature T — divides logits before softmax. T = 1: the model’s honest distribution. T < 1 sharpens (more deterministic); T > 1 flattens (more random). T → 0 equals argmax.
• Top-k — zero out all but the k highest-probability tokens, renormalize, sample.
• Top-p (nucleus) — keep the smallest set of tokens whose cumulative probability ≥ p, renormalize, sample. Adapts the cut: 2 candidates when the model is confident, 50 when unsure.

Technical

• Objective: next-token prediction over trillions of web/book/code tokens — self-supervised (labels are the data, Stage 1 §2; mechanics Stage 2 §9).
• What it buys: grammar, world knowledge, style transfer, in-context pattern completion — compressed statistically into weights. No labels, no tasks, just continuation.
• Result — a base model: a brilliant autocomplete with no notion of being an assistant. Prompt “The capital of France is” → likely completes fine; prompt with a question and it may continue with more questions, because that’s what pages of Q&A look like.
• Scale figures (published): GPT-3: 175B params / ~300B tokens. Chinchilla: 70B / ~1.4T tokens — compute-optimal, and it beat the 10× larger Gopher. Modern models train far beyond Chinchilla-optimal for deployment efficiency (§1’s check question).

Connect to your repo: TinyShakespeare pretraining is exactly this loop at 1/10⁶ scale — loss ln(65)≈4.17 → ~1.5, samples emerging from noise.

Technical — stage 3 without the folklore

• RLHF: humans rank answer pairs → train a reward model to predict preferences → optimize the policy (the LLM) against that reward with RL (PPO), tethered to the SFT model by a KL penalty so it doesn’t drift into reward-hacking gibberish.
• DPO: a closed-form alternative — skip the reward model, optimize the policy directly on preference pairs with a classification-style loss. Simpler, stable, widely used in open models. Simplification: DPO and PPO optimize related but not identical objectives; each has failure modes (reward hacking vs off-policy limits).
• Parameter-efficient variants: LoRA freezes weights, trains small low-rank adapters — a fraction of the memory, mergeable afterwards (your Phase 2 plan; Stage 2 §10 is the same principle for CNN backbones).

Technical — three honest caveats about “1M context”

• Effective use ≠ nominal length: models recall best at the ends; long middles are under-used (“lost in the middle” effect). Marketing window ≠ reliable working memory.
• Cost scales with length: attention is quadratic in context per pass; you pay in latency and money for every token, every step.
• No persistence: between requests, only what your app re-sends exists. “Memory” features are engineering (store & re-inject), not weight changes.

Technical — the main failure families

Positioning: none of these are “bugs” in one implementation — they are properties of the paradigm. Engineering (retrieval, tools, evals — Chapters 4–5) is how products survive them.

Technical — store weights in fewer bits

• Post-training quantization (PTQ): convert trained weights to INT8/INT4 with careful scaling (methods: GPTQ, AWQ, llama.cpp’s k-quants). Small quality dips, huge memory wins; usually negligible for chat use.
• QLoRA: quantize the frozen base to 4-bit, train LoRA adapters on top — fine-tuning on consumer GPUs (your Phase 2 compute plan).
• KV cache: past keys/values stored per layer to avoid recomputing them each token — memory grows with context length; this (not weights) often dominates long-context serving.
• Other levers (one-line each): batching many requests, speculative decoding (a small draft model proposes, the big model verifies), FlashAttention kernels.

Technical — the three options compared

Connect to our repo: our minimal rag.py (FastAPI + pgvector + sentence-transformers) implements exactly the query pipeline of §2 — this sheet is that file, explained and upgraded.

Technical — the three stages

• Extraction — pull text out of the container. PDFs are a layout format, not a text format: reading order can scramble multi-column pages, tables arrive as mangled lines, and scans need OCR (which makes errors). Prefer structure-aware parsers (pdfplumber, docling-style extractors) and keep page numbers for citations.
• Cleaning — drop repeating headers/footers, page numbers, legal boilerplate; fix hyphenation; normalize whitespace and encodings. Junk chunks don’t just waste space — they win searches for common words and crowd out real answers.
• Deduplication — near-duplicate documents (versioned files, forwarded copies) create redundant chunks that fragment retrieval. Detect by content hash or embedding similarity.

Technical — strategies and trade-offs

Size trade-off: small chunks (≈100–300 tokens) = precise matching but context-poor; large (≈600–1000) = self-contained but diluted. Typical sweet spot 300–600 tokens. Overlap prevents answers that straddle a cut:

stride = chunk − overlap  ·  chunks = ⌈(N − chunk) / stride⌉ + 1

Worked: 2000-token doc, chunk 400, overlap 80 → stride 320 → ⌈1600/320⌉ + 1 = 6 chunks at offsets 0, 320, 640, 960, 1280, 1600.

Practical upgrade: embed the chunk with context prefixed — “Chapter 4 · Side Effects — Nausea is the most common…” — so the vector carries information the bare text lost.

Technical

• A bi-encoder maps each text independently to a fixed vector (LLM Fundamentals §4, Stage 3). Trained contrastively: relevant query–passage pairs pulled together, irrelevant pushed apart (DPR, Sentence-BERT). This is what enables precomputing an index.
• Dimensions & models: all-MiniLM-L6-v2 → 384-dim (our repo); common open models 768–1024-dim; commercial up to 3072-dim (text-embedding-3-large). Bigger ≠ automatically better for your domain — check a retrieval benchmark (MTEB-style) or your own eval set.
• Normalize vectors to unit length: then dot product is cosine similarity — one multiply, no norm computation at query time.
• The one inviolable rule: index and query with the same model (and same preprocessing). Vectors from different models live in unrelated spaces — similarity between them is noise.

Technical — cosine similarity

cos(q, d) = (q · d) / (‖q‖ · ‖d‖)   ∈  [−1, 1]

Measures the angle between vectors — direction = meaning, ignoring magnitude (length ≠ importance). 1 = same direction, 0 = unrelated, negative = opposing.

Scaling up: ANN indexes

Exact search compares the query to all N vectors — O(N) per query — fine to ~10⁵ chunks. Beyond that, approximate nearest neighbor (ANN) indexes: HNSW (a navigable small-world graph — skip-list-like greedy descent) or IVF (cluster vectors, search only the nearest clusters). They trade a tunable sliver of recall for orders of magnitude speed. You configure them once; the math of §6 is unchanged. Simplification: HNSW/IVF parameter tuning (ef, nprobe…) is its own craft; start with library defaults and measure recall on your eval set.

Technical

• What a chunk row stores: the vector, the original text, and metadata — doc_id, chunk_id, source/page/url, section title, timestamp, access-control tags. The text is not optional: you must return it to the LLM (§11) — vectors are only the address.
• Metadata filtering is a first-class feature: “only chunks from docs this user may read, updated after 2024-01” — applied with the ANN search, not after. This is how access control and freshness work (§12).
• Lifecycle operations: upsert on re-ingestion and delete old chunks by doc_id — the classic production bug is re-indexing a document and leaving the previous version’s chunks in place: stale duplicates that outrank fresh text.
• Options: pgvector (Postgres extension — one database for app + vectors; our choice), Qdrant, Pinecone, Chroma, Weaviate, Milvus. They differ in ANN implementation, filtering power, and ops burden — not in the conceptual model above.

Technical — BM25 in one look

score(q, d) = Σ_t∈q IDF(t) · tf(t,d)·(k₁+1) / ( tf + k₁·(1 − b + b·|d|/avgdl) )

• IDF(t) — rare terms weigh more (“sclerotinia” ≫ “the”).
• TF saturation (k₁ ≈ 1.2–2.0) — the 10th occurrence of a term adds little: relevance doesn’t scale linearly with shouting.
• Length normalization (b ≈ 0.75) — long documents don’t win just by containing everything.

Where BM25 wins: exact identifiers, error codes, SKU numbers, rare names, acronyms — strings that must match literally and that embedding models, which happily paraphrase, blur. Where embeddings win: paraphrase and meaning (“car insurance” ↔ “auto coverage”). Simplification: hybrid scoring schemes also exist beyond RRF (weighted score fusion); RRF is the robust default because it needs no score calibration — §9.

Technical — Reciprocal Rank Fusion

RRF(d) = Σ_lists 1 / (k + rank_list(d))    with k = 60

RRF uses ranks only — no score calibration needed between a cosine and a BM25 score (they live on incomparable scales). k = 60 dampens the tyranny of rank 1 (Cormack et al., 2009).

Technical

• Cross-encoder input: [CLS] query [SEP] document [SEP] → one relevance score — it cannot precompute anything, which is exactly why it’s accurate and why it’s slow.
• Typical gains are largest on hard, near-miss candidates — precisely the ones the top-k cut would get wrong.
• Our repo upgrade path: retrieve(k=4) → retrieve(k=50) → cross-encoder → top-4. Same interface, one added stage.

Technical — the prompt assembly

• Order matters: models attend best to the start and end of context (“lost in the middle,” LLM Fundamentals §8, Stage 3). Put the best-scoring chunk first or last, not buried third.
• Number your sources: [1] … [2] … so the model can cite and you can verify (§12).
• State the contract: “Answer ONLY using the context below; if it’s not there, say I don’t know based on the provided documents.” This single sentence is the honest grounding mechanism — it gives refusal a legitimate output.
• Count tokens: chunks + question + history + generation budget must fit the window (Stage 3 §8). Overflow handling — drop lowest-ranked chunks first, never the question.
• Our rag.py does exactly this: numbered context + contract + question → gpt-4o-mini (or a local Ollama model) → answer + sources array.

Technical

• Provenance rides on the chunk row (§7): doc title, page, URL, section, chunk_id. The API returns {answer, sources:[{id, source, preview, distance}]} — exactly the shape our rag.py emits.
• Citation discipline: prompt the model to cite [1]-style per claim, and validate: every cited id must exist among the chunks actually sent (a model citing chunk [3] that was never provided is a generation bug, caught mechanically).
• Access control is a retrieval-time filter: WHERE acl @> user_roles before ANN search. Filtering the answer afterward is too late — the model already saw the text. Security is a query-plan property, not an output filter.
• Quote-level grounding (advanced): require the model to quote the supporting span per claim; verify the quote actually appears in the cited chunk (string/embedding match) — cheap faithfulness enforcement.

Technical — failure modes and where they show

Technical — precise definitions

• LLM call — text in, text out. No state, no tools, no loop.
• Workflow — orchestration code defines the steps; LLMs fill in the steps (always retrieve → always summarize → always answer). Deterministic structure, testable stages.
• Agent — the LLM chooses the sequence of tool calls and when to stop, based on observations it receives. Adaptive to what it finds; less predictable; needs budgets and guardrails (§8, §10).

This matches the widely-used workflow/agent distinction (e.g., Anthropic’s “Building effective agents”): start with the simplest pattern that works, reach for agents only when the step sequence is genuinely unknowable at build time (§7).

Technical — the contract

A tool is three things the model is told about:

• name — e.g., search, calculator, send_email;
• description — when to use it (the model picks tools from these descriptions!);
• parameters schema — a JSON Schema: types, required fields, enums, ranges.

The round trip: the model emits a structured call (function calling / JSON mode) — {"name":"calculator","arguments":{"expr":"68000000/1000"}} — your code validates it (§8), executes the real function, and returns the result as a message. The model never executes anything. It proposes; the runtime disposes. This separation is exactly where validation and permissions live.

Simplification: providers differ in wire format (OpenAI-style function calling, tool-use blocks, MCP servers), but the model-proposes/runtime-executes contract is universal.

Technical — three kinds, three mechanisms

• Consequence 1: truncating a long scratchpad deletes the agent’s only working memory of those steps — summarize instead of silently dropping (Stage 3 §8’s context rules apply verbatim).
• Consequence 2: “memory” across sessions is your engineering — stored rows re-injected into prompts. Weights never change. Any product claiming otherwise is describing a database.
• Checkpoints & resumability: serializing the scratchpad per step lets you replay, audit, and resume failed runs — the backbone of observability (§10).

Technical — the spectrum

• Chain-of-Thought (CoT) — eliciting intermediate reasoning before an answer improves complex tasks (Wei et al., 2022). In agents, the “Thought” line is CoT applied to tool choice.
• Interleaved (ReAct) — think → act → observe → think… Plans are implicit in the trace; the agent adapts at every step. Default for tool tasks.
• Plan-then-execute — one call produces an explicit step list; an executor runs it. Better for long tasks with stable structure; brittle when observations contradict the plan (re-planning needed).
• Reflection — after failure (or before retry), the model critiques its own trace and stores the lesson as verbal feedback for the next attempt (Reflexion, Shinn et al., 2023) — memory-driven self-improvement within a session, no weight updates.

Orchestration = whatever code drives these patterns: the loop of §3 plus budgets, retries, and (for multi-agent, §6) message routing between roles.

Technical

• Default: single agent, good tools. One loop, one scratchpad — simplest to debug, cheapest, no inter-agent protocol to fail.
• Split when you have a concrete reason: (1) context isolation — a critic that never sees the author’s reasoning judges the output fresh; (2) distinct roles/tool sets — researcher vs coder vs tester, each with its own tool allowlist; (3) parallelism — independent subtasks fan out concurrently.
• Common topology: orchestrator–worker. A lead agent decomposes the task, dispatches to specialists, merges results. (Generative Agents, Park et al. 2023, showed rich emergent behavior — and equally the cost: many agents, many calls.)
• The price list: message-passing bugs, doubled latency, harder error attribution (which agent failed?), and compounding context loss at each handoff. Every handoff is a translation step that can drop information.

Technical — the decision table

The cost side of the ledger: an agent runs N model calls where a workflow runs 1–3 — with growing context each step (§10). Latency, spend, and failure surface all scale with N. “Agent” is the most expensive pattern in this sheet; it must earn that premium per use case.

Technical — the four gates

1. Schema validation — every call checked against the tool’s JSON Schema before execution: types, required fields, ranges, enums. Malformed call → structured error back to the model (as an observation), never an exception into the void.
2. Permissions & allowlists — which tools exist for this agent at all; per-tool scopes (read vs write); human-in-the-loop gates for destructive or irreversible actions (payments, deletes, emails). Least privilege per agent, per session.
3. Input sanitization — arguments treated as hostile: the calculator in §12 accepts only ^[0-9+\-*/(). ]+$; a SQL tool uses parameterized queries; a shell tool… you probably shouldn’t have a shell tool.
4. Failure handling — timeouts per tool, bounded retries with backoff (only for idempotent operations — never retry a payment!), a max_steps budget, and every failure surfaced to the model as an honest observation it can react to. The loop must always terminate: finish, budget, or hard error — never spin.

Simplification: “sandboxing” ranges from containers to capability systems; the principle is constant — the blast radius of any single bad call must be bounded by design, not by hope.

Technical — why this is structural

• The vulnerability lives in the architecture: everything is context (Stage 3 §8). System prompt, user message, tool output — all enter as tokens; no cryptographically signed boundary exists between them.
• Defense is layered privilege, not clever prompting: the agent that reads email should not also hold the payment tool (privilege separation); tool arguments are validated regardless of where they came from (§8); destructive actions need a human gate; and the blast radius per agent is capped by its allowlist.
• Honest framing: mitigations reduce probability and impact; they do not abolish the risk. Security review for agentic features is a release requirement, not a nice-to-have — see OWASP’s LLM Top 10 (prompt injection is LLM01).

Technical — the four disciplines

• Evaluation: a fixed task suite with checkable outcomes (deterministic fixtures where possible). Track task success rate, steps per task, tool error rate, and regression-test whole trajectories — an agent that succeeds in 12 steps instead of 5 is a regression even when it “works.”
• Observability: log every Thought / Action / Observation as a structured trace with IDs. This is non-optional: agents fail in sequences, and only traces show you which step went wrong. Replay failed traces against fixed models/tools to debug deterministically.
• Latency: N steps = N sequential LLM calls plus tool time — seconds to minutes. Stream progress to users, run independent tool calls in parallel, and cache tool results within a run.
• Cost: context grows every iteration, so token spend grows super-linearly with steps: step k re-sends the whole history. max_steps is a cost budget as much as a safety rail. Track cost-per-completed-task as the headline number.

cost(task) ≈ Σ_k=1..N tokens(context_k) · price — grows with k², roughly, for constant-size steps

Technical — what changes at the transition

Connect to our repo: train.py + Colab is the left loop. This sheet builds the right loop around rag.py / a model checkpoint.

Technical — a layout that scales

my-ai-service/
├── pyproject.toml          # dependencies + metadata (single source of truth)
├── Dockerfile              # built in CI, never by hand (§7)
├── src/service/
│   ├── main.py             # FastAPI app (§4) — thin, imports the rest
│   ├── inference.py        # load artifact, predict — pure, testable
│   ├── schemas.py          # Pydantic request/response models (§5)
│   └── settings.py         # env-var config, no hardcoded secrets (§8)
├── tests/
│   ├── test_api.py         # fast tests via TestClient (§11)
│   └── fixtures/           # tiny model copy for tests — never the 4 GB one
└── .github/workflows/ci.yml

• Importable package (src/ layout) — tests and server import the same code the notebook proved out; no copy-pasted preprocessing.
• Notebooks are for exploration only — anything a service needs gets promoted into src/ with tests. Notebook code is a draft, not a dependency.
• Config lives in settings.py reading env vars (§8); behavior changes without code edits.

Technical — the formats

Ship a bundle, not a file: weights + preprocessing artifacts (the fitted scaler! §6) + config + a version string. Versioned in object storage (S3 §10), referenced by exact path or hash — “latest” is how 3 a.m. incidents start.

artifact size = params × bytes/param — our GPT ≈ 10.8M × 4 B ≈ 43 MB (fp32) → ≈ 21 MB (fp16)

Technical — the minimal serving core

# main.py — the whole skeleton; sections below harden it
from fastapi import FastAPI
from pydantic import BaseModel, Field
from service.inference import Predictor

app = FastAPI(title="my-ai-service", version="1.2.0")
predictor = Predictor()          # loads the artifact bundle once, at startup

class PredictIn(BaseModel):
    text: str = Field(min_length=1, max_length=8000)

class PredictOut(BaseModel):
    label: str
    confidence: float

@app.get("/healthz")
def healthz():
    return {"status": "ok"}          # liveness: process is up

@app.get("/readyz")
def readyz():
    return {"ready": predictor.ready}  # readiness: model actually loaded

@app.post("/v1/predict", response_model=PredictOut)
def predict(body: PredictIn):
    return predictor.predict(body.text)

• REST in one line: resources at URLs, manipulated with methods (GET/POST/…), JSON bodies, status codes as the contract. Version the path (/v1/) so breaking changes get a new route, not angry clients.
• FastAPI gives Pydantic validation (§5), automatic OpenAPI docs at /docs, and dependency injection — the framework choice here because the learning curve is small and the guardrails are real.
• Load the model once at startup (module scope or lifespan event) — reloading per request is the classic first-deploy performance bug.

Technical — the hardening layers

1. Schema validation (Pydantic) — types, lengths, ranges, required fields. Invalid requests fail before touching the model with 422 Unprocessable Entity and a precise error message. Free, declarative, self-documenting.
2. Body-size and rate limits — at the gateway/proxy (§9), not just app code: reject absurd payloads and abusive clients cheaply.
3. Timeouts everywhere — per-request server timeout, per-dependency timeouts. A slow downstream must degrade one endpoint, not hang the fleet.
4. Correct status codes: 400/422 client error · 401/403 auth · 503 when the model isn’t ready or is overloaded (lets load balancers route around you) · 500 for genuine bugs. Never return 200 with {"error": ...} — monitoring can’t see it (§12).
5. Fail with a contract: {"error": {"code": "MODEL_OVERLOADED", "message": "retry after N"}} — machine-readable errors are part of your API.

Technical — the pipeline and its one law

request → validate → preprocess(exactly as training) → model → postprocess → response

Train/serve skew = any difference between the training-time and serving-time transformations. The classic killers:

• Recomputed statistics: serving refits the scaler per request/batch instead of loading the training-fitted one (§3’s artifact bundle exists to prevent this).
• Two implementations: notebook preprocessing pasted into the API and edited once (§2’s layout exists to prevent this).
• Version drift: serving upgrades a tokenizer/library that training used differently — pin and test both sides together.
• Distribution shift (real-world inputs moving — not a bug, a fact of life): monitored in the Evaluation sheet §10; skew is your bug, shift is time’s. Diagnose them separately.

Test it like you mean it: a golden-set test that feeds known inputs through the served endpoint and asserts predictions match the training-time outputs bit-for-bit (or within tolerance). This single test catches most skew classes automatically.

Technical — the image is the environment

# Dockerfile — pin the base; small context; non-root; one job per container
FROM python:3.11-slim-bookworm

WORKDIR /app
COPY pyproject.toml ./
RUN pip install --no-cache-dir .          # deps layer cached between builds
COPY src/ ./src/

RUN useradd -m appuser && USER appuser     # never root in production
EXPOSE 8000
CMD ["uvicorn", "service.main:app", "--host", "0.0.0.0", "--port", "8000"]

• Why containers: the image bundles interpreter + libraries + code + system deps — the same bytes run on your laptop, CI, and the cloud. “Works on my machine” becomes “works on every machine, identically.”
• Layers & caching: each instruction is a cached layer; copying pyproject.toml before source means dependency installs cache while you iterate on code — minutes to seconds per build.
• Hygiene: .dockerignore (never ship .git/, datasets, local .env); pin base images (digest for real rigor); one process per container; the model artifact usually loads from S3 at startup (§9) rather than bloating the image — decide deliberately per size.

Technical — 12-factor config

• Config = environment: the same image runs dev/staging/prod; only env vars change (Twelve-Factor App, config principle). Code never branches per environment — config does.
• Hierarchy of homes for secrets: local dev → .env file, git-ignored forever; cloud → a secrets manager or the platform’s secret store injected at task start; never in code, never in the image, never in git history (history is forever and public scanners crawl it).
• Least privilege pairs with secrets: a secret only earns its keep with an IAM role that limits what it can touch (§10) — a leaked key for a read-only bucket is a bad afternoon; a leaked admin key is a breach.
• Rotation: treat secrets as perishable — rotation story exists on day one, not after an incident.

# settings.py — typed config, fail fast at boot, not at 3 a.m.
import os
MODEL_URI = os.environ["MODEL_URI"]            # crashes at startup if missing — good
LOG_LEVEL = os.environ.get("LOG_LEVEL", "INFO")
MAX_TEXT  = int(os.environ.get("MAX_TEXT", "8000"))

Technical — one row per service, tied to the diagram

Simplification: each service has dozens of features; this table is the 10% that carries 90% of AI-serving deployments. The concepts (network tiers, least privilege, versioned artifacts, telemetry) transfer to GCP/Azure verbatim.

Technical — the pipeline stages

# .github/workflows/ci.yml (essentials)
jobs:
  test:
    steps:
      - run: pip install -e .[dev] && pytest -q        # unit + API tests
  build-push:
    needs: test
    steps:
      - run: docker build -t $ECR/repo:$GIT_SHA .       # sha-tagged, never "latest"
      - run: docker push $ECR/repo:$GIT_SHA
  deploy-staging:
    needs: build-push
    steps:
      - run: aws ecs update-service --task-def new ...  # then smoke tests hit /healthz

• Test pyramid for ML services: unit (preprocessing, schemas — fast, no model), integration (API + tiny fixture model end-to-end via TestClient), golden-set (served predictions match training-time outputs — the skew test, §6), smoke (the deployed staging endpoint answers, healthy, in budget).
• CI = Continuous Integration: every push runs the tests. CD = Continuous Delivery/Deployment: the green build becomes a running version automatically — through staging at minimum, canary in production (§13).
• Tag images with the git SHA — the running version and the code version become the same thing, which makes rollbacks a tag flip, not an archaeology dig.

Technical — the observability trio

• Logs — structured JSON per request: {request_id, route, latency_ms, status, confidence}. Structured beats string logs because it’s queryable; request IDs stitch a request’s journey across boxes.
• Metrics — aggregated numbers over time: request rate, error rate, latency percentiles, CPU/memory. Percentiles, not averages: an average of 150 ms is compatible with 5% of users waiting 2 s — and those users complain the loudest. p95/p99 are the contract.
• Alarms — on symptoms users feel (p95 over budget, 5xx rate), not on every cause (single CPU spikes). Every alarm must have an owner and a runbook, or it trains everyone to ignore alarms.

Scalability — the two levers

capacity ≈ replicas × (1 / p50_service_time) · 4 replicas × (1/0.14 s) ≈ 28 req/s

• Vertical — bigger machines (more CPU/RAM/GPU). Quick, capped, and wastes money at 3 a.m. traffic lows.
• Horizontal — more replicas, autoscaled on CPU or request rate; stateless services (ours is — all state is in S3/RDS, §9) scale horizontally for free. This is the default.
• The batching trade: grouping requests raises throughput but adds queue wait — tune max batch size and max wait against the p95 budget, not vibes.

Technical — the registry + the rollout patterns

• Model registry (MLflow-style): every trained model registered with its version, metrics, data hash, and artifact URI. Promotion is a decision on record: candidate → staging → production. “Which model is live?” must be a query, not a memory.
• Rollback is a feature you build before you need it: keep the previous task definition one command away; re-point to the prior S3 artifact. Rehearse it in staging.

Technical — three structural reasons

1. Base rates: with 0.5% prevalence, “always negative” scores 99.5% while catching nothing. Accuracy conflates class frequency with model skill.
2. Asymmetric costs: a missed tumor and a false alarm both subtract one from accuracy — and are not remotely the same event. One number cannot encode a cost ratio of 100:1.
3. Aggregation hides slices: 90% overall can hide a 30-point subgroup gap (§7). Any single number is an average over distributions you haven’t examined.

What replaces it: a metric set chosen from the problem (§2), evaluated per slice (§5, §7), with calibration checked when probabilities drive decisions (§6), and a decision threshold justified in writing (§2’s worked cost).

Technical — expected cost of a threshold

E[cost](t) = FN(t)·C_FN + FP(t)·C_FP

For each candidate threshold t, count the confusion-matrix cells and price them. Pick t that minimizes expected cost. This is the entire method — and it must be done with numbers written down.

Technical — the additions

• PR-AUC vs ROC-AUC under imbalance: ROC-AUC’s false-positive rate is computed against the huge negative class, flattering weak models on rare positives; PR-AUC stays honest about the class you care about (Saito & Rehmsmeier, 2015). Rule: rare positives → report PR-AUC.
• Macro vs micro averaging: macro = unweighted mean of per-class metrics (every class counts equally); micro = pooled counts (frequent classes dominate). Worked: class A (990 examples): P=R=0.99 → F1≈0.99; class B (10 examples): P=0.5, R=0.3 → F1 = 2·0.15/0.8 = 0.375. Macro F1 = (0.99+0.375)/2 ≈ 0.68 — the rare class drags it down, honestly. Micro ≈ 0.985 — nearly invisible. Report macro when rare classes matter.
• Regression beyond a single number: MAE/RMSE/R² (Stage 1 §6) plus the residual plot — errors vs prediction, vs each feature, over time. Patterns in residuals (funnels, curvature, seasonality) are diagnoses a scalar can’t give: heteroscedasticity, missing interaction, drift.

Technical — the shift taxonomy

• Robustness testing before deployment: evaluate on deliberately perturbed inputs — image noise/brightness/crops; text paraphrases and typos; missing values. A model that collapses under mild perturbation memorized textures, not features (CV sheet’s Grad-CAM cross-check, Stage 3).
• Time-based splits as the honest rehearsal for deployment: train on month 1–11, test on month 12. Random splits let the model peek at the future (Stage 1 §3’s temporal leakage).
• OOD inputs: decide the behavior by design — detect-and-abstain beats confident nonsense (LLM Fundamentals §9’s “I don’t know” is the same principle).

Technical — the categorization that matters

• Label noise — the “truth” was wrong; fix the dataset (and the annotation process), not the model.
• Genuine ambiguity — humans disagree too; define policy (collect more labels, add an “uncertain” class, or accept and document).
• Capability gap — consistent, systematic errors the model should get: the real improvement target. Sub-categorize by feature (lighting, dialect, rare class).
• Leakage/eval bugs — errors that appear when you fix the split (Stage 1 §3): sometimes the “errors” were a broken test all along.

The rule: no architecture change until 50 errors are read and categorized. Most “we need a bigger model” conclusions dissolve into “we need cleaner labels for slice X.”

Technical — the two measures

ECE = Σ_bins (n_b/N) · |accuracy_b − confidence_b|   ·   Brier = (1/N) Σ (p_i − y_i)²

ECE (expected calibration error): bin predictions by confidence, weigh each bin’s |accuracy − confidence| gap by its size. Brier: mean squared error of probabilities — a proper scoring rule that rewards both discrimination and honesty.

Technical — where bias enters, and the metric families

• Entry points: unrepresentative data collection (who got sampled?), label bias (historical decisions as ground truth), proxy features (postcode standing in for race, name for gender), and measurement differences across groups.
• Common metric families (each encodes a value judgment): demographic parity (similar positive rates across groups), equal opportunity (similar true-positive rates), equalized odds (similar TPR and FPR). These can be mutually unsatisfiable with different base rates — choosing one is a policy decision with stakeholders, not a coding task.
• The operational habit: report every headline metric per group, plus the gaps; audit the errors per group (who gets false positives? who gets missed?). Fairness work starts as measurement, and measurement starts with the slices.

Simplification: “fairness” has no single formal definition — the field maps which definitions conflict and when (fairmlbook.org). Anyone selling “the fairness metric” is selling one value judgment as neutral.

Technical — the working rules

• Data minimization: collect and retain only what the task needs. Every extra field is liability with a schema. Define retention periods and enforce deletion.
• Anonymization vs pseudonymization: anonymization removes re-identification paths (hard — re-identification research keeps proving “anonymized” sets re-identifiable); pseudonymization replaces identifiers with keys you hold (useful, still personal data under GDPR-style regulation — treat it accordingly).
• Models memorize: trained models can leak training data (membership-inference and extraction attacks). Consequences: don’t train on data you can’t defend having; consider privacy techniques (differential privacy, PII scrubbing) for sensitive corpora; LLM-specific surfaces (injection, exfiltration via tools) are AI Agents §9.
• Baseline hygiene: encryption in transit and at rest, strict access control (§10’s IAM), audit logs of who accessed what — the boring 90% of “AI security.”

Technical — the five pins

1. Code version — the exact commit (git SHA), including config files.
2. Data version — dataset hash or versioned snapshot (DVC-style); data changes are the silent killer of “same code, different number.”
3. Environment — pinned dependencies (lockfile) or the container digest (§7); “pip install torch” is not a specification.
4. Seeds & determinism — set seeds for python/numpy/framework; honest limits: GPU nondeterminism (atomic adds in some kernels) can persist; bit-exactness may require deterministic-algorithm flags and still isn’t guaranteed across hardware. Aim for statistically equivalent, and say which you achieved.
5. Experiment tracking — log params, metrics, artifacts per run (MLflow/W&B) so every result links to its five pins automatically. The query “which run produced the shipped model?” must be one click.

Technical — the production evaluation loop

• Input monitoring (no labels needed): feature/input distributions vs training reference — drift scores (e.g., PSI-style comparisons per feature, KS tests) alert when the world leaves the training distribution (§4’s covariate shift detector).
• Output monitoring: prediction distributions and confidence histograms — a slow slide toward all-one-class, or calibration sagging, is visible before any label arrives.
• Performance monitoring (labels when they arrive): delayed ground truth (chargebacks, diagnoses, human review samples) evaluated on rolling windows, per slice (§7). §12’s service metrics tell you the system is healthy; this tells you the model still is.
• The retraining decision is a policy: define triggers (drift threshold, performance floor, calendar) and the full path: retrain → this sheet’s evaluation on fresh data → §13’s canary. No ad-hoc “someone noticed.”
• Know the ceiling: document limitations explicitly — known OOD failure modes, groups with thin data, abstention behavior. The limitations section of §11’s template is a deliverable, not a confession.

Technical — the three failure modes this sheet fixes

• The linear slog — reading front-to-back, drowning in section 3’s math before knowing why it matters. Fix: passes (§2), anatomy (§3).
• The abstract believer — accepting claims without auditing evidence. Fix: baselines/ ablations (§6), reproducibility check (§7).
• The vanishing read — papers read, nothing retained, nothing built. Fix: template (§9), experiment log (§10), portfolio path (§11).

Technical — what each pass produces

• Pass 1 — triage (5 questions): Category (method/analysis/survey/benchmark)? Context (what line of work)? Claims? Contributions (§4)? Clarity (is the writing itself a red flag)? Output: a one-sentence answer to “what does this paper claim to add?”
• Pass 2 — evidence audit: read the method with care but skip long proofs; study every figure and table (§5); check baselines and ablations (§6); note unread references. Output: the §9 template mostly filled.
• Pass 3 — rebuild: for the few papers that anchor your area: assume you must reimplement it (§8) and identify every step the paper glosses over. Output: implementation notes — the hidden decisions that never make it into the text.

Technical — the section map

Real anchors, from papers you already know: in “Attention Is All You Need” (Vaswani et al., 2017) the contribution is the architecture itself — recurrence replaced by attention, in Fig. 1 and §3; in “Language Models are Few-Shot Learners” (Brown et al., GPT-3, 2020) the contribution is an empirical claim about scale — carried almost entirely by tables and figures, not equations. Same anatomy, opposite payload — the section map tells you where to hunt in each.

Technical — the hunt

• Where: the last one or two paragraphs of the introduction — usually opened by “In this paper, we…” / “Our contributions are…”. Secondary sources: the abstract’s third sentence, the first paragraph of the method.
• Classify it: new architecture (Transformer) · new training recipe/objective (Chinchilla’s compute-optimal scaling, Hoffmann et al. 2022) · new capability claim (GPT-3 few-shot) · new analysis/benchmark · a survey. The class determines which section carries the burden of proof.
• The one-sentence test: “Compared to [prior approach], this paper adds [mechanism/claim], shown by [evidence type].” If you can’t fill the blanks, you don’t understand the paper yet — and now you know precisely what to hunt in pass 2.

Technical — the extraction rules

• Equations — identify before deriving: (1) what are the inputs and outputs, with shapes? (2) which symbols are learned parameters vs hyperparameters vs data? (3) what does each term contribute intuitively? The scaled dot-product attention formula is the example from your Transformers sheet: Q·Kᵀ scores compatibility, √d_k stabilizes, softmax normalizes, ×V delivers content — four operations, no derivation needed to use it.
• Tables — read headers first, then hunt deltas: what are rows (methods?) and columns (datasets/metrics?)? Which cell is “best” and what comparison supports the bolding? Compute the delta yourself: relative gain = (new − baseline) / baseline
• Figures — ask what claim each one carries: architecture figures answer “how does it work”; loss/scaling curves answer “does the trend hold”; qualitative examples answer “what does it look like.” If a figure answers none of these for you, note it and move on.
• Cross-check figures vs text: the text says “significant improvement” — the y-axis says +0.3. Papers sell; axes testify. Always read the axes.

Technical — the two audit tools

• Baseline = “what existed before / what’s trivial.” You need both kinds: prior state-of-the-art (is this progress?) and trivial (majority class, a simple heuristic — is the task even hard?).
• Ablation = remove one component at a time and re-measure. This is how a paper proves each piece matters, rather than asserting it. A model with five claimed innovations and no ablation table is asking for faith, not granting evidence.

The fair-comparison checklist: same data and split? same evaluation protocol? same compute/tuning budget? was the baseline tuned, or used out-of-the-box? multiple seeds with variance reported, or one lucky run?

Technical — the pre-implementation audit

• Read the limitations section first (top venues now encourage requiring one). Authors telling you where their method breaks is a credibility signal, not weakness — no limitations section on a methods paper is itself a finding.
• The reproducibility scan (five checks): code released? data available (or at least the split documented)? hyperparameters complete (appendix/configs)? compute reported (how many GPUs for how long)? multiple seeds? Each missing item is a tax you’ll pay if you build on it.
• Scope honesty: “improves on benchmark X with setting Y” ≠ “works everywhere.” Write the scope sentence into your template (§9) verbatim — future-you will otherwise over-generalize.
• Replication signal: has anyone else reproduced/built on it? (Citations that use it, open issues on the repo, follow-up papers.) A flashy result with zero reuse after a year deserves suspicion.

Technical — the mapping and the ladder

The paper→repo mapping you already know from our GPT project:

The implementation ladder (climb one rung at a time):

1. Core mechanism at toy scale — smallest runnable version (nanoGPT is Karpathy’s pass-3 of GPT-2; our repo is yours of the same idea). Goal: the mechanism works at all.
2. Overfit a tiny batch — the sanity gate from our training notes: if you can’t drive loss ≈ 0 on 64 samples, nothing else is debugging.
3. Match a published number you can afford — small dataset, small config; expect proximity, not equality (different hardware/library versions — Evaluation sheet §9’s limits).
4. Extend — your own ablation (§6!) on the mechanism: change one thing, measure.

Technical — one line per run, forever

# experiments.jsonl — one JSON object per run (append-only, greppable)
{"run_id": "2025-01-15_a1b2c3d4",
 "paper": "attention-is-all-you-need",
 "hypothesis": "removing the scaling factor destabilizes softmax at d=512",
 "code_sha": "a1b2c3d4",
 "config_hash": "9f8e7d...",
 "seed": 1337,
 "metrics": {"val_loss": 2.31, "steps_to_diverge": 140},
 "verdict": "hypothesis supported — divergence without scaling",
 "next_action": "repeat with 2 more seeds before claiming"}

• Why JSONL: append-only (history is never edited), one run per line (diff-able, script-readable), greppable with the tools you already have.
• The five pins ride along (Evaluation sheet §9): code SHA, config hash, seed, plus the hypothesis written before the result — pre-registration at desk scale (Evaluation §11’s principle, one run at a time).
• The habit that matters: hypothesis and verdict as separate fields — you wrote what you expected first, and the log preserves whether reality agreed. That’s how “I tried some things” becomes evidence.

Technical — the five contribution levels

1. Reading notes — filled §9 templates, published (repo/blog). Demonstrates judgment.
2. Annotated walkthrough — a paper explained with your diagrams (the sketch sheets in this Lab are exactly this for the Transformer and RAG literature).
3. Toy reimplementation — runnable code + honest comparison notes (§8’s ladder).
4. Extension/ablation — your own experiment on the mechanism, logged (§10). Demonstrates research behavior, not just consumption.
5. Application — the idea transferred to a domain you own (e.g., retrieval ideas applied to a RAG system — Chapter 4’s territory).

Honest framing (per this Lab’s rules): label each artifact by level — “toy-scale reimplementation of X’s mechanism” is a strong, true portfolio line; “reproduced X” is not, unless you matched published numbers at published scale. The Lab’s provenance footers model the wording.

Technical

• HTTP — a text-based request/response protocol: METHOD /path HTTP/1.1, headers (key: value), a blank line, an optional body. Stateless by design: each request carries everything needed — which is why state must live somewhere (headers, cookies, tokens, a database).
• HTTPS = HTTP inside TLS: the client and server negotiate encryption before any application data flows; the server presents a certificate (signed by a certificate authority) proving it owns the domain. Guarantees: encryption (nobody on the wire reads it), integrity (nobody modifies it undetected), authenticity (the server is who the certificate says).
• What TLS does not fix: a malicious server, bad application logic, leaked secrets, XSS in your frontend. HTTPS protects the road, not the destination.
• Where TLS ends: at the load balancer in Stage 5 §9’s architecture — internal traffic then rides the private VPC. That’s the deliberate design: one encrypted public door, private interior.

Technical — the methods and their safety contracts

• Safe = read-only (GET/HEAD/OPTIONS). Idempotent = doing it N times leaves the same state as once. POST is the odd one out — which is exactly why the Agents sheet’s retry rules (§8) and payment idempotency keys exist.
• Consequences in code: GETs can be cached, pre-fetched, retried blindly. POSTs need dedupe keys or careful retry policy. Routers map methods to handlers; using POST for reads (or GET for writes) breaks caches and expectations.

Technical — the working set (real codes, official semantics)

The triage rule: 4xx = the request was wrong (fix the client or the docs); 5xx = the server broke its promise (fix the server, page someone). Stage 5 §5 adds the discipline: never 200-with-error-body — monitoring reads statuses, not vibes.

Technical

• JSON — the exchange format: objects, arrays, strings, numbers, booleans, null. One gotcha that reaches AI work: no native float32, no dates — NumPy types must become native before serialization (Stage 5 §5’s check), and timestamps are strings (ISO 8601 by convention).
• REST conventions: URLs name resources (nouns): /v1/predictions, /v1/models/7; methods express the verb; status codes express the outcome; requests are stateless. The API contract = methods × routes × schemas × statuses.

Technical — the two laws

1. Never trust the client. Frontend validation is UX; server validation is security (Stage 5 §5). Anything arriving over HTTP is attacker-controlled until validated.
2. Secrets never cross the line. API keys live in backend env vars (§9). Anything in frontend code/bundles is public the moment you deploy.

Technical — the vocabulary and the token

• Authentication (AuthN): who are you? — password check, OAuth login, API key.
• Authorization (AuthZ): what may you do? — roles, scopes, per-resource rules (the RAG sheet’s ACL-at-retrieval is AuthZ).
• Sessions vs tokens: classic sessions store state server-side (cookie → session store). JWT (JSON Web Token, RFC 7519) carries identity inside the token — stateless verification, ideal for APIs.

Technical — durable state vs fast answers

• Database — the durable source of truth (Stage 5 §10’s RDS/pgvector). Every write lands here; consistency rules apply.
• Cache layers, fastest first: browser cache → CDN (edge) → application cache (Redis-style, keyed lookups) → DB. Each layer answers “do I already have this?” so the expensive layer (DB, or a model call!) is spared.
• HTTP caching — the two headers that run the web: Cache-Control: max-age=3600 (“this response is fresh for an hour”) and validators: the server stamps ETag: "v7"; the client replays it with If-None-Match: "v7"; unchanged content → 304 Not Modified — no body, just the cheap answer.

Technical — compressed from Stage 5 §8

• Config via environment: DB URLs, model URIs, log levels — read from env vars (os.environ / settings.py), one image, many environments.
• Secrets hierarchy: local .env (git-ignored) → platform secret store → never in code, never in the repo, never in frontend bundles (§6’s law).
• Fail fast: required vars crash at boot, not on request 4,000 (Stage 5 §8’s os.environ[...] vs .get() distinction).

Technical — same-origin and its escape hatch

• The browser’s same-origin policy: scripts may read responses only from the same origin (scheme + host + port all equal). localhost:3000 vs localhost:8000 = different origins, full stop.
• CORS (Cross-Origin Resource Sharing) is the server-side opt-out: the API declares which origins may read its responses via Access-Control-Allow-Origin. It’s enforced by the browser; curl doesn’t care — which is exactly why APIs “work in Postman but fail in the app.”
• Simple vs preflighted requests: GET/HEAD and simple POSTs (form-encoded) go straight through; a JSON POST (Content-Type: application/json) is not simple — the browser first sends an OPTIONS preflight and demands permission.

Technical — the layered habit

• Manual exploration: curl / httpie / Postman — curl -X POST localhost:8000/v1/predict -H "Content-Type: application/json" -d '{"text":"hi"}'. Great for the first look, worthless as a safety net.
• Contract tests in CI: FastAPI’s TestClient — assert status codes and response schemas (Stage 5 §15’s four tests are the template). The contract is status + schema, so that’s what the tests pin.
• Negative tests first: empty body, wrong types, oversize input, missing auth — the 4xx half of the contract is where APIs actually break.
• Public playgrounds: httpbin.org echoes your requests — perfect for seeing exactly what headers/bodies you really send while learning.

Technical — the pattern, assembled

An AI endpoint is a normal REST endpoint whose handler calls a model (Stage 5 §4’s FastAPI core). What’s special about AI endpoints:

• Schema = the product: the request schema (what the model accepts), response schema (label + confidence + model_version — always ship the version so clients can react, Stage 5 §13).
• Latency is part of the contract: state the budget; stream when answers are long — token-by-token via SSE (text/event-stream: chunks sent as generated, the chat-typing effect) instead of one blocked 200 (LLM Fundamentals §2’s per-token loop, exposed).
• Statelessness & idempotency: inference is a pure function of (input, model version) — cacheable at temperature 0 (§8), retryable with idempotency keys for POSTs (§3, Agents sheet §8).
• Security stack, all of it: auth on every endpoint (§7), rate limits (429 + backoff), input validation before any model compute (Stage 5 §5), secrets behind the backend (§9), CORS pinned to your frontend origin (§10), and never trusting client data — including text destined for a prompt (Agents sheet §9’s injection warning applies to your API’s callers too).