Technical — what changes at the transition

Connect to our repo: train.py + Colab is the left loop. This sheet builds the right loop around rag.py / a model checkpoint.

Technical — a layout that scales

my-ai-service/
├── pyproject.toml          # dependencies + metadata (single source of truth)
├── Dockerfile              # built in CI, never by hand (§7)
├── src/service/
│   ├── main.py             # FastAPI app (§4) — thin, imports the rest
│   ├── inference.py        # load artifact, predict — pure, testable
│   ├── schemas.py          # Pydantic request/response models (§5)
│   └── settings.py         # env-var config, no hardcoded secrets (§8)
├── tests/
│   ├── test_api.py         # fast tests via TestClient (§11)
│   └── fixtures/           # tiny model copy for tests — never the 4 GB one
└── .github/workflows/ci.yml

• Importable package (src/ layout) — tests and server import the same code the notebook proved out; no copy-pasted preprocessing.
• Notebooks are for exploration only — anything a service needs gets promoted into src/ with tests. Notebook code is a draft, not a dependency.
• Config lives in settings.py reading env vars (§8); behavior changes without code edits.

Technical — the formats

Ship a bundle, not a file: weights + preprocessing artifacts (the fitted scaler! §6) + config + a version string. Versioned in object storage (S3 §10), referenced by exact path or hash — “latest” is how 3 a.m. incidents start.

artifact size = params × bytes/param — our GPT ≈ 10.8M × 4 B ≈ 43 MB (fp32) → ≈ 21 MB (fp16)

Technical — the minimal serving core

# main.py — the whole skeleton; sections below harden it
from fastapi import FastAPI
from pydantic import BaseModel, Field
from service.inference import Predictor

app = FastAPI(title="my-ai-service", version="1.2.0")
predictor = Predictor()          # loads the artifact bundle once, at startup

class PredictIn(BaseModel):
    text: str = Field(min_length=1, max_length=8000)

class PredictOut(BaseModel):
    label: str
    confidence: float

@app.get("/healthz")
def healthz():
    return {"status": "ok"}          # liveness: process is up

@app.get("/readyz")
def readyz():
    return {"ready": predictor.ready}  # readiness: model actually loaded

@app.post("/v1/predict", response_model=PredictOut)
def predict(body: PredictIn):
    return predictor.predict(body.text)

• REST in one line: resources at URLs, manipulated with methods (GET/POST/…), JSON bodies, status codes as the contract. Version the path (/v1/) so breaking changes get a new route, not angry clients.
• FastAPI gives Pydantic validation (§5), automatic OpenAPI docs at /docs, and dependency injection — the framework choice here because the learning curve is small and the guardrails are real.
• Load the model once at startup (module scope or lifespan event) — reloading per request is the classic first-deploy performance bug.

Technical — the hardening layers

1. Schema validation (Pydantic) — types, lengths, ranges, required fields. Invalid requests fail before touching the model with 422 Unprocessable Entity and a precise error message. Free, declarative, self-documenting.
2. Body-size and rate limits — at the gateway/proxy (§9), not just app code: reject absurd payloads and abusive clients cheaply.
3. Timeouts everywhere — per-request server timeout, per-dependency timeouts. A slow downstream must degrade one endpoint, not hang the fleet.
4. Correct status codes: 400/422 client error · 401/403 auth · 503 when the model isn’t ready or is overloaded (lets load balancers route around you) · 500 for genuine bugs. Never return 200 with {"error": ...} — monitoring can’t see it (§12).
5. Fail with a contract: {"error": {"code": "MODEL_OVERLOADED", "message": "retry after N"}} — machine-readable errors are part of your API.

Technical — the pipeline and its one law

request → validate → preprocess(exactly as training) → model → postprocess → response

Train/serve skew = any difference between the training-time and serving-time transformations. The classic killers:

• Recomputed statistics: serving refits the scaler per request/batch instead of loading the training-fitted one (§3’s artifact bundle exists to prevent this).
• Two implementations: notebook preprocessing pasted into the API and edited once (§2’s layout exists to prevent this).
• Version drift: serving upgrades a tokenizer/library that training used differently — pin and test both sides together.
• Distribution shift (real-world inputs moving — not a bug, a fact of life): monitored in the Evaluation sheet §10; skew is your bug, shift is time’s. Diagnose them separately.

Test it like you mean it: a golden-set test that feeds known inputs through the served endpoint and asserts predictions match the training-time outputs bit-for-bit (or within tolerance). This single test catches most skew classes automatically.

Technical — the image is the environment

# Dockerfile — pin the base; small context; non-root; one job per container
FROM python:3.11-slim-bookworm

WORKDIR /app
COPY pyproject.toml ./
RUN pip install --no-cache-dir .          # deps layer cached between builds
COPY src/ ./src/

RUN useradd -m appuser && USER appuser     # never root in production
EXPOSE 8000
CMD ["uvicorn", "service.main:app", "--host", "0.0.0.0", "--port", "8000"]

• Why containers: the image bundles interpreter + libraries + code + system deps — the same bytes run on your laptop, CI, and the cloud. “Works on my machine” becomes “works on every machine, identically.”
• Layers & caching: each instruction is a cached layer; copying pyproject.toml before source means dependency installs cache while you iterate on code — minutes to seconds per build.
• Hygiene: .dockerignore (never ship .git/, datasets, local .env); pin base images (digest for real rigor); one process per container; the model artifact usually loads from S3 at startup (§9) rather than bloating the image — decide deliberately per size.

Technical — 12-factor config

• Config = environment: the same image runs dev/staging/prod; only env vars change (Twelve-Factor App, config principle). Code never branches per environment — config does.
• Hierarchy of homes for secrets: local dev → .env file, git-ignored forever; cloud → a secrets manager or the platform’s secret store injected at task start; never in code, never in the image, never in git history (history is forever and public scanners crawl it).
• Least privilege pairs with secrets: a secret only earns its keep with an IAM role that limits what it can touch (§10) — a leaked key for a read-only bucket is a bad afternoon; a leaked admin key is a breach.
• Rotation: treat secrets as perishable — rotation story exists on day one, not after an incident.

# settings.py — typed config, fail fast at boot, not at 3 a.m.
import os
MODEL_URI = os.environ["MODEL_URI"]            # crashes at startup if missing — good
LOG_LEVEL = os.environ.get("LOG_LEVEL", "INFO")
MAX_TEXT  = int(os.environ.get("MAX_TEXT", "8000"))

Technical — one row per service, tied to the diagram

Simplification: each service has dozens of features; this table is the 10% that carries 90% of AI-serving deployments. The concepts (network tiers, least privilege, versioned artifacts, telemetry) transfer to GCP/Azure verbatim.

Technical — the pipeline stages

# .github/workflows/ci.yml (essentials)
jobs:
  test:
    steps:
      - run: pip install -e .[dev] && pytest -q        # unit + API tests
  build-push:
    needs: test
    steps:
      - run: docker build -t $ECR/repo:$GIT_SHA .       # sha-tagged, never "latest"
      - run: docker push $ECR/repo:$GIT_SHA
  deploy-staging:
    needs: build-push
    steps:
      - run: aws ecs update-service --task-def new ...  # then smoke tests hit /healthz

• Test pyramid for ML services: unit (preprocessing, schemas — fast, no model), integration (API + tiny fixture model end-to-end via TestClient), golden-set (served predictions match training-time outputs — the skew test, §6), smoke (the deployed staging endpoint answers, healthy, in budget).
• CI = Continuous Integration: every push runs the tests. CD = Continuous Delivery/Deployment: the green build becomes a running version automatically — through staging at minimum, canary in production (§13).
• Tag images with the git SHA — the running version and the code version become the same thing, which makes rollbacks a tag flip, not an archaeology dig.

Technical — the observability trio

• Logs — structured JSON per request: {request_id, route, latency_ms, status, confidence}. Structured beats string logs because it’s queryable; request IDs stitch a request’s journey across boxes.
• Metrics — aggregated numbers over time: request rate, error rate, latency percentiles, CPU/memory. Percentiles, not averages: an average of 150 ms is compatible with 5% of users waiting 2 s — and those users complain the loudest. p95/p99 are the contract.
• Alarms — on symptoms users feel (p95 over budget, 5xx rate), not on every cause (single CPU spikes). Every alarm must have an owner and a runbook, or it trains everyone to ignore alarms.

Scalability — the two levers

capacity ≈ replicas × (1 / p50_service_time) · 4 replicas × (1/0.14 s) ≈ 28 req/s

• Vertical — bigger machines (more CPU/RAM/GPU). Quick, capped, and wastes money at 3 a.m. traffic lows.
• Horizontal — more replicas, autoscaled on CPU or request rate; stateless services (ours is — all state is in S3/RDS, §9) scale horizontally for free. This is the default.
• The batching trade: grouping requests raises throughput but adds queue wait — tune max batch size and max wait against the p95 budget, not vibes.

Technical — the registry + the rollout patterns

• Model registry (MLflow-style): every trained model registered with its version, metrics, data hash, and artifact URI. Promotion is a decision on record: candidate → staging → production. “Which model is live?” must be a query, not a memory.
• Rollback is a feature you build before you need it: keep the previous task definition one command away; re-point to the prior S3 artifact. Rehearse it in staging.

Technical — three structural reasons

1. Base rates: with 0.5% prevalence, “always negative” scores 99.5% while catching nothing. Accuracy conflates class frequency with model skill.
2. Asymmetric costs: a missed tumor and a false alarm both subtract one from accuracy — and are not remotely the same event. One number cannot encode a cost ratio of 100:1.
3. Aggregation hides slices: 90% overall can hide a 30-point subgroup gap (§7). Any single number is an average over distributions you haven’t examined.

What replaces it: a metric set chosen from the problem (§2), evaluated per slice (§5, §7), with calibration checked when probabilities drive decisions (§6), and a decision threshold justified in writing (§2’s worked cost).

Technical — expected cost of a threshold

E[cost](t) = FN(t)·C_FN + FP(t)·C_FP

For each candidate threshold t, count the confusion-matrix cells and price them. Pick t that minimizes expected cost. This is the entire method — and it must be done with numbers written down.

Technical — the additions

• PR-AUC vs ROC-AUC under imbalance: ROC-AUC’s false-positive rate is computed against the huge negative class, flattering weak models on rare positives; PR-AUC stays honest about the class you care about (Saito & Rehmsmeier, 2015). Rule: rare positives → report PR-AUC.
• Macro vs micro averaging: macro = unweighted mean of per-class metrics (every class counts equally); micro = pooled counts (frequent classes dominate). Worked: class A (990 examples): P=R=0.99 → F1≈0.99; class B (10 examples): P=0.5, R=0.3 → F1 = 2·0.15/0.8 = 0.375. Macro F1 = (0.99+0.375)/2 ≈ 0.68 — the rare class drags it down, honestly. Micro ≈ 0.985 — nearly invisible. Report macro when rare classes matter.
• Regression beyond a single number: MAE/RMSE/R² (Stage 1 §6) plus the residual plot — errors vs prediction, vs each feature, over time. Patterns in residuals (funnels, curvature, seasonality) are diagnoses a scalar can’t give: heteroscedasticity, missing interaction, drift.

Technical — the shift taxonomy

• Robustness testing before deployment: evaluate on deliberately perturbed inputs — image noise/brightness/crops; text paraphrases and typos; missing values. A model that collapses under mild perturbation memorized textures, not features (CV sheet’s Grad-CAM cross-check, Stage 3).
• Time-based splits as the honest rehearsal for deployment: train on month 1–11, test on month 12. Random splits let the model peek at the future (Stage 1 §3’s temporal leakage).
• OOD inputs: decide the behavior by design — detect-and-abstain beats confident nonsense (LLM Fundamentals §9’s “I don’t know” is the same principle).

Technical — the categorization that matters

• Label noise — the “truth” was wrong; fix the dataset (and the annotation process), not the model.
• Genuine ambiguity — humans disagree too; define policy (collect more labels, add an “uncertain” class, or accept and document).
• Capability gap — consistent, systematic errors the model should get: the real improvement target. Sub-categorize by feature (lighting, dialect, rare class).
• Leakage/eval bugs — errors that appear when you fix the split (Stage 1 §3): sometimes the “errors” were a broken test all along.

The rule: no architecture change until 50 errors are read and categorized. Most “we need a bigger model” conclusions dissolve into “we need cleaner labels for slice X.”

Technical — the two measures

ECE = Σ_bins (n_b/N) · |accuracy_b − confidence_b|   ·   Brier = (1/N) Σ (p_i − y_i)²

ECE (expected calibration error): bin predictions by confidence, weigh each bin’s |accuracy − confidence| gap by its size. Brier: mean squared error of probabilities — a proper scoring rule that rewards both discrimination and honesty.

Technical — where bias enters, and the metric families

• Entry points: unrepresentative data collection (who got sampled?), label bias (historical decisions as ground truth), proxy features (postcode standing in for race, name for gender), and measurement differences across groups.
• Common metric families (each encodes a value judgment): demographic parity (similar positive rates across groups), equal opportunity (similar true-positive rates), equalized odds (similar TPR and FPR). These can be mutually unsatisfiable with different base rates — choosing one is a policy decision with stakeholders, not a coding task.
• The operational habit: report every headline metric per group, plus the gaps; audit the errors per group (who gets false positives? who gets missed?). Fairness work starts as measurement, and measurement starts with the slices.

Simplification: “fairness” has no single formal definition — the field maps which definitions conflict and when (fairmlbook.org). Anyone selling “the fairness metric” is selling one value judgment as neutral.

Technical — the working rules

• Data minimization: collect and retain only what the task needs. Every extra field is liability with a schema. Define retention periods and enforce deletion.
• Anonymization vs pseudonymization: anonymization removes re-identification paths (hard — re-identification research keeps proving “anonymized” sets re-identifiable); pseudonymization replaces identifiers with keys you hold (useful, still personal data under GDPR-style regulation — treat it accordingly).
• Models memorize: trained models can leak training data (membership-inference and extraction attacks). Consequences: don’t train on data you can’t defend having; consider privacy techniques (differential privacy, PII scrubbing) for sensitive corpora; LLM-specific surfaces (injection, exfiltration via tools) are AI Agents §9.
• Baseline hygiene: encryption in transit and at rest, strict access control (§10’s IAM), audit logs of who accessed what — the boring 90% of “AI security.”

Technical — the five pins

1. Code version — the exact commit (git SHA), including config files.
2. Data version — dataset hash or versioned snapshot (DVC-style); data changes are the silent killer of “same code, different number.”
3. Environment — pinned dependencies (lockfile) or the container digest (§7); “pip install torch” is not a specification.
4. Seeds & determinism — set seeds for python/numpy/framework; honest limits: GPU nondeterminism (atomic adds in some kernels) can persist; bit-exactness may require deterministic-algorithm flags and still isn’t guaranteed across hardware. Aim for statistically equivalent, and say which you achieved.
5. Experiment tracking — log params, metrics, artifacts per run (MLflow/W&B) so every result links to its five pins automatically. The query “which run produced the shipped model?” must be one click.

Technical — the production evaluation loop

• Input monitoring (no labels needed): feature/input distributions vs training reference — drift scores (e.g., PSI-style comparisons per feature, KS tests) alert when the world leaves the training distribution (§4’s covariate shift detector).
• Output monitoring: prediction distributions and confidence histograms — a slow slide toward all-one-class, or calibration sagging, is visible before any label arrives.
• Performance monitoring (labels when they arrive): delayed ground truth (chargebacks, diagnoses, human review samples) evaluated on rolling windows, per slice (§7). §12’s service metrics tell you the system is healthy; this tells you the model still is.
• The retraining decision is a policy: define triggers (drift threshold, performance floor, calendar) and the full path: retrain → this sheet’s evaluation on fresh data → §13’s canary. No ad-hoc “someone noticed.”
• Know the ceiling: document limitations explicitly — known OOD failure modes, groups with thin data, abstention behavior. The limitations section of §11’s template is a deliverable, not a confession.