Technical — the three options compared

Connect to our repo: our minimal rag.py (FastAPI + pgvector + sentence-transformers) implements exactly the query pipeline of §2 — this sheet is that file, explained and upgraded.

Technical — the three stages

• Extraction — pull text out of the container. PDFs are a layout format, not a text format: reading order can scramble multi-column pages, tables arrive as mangled lines, and scans need OCR (which makes errors). Prefer structure-aware parsers (pdfplumber, docling-style extractors) and keep page numbers for citations.
• Cleaning — drop repeating headers/footers, page numbers, legal boilerplate; fix hyphenation; normalize whitespace and encodings. Junk chunks don’t just waste space — they win searches for common words and crowd out real answers.
• Deduplication — near-duplicate documents (versioned files, forwarded copies) create redundant chunks that fragment retrieval. Detect by content hash or embedding similarity.

Technical — strategies and trade-offs

Size trade-off: small chunks (≈100–300 tokens) = precise matching but context-poor; large (≈600–1000) = self-contained but diluted. Typical sweet spot 300–600 tokens. Overlap prevents answers that straddle a cut:

stride = chunk − overlap  ·  chunks = ⌈(N − chunk) / stride⌉ + 1

Worked: 2000-token doc, chunk 400, overlap 80 → stride 320 → ⌈1600/320⌉ + 1 = 6 chunks at offsets 0, 320, 640, 960, 1280, 1600.

Practical upgrade: embed the chunk with context prefixed — “Chapter 4 · Side Effects — Nausea is the most common…” — so the vector carries information the bare text lost.

Technical

• A bi-encoder maps each text independently to a fixed vector (LLM Fundamentals §4, Stage 3). Trained contrastively: relevant query–passage pairs pulled together, irrelevant pushed apart (DPR, Sentence-BERT). This is what enables precomputing an index.
• Dimensions & models: all-MiniLM-L6-v2 → 384-dim (our repo); common open models 768–1024-dim; commercial up to 3072-dim (text-embedding-3-large). Bigger ≠ automatically better for your domain — check a retrieval benchmark (MTEB-style) or your own eval set.
• Normalize vectors to unit length: then dot product is cosine similarity — one multiply, no norm computation at query time.
• The one inviolable rule: index and query with the same model (and same preprocessing). Vectors from different models live in unrelated spaces — similarity between them is noise.

Technical — cosine similarity

cos(q, d) = (q · d) / (‖q‖ · ‖d‖)   ∈  [−1, 1]

Measures the angle between vectors — direction = meaning, ignoring magnitude (length ≠ importance). 1 = same direction, 0 = unrelated, negative = opposing.

Scaling up: ANN indexes

Exact search compares the query to all N vectors — O(N) per query — fine to ~10⁵ chunks. Beyond that, approximate nearest neighbor (ANN) indexes: HNSW (a navigable small-world graph — skip-list-like greedy descent) or IVF (cluster vectors, search only the nearest clusters). They trade a tunable sliver of recall for orders of magnitude speed. You configure them once; the math of §6 is unchanged. Simplification: HNSW/IVF parameter tuning (ef, nprobe…) is its own craft; start with library defaults and measure recall on your eval set.

Technical

• What a chunk row stores: the vector, the original text, and metadata — doc_id, chunk_id, source/page/url, section title, timestamp, access-control tags. The text is not optional: you must return it to the LLM (§11) — vectors are only the address.
• Metadata filtering is a first-class feature: “only chunks from docs this user may read, updated after 2024-01” — applied with the ANN search, not after. This is how access control and freshness work (§12).
• Lifecycle operations: upsert on re-ingestion and delete old chunks by doc_id — the classic production bug is re-indexing a document and leaving the previous version’s chunks in place: stale duplicates that outrank fresh text.
• Options: pgvector (Postgres extension — one database for app + vectors; our choice), Qdrant, Pinecone, Chroma, Weaviate, Milvus. They differ in ANN implementation, filtering power, and ops burden — not in the conceptual model above.

Technical — BM25 in one look

score(q, d) = Σ_t∈q IDF(t) · tf(t,d)·(k₁+1) / ( tf + k₁·(1 − b + b·|d|/avgdl) )

• IDF(t) — rare terms weigh more (“sclerotinia” ≫ “the”).
• TF saturation (k₁ ≈ 1.2–2.0) — the 10th occurrence of a term adds little: relevance doesn’t scale linearly with shouting.
• Length normalization (b ≈ 0.75) — long documents don’t win just by containing everything.

Where BM25 wins: exact identifiers, error codes, SKU numbers, rare names, acronyms — strings that must match literally and that embedding models, which happily paraphrase, blur. Where embeddings win: paraphrase and meaning (“car insurance” ↔ “auto coverage”). Simplification: hybrid scoring schemes also exist beyond RRF (weighted score fusion); RRF is the robust default because it needs no score calibration — §9.

Technical — Reciprocal Rank Fusion

RRF(d) = Σ_lists 1 / (k + rank_list(d))    with k = 60

RRF uses ranks only — no score calibration needed between a cosine and a BM25 score (they live on incomparable scales). k = 60 dampens the tyranny of rank 1 (Cormack et al., 2009).

Technical

• Cross-encoder input: [CLS] query [SEP] document [SEP] → one relevance score — it cannot precompute anything, which is exactly why it’s accurate and why it’s slow.
• Typical gains are largest on hard, near-miss candidates — precisely the ones the top-k cut would get wrong.
• Our repo upgrade path: retrieve(k=4) → retrieve(k=50) → cross-encoder → top-4. Same interface, one added stage.

Technical — the prompt assembly

• Order matters: models attend best to the start and end of context (“lost in the middle,” LLM Fundamentals §8, Stage 3). Put the best-scoring chunk first or last, not buried third.
• Number your sources: [1] … [2] … so the model can cite and you can verify (§12).
• State the contract: “Answer ONLY using the context below; if it’s not there, say I don’t know based on the provided documents.” This single sentence is the honest grounding mechanism — it gives refusal a legitimate output.
• Count tokens: chunks + question + history + generation budget must fit the window (Stage 3 §8). Overflow handling — drop lowest-ranked chunks first, never the question.
• Our rag.py does exactly this: numbered context + contract + question → gpt-4o-mini (or a local Ollama model) → answer + sources array.

Technical

• Provenance rides on the chunk row (§7): doc title, page, URL, section, chunk_id. The API returns {answer, sources:[{id, source, preview, distance}]} — exactly the shape our rag.py emits.
• Citation discipline: prompt the model to cite [1]-style per claim, and validate: every cited id must exist among the chunks actually sent (a model citing chunk [3] that was never provided is a generation bug, caught mechanically).
• Access control is a retrieval-time filter: WHERE acl @> user_roles before ANN search. Filtering the answer afterward is too late — the model already saw the text. Security is a query-plan property, not an output filter.
• Quote-level grounding (advanced): require the model to quote the supporting span per claim; verify the quote actually appears in the cited chunk (string/embedding match) — cheap faithfulness enforcement.

Technical — failure modes and where they show

Technical — precise definitions

• LLM call — text in, text out. No state, no tools, no loop.
• Workflow — orchestration code defines the steps; LLMs fill in the steps (always retrieve → always summarize → always answer). Deterministic structure, testable stages.
• Agent — the LLM chooses the sequence of tool calls and when to stop, based on observations it receives. Adaptive to what it finds; less predictable; needs budgets and guardrails (§8, §10).

This matches the widely-used workflow/agent distinction (e.g., Anthropic’s “Building effective agents”): start with the simplest pattern that works, reach for agents only when the step sequence is genuinely unknowable at build time (§7).

Technical — the contract

A tool is three things the model is told about:

• name — e.g., search, calculator, send_email;
• description — when to use it (the model picks tools from these descriptions!);
• parameters schema — a JSON Schema: types, required fields, enums, ranges.

The round trip: the model emits a structured call (function calling / JSON mode) — {"name":"calculator","arguments":{"expr":"68000000/1000"}} — your code validates it (§8), executes the real function, and returns the result as a message. The model never executes anything. It proposes; the runtime disposes. This separation is exactly where validation and permissions live.

Simplification: providers differ in wire format (OpenAI-style function calling, tool-use blocks, MCP servers), but the model-proposes/runtime-executes contract is universal.

Technical — three kinds, three mechanisms

• Consequence 1: truncating a long scratchpad deletes the agent’s only working memory of those steps — summarize instead of silently dropping (Stage 3 §8’s context rules apply verbatim).
• Consequence 2: “memory” across sessions is your engineering — stored rows re-injected into prompts. Weights never change. Any product claiming otherwise is describing a database.
• Checkpoints & resumability: serializing the scratchpad per step lets you replay, audit, and resume failed runs — the backbone of observability (§10).

Technical — the spectrum

• Chain-of-Thought (CoT) — eliciting intermediate reasoning before an answer improves complex tasks (Wei et al., 2022). In agents, the “Thought” line is CoT applied to tool choice.
• Interleaved (ReAct) — think → act → observe → think… Plans are implicit in the trace; the agent adapts at every step. Default for tool tasks.
• Plan-then-execute — one call produces an explicit step list; an executor runs it. Better for long tasks with stable structure; brittle when observations contradict the plan (re-planning needed).
• Reflection — after failure (or before retry), the model critiques its own trace and stores the lesson as verbal feedback for the next attempt (Reflexion, Shinn et al., 2023) — memory-driven self-improvement within a session, no weight updates.

Orchestration = whatever code drives these patterns: the loop of §3 plus budgets, retries, and (for multi-agent, §6) message routing between roles.

Technical

• Default: single agent, good tools. One loop, one scratchpad — simplest to debug, cheapest, no inter-agent protocol to fail.
• Split when you have a concrete reason: (1) context isolation — a critic that never sees the author’s reasoning judges the output fresh; (2) distinct roles/tool sets — researcher vs coder vs tester, each with its own tool allowlist; (3) parallelism — independent subtasks fan out concurrently.
• Common topology: orchestrator–worker. A lead agent decomposes the task, dispatches to specialists, merges results. (Generative Agents, Park et al. 2023, showed rich emergent behavior — and equally the cost: many agents, many calls.)
• The price list: message-passing bugs, doubled latency, harder error attribution (which agent failed?), and compounding context loss at each handoff. Every handoff is a translation step that can drop information.

Technical — the decision table

The cost side of the ledger: an agent runs N model calls where a workflow runs 1–3 — with growing context each step (§10). Latency, spend, and failure surface all scale with N. “Agent” is the most expensive pattern in this sheet; it must earn that premium per use case.

Technical — the four gates

1. Schema validation — every call checked against the tool’s JSON Schema before execution: types, required fields, ranges, enums. Malformed call → structured error back to the model (as an observation), never an exception into the void.
2. Permissions & allowlists — which tools exist for this agent at all; per-tool scopes (read vs write); human-in-the-loop gates for destructive or irreversible actions (payments, deletes, emails). Least privilege per agent, per session.
3. Input sanitization — arguments treated as hostile: the calculator in §12 accepts only ^[0-9+\-*/(). ]+$; a SQL tool uses parameterized queries; a shell tool… you probably shouldn’t have a shell tool.
4. Failure handling — timeouts per tool, bounded retries with backoff (only for idempotent operations — never retry a payment!), a max_steps budget, and every failure surfaced to the model as an honest observation it can react to. The loop must always terminate: finish, budget, or hard error — never spin.

Simplification: “sandboxing” ranges from containers to capability systems; the principle is constant — the blast radius of any single bad call must be bounded by design, not by hope.

Technical — why this is structural

• The vulnerability lives in the architecture: everything is context (Stage 3 §8). System prompt, user message, tool output — all enter as tokens; no cryptographically signed boundary exists between them.
• Defense is layered privilege, not clever prompting: the agent that reads email should not also hold the payment tool (privilege separation); tool arguments are validated regardless of where they came from (§8); destructive actions need a human gate; and the blast radius per agent is capped by its allowlist.
• Honest framing: mitigations reduce probability and impact; they do not abolish the risk. Security review for agentic features is a release requirement, not a nice-to-have — see OWASP’s LLM Top 10 (prompt injection is LLM01).

Technical — the four disciplines

• Evaluation: a fixed task suite with checkable outcomes (deterministic fixtures where possible). Track task success rate, steps per task, tool error rate, and regression-test whole trajectories — an agent that succeeds in 12 steps instead of 5 is a regression even when it “works.”
• Observability: log every Thought / Action / Observation as a structured trace with IDs. This is non-optional: agents fail in sequences, and only traces show you which step went wrong. Replay failed traces against fixed models/tools to debug deterministically.
• Latency: N steps = N sequential LLM calls plus tool time — seconds to minutes. Stream progress to users, run independent tool calls in parallel, and cache tool results within a run.
• Cost: context grows every iteration, so token spend grows super-linearly with steps: step k re-sends the whole history. max_steps is a cost budget as much as a safety rail. Track cost-per-completed-task as the headline number.

cost(task) ≈ Σ_k=1..N tokens(context_k) · price — grows with k², roughly, for constant-size steps