An agent that “seems to work” is an agent nobody has measured. LLMs are non-deterministic and fail quietly: the demo shines, then week two produces a nonsense answer nobody saw coming. Evaluation and observability are how you replace “it seems fine” with numbers.

The analogy

The dashboard and the crash test. Observability is the dashboard: speed, fuel, warning lights — you see what the system is doing while it drives. Evals are the crash tests: before shipping a change, you run it into the same wall as last time and check it survives. No carmaker ships on “the test driver felt good about it”.

The principle

flowchart LR
    P["prompt or code change"] --> E["run the eval set"]
    E --> S{"scores vs baseline"}
    S -->|better or equal| SHIP([ship])
    S -->|regression| P
    PROD["production traces"] -->|"failures become new test cases"| E

Two habits, one loop:

• Observe everything. Log each step of every run: the exact prompt, the tool calls and their results, tokens spent, the final output. When something breaks, the trace tells you which step drifted — without it you’re debugging a black box.
• Evaluate on a fixed set. A few dozen real cases with expected outcomes. Score them the same way after every change. Where checks can’t be programmatic (“is this summary faithful?”), use a second model as judge — an evaluator with an explicit rubric, spot-checked by a human.

A concrete example

A team maintains a contract-analysis assistant:

eval set → 50 real contracts + expected key clauses
change   → someone "improves" the extraction prompt
run      → clause recall drops 92% → 71% on renewal dates
verdict  → the new prompt is better at summaries,
           worse at dates — rejected, fixed, re-run: 94%

Without the eval set, that regression ships silently and a customer finds it.

The starter kit

• Start with 20 cases, not 500 — real inputs, drawn from actual usage, including the ugly ones.
• Every production failure becomes a test case. Your eval set grows from reality, like a regression suite.
• Score simply: exact-match where possible, rubric-based judge where not, human sampling to calibrate the judge.
• Track cost and latency too — a prompt that’s 2% better and 3× more expensive is usually a regression (see budgets & limits).

The classic trap

Grading on vibes — including the model grading itself. “I read three answers and they looked good” doesn’t survive contact with a prompt change. And an LLM judging with no rubric says “looks good” almost as often as a human skimming. Fixed set, explicit criteria, same measurement every time — that’s the whole trick.