What causes LLM regressions. Three distinct sources: (1) Your changes — prompt edits, retrieval pipeline modifications, system prompt updates, dependency upgrades. (2) Provider changes — OpenAI...

Build a golden dataset. Sample production traffic, select representative examples across use case categories, add ground truth labels (human annotation or verified outputs), include known edge cases...

"Walk me through how you'd detect and handle a quality regression in a production LLM system." Three layers: (1) Pre-deployment — eval gate in CI runs golden dataset on every change, blocks deploys...