# A practical LLM evaluation loop for AI features that need to ship

## Prompt testing is not an evaluation strategy

An AI feature can look excellent in a demo and still fail the first week of real use. A user asks a question in a different way, a retrieved document is incomplete, a tool returns an unexpected shape, or a model update changes the tone of the answer. The team then opens the prompt, changes a sentence, and tests three examples by hand.

That process feels productive because the output changes immediately. It is not a reliable way to know whether the product improved.

A production AI feature needs an evaluation loop. The loop should make a change observable, repeatable, and reversible.

---

*This is an excerpt. Read the full post at [otf-kit.dev/blog/llm-evaluation-loop](https://otf-kit.dev/blog/llm-evaluation-loop) — full-stack kits your AI coding agent can actually ship to production. [Browse the kits →](https://otf-kit.dev)*