Checking AI for Mistakes, in Plain English
On this page
How to tell whether an AI answer, feature, or system is wrong, and what to do about it without becoming an ML engineer.
The glossary covered the vocabulary. This guide covers the part that decides whether an AI product survives contact with real users: finding out where it's wrong before they do.
AI systems add a new kind of failure on top of the ordinary kind. A broken retrieval filter or a wrong tool definition reproduces every time, and you fix it the way you'd fix any bug. The probabilistic failures are different: a mistake shows up in one answer out of forty, disappears when you retry, and comes back in a slightly different shape after you change the prompt. For those, the job isn't "find the bug and fix it." The job is to build habits and tooling that keep catching mistakes as the system changes, and I've laid it out in the order I'd actually work: what mistakes look like, where to look first, how to check a single answer by hand, how to use one tool to check another, and how to check the whole system over time.
1. What a mistake looks like#
Engineers have names for these. Here's what each one looks like from the outside.
Hallucination (fabrication). The model states something unsupported as if it were true: a policy that doesn't exist, a citation to a paper nobody wrote, a function in a library that was never there. The tell is fluency. Fabricated answers usually read better than true ones, because nothing constrained them.#
Retrieval failure (wrong evidence). In a RAG system the model answered honestly from the documents it was handed, and the documents were the wrong ones. From the outside this looks identical to hallucination. From the inside it's a search problem, and that distinction decides what you fix.#
Instruction drift. The system prompt said "never quote prices," and for 200 turns it didn't, and then it did. Long conversations and long contexts make this more likely; instructions at the top of a 30,000-token context are competing with everything that came after them.#
Format failure. You asked for JSON with four fields and got prose, or five fields, or a number formatted as a string. Trivial for a person to read, fatal for the software downstream that expected the schema.#
Tool misuse. An agent called the wrong tool, called the right tool with wrong arguments, or called it and ignored the result. The output can look fine. The calendar was never actually checked. Two agent-specific versions: the agent decides not to search at all and answers from memory, and the agent loops, searching again and again without converging on an answer. Both are visible in a trace and invisible in the output.#
Stale knowledge. The model's training has a cutoff and the question is about something after it. It won't always say so; it'll often answer confidently from the old snapshot, which is why "is this still true?" questions are dangerous.#
Silent partial completion. You asked for ten items and got seven, well formatted, with no note that three are missing. Or a summary that quietly dropped the one paragraph that contradicted the rest. These are the hardest to catch because there's no error to see, only an absence.#
Overreach. The model was asked a narrow question and answered a broader one, adding recommendations, assumptions, or "next steps" nobody asked for. Sometimes harmless. In a legal or medical context, not.#
Ungrounded fallback (the generic answer). Ask a bare model "can I still join the course?" and it says "you can usually join, check the website." Not false, not grounded, not useful. This is what a RAG product sounds like when retrieval has quietly stopped working: the model falls back to general knowledge and produces something plausible for any company instead of something true for yours. It's easy to miss because nothing in the answer is wrong. It's just not about you.#
2. Where to look first#
When something's wrong, most people jump to one of two conclusions: the prompt needs rewording or the model isn't smart enough. Both are usually the last place I'd look.
Start with the trace. A trace is the record of what actually happened on that request: what was retrieved, what the full prompt looked like after everything was assembled, which tools were called with which arguments, what came back, and what the model produced at each step. If your team can't show you this for a failing request, that's the first thing to fix, because everything below depends on it. Langfuse, Arize Phoenix, LangSmith, and Braintrust all do this; so does a homemade log file if that's what you have.#
Then work up the stack, bottom first:
- Retrieval. Did the system find the right documents? Open the trace and read what was handed to the model. Retrieval is the first place worth checking in any RAG product, because it's the one failure that no amount of prompt editing can substitute for. And it's often an ordinary bug: a search filter set to the wrong product line, a field boost that ranks section titles above answers. Set the filter to the wrong course in a course FAQ bot and you get confident, well-cited answers from the wrong course, every time, reproducibly.
- The query. In an agentic system the model writes the search query itself, which means the query is a model output and can be wrong on its own. A bad answer often traces to a bad query rather than a bad index, and this is the step most people skip when reading a trace.
- Context. Was the right information in there but buried, contradicted by something else, or outdated? A policy document from two versions ago sitting next to the current one will produce answers that are half right.
- Tools. Did the agent call what it should have, and did it use the result? Wrong arguments and ignored results are both common.
- Prompt. Only now. Is the instruction ambiguous, missing a case, or asking for two things that conflict?
- Model. Last. If retrieval, context, tools, and prompt all check out and the model still fails, then you have a model problem, and the fix is a different model, not a cleverer prompt.
This order matters because each layer down is cheaper to fix and more likely to be the cause. Swapping models is expensive, disruptive, and often changes nothing because the wrong documents are still being retrieved.
The method that makes the order usable: swap one component, hold the rest. If the pipeline is built so the search index and the model are interchangeable parts (the way the DataTalksClub course builds it, with both passed in as arguments), you can run the same failing question three ways: original, with a different index, with a different model. Whichever swap changes the answer tells you where the problem lives. This is faster than reasoning about it, and it's a question to put to your engineers if the system can't do it: how hard is it to swap the index?#
3. Checking a single answer by hand#
Before any tooling, there are habits that catch a lot. These work in ChatGPT, Claude, an internal assistant, anywhere.
Claim, evidence, check. Every factual sentence in an AI answer is a claim. For each one you'll act on, ask what evidence supports it, then check that evidence yourself. That's the whole method; everything below is a way of doing it faster.#
Ask for the evidence, then check one piece of it. "Which document did that come from? Quote the sentence." If the system can't, or quotes something that doesn't say what it claimed, you've found the problem. You don't need to check all of it. One fabricated source tells you the rest needs checking too.#
A citation existing is not a citation supporting. The most common trap for careful readers: the model cites a real paper, a real page, a real clause, and the source doesn't say what the model claims. Fabricated citations are easy to catch because the link is dead. Real citations that don't support the claim look perfectly legitimate until you open them. Open them.#
Check against the right source. Pricing comes from the vendor's current pricing page, not a blog post about it. Research claims come from the paper, not the press release. Contract terms come from the contract. A legal or policy answer comes from the policy document, not a summary of it. The model will often cite the secondary source because that's what was easiest to find; go one level up.#
Ask something it can't know. Ask the system a question your knowledge base doesn't cover and watch what it does. The right behavior is to say the information isn't available. The common behavior is to improvise something reasonable. Every well-built RAG prompt contains an instruction like "if the context doesn't contain the answer, say so," and almost nobody tests whether the system obeys it.#
Ask the counter-question. If it said the policy allows X, ask "what in the policy would prohibit X?" A grounded answer will point to the text either way. A fabricated one will happily argue both sides.#
Make it separate what it knows from what it's guessing. "Split this into claims supported by the material I gave you, claims you're inferring, and claims you're not sure about." Models handle this instruction reasonably well, and the third bucket is where the mistakes live.#
Run it again. Same question, fresh session. If the answer changes materially, the first one wasn't reliable either. The evidence is asymmetric: three matching answers are weak evidence of correctness, since a model can be consistently wrong, but three different answers are strong evidence that you can't rely on any of them.#
Change one thing. Rephrase the question slightly, or remove a document from the context, and see whether the answer swings. Answers that flip on small input changes are answers you shouldn't trust.#
The new-hire test. Would you accept this from a smart person on their first week, without checking? If the answer is no, don't accept it from the model either.#
4. Using one tool to check another#
Checking by hand doesn't scale past a few dozen outputs. This is where you point a second tool at the first one, and it's the part most non-engineers don't realize is available to them.
A second opinion from a different model. Paste the answer into a different model and ask it to find errors, unsupported claims, and missing pieces. The key word is different. Panickssery, Bowman, and Feng showed in "LLM Evaluators Recognize and Favor Their Own Generations" (NeurIPS 2024) that models can identify their own output and rate it higher than humans do, so don't let a model grade itself. Beyond that, "different" is about the model, not the logo; two models from one provider can differ more than two from different providers. Whatever you pick, the rule that survives is the one in the next paragraph: calibrate the judge against human decisions. The checker doesn't need to be smarter than the writer. It needs to have different blind spots.#
LLM-as-a-judge. The scaled version of the second opinion: a model scores outputs against a written rubric, hundreds at a time. It works, with three cautions I'd treat as non-negotiable. First, write the rubric as specific yes/no questions ("does the answer cite a document from the provided set?") rather than "rate quality 1 to 10," because judges are far more consistent on the former. Second, know the documented biases: Zheng et al. ("Judging LLM-as-a-Judge," 2023) found judges prefer longer answers and, when comparing two, prefer whichever comes first, so shuffle order and don't reward length. Third, before trusting the judge, grade fifty outputs yourself and check that its scores match yours. If they don't, fix the rubric or change the judge. Until you've done this, the judge's scores are numbers, not evidence.#
Deterministic checkers first, model checkers second. Whenever a mistake can be caught by something that isn't a model, use that. A schema validator deterministically catches structural violations: missing fields, wrong types, extra keys. It can't tell you the value is right; refund_amount: 5000 passes a numeric schema whether the correct refund was $50 or $5,000, so validation is a floor, not a verdict. A calculator or a bit of code checks arithmetic the model did in its head. A unit test checks generated code. A web search verifies a date or a price. These are cheap, they never hallucinate, and they should run before any model-based judge does. The model judge is for what's left: was the summary faithful, was the tone right, did it answer the actual question.#
Self-consistency. Run the same input five times and compare. If the system gives the same answer five times, you've learned little. If it gives three different ones, you've learned the input is a weak spot, and that's worth knowing before a user finds it.#
Cross-model comparison. Tools like Promptfoo run the same prompts across several models and show the results side by side. This is how you find out that the model you're paying for fails the invoice cases a cheaper one passes, or the reverse. It's also the fastest way to check whether a "prompt problem" is really a "this model can't do this" problem.#
Layering. None of these replaces the others. A reasonable checking stack for a support assistant is: schema validation on every response, a self-consistency check on a sample, a different-provider judge scoring faithfulness against the retrieved documents, and a human reading twenty outputs a week. Each layer catches what the previous one can't.#
What tools can't catch. A judge scores what you put in front of it. If your test set doesn't contain the failure that hurts you, every dashboard stays green while users suffer. And two models trained on similar data can share a blind spot, so agreement between them is evidence, not proof. The tools narrow the space you have to check by hand. They don't close it.#
5. Checking the system, not the answer#
A single answer being right tells you almost nothing. What you want to know is whether the system is right across the cases that matter, and whether it stays right after the next change.
Build the golden set from real failures. Don't start by inventing test cases. Start by collecting the questions that already went wrong: from support tickets, from users' thumbs-down, from your own testing. Pair each with the correct answer or the correct source document. Then add synthetic cases for the risks you haven't seen yet: the adversarial question, the empty document, the request that should be refused. Real failures tell you what happens; synthetic ones let you test what would be catastrophic before a user finds it. And make sure the set includes questions the knowledge base cannot answer, with "says it doesn't know" as the expected result. Most golden sets contain only answerable questions, which means the no-answer path never gets tested until a user hits it.#
Do error analysis before building metrics. Hamel Husain's "Your AI Product Needs Evals" (2024) makes the point better than I can: read a few hundred outputs, write a short note on each failure, group the notes into categories, and fix the biggest category first. Most teams skip this and go straight to dashboards, then spend weeks optimizing a metric that doesn't correspond to what's actually breaking. The reading is tedious. It's also the hour that pays back most.#
Rerun the golden set after every change. Every change: new prompt, new model, new retrieval settings, updated tool description, new chunk size. A regression is a change that fixed one thing and broke another, and you won't see it unless the old cases run again. Tools like Promptfoo and DeepEval run this in CI so it happens automatically instead of when someone remembers.#
Watch production, not just tests. Tests cover what you thought of. Observability shows what users are doing: which questions come up, where the thumbs-downs cluster, which tool calls fail, where latency spikes. Sample real conversations weekly and read them. New failure categories show up here first, and they become next month's golden set.#
Decide when a human is required. Not every output needs review, and some outputs always do. The rule of thumb I use: match the review to the cost of being wrong. An internal summary can go out unchecked. A refund, a contract clause, a medical statement, or anything a customer will act on gets a person in the loop until the eval numbers have earned their way out of it.#
6. A routine that fits in a week#
You don't need all of this on day one. This is the minimum I'd want on any AI feature that real people use. The numbers are starting points for a low-traffic feature; scale them with volume and with the cost of being wrong.
Before shipping: a golden set of at least thirty real cases, a schema validator on every structured output, and one person who has read a hundred outputs and can name the top three failure types.
After every change: rerun the golden set, compare against the last run, and don't ship if any previously passing case now fails without someone explaining why.
Weekly: read twenty real production conversations, add any new failure to the golden set, and spot-check the LLM judge against your own grading on ten of them.
When someone reports "the AI is wrong": pull the trace, check retrieval, check the query, check context, check tools, check the prompt, and only then blame the model.
7. Prompts you can paste#
For a single answer you're about to rely on:
For each factual claim in your last answer, tell me whether it came from the material I provided, from your general knowledge, or is an inference. For anything from the material, quote the supporting sentence.
For a second-opinion check in a different model:
Below is an answer produced by another AI system, followed by the source material it was given. Identify every claim in the answer that is not supported by the source material, every claim that contradicts it, and anything important in the source material the answer left out. Do not rewrite the answer. List problems only.
For a judge rubric (adapt the questions to your task):
Score the response on these questions, answering yes or no to each and citing the evidence: (1) Does every factual claim trace to the provided documents? (2) Does the response answer the question that was asked, not a related one? (3) Does it avoid stating prices, dates, or policy details that don't appear in the documents? (4) Is anything the user needed to know omitted? Output the four answers as JSON.
For a stability check:
I'm going to ask this question three times in separate sessions. Answer as precisely as you can, and if there's anything you'd want to caveat, put it in a separate final line beginning "Uncertain:".
Resources#
The tools mentioned above, with what each is for. Licensing checked September 2026; re-check before relying on it, since this category changes hands often.
- Promptfoo. Open-source, runs from the command line, built for comparing prompts and models side by side and gating deployments in CI.
- DeepEval. Open-source Python library for writing evals as tests; the broadest set of built-in metrics for RAG, agents, and safety checks.
- Ragas. Open-source library focused specifically on RAG metrics: faithfulness, answer relevance, context precision and recall.
- Langfuse and Arize Phoenix. Open-source tracing and observability; the "show me what actually happened on this request" layer. Phoenix is source-available under Elastic License 2.0 rather than fully open.
- Braintrust and LangSmith. Commercial platforms with free tiers that combine datasets, judges, human annotation, and production monitoring in one place.
Reading, in the order I'd suggest:
- DataTalksClub, LLM Zoomcamp, module 1 (Agentic RAG). Lessons 3, 5, 6, and 12 in particular: you watch the generic-answer failure happen, watch a search filter change the answer, see the "say you don't know" instruction written, and then see the model take over the search decision. Free, in plain Python, and the best way to feel these failures rather than read about them.
- Hamel Husain, "Your AI Product Needs Evals" (2024). The clearest argument for error analysis before metrics, from someone who's done it for many teams.
- Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (2023). Where the position and verbosity biases of model judges were first measured carefully.
- Panickssery, Bowman, and Feng, "LLM Evaluators Recognize and Favor Their Own Generations" (NeurIPS 2024). Why the checker should be a different model from the writer.