Run the same test twice against a traditional system and you expect the same result twice. Run the same prompt twice against an LLM and you may get two different answers – both arguably correct.
That single fact breaks the assumption underneath almost every test case we have ever written. As AI features move from demos into production checkout flows, support agents, and underwriting decisions, QC has to answer an uncomfortable question: how do you sign off on a system that is allowed to be different every time?
Here is how I approach it.
What is AI/LLM application testing?
It is the discipline of validating software whose output is probabilistic rather than deterministic – chatbots, RAG-based assistants, summarisers, classifiers, copilots, and agentic workflows.
The key mental shift: you stop asserting on exact output and start asserting on properties of the output.
A realistic scope covers:
- Correctness and groundedness – is the answer factually right, and is it supported by the retrieved source rather than invented?
- Relevance and completeness – does it actually answer the question asked, without omitting critical information?
- Safety and guardrails – does it refuse harmful, out-of-scope, or policy-violating requests?
- Security – prompt injection, jailbreaks, and leakage of system prompts, PII, or other users’ data.
- Bias and fairness – does answer quality degrade for certain demographics, dialects, or languages?
- Structural validity – for tool-calling and agents: is the JSON well-formed, is the right function called with the right arguments?
- Non-functional – latency, token cost per interaction, rate limits, and graceful degradation when the model provider is down.
Crucially, “the system” is not just the model. It is the prompt template, the retrieval pipeline, the embedding model, the guardrails, the tool integrations, the fallback logic, and the model version – any of which can change output overnight.
Why does it matter now?
The failure mode is new and expensive. Traditional bugs crash or return a wrong value. LLM failures are fluent and confident – a hallucinated refund policy reads exactly like a correct one. Nobody notices until a customer acts on it.
Silent regressions. A provider updates a model behind the same endpoint, or someone tweaks three words in a system prompt, and behaviour shifts across thousands of interactions. Without an evaluation suite you find out from complaints.
New attack surface. Prompt injection is the SQL injection of this decade – and unlike SQL injection, there is no complete fix, only layered mitigation that must be continuously tested.
Regulatory pressure is arriving. The EU AI Act, ISO/IEC 42001, and the NIST AI Risk Management Framework increasingly expect documented evidence that AI systems were evaluated for accuracy, robustness, and bias. That evidence is a QC deliverable.
Trust is the product. Users forgive a slow page. They do not forgive an assistant that confidently invents their account balance.
Comparison to traditional testing
| Traditional testing | AI / LLM testing | |
| Expected result | One exact value | A range of acceptable outputs |
| Pass criteria | Assertion is true or false | Score against a threshold, over a dataset |
| Repeatability | Same input, same output | Same input, varying output |
| Test data | Fixtures | Curated evaluation datasets (“golden sets”) |
| Failure signal | Exception, wrong value | Degraded quality, subtle drift |
| Coverage measure | Code / requirement coverage | Scenario, risk, and demographic coverage |
| Oracle | Specification | Human judgement, reference answer, or LLM-as-judge |
| Regression trigger | Code change | Code, prompt, data, or model version change |
What does not change: everything around the model. The API contracts, the auth, the database writes, the UI, the error handling when the provider times out – that is ordinary software, and it deserves ordinary, deterministic tests. Do not let the novelty of the model tempt you into treating the whole application as unmeasurable.
How it compares to adjacent practices:
- Classic ML testing (accuracy, precision/recall on a labelled test set) still applies to classifiers and extraction tasks – use it wherever you have ground truth.
- Evaluation frameworks – DeepEval, Ragas, promptfoo, OpenAI Evals, LangSmith, TruLens – are the “test runners” of this space. They score outputs on metrics such as faithfulness, answer relevancy, and context precision.
- Red teaming (Garak, PyRIT, Giskard) is the security-testing counterpart: adversarial prompts, jailbreak suites, injection payloads.
- Observability (Langfuse, Phoenix, LangSmith) closes the loop – production traces become tomorrow’s regression cases.
Best practices
Separate the deterministic shell from the probabilistic core. Mock the model and test your application logic conventionally; then evaluate the model behaviour separately. Two problems, two toolkits.
Build a golden dataset before you build the eval. 50-200 curated cases – real user questions, expected answers or reference documents, and the risk each one represents – is worth more than any metric dashboard. Grow it from production traces and every defect you find.
Assert on properties, not strings. Useful assertions: “contains the correct policy number,” “cites at least one retrieved source,” “does not mention a competitor,” “returns valid JSON matching the schema,” “refuses.” Exact-match assertions will only teach your team to ignore the suite.
Score, then threshold. Run the whole dataset, compute pass rates per metric, and gate the build on aggregate thresholds (for example, faithfulness at or above 0.90 with zero critical-safety failures) rather than on individual case flakiness.
Use LLM-as-judge carefully. It scales evaluation enormously, but it is itself non-deterministic and biased toward verbose answers. Pin the judge model and version, use a detailed rubric, set temperature to 0, and periodically validate the judge against human ratings.
Keep a human in the loop for the top risks. Sample-based human review of the highest-impact scenarios is not a failure of automation; it is the appropriate control for consequential decisions.
Version everything as a release artifact. Prompt, model name and version, temperature, retrieval config, and guardrail rules. If you cannot reproduce a result, you cannot investigate a defect.
Test retrieval separately from generation. In a RAG system most “hallucinations” are actually retrieval failures – the right document never reached the model. Measure context precision and recall on their own before blaming the LLM.
Red team every release. Maintain a standing adversarial suite: prompt injection, role-play jailbreaks, data exfiltration attempts, and out-of-scope requests. Add every new bypass you discover.
Monitor in production, feed it back. Log traces, sample and rate them, watch for drift, and promote interesting failures into the golden dataset. Evaluation is a loop, not a gate.
Addressing common challenges
“Tests fail randomly – is it a bug or just variance?” Set temperature to 0 for reproducibility where the use case allows, then judge stability statistically: run each case several times and require a pass rate, not a single pass. Genuine flakiness at temperature 0 usually points at a non-deterministic retrieval step, not the model.
“We have no ground truth.” Start with what you can verify objectively: schema validity, refusal behaviour, presence of required entities, citation of a source. Then curate reference answers for your top scenarios with a subject-matter expert. Partial ground truth beats none.
“Evaluation runs are slow and expensive.” Tier them. A fast smoke set (20-30 cases) on every pull request; the full golden set nightly; the full red-team suite before release. Cache model responses for unchanged prompts so you only pay for what actually moved.
“The vendor changed the model and everything shifted.” Pin model versions explicitly, never use a floating “latest” alias in production, and treat a model upgrade like a major dependency bump: run the full evaluation suite and compare scores side by side before promoting.
“Prompt injection keeps getting through.” Accept that no single control is sufficient. Layer them – input filtering, strict system-prompt separation, least-privilege tool permissions, output validation, and human confirmation for irreversible actions – and test each layer independently. Then keep testing, because the attacks evolve.
“Sensitive data is going to a third-party model.” This is a QC concern, not only a legal one. Verify PII redaction actually works before the call is made, confirm the vendor’s data-retention and training-opt-out settings, and test that logs and traces do not persist raw customer data.
“Stakeholders want a single ‘is it working?’ number.” Resist a single vanity metric. Report a small scorecard – groundedness, relevance, safety pass rate, cost per interaction, p95 latency – with an explicit statement of what is not covered. Honest uncertainty is more useful than false precision.
Conclusion
Testing AI systems is not a rejection of everything QC already knows – it is an extension of it. Risk-based thinking, curated test data, clear oracles, regression suites, and layered security controls all still apply. What changes is that the assertion becomes a score, the fixture becomes a dataset, and the definition of a regression widens to include prompts, retrieval, and model versions.
The teams getting this right are not the ones with the cleverest metric. They are the ones who wrote down 100 real scenarios, agreed what “good” means for each, automated the objective parts, kept humans on the risky parts, and made the whole thing run before every release.
If your product has an AI feature and no evaluation suite, you do not have a quality problem yet – you have an invisible quality problem. Start with your riskiest user journey, write twenty cases, and measure it. You will learn more in a week than in a month of debate.