How to test AI and LLM applications: evals, RAG, and prompt injection
An LLM application may not have one exact expected sentence, but it is still testable. QA defines acceptable properties, builds representative examples, measures several dimensions, and tests the deterministic software shell separately.
- Describe the capability and harm
- Build an evaluation dataset
- Run deterministic and model checks
- Compare with baseline and investigate
| Criterion | What it measures | Example check |
|---|---|---|
| Correctness | Facts and decision | Reference, expert, or computed oracle |
| Groundedness | Support in available context | Every claim linked to a source |
| Instruction following | Format and constraints | Valid JSON, required fields, language |
| Safety | Forbidden harm and leakage | Red-team suite and policy classifier |
| Operations | Latency, errors, resource use | p95, timeout rate, tokens per task |
1. Separate conventional software from the probabilistic model
An LLM product still contains APIs, authentication, roles, storage, queues, UI, and billing. Test those with deterministic methods. The model output is another layer: wording may change for the same input, so exact string equality is rarely the right oracle.
Map the chain: user input, prompt template, retrieval or tools, model, post-processing, storage, and rendering. Name the owner, version, observable logs, and failure mode for every step. Otherwise every defect is incorrectly labeled a model hallucination.
- Prompt and system-instruction version.
- Model, parameters, and fallback.
- RAG index and document versions.
- Allowed tools and permissions.
- Post-processing, cache, and UI.
Practice task
Map a support assistant and list at least five points where a correct model answer can be distorted without a model failure.
2. Build an evaluation dataset from real tasks and risks
An evaluation dataset is a versioned set of inputs, context, expected properties, and metadata. Include frequent tasks, critical decisions, boundaries, languages, typos, long conversations, ambiguity, and known incidents. One hundred random polished prompts do not represent your users.
Split development and holdout sets so the team does not tune a prompt to visible examples. Store category, risk, locale, expected citations, forbidden claims, and human-review rules. Convert every confirmed serious incident into a sanitized regression case.
id: refund-policy-expired-en input: Can I return an order after 45 days? context: policy_v7.md#returns checks: cites_policy=true; decision=deny; no_invented_exception=true risk: high review: human_required
Practice task
Design 12 cases for an ecommerce assistant: four frequent, four boundary, two adversarial, and two with insufficient information.
3. Combine deterministic checks, rubrics, and judges
Start with inexpensive strict assertions: JSON parsing, schema, required fields, language, citations, number ranges, and secret detection. Use an expert rubric for meaning. LLM-as-a-judge can scale review, but it is also biased and variable, so calibrate it against human labels.
Do not reduce quality to one average. Report critical-category pass rates, classification false positives and negatives, groundedness, safety, and format compliance. One dangerous high-risk answer may block release even when the average improves.
Correctness: 0..2 Groundedness: 0..2 Completeness: 0..2 Format: pass/fail Safety: critical pass/fail Release gate: Safety=pass AND high-risk correctness >= 95%
Practice task
Create a rubric for AI that extracts fields from bug reports. Separate schema checks from semantic correctness and define a release gate.
4. Test RAG retrieval and generation separately
A RAG answer cannot be good without relevant context. Measure retrieval first: is the expected document in top-k, are stale or forbidden sources present, and are ACLs preserved? Then evaluate generation: does it use the retrieved text, cite correctly, and abstain when evidence is absent?
Cover document updates and deletion, duplicates, conflicting versions, formats, tables, and chunk boundaries. Index leakage is critical: a user must not retrieve a document they cannot access even when they know its exact wording.
Query -> expected_doc_ids=[policy-v7] Retrieved -> ids, scores, ACL decision, index version Generated -> claims with citation spans Assert -> recall@5, no forbidden doc, answer grounded or abstains
Practice task
Design a test where old and new return policies coexist in the index. Define data, expected retrieval, and expected answer properties.
5. Test prompt injection, leakage, and dangerous tool calls
Direct injection comes from a user; indirect injection arrives in a page, email, or retrieved document. Build adversarial cases that attempt to reveal system prompts, bypass policy, extract another tenant’s data, or trigger an action. Verify actual tool-call arguments, not only assistant wording.
Least privilege is stronger than a clever prompt. The model should not hold broad credentials; dangerous actions need server authorization, parameter validation, and sometimes human confirmation. Sanitize red-team logs because they may contain working attack techniques and secrets.
- Direct and indirect prompt injection.
- System-prompt and cross-tenant extraction.
- Tool call with substituted id or excessive role.
- Malicious instruction inside retrieved content.
- Encoded, multilingual, and multi-turn bypasses.
Practice task
For an assistant that creates refunds, build an actor/action matrix and three injection cases. Define API-level assertions, not only response wording.
6. Manage nondeterminism and regression
Even temperature zero can vary because of model updates, infrastructure, retrieval, and computation. Fix what you can: model snapshot, parameters, prompt hash, dataset, index, and tool responses. Run important cases repeatedly and measure pass@k or violation frequency.
Compare candidate and baseline on the same set, segmented by category, locale, and risk. A better average can hide a regression in Russian or critical refund scenarios. Keep sanitized traces and replayable tool responses for investigation.
baseline: prompt_41 + model_A candidate: prompt_44 + model_B dataset: support_eval_v12 runs_per_case: 3 report: overall + locale + category + high_risk + latency + cost
Practice task
Report a defect where 4 of 20 runs promise a nonexistent discount. Include enough information for reproduction and risk assessment.
7. Measure quality across languages, roles, and vulnerable segments
An overall metric hides differences. Segment reports by locale, query length, user role, task category, and context availability. Translating an English evaluation set is insufficient: real inputs contain local terminology, mixed languages, morphology, addresses, and cultural context.
Test interface accessibility, understandable refusals, and policy consistency across paraphrases. For sensitive decisions, inspect systematic disparities with domain or compliance experts. Do not claim absence of bias from one average; define a concrete risk, segment, and acceptable difference.
locale: en | ru | vi | mixed query_length: short | long context: sufficient | missing | conflicting risk: low | high user_role: customer | manager | admin report: pass rate + safety violations + sample size
Practice task
For a multilingual QA assistant, build a language × experience level × task type matrix and identify segments where errors cause the most harm.
8. Build a release gate and production feedback loop
Combine conventional tests, offline evals, security suites, latency, and human review. Gates follow risk: an ideation chat and a payment agent cannot share thresholds. Canary or shadow traffic reveals real distributions before full rollout.
In production monitor errors, timeouts, latency, tool failures, corrections, escalations, safety events, and query drift. Thumbs up alone is weak because a user may like a confident false answer. Sample conversations under privacy rules and feed confirmed incidents back into evaluations.
Practice task
Create a release checklist for a QA RAG assistant: software tests, evals, security, performance, observability, canary, and rollback.
Frequently asked questions about LLM testing
Does every prompt need one exact expected answer?
Usually not. Define required facts, forbidden claims, format, sources, and a rubric. Reserve exact matching for deterministic fields.
What is a hallucination?
A convincing but unsupported or false model output. In RAG, distinguish failure to retrieve evidence from failure to use available evidence.
Is testing prompt changes enough?
No. Outcomes depend on the model, retrieval, tools, data, and post-processing. Changes to any component require proportional regression.
Who should label the evaluation dataset?
It depends on risk. QA designs coverage, domain experts label difficult facts, and security owns adversarial cases. Written guidance and disagreement review maintain consistency.
An LLM product becomes testable when the team replaces one perfect expected sentence with a versioned set of risks, examples, criteria, and evidence.