Direct answer
Evaluate an AI system against representative real tasks and explicit acceptance thresholds. Test not only successful answers but unsupported claims, missing evidence, permission boundaries, ambiguous requests, regressions, latency and operating cost.
A successful demonstration is not evidence of production reliability
Traditional software testing often checks whether defined inputs produce expected outputs. Generative AI complicates that model because answers can vary, user requests are open-ended and correctness may depend on context. That does not make rigorous testing impossible. It means the test strategy must reflect the task and the consequences of failure.
The first step is to define what “good enough” means for the intended use. A system summarising low-risk internal documents has a different risk profile from one that recommends financial actions or changes a business record.
Build an evaluation set around real work
Include representative normal cases, difficult cases, edge cases and known failure modes. For a knowledge assistant, this might include questions with direct answers, questions requiring several sources, ambiguous questions, missing evidence and conflicting documents. For an agent, include tool failures, permission boundaries and requests that should be rejected.
| Evaluation dimension | Example question |
|---|---|
| Task success | Did the system complete the user’s intended task? |
| Factual support | Is each material claim supported by the available evidence? |
| Retrieval quality | Was the necessary evidence retrieved? |
| Permission safety | Did the system avoid using or revealing unauthorised data? |
| Robustness | Does it handle paraphrases, incomplete inputs and edge cases? |
| Abstention | Does it ask for clarification or decline when evidence is insufficient? |
| Operational performance | Are latency and cost within acceptable limits? |
| Regression | Did a change make previously passing cases fail? |
Use a repeatable evaluation loop
- Start
Define task and risk
Build representative evaluation set
Set acceptance thresholds
Run system and collect outputs
Score with rules, reviewers and targeted metrics
Analyse failures by category
Change prompt, retrieval, model or workflow
Run regression evaluation
- Decision
Thresholds and risk controls satisfied?
- No↺ back to step 6: Analyse failures by category
- Yes→ step 10: Controlled release and ongoing monitoring
- Outcome
Controlled release and ongoing monitoring
Avoid hiding failures inside an average score
A high overall score can conceal a serious failure in a small but important category. Segment results by task type, user group, data source and severity. Track both frequency and impact. A rare permission leak may matter more than many harmless formatting errors.
Human evaluation should use clear rubrics and, where practical, independent review for ambiguous cases. Automated evaluators can help scale assessment, but they are not infallible and should themselves be checked against human judgement.
Production evaluation is continuous
Models, prompts, source documents and user behaviour change. A system that passed a test set last month can degrade after an index refresh or a workflow change. Maintain a versioned evaluation set, log appropriate operational signals, review user feedback and define rollback or containment conditions.
The Neuromotion perspective: AI testing should be treated as an engineering discipline with explicit tasks, thresholds, failure categories and release controls. “It looked good in our demo” is not an acceptance criterion.
Next step: Before the next AI prototype review, ask for the test set, scoring rubric, failure analysis and release threshold. If those do not exist, the team has not yet demonstrated production readiness.