How to evaluate AI agents before production

Evaluate the journey as well as the answer

An agent can reach a plausible final answer after selecting the wrong tool, sending bad arguments, exposing sensitive information, retrying wastefully, or skipping required approval. Evaluate end-to-end outcomes and the sequence that produced them.

• Task completion and intent resolution
• Tool selection and input accuracy
• Use of tool outputs
• Policy and approval compliance
• Recovery from unavailable or failed tools

Define success before choosing evaluators

Write a short task contract for each scenario: the expected outcome, allowed tools, forbidden actions, evidence requirements, approval points, and resource limits. An evaluator is useful only when it measures a requirement that matters.

• Required result and acceptable alternatives
• Allowed and prohibited tools
• Expected citations or source records
• Maximum steps, latency, and cost
• Conditions that require clarification, refusal, or escalation

Build a representative dataset

Begin with a small curated set drawn from real workflows, then expand it from production failures and user feedback. Keep a stable regression subset so releases remain comparable.

• Normal successful tasks
• Ambiguous and incomplete requests
• Unavailable tools and partial failures
• Conflicting or stale information
• Malicious instructions and indirect prompt injection
• Actions requiring human approval

Layer deterministic and model-based checks

Use deterministic assertions wherever the requirement is objective, such as whether a forbidden tool was called. Use rubric or model-assisted evaluators for qualities such as relevance or completeness, and calibrate them against human review.

• Schema and format validation
• Expected or prohibited tool calls
• Argument and authorization checks
• Citation existence and source matching
• Rubric-based quality and groundedness
• Human review for high-impact edge cases

Measure system evaluation

System evaluation asks whether the complete experience achieved the intended outcome. Useful measures include task completion, intent resolution, groundedness, answer completeness, safety, user effort, and successful escalation.

• Did the user obtain a correct and useful result?
• Was the answer supported by allowed evidence?
• Did the system abstain when it should?
• Was the final state safe and policy compliant?

Measure process evaluation

Process evaluation examines the agent's steps. Microsoft Foundry documents evaluators for tool selection, tool-input accuracy, tool-output utilization, call success, and overall tool-call accuracy.

• Necessary tool selected without redundant calls
• Arguments correct, complete, and authorized
• Tool result used accurately
• Failures handled without unsafe looping
• Approval applied before consequential execution

Include safety and adversarial cases

Test direct and indirect attempts to alter instructions, access unauthorized information, misuse tools, or produce prohibited content. Preview evaluators can be useful, but preview status, regional availability, limitations, and model support must be checked before production reliance.

• Prompt injection in user input and retrieved content
• Unauthorized data and cross-user isolation
• Prohibited actions and approval bypass
• Sensitive information in logs or outputs
• Excessive agency, retries, or resource use

Track operational quality

A quality improvement that doubles latency or cost may not improve the product. Report evaluation scores beside p50 and p95 latency, tool failures, model and tool versions, token use, and cost per completed task.

• Completion rate
• Quality and safety pass rates
• Latency distribution
• Average steps and tool calls
• Cost per attempted and completed task
• Human escalation and rollback rate

Create release gates

Run the suite whenever the model, prompt, instructions, tools, retrieval settings, or orchestration changes. Compare against an approved baseline and prevent promotion when a critical scenario regresses.

• Fast deterministic tests on every change
• Representative offline evaluation before deployment
• Smoke evaluation in the test environment
• Human review for high-impact workflows
• Post-release monitoring and rollback thresholds

A useful first scorecard

For each scenario, record task completion, correct tool selection, valid arguments, approval compliance, groundedness, latency, cost, and reviewer notes. Ten carefully designed scenarios reveal more than hundreds of easy prompts.