Evaluate AI agents

Test the outcome. Inspect every action.

Evaluate task completion, tool selection, arguments, retrieval, trajectories, error recovery, state and final outcomes with a benchmark your team can run repeatedly.

Design an agent evaluation See evaluation coverage

Illustrative agent traceTask → action → outcome

USER REQUESTFind the order and explain the next step.

01
Retrieve context
Order reference and relevant policy
Evidence
02
Select tool
lookup_order · validate required arguments
Action
03
Handle empty result
Ask for the missing reference; preserve state
Recovery
04
Confirm outcome
Explain what is known and what still needs input
Review

Completion is a separate judgment.An appropriate clarification is not a completed order lookup.

Rubric-drivenMeasure what matters

Trace-awareInspect the path

Human-calibratedResolve ambiguity

Built around the real work

Capabilities that move a model from promising to useful.

Choose the parts of the data and quality workflow that match the behavior, market or modality you are building for.

BUILD

Designed for the next decision, not the last task.

Every capability is shaped around downstream use: what the model needs to learn, what reviewers need to judge and what your team needs to ship with confidence.

Talk through your workflow

Task completion

Define successful outcomes and verify the final state, not just the wording of the response.

Tool selection and arguments

Check the chosen tool, required parameters, permissions and policy gates before judging execution.

Retrieval and evidence

Assess relevance, missing context and how the agent handles empty or conflicting results.

Trajectory review

Compare ordered steps or acceptable alternative paths. Identify unnecessary calls and loops.

Recovery and escalation

Test timeouts, tool errors, ambiguous requests and cases that require human confirmation.

Human-calibrated judgments

Combine deterministic checks with reviewer rubrics and adjudication for subjective decisions.

Illustrative agent traceFailure inspection

USER REQUESTFind the order and explain the next step.

01
Retrieve context
Order reference and relevant policy
Evidence
02
Select tool
lookup_order · validate required arguments
Action
03
Handle empty result
Ask for the missing reference; preserve state
Recovery
04
Confirm outcome
Explain what is known and what still needs input
Review

Completion is a separate judgment.An appropriate clarification is not a completed order lookup.

A result is only part of the evidence

Read the trace behind the answer.

Keep observations, tool inputs, tool results and state changes together so reviewers can explain both successful and failed runs.

Tool arguments and returned evidence
State transitions, retries and policy decisions
Final outcome checked against the task brief

See the quality approach

The operating model

Turn failed runs into repeatable tests.

Build a regression suite around the actions your agent is allowed to take and the outcomes users depend on.

Define

Specify success, permissions and acceptable paths.

Capture

Collect tasks, reference states and edge cases.

Execute

Record tool calls, observations and transitions.

Review

Score outcomes and investigate failed steps.

Retest

Compare versions against the same evidence.

Quality that holds up in production

Judge behavior with a clear rubric.

Separate outcome quality from execution quality. A fluent final answer must not hide an invalid action or an incomplete task.

Explicit permissions and confirmation requirements
Acceptable alternate paths, not one brittle reference trace
Human review for ambiguous failures and disagreements

OutcomeCompletion, final state and answer quality

ActionsTool selection, arguments and policy compliance

EvidenceRetrieval relevance and support for claims

RecoveryRetries, escalation, loops and state communication

What arrives at the end

Deliverables your team can use.

Clear outputs, stable structure and the context needed to move the work into training, evaluation or production.

01Task and scenario datasets

02Reference outcomes and acceptable trajectories

03Tool-call validation rules

04Reviewer rubrics and calibration notes

05Scored traces with failure labels

06Version comparisons and known limitations

Where it fits

Support agents

Test resolution, policy checks and appropriate escalation.

Research assistants

Review evidence retrieval, citations and completeness.

Workflow automation

Check state changes and confirmation before consequential actions.

Start with the outcome

Make the next agent release easier to trust.

Bring the task, tools and success criteria. We’ll shape an evaluation program around the behavior that matters.

Design an agent evaluation