Running agent evaluations

Configure evaluations to verify that your agents provide the results that you expect. Create test cases with prompts and expected responses, run evaluations to generate agent responses, then review results to identify where your agents can improve. Use the Cognite CLI for evaluation cases you keep in Git and run from the terminal or CI. You can also run evaluations with Evaluate agents in Cognite Data Fusion (CDF). These paths use different workflows and artifacts; don’t treat them as interchangeable.

Before you start

You must create an agent with the Cognite CLI or in the Agent builder.

CDF stores evaluations created with Evaluate agents as data model instances in the atlas-ai-evals space. Your CDF group must have the datamodelinstances:write capability scoped to that space before you can create an evaluation. See Atlas AI capabilities.

Choose how to evaluate

Evaluate with the Cognite CLI or Evaluate agents in CDF.

Cognite CLI (recommended)
Evaluate agents in CDF

Define cases in eval/eval.yaml, score with cognite agents eval, and keep suites in version control.

Agent evaluation cases: scorers, schema, session context, and tool scorers
Building Atlas AI agents with the Cognite CLI: scaffold → push → evaluate → publish
Evaluation cases JSON Schema: for editors and coding assistants

Requires @cognite/cli 1.10.0 or later.

Use Evaluate agents in CDF for interactive test cases in the browser.

Configure evaluation

Configure an evaluation with test cases that define prompts and expected responses.

Start configuration

In CDF, navigate to Atlas AI > Evaluate agents.
Select + Create new evaluation.

Enter evaluation details

Enter a Name for your evaluation.

Use descriptive names like “Pump maintenance agent - v1” to help you identify your evaluations.

Add a Description to explain the purpose of the evaluation.

Define test cases

Add test cases to define prompts and the expected responses.

In the Test cases section, enter a Prompt.
Enter the Expected response. Atlas AI compares your expected response to the agent’s response to test performance.
Optionally, choose an agent and select Generate answer to use the agent to generate the expected response.

Be specific about what details, like operational status, metrics, or alert information, you want agents to include in their responses.

Run and monitor evaluation

Run an evaluation to test your agent. Monitor real-time progress as the agent processes test cases.

Navigate to evaluation

Go to Evaluation overview.
Select Run evaluation.

Select your agent

Use the search or filter to find your agent by name, status, or latest version.
Select the agent from the list.
Select Run with selected agent to run the evaluation.

Confirm code tool access

If Atlas AI prompts you to allow code tool access for your agent, select Confirm and run to continue running the evaluation or Cancel to go back to the list of agents.

Monitor evaluation progress

When you run the evaluation, Atlas AI sends your prompts to the agent and records the responses in real time. This can take several minutes depending on how many test cases you have.You can select Cancel run to stop the evaluation at any time.

Do not close the browser tab while the evaluation is running. Evaluations run in the browser, and closing or navigating away from the tab will stop the evaluation.

View and analyze results

Review evaluation results to identify where agents need improvement and track performance across multiple runs.

View completed evaluations

On the Evaluation overview tab, review the list of completed evaluations.

Review test case status

Select an evaluation to view the status of your test cases.

Compare responses

Select View details to compare the agent responses to your expected responses for each test case.

Use evaluation insights to improve your agent by refining instructions in the Agent builder, updating tools or data access, or editing test cases.

​Before you start

​Choose how to evaluate

​Configure evaluation

​Run and monitor evaluation

​View and analyze results

​Related information

Before you start

Choose how to evaluate

Configure evaluation

Run and monitor evaluation

View and analyze results

Related information