Agent evaluation cases

Use cognite agents eval to run YAML-defined test cases against a deployed Atlas AI agent. Cases live in eval/eval.yaml inside an agent project created with cognite agents create. This is the CLI / file-based evaluation path. Cognite Data Fusion (CDF) also offers Running agent evaluations in Atlas AI. Use the CLI when you want cases in Git, CI-friendly exit codes, and coding agents that can author YAML against a published schema.

Run evaluations

# From an agent project directory (push first, or use --upsert)
npx @cognite/cli@latest agents eval --upsert

# Single case, verbose
npx @cognite/cli@latest agents eval --case greeting -v

# Cases tagged "ci"
npx @cognite/cli@latest agents eval --tag ci

Eval reads eval/eval.yaml (and optional include fragments). By default it uses externalId from local <name>.agent.yaml. You can instead pass --external-id <id> or --system-agent to evaluate a deployed agent without reading local <name>.agent.yaml. Full flags: command reference: agents eval.

Example suite

eval/eval.yaml

cases:
  - id: greeting
    turns:
      - input: "Hi, what can you help me with?"
        scorers:
          - type: correctness
            criteria: >-
              A friendly greeting that briefly explains what this agent can help
              the user with.

  - id: follow-up
    turns:
      - input: "What can you help me with?"
        scorers:
          - type: correctness
            criteria: "Describes the agent's capabilities."
      - input: "Can you elaborate on the first thing you mentioned?"
        scorers:
          - type: correctness
            criteria: >-
              Expands on a capability mentioned in the previous response.

Schema

Root file (`eval/eval.yaml`)

Field	Description
`include`	Optional list of fragment paths relative to `eval/`.
`cases`	Optional array of cases (may be empty if all cases live in included files).
`dataModels` / `instanceSpaces` / `appContext`	Optional session context defaults for all cases.
`tools`	Optional tools for tool scorers defaults for all cases.

Case

Field	Description
`id`	Required. Unique across the whole suite (root + includes).
`tags`	Optional string tags for `--tag` filtering.
`expectPass`	Optional. Default `true`. Set `false` for negative tests where scorer failure is the expected outcome.
`turns`	Required. At least one turn.
Session context fields	Optional overrides (see below).
`tools`	Optional override of the resolved tool list for this case.

Turn

Field	Description
`input`	Required. User message sent to the agent.
`scorers`	Optional. If present, at least one scorer. At least one turn in each case must have scorers for a meaningful pass/fail.

Session context

Optional fields that scope agent chat. They may appear at root, included file, or case level. Most-specific wins (field replace, not deep merge).

Field	Type	Purpose
`dataModels`	`{ space, externalId, version }[]`	Data models the agent may query
`instanceSpaces`	`string[]`	Instance spaces the agent may query
`appContext`	`string`	Free-text context (for example active plant)

After resolution, dataModels and instanceSpaces must both be present or both absent for a case. appContext may be set alone.

Multi-file suites

eval/eval.yaml

include:
  - cases/maintenance.yaml

eval/
  eval.yaml
  cases/
    maintenance.yaml

Included files must contain a non-empty cases array. They may declare file-level tags that are unioned into each case, and file-level tools / session context that override the root for every case in that file.

Scorers

`type`	Required fields	Purpose
`correctness`	`criteria`	Judge whether the answer meets quality criteria
`faithfulness`	`groundTruth`	Judge whether the answer is grounded in facts (not sent to the agent)
`toolSelection`	—	Judge whether the agent chose appropriate tools
`toolInvocation`	—	Judge tool-call argument quality

- type: correctness
  criteria: "The answer should mention Paris"

- type: faithfulness
  groundTruth: |
    Pump P-101 was shut down on 2024-03-15 due to bearing wear.

- type: toolSelection
- type: toolInvocation

Tool scorers

toolSelection and toolInvocation judge agent tool use against a declared tool list. Both share the same inheritance, skip behavior, and optional per-scorer tools override. Eval tools vs <name>.agent.yaml tools: These are different. <name>.agent.yaml declares tools for the agent runtime (config-time names like query). Eval tools declare what the judge should expect the agent to call at runtime (names like find_assets, execute). Put parameter details in each tool’s description so toolInvocation can check arguments. When to declare eval tools:

Custom tools (config name = runtime name): tool scorers can use <name>.agent.yaml tools when no eval tools are set.
Platform tools (query, ask_document, and similar): the runtime often exposes sub-tools (find_assets, list_views, …). Declare those runtime names in eval tools, or score outcomes with correctness / faithfulness instead.
System tools (sandbox execute, and similar): not in <name>.agent.yaml — declare them in eval tools.
--external-id / --system-agent: local <name>.agent.yaml is not loaded — supply eval tools if you use tool scorers.

Discover runtime names by running once with --verbose or --report-dir and inspecting toolsCalled:. Empty tool list: If no tools resolve after the cascade, the tool scorer is skipped with a stderr warning; other scorers on the turn still run. A case whose only scorers were skipped tool scorers fails the verdict.

Tools for tool scorers

Optional tools field for toolSelection and toolInvocation. Each entry needs non-empty name and description. Empty tools: [] is rejected. Inheritance matches session context: most-specific-wins, whole-array replace (not merge-by-name):

Root eval/eval.yaml
Included file
Individual case
Scorer on a turn (overrides everything above for that scorer only)

If no eval level declares tools, eval falls back to tools[] in local <name>.agent.yaml when that file is loaded.

eval/eval.yaml

tools:
  - name: find_assets
    description: |
      Find assets in the knowledge graph.
      Parameters: filter (string, required), limit (integer, optional).

cases:
  - id: find-pumps
    turns:
      - input: "Find pumps"
        scorers:
          - type: toolSelection
          - type: toolInvocation
            tools:
              - name: find_assets
                description: Must use find_assets with a name filter for pumps.

Machine-readable schema

Coding agents and editors can validate suites against the published JSON Schema:

agent-eval-cases.schema.json

Keep local cases compatible with this schema when you edit eval/eval.yaml. Loader rules that are harder to express in JSON Schema (unique ids across includes, scored-turn requirements, dataModels/instanceSpaces pairing) are still enforced by cognite agents eval at runtime.

Develop

SDKs and toolkits

Run evaluations

Example suite

Schema

Root file (`eval/eval.yaml`)

Case

Turn

Session context

Multi-file suites

Scorers

Tool scorers

Tools for tool scorers

Machine-readable schema

​Run evaluations

​Example suite

​Schema

​Root file (eval/eval.yaml)

​Case

​Turn

​Session context

​Multi-file suites

​Scorers

​Tool scorers

​Tools for tool scorers

​Machine-readable schema

​Related information

Run evaluations

Example suite

Schema

Root file (`eval/eval.yaml`)

Case

Turn

Session context

Multi-file suites

Scorers

Tool scorers

Tools for tool scorers

Machine-readable schema

Related information