> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cognite.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Agent evaluation cases

> Complete reference for eval/eval.yaml used by `cognite agents eval`: criteria, groundTruth, tools, and session context.

Use `cognite agents eval` to run YAML-defined test cases against a **deployed** Atlas AI agent. Cases live in `eval/eval.yaml` inside an agent project created with `cognite agents create`.

This is the **CLI / file-based** evaluation path. Cognite Data Fusion (CDF) also offers [Running agent evaluations](/cdf/atlas_ai/guides/atlas_ai_agent_evaluating) in Atlas AI. Use the CLI when you want cases in Git, CI-friendly exit codes, and coding agents that can author YAML against a published schema.

## Run evaluations

```bash theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
# From an agent project directory (push first, or use --upsert)
npx @cognite/cli@latest agents eval --upsert

# Single case, verbose
npx @cognite/cli@latest agents eval --case greeting -v

# Cases tagged "ci"
npx @cognite/cli@latest agents eval --tag ci
```

Eval reads `eval/eval.yaml` (and optional `include` fragments). By default it uses `externalId` from local `<name>.agent.yaml`. You can instead pass `--external-id <id>` or `--system-agent` to evaluate a deployed agent without reading local `<name>.agent.yaml`.

Full flags: [command reference: agents eval](./reference#agents-eval).

## Example suite

```yaml title="eval/eval.yaml" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
cases:
  - id: greeting
    turns:
      - input: "Hi, what can you help me with?"
        scorers:
          - type: correctness
            criteria: >-
              A friendly greeting that briefly explains what this agent can help
              the user with.

  - id: follow-up
    turns:
      - input: "What can you help me with?"
        scorers:
          - type: correctness
            criteria: "Describes the agent's capabilities."
      - input: "Can you elaborate on the first thing you mentioned?"
        scorers:
          - type: correctness
            criteria: >-
              Expands on a capability mentioned in the previous response.
```

## Schema

### Root file (`eval/eval.yaml`)

| Field                                          | Description                                                                        |
| ---------------------------------------------- | ---------------------------------------------------------------------------------- |
| `include`                                      | Optional list of fragment paths **relative to `eval/`**.                           |
| `cases`                                        | Optional array of cases (may be empty if all cases live in included files).        |
| `dataModels` / `instanceSpaces` / `appContext` | Optional [session context](#session-context) defaults for all cases.               |
| `tools`                                        | Optional [tools for tool scorers](#tools-for-tool-scorers) defaults for all cases. |

### Case

| Field                  | Description                                                                                                |
| ---------------------- | ---------------------------------------------------------------------------------------------------------- |
| `id`                   | Required. Unique across the whole suite (root + includes).                                                 |
| `tags`                 | Optional string tags for `--tag` filtering.                                                                |
| `expectPass`           | Optional. Default `true`. Set `false` for negative tests where scorer **failure** is the expected outcome. |
| `turns`                | Required. At least one turn.                                                                               |
| Session context fields | Optional overrides (see below).                                                                            |
| `tools`                | Optional override of the resolved tool list for this case.                                                 |

### Turn

| Field     | Description                                                                                                             |
| --------- | ----------------------------------------------------------------------------------------------------------------------- |
| `input`   | Required. User message sent to the agent.                                                                               |
| `scorers` | Optional. If present, at least one scorer. At least one turn in each case must have scorers for a meaningful pass/fail. |

### Session context

Optional fields that scope agent chat. They may appear at **root**, **included file**, or **case** level. **Most-specific wins** (field replace, not deep merge).

| Field            | Type                               | Purpose                                      |
| ---------------- | ---------------------------------- | -------------------------------------------- |
| `dataModels`     | `{ space, externalId, version }[]` | Data models the agent may query              |
| `instanceSpaces` | `string[]`                         | Instance spaces the agent may query          |
| `appContext`     | `string`                           | Free-text context (for example active plant) |

After resolution, `dataModels` and `instanceSpaces` must both be present or both absent for a case. `appContext` may be set alone.

### Multi-file suites

```yaml title="eval/eval.yaml" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
include:
  - cases/maintenance.yaml
```

```text theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
eval/
  eval.yaml
  cases/
    maintenance.yaml
```

Included files must contain a non-empty `cases` array. They may declare file-level `tags` that are unioned into each case, and file-level `tools` / session context that override the root for every case in that file.

## Scorers

| `type`           | Required fields | Purpose                                                               |
| ---------------- | --------------- | --------------------------------------------------------------------- |
| `correctness`    | `criteria`      | Judge whether the answer meets quality criteria                       |
| `faithfulness`   | `groundTruth`   | Judge whether the answer is grounded in facts (not sent to the agent) |
| `toolSelection`  | —               | Judge whether the agent chose appropriate tools                       |
| `toolInvocation` | —               | Judge tool-call argument quality                                      |

```yaml theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
- type: correctness
  criteria: "The answer should mention Paris"

- type: faithfulness
  groundTruth: |
    Pump P-101 was shut down on 2024-03-15 due to bearing wear.

- type: toolSelection
- type: toolInvocation
```

### Tool scorers

`toolSelection` and `toolInvocation` judge agent **tool use** against a declared tool list. Both share the same inheritance, skip behavior, and optional per-scorer `tools` override.

**Eval `tools` vs `<name>.agent.yaml` `tools`:** These are different. `<name>.agent.yaml` declares tools for the agent runtime (config-time names like `query`). Eval `tools` declare what the *judge* should expect the agent to *call* at runtime (names like `find_assets`, `execute`). Put parameter details in each tool's `description` so `toolInvocation` can check arguments.

**When to declare eval `tools`:**

* **Custom tools** (config name = runtime name): tool scorers can use `<name>.agent.yaml` tools when no eval `tools` are set.
* **Platform tools** (`query`, `ask_document`, and similar): the runtime often exposes sub-tools (`find_assets`, `list_views`, …). Declare those runtime names in eval `tools`, or score outcomes with `correctness` / `faithfulness` instead.
* **System tools** (sandbox `execute`, and similar): not in `<name>.agent.yaml` — declare them in eval `tools`.
* **`--external-id` / `--system-agent`:** local `<name>.agent.yaml` is not loaded — supply eval `tools` if you use tool scorers.

Discover runtime names by running once with `--verbose` or `--report-dir` and inspecting `toolsCalled:`.

**Empty tool list:** If no tools resolve after the cascade, the tool scorer is **skipped** with a stderr warning; other scorers on the turn still run. A case whose only scorers were skipped tool scorers fails the verdict.

### Tools for tool scorers

Optional `tools` field for `toolSelection` and `toolInvocation`. Each entry needs non-empty `name` and `description`. Empty `tools: []` is rejected.

Inheritance matches [session context](#session-context): **most-specific-wins**, whole-array **replace** (not merge-by-name):

1. Root `eval/eval.yaml`
2. Included file
3. Individual case
4. Scorer on a turn (overrides everything above for that scorer only)

If no eval level declares `tools`, eval falls back to `tools[]` in local `<name>.agent.yaml` when that file is loaded.

```yaml title="eval/eval.yaml" theme={"languages":{"custom":["/_languages/kuiper.json","../_languages/kuiper.json"]}}
tools:
  - name: find_assets
    description: |
      Find assets in the knowledge graph.
      Parameters: filter (string, required), limit (integer, optional).

cases:
  - id: find-pumps
    turns:
      - input: "Find pumps"
        scorers:
          - type: toolSelection
          - type: toolInvocation
            tools:
              - name: find_assets
                description: Must use find_assets with a name filter for pumps.
```

## Machine-readable schema

Coding agents and editors can validate suites against the published JSON Schema:

* [agent-eval-cases.schema.json](/assets/schemas/agent-eval-cases.schema.json)

Keep local cases compatible with this schema when you edit `eval/eval.yaml`. Loader rules that are harder to express in JSON Schema (unique ids across includes, scored-turn requirements, `dataModels`/`instanceSpaces` pairing) are still enforced by `cognite agents eval` at runtime.

## Related information

* [Building Atlas AI agents with the Cognite CLI](./agents)
* [Agent definition (`<name>.agent.yaml`)](./agent-yaml)
* [Running agent evaluations](/cdf/atlas_ai/guides/atlas_ai_agent_evaluating)
* [Command reference: agents eval](./reference#agents-eval)