> ## Documentation Index
> Fetch the complete documentation index at: https://agno-v2-codex-homepage-20260719-015142.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Evals

> Turn expected agent behavior into repeatable checks for development, CI, and scheduled runs.

Probes help you discover weaknesses. Evals turn the behavior you care about into repeatable checks that can run before release, in CI, and on a schedule.

## Cases

Cases live in `evals/cases.py`. Each case sends one input to an agent (`agent=`) and can run up to two checks:

* **judge**: `AgentAsJudgeEval` scores the response against `criteria` (binary pass/fail) using an LLM.
* **reliability**: `ReliabilityEval` checks which tools fired against `expected_tool_calls`.

A case looks like this:

```python evals/cases.py theme={null}
from os import getenv

# The WebSearch agent calls parallel_search when PARALLEL_API_KEY is set, web_search otherwise.
_WEB_SEARCH_TOOL = "parallel_search" if getenv("PARALLEL_API_KEY") else "web_search"

CASES: tuple[Case, ...] = (
    Case(
        name="web_search_recent_anthropic_research",
        agent=web_search,
        input="What did Anthropic publish about agent research recently?",
        tags=("live",),
        timeout_seconds=120,
        criteria=(
            "Answers the question by citing at least one real Anthropic URL "
            "(anthropic.com domain). The response is grounded in fetched content "
            "rather than refusing to answer."
        ),
        expected_tool_calls=(_WEB_SEARCH_TOOL,),
    ),
    # add more cases here
)
```

A case can use either check or both. If both are set, the agent runs once and feeds the same response into both.

Add tags to group your cases into suites. The template uses three tags: `smoke`, `release`, and `live`. This case uses the `live` tag because its answer depends on the open web.

## Run the suite

The suite imports agents on the host and writes results to Postgres through `eval_db`. Start the database, activate the local virtual environment, and set `OPENAI_API_KEY` in `.env`.

<Steps>
  <Step title="Start Postgres">
    ```bash theme={null}
    docker compose up -d agentos-db
    ```
  </Step>

  <Step title="Create a virtual environment">
    The eval suite runs on the host and needs a local virtual environment:

    ```bash theme={null}
    ./scripts/venv_setup.sh
    ```

    Activate it:

    ```bash theme={null}
    source .venv/bin/activate
    ```
  </Step>

  <Step title="Run the eval suite">
    ```bash theme={null}
    python -m evals --tag smoke      # fast checks
    python -m evals --tag release    # pre-release suite
    python -m evals --tag live       # checks that depend on live sources
    python -m evals                  # all cases
    ```

    Other options:

    ```bash theme={null}
    python -m evals --name <case>                       # one case while iterating
    python -m evals --tag release --json-output out.json
    python -m evals -v                                  # stream full run panels
    ```
  </Step>
</Steps>

Each case prints its response and the verdicts for the checks it defines. The run ends with an `Eval Summary` table.

Results are written to Postgres via `eval_db`. The eval history appears on [os.agno.com](https://os.agno.com) alongside your sessions and traces, so you can compare inputs, responses, and verdicts across runs.

## Diagnose failures with your coding agent

Open your coding agent and ask it to run:

```text theme={null}
Run the eval-and-improve skill in .agents/skills.
```

The coding agent runs the suite and classifies failures as bad criteria, real regressions, or unreliable LLM judgments. It proposes scoped fixes, updates the agent or case, and re-runs the affected checks.

## Choose when to run evals

These behavioral checks call models, so choose a cadence that fits the cost and confidence you need.

| Situation                   | Suggested run                                                  |
| --------------------------- | -------------------------------------------------------------- |
| While changing one behavior | Run the affected case with `--name <case>`.                    |
| Before a release            | Run the `release` tag when you want broader confidence.        |
| In CI                       | Use stable `smoke` or `release` cases. Leave `live` cases out. |
| After changing a model      | Run the cases for affected agents.                             |
| On a schedule               | Run `smoke` cases when recurring model calls are useful.       |

The template ships a `run_evals` workflow for scheduled checks. Scheduled evals are off by default because they make model calls. Set `ENABLE_SCHEDULED_EVALS=True` to run the `smoke`-tagged cases daily. See [scheduling](/features/scheduling) for the cron API.

## What good cases look like

* **Specific.** "Returns a JSON object with `ticker` and `price`" beats "Returns the right answer".
* **Stable.** Avoid prompts whose correct answer changes daily. Use phrasing like "describes a real, recent..." so the case remains useful as the answer changes.
* **Scoped to one behavior.** One case per behavior makes failures easy to read.
* **Anchored to tools.** `expected_tool_calls` catches responses that skip a required tool call.

## Next

[Run your platform on Railway →](/agent-platform/run-railway)