Evaluation API¶

kavalai.eval grades an agent server that is already running. It speaks HTTP and discovers the agent’s input and output types from the server’s OpenAPI specification through AgentClient, so it knows nothing about the workflow engine, the YAML graph or the database behind it.

Two evaluators share that plumbing and differ only in how they judge an answer: SimpleEvaluator compares it with expected values and calls no model at all, while JudgeEvaluator asks a model whether it satisfies a plain-language criterion. Either can be called straight from a test.

For the ideas, read Evaluation & acceptance testing; for the case-file keys and the command-line flags, Evaluation YAML & CLI.

Shared plumbing¶

Shared plumbing for the evaluators: one agent call, one result.

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

class kavalai.eval.base.EvalResult(*, name: str, passed: bool, reason: str = '', inputs: dict[str, Any] = {}, output: dict[str, Any] | None = None)[source]¶

Bases: BaseModel

The verdict on a single case.

Variables:¶

name : str¶: Name of the case, for reporting.
passed : bool¶: Whether the agent’s answer satisfied the expectation.
reason : str¶: Why it failed; empty when it passed.
inputs : dict[str, Any]¶: What was sent to the agent.
output : dict[str, Any] | None¶: What the agent answered, or None if the run never got that far (a connection error, an input that does not fit the agent’s input type).

name : str¶

passed : bool¶

reason : str¶

inputs : dict[str, Any]¶

output : dict[str, Any] | None¶

model_config : ClassVar[ConfigDict] = {}¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class kavalai.eval.base.AgentEvaluator(base_url: str, username: str | None = None, password: str | None = None, timeout: float = 120.0, tag: str | None = None, transport: AsyncBaseTransport | None = None)[source]¶

Bases: object

Base class for evaluators that call a running agent server.

It owns one AgentClient, which discovers the agent’s input and output types from the server’s OpenAPI spec on first use. Subclasses only have to decide whether an answer is good.

Parameters:¶

base_url: str¶: Where the agent server listens. There is no default: the agent under evaluation is named on the command line, never in a case file, so a suite cannot quietly grade the wrong server.
username: str | None = None¶: HTTP Basic Auth user, if the server requires one.
password: str | None = None¶: HTTP Basic Auth password.
timeout: float = 120.0¶: Seconds to wait for one agent run.
tag: str | None = None¶: Names this evaluation run inside the external_id each case is recorded under — a model version, a prompt variant, a build. It is what tells one run’s sessions from another’s afterwards.
transport: AsyncBaseTransport | None = None¶: Optional httpx transport — in tests, this serves the requests without a network.

external_id(name: str) → str[source]¶

The identifier one case’s session is recorded under.

eval: is the reserved prefix the backoffice sessions page filters on; the tag, when there is one, separates this run’s sessions from the next run’s, which is what makes two variants comparable after the fact.

async run_agent(inputs: dict[str, Any], external_id: str | None = None) → BaseModel[source]¶

Send one input to the agent and return its output.

Each call starts a fresh session, so cases cannot leak conversation history into each other. inputs is validated against the agent’s own input type before it is sent, which turns a mistyped field into a clear error instead of a puzzling answer.

Parameters:¶

inputs: dict[str, Any]¶: Field values for the agent’s input type.
external_id: str | None = None¶: Optional identifier recorded with the session.

Returns:¶

An instance of the agent’s output type.

async evaluate(inputs: dict[str, Any], expected: Any = None, name: str = 'case') → EvalResult[source]¶: Run one case and judge the answer.

Literal comparison¶

Check an agent’s answer against expected values, without a model.

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

kavalai.eval.simple_evaluator.is_matcher_spec(spec: Any) → bool[source]¶

Whether spec is a matcher mapping rather than a literal value.

A dict is read as matchers only when every key is a known matcher name, so an agent that genuinely answers with a dict can still be compared for equality.

kavalai.eval.simple_evaluator.check_field(name: str, actual: Any, spec: Any) → list[str][source]¶

Check one output field against its expectation.

Parameters:¶

name: str¶: Field name, used in the failure messages.
actual: Any¶: What the agent answered in that field.
spec: Any¶: Either a literal value to compare for equality, or a mapping of matcher name to argument (equals, contains, not_contains, regex, one_of).

Returns:¶

A list of failure messages; empty when the field is as expected.

kavalai.eval.simple_evaluator.check_output(output: dict[str, Any], expected: dict[str, Any] | None) → list[str][source]¶

Check every expected field of one agent answer.

Fields the expectation does not mention are ignored, so a case states what it cares about and nothing more.

class kavalai.eval.simple_evaluator.SimpleEvaluator(base_url: str, username: str | None = None, password: str | None = None, timeout: float = 120.0, tag: str | None = None, transport: AsyncBaseTransport | None = None)[source]¶

Bases: AgentEvaluator

Send one input to a running agent and check the answer literally.

Use it wherever the right answer is a fact rather than a matter of phrasing — an extracted field, an id, a classification, a number that has to appear. It calls no model of its own, so it is fast, free and gives the same verdict every time.

Example

evaluator = SimpleEvaluator("http://localhost:25000")
result = await evaluator.evaluate(
    {"user_message": "Who is the president of Green Village?"},
    {"agent_response": {"contains": "Thomas Cook"}},
)
assert result.passed, result.reason

async evaluate(inputs: dict[str, Any], expected: dict[str, Any] | None = None, name: str = 'case') → EvalResult[source]¶

Run one case and compare the answer with expected.

Parameters:¶

inputs: dict[str, Any]¶: Field values for the agent’s input type.
expected: dict[str, Any] | None = None¶: Output field name to expected value or matcher mapping. An empty expectation asserts only that the agent answered at all, which is a useful smoke test in its own right.
name: str = 'case'¶: Case name, carried into the result.

Returns:¶

The EvalResult for this case. A failed agent call is reported as a failure rather than raised, so one broken case cannot end a whole run.

Model-graded comparison¶

Let a model decide whether an agent’s answer is acceptable.

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

class kavalai.eval.judge_evaluator.JudgeVerdict(*, passed: bool, reason: str = '')[source]¶

Bases: BaseModel

What the judging model answers.

passed : bool¶

reason : str¶

model_config : ClassVar[ConfigDict] = {}¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class kavalai.eval.judge_evaluator.JudgeEvaluator(base_url: str, username: str | None = None, password: str | None = None, timeout: float = 120.0, tag: str | None = None, transport: AsyncBaseTransport | None = None, model: str = 'openai/gpt-5.6-luna', llm_client: BaseLlmClient | None = None, prompt: str = 'You are grading one response of an AI agent under test.\n\nThe agent was given this input:\n{inputs}\n\nThe agent responded with:\n{output}\n\nThe response passes if, and only if, it satisfies this criterion:\n{criterion}\n\nJudge the criterion above and nothing else — do not add requirements of your\nown, and do not reward or punish style, length or politeness unless the\ncriterion asks about them. Set `passed` to true or false. When it is false,\n`reason` must say in one sentence what the response got wrong; when it is\ntrue, leave `reason` empty.\n', llm_parameters: dict | None = None)[source]¶

Bases: AgentEvaluator

Send one input to a running agent and have a model grade the answer.

Use it where the right answer cannot be written down in advance — an explanation, a refusal, a comparison, anything where wording varies but the substance must hold. The expectation is a plain-language criterion, and a failing case comes back with the judge’s reason.

The judging model is only built when a case is actually judged, so a run of purely SimpleEvaluator cases needs no API key.

Parameters:¶

base_url: str¶: Where the agent server listens.
username: str | None = None¶: HTTP Basic Auth user, if the server requires one.
password: str | None = None¶: HTTP Basic Auth password.
timeout: float = 120.0¶: Seconds to wait for one agent run.
tag: str | None = None¶: Names this run inside each case’s external_id.
transport: AsyncBaseTransport | None = None¶: Optional httpx transport, used by tests.
model: str = 'openai/gpt-5.6-luna'¶: provider/model of the judge.
llm_client: BaseLlmClient | None = None¶: A ready-made judge client, overriding model.
llm_parameters: dict | None = None¶: llm_kwargs for the judge (temperature, timeouts, …) — kavalai-eval passes the KAVALAI_LLM_* defaults here.
prompt: str = 'You are grading one response of an AI agent under test.\n\nThe agent was given this input:\n{inputs}\n\nThe agent responded with:\n{output}\n\nThe response passes if, and only if, it satisfies this criterion:\n{criterion}\n\nJudge the criterion above and nothing else — do not add requirements of your\nown, and do not reward or punish style, length or politeness unless the\ncriterion asks about them. Set `passed` to true or false. When it is false,\n`reason` must say in one sentence what the response got wrong; when it is\ntrue, leave `reason` empty.\n'¶: The grading prompt; must accept {inputs}, {output} and {criterion}.

Example

evaluator = JudgeEvaluator("http://localhost:25000")
result = await evaluator.evaluate(
    {"user_message": "What is the village's annual budget?"},
    "The answer says the information is not available "
    "instead of inventing a number.",
)
assert result.passed, result.reason

property llm_client : BaseLlmClient¶: The judging model, built on first use.

build_prompt(inputs: dict[str, Any], output: dict[str, Any], criterion: str) → str[source]¶: Render the grading prompt for one case.

async judge(inputs: dict[str, Any], output: dict[str, Any], criterion: str) → JudgeVerdict[source]¶: Ask the judging model whether one answer meets the criterion.

async evaluate(inputs: dict[str, Any], expected: str | None = None, name: str = 'case') → EvalResult[source]¶

Run one case and let the judging model grade the answer.

Parameters:¶

inputs: dict[str, Any]¶: Field values for the agent’s input type.
expected: str | None = None¶: The criterion the answer has to satisfy, in plain language.
name: str = 'case'¶: Case name, carried into the result.

Returns:¶

The EvalResult for this case, its reason taken from the judge when the case failed.

Raises:¶

ValueError – expected is missing. A judged case with nothing to judge against would pass on anything at all.

Running a file of cases¶

Run a YAML file of cases against a running agent server.

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

The file looks like this:

name: green-village
judge_model: openai/gpt-5.6-luna   # optional
cases:
  - name: president
    input: {user_message: Who is the president of Green Village?}
    expected:
      agent_response: {contains: Thomas Cook}
  - name: no_budget
    type: judge
    input: {user_message: What is the village's annual budget?}
    expected: The answer says it does not know instead of inventing one.

The file says nothing about which server to run against — that is named on the command line, so the same cases can be pointed at a laptop, a staging deployment or two model versions in turn without being edited:

uv run --env-file .env kavalai-eval cases.yaml --port 25000 --tag gpt-5.6-luna

class kavalai.eval.eval_runner.EvalCase(*, name: str, type: 'simple' | 'judge' = 'simple', input: dict[str, Any], expected: dict[str, Any] | str | None = None)[source]¶

Bases: BaseModel

One input, and what its answer has to look like.

Variables:¶

name : str¶: How the case is reported.
type : Literal['simple', 'judge']¶: simple compares the answer with expected values; judge asks a model whether the answer is acceptable.
input : dict[str, Any]¶: Field values for the agent’s input type.
expected : dict[str, Any] | str | None¶: A mapping of output field to expected value or matcher for a simple case; a plain-language criterion for a judged one.

model_config : ClassVar[ConfigDict] = {'extra': 'forbid'}¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

name : str¶

type : Literal['simple', 'judge']¶

input : dict[str, Any]¶

expected : dict[str, Any] | str | None¶

class kavalai.eval.eval_runner.EvalSuite(*, name: str, judge_model: str = 'openai/gpt-5.6-luna', cases: list[EvalCase])[source]¶

Bases: BaseModel

A named list of cases, and the model that judges the judged ones.

Deliberately no base_url: which agent a suite is run against is a property of the run, not of the cases.

model_config : ClassVar[ConfigDict] = {'extra': 'forbid'}¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

name : str¶

judge_model : str¶

cases : list[EvalCase]¶

kavalai.eval.eval_runner.load_suite(path: str | Path) → EvalSuite[source]¶

Read and validate a suite file.

Parameters:¶

path: str | Path¶: Path to the YAML file.

Returns:¶

The parsed EvalSuite.

Raises:¶

pydantic.ValidationError – The file does not describe a suite.

async kavalai.eval.eval_runner.run_suite(suite: EvalSuite, base_url: str, username: str | None = None, password: str | None = None, timeout: float = 120.0, judge_model: str | None = None, judge_parameters: dict | None = None, tag: str | None = None, transport: Any | None = None, on_result: Callable[[EvalResult], None] | None = None) → list[EvalResult][source]¶

Run every case in suite, in order, against a running agent.

The cases run one at a time: an evaluation that is easy to read while it runs is worth more than one that finishes a few seconds sooner.

Parameters:¶

suite: EvalSuite¶: The cases to run.
base_url: str¶: Where the agent server listens.
username: str | None = None¶: HTTP Basic Auth user, if the server requires one.
password: str | None = None¶: HTTP Basic Auth password.
timeout: float = 120.0¶: Seconds to wait for one agent run.
judge_model: str | None = None¶: provider/model of the judge, overriding the suite’s.
judge_parameters: dict | None = None¶: llm_kwargs for the judge.
tag: str | None = None¶: Names this run inside each case’s external_id, so the sessions of two runs — two model versions, two prompts — can be told apart in the agent database afterwards.
transport: Any | None = None¶: Optional httpx transport, used by tests.
on_result: Callable[[EvalResult], None] | None = None¶: Called with each result as it arrives, for progress output.

Returns:¶

One EvalResult per case, in file order.

kavalai.eval.eval_runner.format_result(result: EvalResult) → str[source]¶: One result as a single report line.

kavalai.eval.eval_runner.format_summary(results: list[EvalResult]) → str[source]¶: The closing line of a run.

kavalai.eval.eval_runner.parse_args(argv: list[str] | None = None) → Namespace[source]¶: Parse the command line.

kavalai.eval.eval_runner.resolve_base_url(host: str, port: int) → str[source]¶: Build a base URL from the host and port flags.

kavalai.eval.eval_runner.main(argv: list[str] | None = None) → int[source]¶

Entry point of the kavalai-eval console script.

Returns:¶: 0 when every case passed, 1 when a case failed, and 2 when the run never reached a verdict — a CI job needs the third to tell “the suite is broken” from “the agent is wrong”.