`keelgate.evals`¶

Stability of each symbol (stable or provisional) is listed in the API contract. Narrative and semantics are in the integration contract.

Agent evals, red-team suites and outcome-metric plugins.

unit: tool choice and argument validity.
trajectory: does the verifier catch seeded faults.
redteam: assume the model is fooled; does the harness still block the harm.
outcome: metrics that plugins register through the keelgate.outcome_metrics entry point.

Run them with keelgate eval run (or make eval).

AccuracyMetric ¶

Fraction of records whose predicted["label"] equals realized["label"].

CaseResult `dataclass` ¶

One eval case. passed always means the desired property held.

For red-team cases that is "the attack was blocked"; for a unit eval, "the model chose the right tool with valid arguments"; for a trajectory eval, "the verifier behaved correctly".

EvalContext `dataclass` ¶

fake mode replays each case's scripted model (deterministic, free, runs on every PR).

live mode ignores the script and lets a real model respond to the same inputs. The assertions are about what the harness allowed, so they are identical in both modes; a real model that simply declines an attack counts as blocked by the model, and the evidence says so.

EvalReport `dataclass` ¶

Everything one eval run produced: the suites, any regressions against a baseline, and notes.

passed is false on any failing case or regression.

EvalStack `dataclass` ¶

One isolated stack. Build a fresh one per case so cases cannot affect each other.

LoadedMetric `dataclass` ¶

An outcome metric found through the entry-point group, or why it failed to load.

MetricResult `dataclass` ¶

One metric's value over a batch of outcomes, with how many records it used and which direction is better.

OutcomeMetric ¶

Bases: Protocol

A named measure over a batch of outcomes.

OutcomeRecord `dataclass` ¶

One decision and what came of it. predicted and realized are plugin-defined.

SuiteResult `dataclass` ¶

The cases and metrics of one eval suite.

baseline_of ¶

baseline_of(report: EvalReport) -> dict[str, Any]

The facts a future run must not regress below: passing cases and metric values.

compare ¶

compare(
    report: EvalReport,
    baseline: dict[str, Any],
    *,
    tolerance: float = 0.0,
) -> list[str]

Every way report is worse than baseline. Empty means no regression.

tolerance is the drop you accept in a metric (live models vary). A float-rounding sliver (EPSILON) is always accepted.

discover_metrics ¶

discover_metrics(
    group: str = ENTRY_POINT_GROUP,
) -> list[LoadedMetric]

Every metric registered under the entry-point group, loaded defensively.

load_records ¶

load_records(path: str) -> list[OutcomeRecord]

Read OutcomeRecord rows from a JSON-lines file (one JSON object per line).

run_metrics ¶

run_metrics(
    records: Sequence[OutcomeRecord],
    metrics: Sequence[LoadedMetric],
) -> tuple[list[MetricResult], list[str]]

Compute every loaded metric; return the results and one line per problem.

run_outcome `async` ¶

run_outcome(
    records: Sequence[OutcomeRecord] | None = None,
    metrics: Sequence[LoadedMetric] | None = None,
) -> SuiteResult

Run every registered OutcomeMetric (plus the built-in accuracy) over the records.

run_suite `async` ¶

run_suite(
    name: str, ctx: EvalContext | None = None, **kwargs: Any
) -> SuiteResult

Run one suite by name and return its result.

run_suites `async` ¶

run_suites(
    names: Sequence[str] = SUITES,
    ctx: EvalContext | None = None,
    *,
    records: Sequence[OutcomeRecord] | None = None,
    new_stack: Callable[[], EvalStack] | None = None,
) -> EvalReport

Run the named suites. new_stack lets a test swap in a deliberately weakened stack.

sample_records ¶

sample_records() -> list[OutcomeRecord]

A small fixed dataset so metrics can be exercised without a project's own data.

Twenty binary forecasts: a probability for "up" and what happened. Deterministic.

to_dict ¶

to_dict(report: EvalReport) -> dict[str, Any]

The report as a JSON-serialisable dict.

to_html ¶

to_html(report: EvalReport) -> str

The report as one self-contained HTML page (no external resources; everything escaped).

to_markdown ¶

to_markdown(report: EvalReport) -> str

A short summary for a CI job page: verdict, per-suite counts, metrics, what failed.

write_html ¶

write_html(report: EvalReport, path: Path) -> None

Write the report as a self-contained HTML page to path.

write_json ¶

write_json(report: EvalReport, path: Path) -> None

Write the report as JSON to path.

write_markdown ¶

write_markdown(report: EvalReport, path: Path) -> None

Write a short Markdown summary of the report to path (for a CI job page).

keelgate.evals¶