Evals
Settings → Evals (/admin/settings/ai/evals) configures how the server-side LLM-as-judge runner behaves once judge-vendor credentials are in place: which judge vendor and model to use, how long a run is allowed to take, how much trace it sees, which repos auto-evaluate, and whether judge findings get deduplicated with embeddings. Judge-vendor API keys live on AI Providers — this page picks the active judge vendor and sets behavior, not credentials.
Evals agent
Section titled “Evals agent”Selects the judge vendor and model the server-side runner uses by default. Credentials for the chosen judge vendor come from AI Providers.
| Key | Type | Default | Description |
|---|---|---|---|
Evals:Agent:Provider | enum Anthropic / OpenAI / KurrentAI / Cohere / OpenRouter | Anthropic (runtime) | Default judge vendor for server eval runs. |
Evals:Agent:Model | string | claude-sonnet-4-6 | Default judge model. |
Both fields render blank in the form and fall back at runtime:
- Provider — an unset value falls back to the legacy
Evals:AutoEval:Providerif present, otherwise Anthropic. - Model — an unset value falls back to the selected judge vendor’s configured judge model, otherwise that judge vendor’s built-in default.
Run limits
Section titled “Run limits”Per-run budgets for the server-side runner:
| Key | Range | Default | Description |
|---|---|---|---|
Evals:ServerRunner:RunTimeoutSeconds | 60–3600 | 1800 (30 min) | Wall-clock budget for an entire eval run. On expiry the run surfaces as EvalFailed("cancelled"). |
Evals:ServerRunner:JudgeTimeoutSeconds | 30–3600 | 300 (5 min) | Wall-clock budget for a single judge call. |
Evals:ServerRunner:RetrospectiveReserveSeconds | 0–900 | 120 | Seconds reserved out of the run timeout for the retrospective synthesis pass. |
Evals:ServerRunner:MaxTraceChars | 10000–2000000 | 400000 (~100k tokens) | Upper bound on the formatted trace passed to each judge prompt. When exceeded, the orchestrator keeps the most recent events and prepends a marker noting how many were elided. |
Per-repository settings
Section titled “Per-repository settings”A table of the repositories visible to you, each with two per-repository overrides:
- Auto-evaluate on close — runs an eval automatically when a session on this repo closes. Disabled until a server eval runner is configured (see the Evals agent above).
- Fact-promotion threshold (integer ≥ 1) — the minimum weight a judge fact needs before it’s promoted into repo guidelines. The placeholder shows the default minimum weight.
Judge-fact deduplication
Section titled “Judge-fact deduplication”JudgeFactDedup:Enabled is the master switch for the semantic dedup pipeline:
- Off — eval-generated judge facts are retained verbatim, and no embeddings are computed.
- On — candidate facts are embedded and compared against the existing pool; near-duplicates collapse into clusters. Toggling this on also triggers a background backfill of any facts that accumulated while dedup was off.
Deduplication requires an active embedding provider with its API key set — configure that on Embeddings.
The eval catalog
Section titled “The eval catalog”Evals also has a link out to the eval catalog — the set of evals the server-side runner can execute:
Edit the eval catalog → /admin/evals.
Related
Section titled “Related”- AI providers — judge-vendor credentials for the runner.
- Embeddings — the embedding provider that powers judge-fact deduplication.
- Eval catalog — the question and template library evals draw from.
- Evaluations — the user-facing feature these settings support.