Evals & queue
Open Evals → Evals (/admin/settings/ai/evals) to configure the server evaluation runner. Credentials live on AI Providers; questions and prompt templates live in the Eval Catalog.
Provider and judge models
Section titled “Provider and judge models”In Runner → Evals agent, choose the Provider used for automatic evaluations. Evals:Agent:Provider defaults to Auto (Kurrent AI preferred). Auto uses Kurrent AI when available, otherwise the first available provider in this order: Anthropic, OpenAI, OpenRouter, Cohere.
An explicit provider selection is a pin: if it is unavailable, evaluation does not silently switch vendors. Existing Evals:AutoEval:Provider values remain a fallback when the new provider key is unset; saving Auto overrides that legacy choice.
Under Eval judge, choose the model for each configured vendor. A vendor appears after its credentials are available. Models are per vendor; there is no shared Evals:Agent:Model field.
| Configuration key | Built-in default |
|---|---|
Evals:ServerRunner:Anthropic:Model | claude-sonnet-4-6 |
Evals:ServerRunner:OpenAI:Model | gpt-5-mini |
Evals:ServerRunner:Cohere:Model | command-a-03-2025 |
Evals:ServerRunner:OpenRouter:Model | google/gemini-2.5-flash |
Kurrent AI’s judge model is managed at deployment. The local daemon runner uses its local agent configuration. Managed Kurrent AI evaluations are available on Free as well as paid plans.
Run limits
Section titled “Run limits”Set these under Runner → Run limits, then click Save.
| Configuration key | Range | Default | Meaning |
|---|---|---|---|
Evals:ServerRunner:RunTimeoutSeconds | 60–3600 seconds | 1800 | Total run budget. |
Evals:ServerRunner:JudgeTimeoutSeconds | 30–3600 seconds | 300 | Budget for one question’s judge invocation. |
Evals:ServerRunner:RetrospectiveReserveSeconds | 0–900 seconds | 120 | Time reserved within the run budget for the retrospective. |
Evals:ServerRunner:MaxTraceChars | 10000–2000000 | 400000 | Maximum trace characters substituted into a judge prompt. |
One question timing out does not stop the other judges. At the retrospective boundary, completed verdicts can be retained while unanswered questions are marked failed. Trace truncation keeps the most recent events and marks omitted content.
Automatic evaluations and promotion
Section titled “Automatic evaluations and promotion”Per-repository settings has two controls:
- Auto-evaluate on close queues evaluations when sessions end. It requires a configured server eval provider. Turning it on also queues eligible ended sessions that have not been evaluated.
- Promotion floor sets the minimum judge-fact weight for promotion to repository guidelines. Clear it to use the displayed default; an explicit value must be at least 1.
Scopes without an explicit override default to auto-evaluation on when Kurrent AI is the effective provider. A default chip identifies inherited defaults. Toggle a scope off to opt out.
On plans with Projects, Per-project settings governs every repository in a confirmed project. Those repositories show via project in their rows; change the project setting instead. Enabling a project also queues eligible unevaluated ended sessions in its repositories.
These table controls save immediately, independently of the Runner’s Save button.
Judge-fact deduplication
Section titled “Judge-fact deduplication”Under Runner → Judge-fact deduplication, enable the switch and save to embed findings and combine near-duplicates into clusters. Configure Embeddings first. Enabling deduplication backfills findings retained while it was off; disabling it keeps raw findings without computing new deduplication embeddings.
Eval Queue
Section titled “Eval Queue”Open Evals → Eval Queue (/admin/eval-queue) to inspect waiting, in-flight, failed, and cancelled server runs. The server processes one queued evaluation at a time, with manual runs ahead of automatic runs; each run can still execute multiple question judges concurrently.
| Section | What to do |
|---|---|
| Breakers | Inspect vendor-wide failures such as invalid credentials or exhausted quota. Correct the cause, then use Reset to retry immediately, or wait for the next cooldown probe. Held runs stay pending without consuming attempts. |
| Queue | Check manual and automatic pending counts, the current run, and the oldest waiting timestamp. |
| Pending | Inspect priority, attempts, next-attempt time, and vendor/model. Cancel removes a waiting run and suppresses that session from automatic evaluation until re-enqueued. In-flight runs cannot be cancelled here. |
| Dead letter | Review runs that exhausted retries. Re-enqueue an individual run or use Re-enqueue all dead; bulk recovery skips sessions that already have an evaluation. |
| Cancelled | Re-enqueue a cancelled run to lift its automatic-evaluation suppression, individually or in bulk. |
See Evaluations for running and reading evaluations as a user.