Eval catalog

Open Evals → Eval Catalog (/admin/evals) to manage the questions and prompts used by server evaluations. The page has Questions and Catalog prompts tabs. Changes apply to subsequent evaluations without a deploy.

Provider selection, models, budgets, and automatic evaluations belong on Evals.

Add sections and questions

On Questions, click Add section, enter Display name (Required), and click Create. You do not need to supply a key or order number.

Click Add question, choose its Section, enter its Title, optionally enter Question text, and click Create. At least one live section is needed before adding a question.

The table shows each question’s section, title, and enabled state. Use the section selector to move a question to another live section. Rename changes a section’s display name. Remove section is offered once the section has no live questions; removed sections can be restored.

Edit a question

Select a question row to open Edit question. Edit its title and question text, inspect the full prompt preview, then click Save. Discard changes restores the saved draft; Close closes the editor.

The preview substitutes the question text into the shared question template, so it shows the effective prompt rather than only your question. The Version selector loads prior question text for review or reuse. Saving older text creates another version instead of deleting later history.

Use the row’s Enabled switch to exclude a question from future runs without deleting it. Delete soft-deletes it; Restore brings it back. Deleted questions cannot be enabled until restored. These structural controls apply immediately.

Question template

On Catalog prompts, the Question template wraps every question’s text. It must include {QUESTION_TEXT}; the editor will not save a template without that placeholder.

Changing this template creates new effective prompt versions for all questions. Review its effect carefully, since the next evaluations will use that shared framing. The template has its own version selector, Save, and Discard changes controls.

Retrospective

The Retrospective prompt controls the synthesis pass that produces evaluation suggestions, issues, and strengths. It has a separate editor and version history from the question template.

Concurrent edits

Catalog changes update other open admin views. Save or discard an in-progress draft before switching what you are editing. An indicator on each tab identifies unsaved changes.

See Evaluations for the resulting session readout and Eval Queue for run status.