Web UI

What it is #

Running EvalDetectBench from the CLI is fine for one experiment. The UI exists for the cases the CLI doesn't handle gracefully: iterating on configs, watching long runs and reacting to failures, comparing two completed runs, and debugging when a run silently produces nothing (typically a hidden judge model inside an inspect task, a missing key, or a provider policy block).

Install & launch #

uv sync --extra ui                    # adds fastapi + uvicorn
python ui/server.py                   # serves on 127.0.0.1:8000
python ui/server.py --port 9000       # different port
python ui/server.py --host 0.0.0.0    # expose on LAN
python ui/server.py --reload          # dev mode, auto-restart on edits

You also need API keys for whichever providers your runs touch. Drop them in a .env at the repo root (auto-loaded by run.py):

OPENROUTER_API_KEY=sk-or-...
ANTHROPIC_API_KEY=sk-ant-...
OPENAI_API_KEY=sk-...

Layout #

The sidebar has two tabs — Runs (configs) and Probes — plus a top-bar Compare action. Everything is one page; clicking a sidebar item swaps the main panel. Each sidebar entry has a colored status dot:

Building a config #

Click + next to Runs to open the config builder. The form has five sections: Name, Data Sources, Target Models, Probe Questions, Settings.

Hidden judge / scorer detection #

Many inspect tasks include grader or judge model parameters that default to openai/... and silently bypass whatever generator you configured. The UI parses every task's AST, finds string parameters with model-ID-like defaults, and surfaces them as override fields with the OpenRouter-prefixed equivalent pre-filled. Twelve tasks in the standard inspect_evals install have such hidden defaults — among them agentharm, coconot, healthbench, simpleqa, instrumentaleval, and fortress_benign.

Target models #

This is the model that answers your probe questions about each transcript. Three-tier filter:

• Routing — OpenRouter / Direct / All
• Provider — OpenAI / Anthropic / Google / …
• Model list — multi-select checkboxes within the current filter

Validation is live against OpenRouter's /api/v1/models catalog. Models that appear in past configs but aren't in OpenRouter's current catalog get a ⚠ warning chip — typically typos (e.g. dash-form claude-opus-4-5 vs. canonical dot-form claude-opus-4.5) or retired model IDs. The custom-ID input below the list lets you type any model name not in the dropdown; unvalidated openrouter/* IDs trigger a confirm dialog before being added.

Per-source generator #

Each inspect source needs a generator (the model that produces trajectories). Same three-tier picker, inline in the source row. The generator is mandatory for inspect-backend sources — trying to preview or save without one shows a red border on the offending source and an explanatory alert.

Settings, preview, create #

Epochs, temperature, max tasks, cache toggle, batch mode — defaults match the recommended CLI defaults. Preview YAML renders the generated YAML inline; Create Config saves to configs/ and switches you to that config's overview page.

Running #

Selecting a config from the sidebar opens its overview: a header with edit / delete / run buttons, plus four cards summarising what'll happen — Models, Probes, Sources, Settings. Below that, the list of past runs.

The Run button expands an inline options bar with:

• Skip collect / Skip PQ sel. / Skip measure / Skip analyze — re-use existing stage outputs. Each is auto-disabled when no prior outputs exist for that stage (so you can't accidentally skip something that hasn't run yet).
• Force re-collect — even with cache enabled, regenerate trajectories from scratch.
• Allow dirty log dir — let inspect run on top of a pre-existing log directory without bailing.

A pre-flight check runs on Start: the server validates the config and catches model-dependent sources (e.g. inspect) that are missing a generator, refusing to launch with a clear error. The pipeline doesn't quietly drop sources and crash three stages later.

After Start, the page does not navigate away. The new run appears in the config's run list and auto-expands inline with the live pipeline strip and streaming log right where you clicked.

Inline run view #

Clicking the chevron on any run entry (or starting a new run) expands the run in place. You see the live pipeline strip, the full log panel, and — for completed runs — the per-combo results.

Pipeline strip #

Four boxes (or three when probe_selection isn't configured): collect → probe_selection → measure → analyze. Done is green with a check; running is yellow with a spinning arrow and a pulse; pending is grey.

Live log #

Tails the last 500 lines via SSE, refreshing whenever the file changes. ERROR / Traceback / Exception lines are colored red, WARNING in yellow, timestamps dimmed. A Load full log button appears if the log is longer than what's currently shown — clicking it fetches the whole file.

Sample progress #

When the measure or probe_selection stage is active, a progress bar reads the in-flight .eval files for sample-level progress and shows a percentage plus per-task mini-bars.

Editing YAML #

The header of any config view has a </> button that opens the in-browser YAML editor. Any run view has a dedicated Edit Config button alongside Stop / Resume.

Live pydantic validation underlines the offending line on error with the field name. Validation runs the same checks as run.py — if a save fails, the error names the offending field.

Probe questions #

The Probes tab in the sidebar lists every probe-question set under probe_questions/. Each set is a YAML list of closed-ended questions appended to transcripts during the measure stage; each question has a real / eval answer that the judge model picks between (binary), or a scale rating.

Each probe item is one of:

• Binary question with answer: { real: '0', eval: '1' } or similar. The judge picks one.
• Scale question with answer_type: scale, scale_min, scale_max. The judge returns a rating.

The optional template_config block controls how the question is appended to the transcript (method: append vs. inline, context-dependent prefixes, etc.). The downstream measure stage just executes whichever pq_file the config points at, filtered to pq_ids if provided.

Comparing runs #

The top-bar Compare button opens a side-by-side picker. Pick two completed runs and you get a table of per-combo AUROC / AUPRC / Accuracy with delta columns (green when B is better, red when worse).

Failure diagnoses #

Failures surface in three places:

1. Toast (top-right) — red toast with a diagnosed cause and one-line hint. Click to open the full error modal.
2. Native browser notification — same diagnosis, fires when the tab is backgrounded so long-running jobs can notify you while you're elsewhere.
3. Error modal — opens automatically when you visit a failed run's view. Shows the diagnosis as a yellow callout at the top with a clear title + hint, then the full Python traceback in a collapsed <details> block.

Patterns detected #

The server scans both the run's stdout log and any inspect .eval log files (where the real provider response lives, since stdout wraps provider errors in opaque RetryError). Detected patterns:

Architecture #

ui/
├── server.py       FastAPI app + subprocess management
├── frontend.html   Single-page client (Bootstrap 5 + vanilla JS, no build step)
├── discover.py     Live discovery: models, datasets, inspect tasks
└── README.md       Quickstart

Key design decisions:

• Subprocess, not in-process. The server shells out to python run.py --config ... as a subprocess and tracks PID + stage outputs. The UI restarting never affects in-flight runs.
• State on disk. Stage status is read from output/{experiment}/runs/{name}/{collect,probe_selection,measure,analyze}/, and per-run state from .run_state.json in the same dir. The UI is effectively a stateless reader plus a process launcher.
• SSE for live progress. /api/runs/{name}/progress streams stage status + sample progress + log tail every 2 seconds while a run is live; closes on terminal status.
• No build step. frontend.html is one file with inline CSS + JS; Bootstrap and CodeMirror load from CDN. Editing the HTML and refreshing the browser is the dev loop.
• UI independent of aware_bench/. All UI logic lives in ui/; the pipeline package is treated as an opaque CLI dependency.
• Validation is layered. Frontend blocks obvious mistakes (missing generator on inspect sources); server pre-flight re-validates so raw YAML edits can't slip past; aware_bench itself is the final source of truth.

Troubleshooting #

End-to-end demo #

The repo ships with a canonical demo config that exercises all four pipeline stages within a small budget (~$5–10 on gpt-5.2 via OpenRouter):

python run.py --config configs/examples/config_demo_e2e.yaml

Or from the UI: open the demo_e2e config in the Runs tab and click Run (top-right). Watch the pipeline strip turn green stage by stage; results appear when analyze completes. Validates your install end-to-end.