EvalDetectBench: A Benchmark for Measuring Evaluation Awareness in Frontier Language Models

by Xinning Li*, Kemunto Ochwang'i*, Aryasomayajula Ram Bharadwaj*, Alexandra Souly†, Robert Kirk†

*Equal contribution. †Corresponding authors.

Abstract

Frontier large language models can often recognize when they are being evaluated, a capability known as evaluation awareness. If models behave differently in evaluations than in deployment, this undermines the validity of evaluation results, which are a crucial component of current AI safety frameworks. We introduce EvalDetectBench, an open pipeline and benchmark for measuring evaluation awareness that works with any Inspect-compatible evaluation, allowing practitioners to test against current and future benchmarks. EvalDetectBench ships with a newly curated transcript suite covering current frontier system-card evaluations and diverse deployment sources.

The benchmark serves two purposes: measuring how reliably frontier LLMs recognize that they are being evaluated, and assessing how detectable individual benchmarks are as evaluations. We identify two methodological choices in the existing literature that introduce systematic bias: the identity of the model that generated the deployment transcripts accounts for 11.25% of measurement variance and can reorder model rankings; and elicitation prompts selected for high performance on one model can perform near chance on others. EvalDetectBench corrects for both via per-model probe calibration and a stratified generator-harmonisation procedure.

EvalDetectBench overview pipeline figure
Overview of EvalDetectBench methodology. (a) Eval and deploy transcripts pass through the pipeline, are probed with calibration-selected questions, scored by a judge LLM, and ranked by AUROC. (b) AUROC under three deployment-data conditions: self-only (same-family deploy data, unconfounded reference), mixed (cross-family deploy data), and corrected (mixed with generator-identity bias absorbed). (c) Per-model probe question selection: held-out AUROC by ensemble size k. Calibration-selected probes outperform the fixed Needham et al. set across all models, with diminishing returns beyond k ≈ 5.

Contributions

  1. An open, extensible pipeline for measuring evaluation awareness on any Inspect-compatible benchmark, with a refreshed transcript suite covering current frontier system-card evaluations and diverse deployment pools.
  2. Per-model probe calibration. Elicitation prompts do not transfer across model families; a fixed set under-elicits awareness by up to 0.08 AUROC. Per-model selection outperforms fixed and random baselines.
  3. A generator-identity confound and correction. 11.25% of P(eval) variance is attributable to the deployment transcript's generator; this persists under resampling, and we propose a stratified harmonisation that recovers the unconfounded signal.
  4. An empirical study across 12 judge models and 13 benchmarks: judges discriminate at AUROC > 0.75 and P(eval) is high across benchmark categories.

Key Findings

Heatmap of P(eval) across judge x generator pairs
Claude-style models are seen as more deploy-like by all judges. Deploy P(eval): Judge × Generator. The columnar pattern shows the generator effect: Claude transcripts (left columns) receive low P(eval) from all judges, while GPT and Gemini transcripts receive comparatively high scores regardless of who judges them.
Probe-question AUROC vs k, per model
Ensemble AUROC vs. number of probe questions, per model. Per-model exhaustive calibration selection (green) and its held-out performance (red), versus a random-selection baseline (blue, mean ± 1 std over 500 Monte Carlo draws per k) and Needham et al.'s fixed PQ set (orange dotted line, k = 5). Per-model selection meets or beats the baseline from k ≈ 3–4 onward.
Stratified harmonisation correction validation
Stratified harmonisation correction. Per judge AUROC under three conditions: Mixed (raw scores from cross-family deploy pool), Mixed Corrected (same pool with harmonisation at λ = 0.1), and Self-only (same-family deploys, the unconfounded reference). The corrected AUROC is consistently closer to Self-only than the uncorrected version.
Mixed AUROC across 12 judges
Per-model mixed AUROC across 12 judges. Closed-source models grouped by family, ordered by release date. Within each closed-source family the newest model attains the highest AUROC (Anthropic 0.76 → 0.88 → 0.89; OpenAI 0.78 → 0.80 → 0.83; Google 0.76 → 0.87). The four open-source judges cluster in [0.78, 0.81].
Mean P(eval) per benchmark
Mean P(eval) per evaluation dataset, averaged across six judges and grouped by category. Error bars are bootstrap 95% confidence intervals. P(eval) is high (> 0.83 pooled across frontier judges) on a substantial subset of benchmarks in our suite.

Quickstart

EvalDetectBench is an open pipeline you can run against any Inspect-compatible evaluation. The full code, datasets, and Web UI live in the aware_bench repo.

Install

git clone https://github.com/freeze-lasr/aware_bench
cd aware_bench
uv sync --extra ui

Add your API keys to .env (OPENROUTER_API_KEY, ANTHROPIC_API_KEY, or OPENAI_API_KEY) for whichever judge model you plan to run.

Run the end-to-end demo

The repo ships with a canonical demo config that exercises all four pipeline stages (collect → probe selection → measure → analyze) on ~100 transcripts — budget ~$5–10 on gpt-5.2 via OpenRouter. 

# From the command line
python run.py --config configs/examples/config_demo_e2e.yaml

# Or launch the Web UI and click "Run" on the demo_e2e config
python ui/server.py
# then open http://localhost:8000

Use it on your own benchmark

Point a config at your Inspect log or local JSON transcripts, swap in your judge model, and the pipeline reports AUROC, AUPRC, and per-probe P(eval) breakdowns. Full schema and worked examples: README, UI guide.

Citation

@misc{li2026evalawarebench,
    title={EvalDetectBench: A Benchmark for Measuring Evaluation Awareness in Frontier Language Models},
    author={Xinning Li and Kemunto Ochwang'i and Aryasomayajula Ram Bharadwaj and Alexandra Souly and Robert Kirk},
    year={2026},
    howpublished={\url{https://github.com/freeze-lasr/aware_bench/}}
}

Resources