ProofJudge: Can we align vibe-proving with human taste?
Towards measuring alignment with human taste in autoformalization with judge agents.
Evals
The Tests All Pass
METR’s SWE-bench analysis shows us taste isn’t verifiable.
Offsec Evals: Growing Up In The Dark Forest
If you contribute a public benchmark, are you giving free capability to your competitors?
GPT-5 is Good, Actually: The Agony and Ecstasy of Public Benchmarks
An attempt to explain why benchmarks are either bad or secret, and why the bar charts don’t matter so much.