Evals on Shane Caldwell https://hackbot.dad/tags/evals/ Recent content in Evals on Shane Caldwell Shane Caldwell https://hackbot.dad/ https://hackbot.dad/ Hugo -- 0.146.2 en-us Mon, 20 Apr 2026 00:00:00 +0000 No Autonomy Without Scalable Oversight https://hackbot.dad/writing/no-autonomy-without-scalable-oversight/ Mon, 20 Apr 2026 00:00:00 +0000 https://hackbot.dad/writing/no-autonomy-without-scalable-oversight/ What to expect as we enter the Year of The Judge ProofJudge: Can we align vibe-proving with human taste? https://hackbot.dad/writing/proofjudge-and-taste/ Sun, 29 Mar 2026 00:00:00 +0000 https://hackbot.dad/writing/proofjudge-and-taste/ Towards measuring alignment with human taste in autoformalization with judge agents. The Tests All Pass https://hackbot.dad/writing/tests-all-pass/ Sat, 14 Mar 2026 00:00:00 +0000 https://hackbot.dad/writing/tests-all-pass/ METR's SWE-bench analysis shows us taste isn't verifiable. Offsec Evals: Growing Up In The Dark Forest https://hackbot.dad/writing/offsec-evals-dark-forest/ Tue, 28 Oct 2025 00:00:00 +0000 https://hackbot.dad/writing/offsec-evals-dark-forest/ If you contribute a public benchmark, are you giving free capability to your competitors? GPT-5 is Good, Actually: The Agony and Ecstasy of Public Benchmarks https://hackbot.dad/writing/agony-and-ecstasy-evals/ Sun, 17 Aug 2025 00:00:00 +0000 https://hackbot.dad/writing/agony-and-ecstasy-evals/ An attempt to explain why benchmarks are either bad or secret, and why the bar charts don't matter so much.