Evals Aren't Dead, But They're Low Res
Put down that ruler and go build a microscope.
AI security researcher · Dreadnode
Full Contact
Alignment
I build scalable oversight and control systems for autonomous agents operating in adversarial, high-consequence environments.
Shane CaldwellResearcher & engineer
Latest research
September 2026 · arXiv
ScopeBench: Do Agents Preserve Engagement Boundaries Under Goal Pressure?
We introduce ScopeBench, a methodological pilot that measures whether autonomous offensive-security agents respect engagement boundaries when completing an objective requires violating scope. We find scope adherence and capability can be measured independently.
Notes from the work
Recent writing
No Autonomy Without Scalable Oversight
What to expect as we enter the Year of The Judge.
ProofJudge: Can we align vibe-proving with human taste?
Towards measuring alignment with human taste in autoformalization with judge agents.
The Tests All Pass
METR’s SWE-bench analysis shows us taste isn’t verifiable.
Browse by thread