No Autonomy Without Scalable Oversight
What to expect as we enter the Year of The Judge.
AI security researcher · Dreadnode
Full Contact
Alignment
I build scalable oversight and control systems for autonomous agents operating in adversarial, high-consequence environments.
Shane CaldwellResearcher & engineer
Latest research
August 2025 · arXiv
PentestJudge: Judging Agent Behavior Against Operational Requirements
A system for evaluating whether penetration-testing agents satisfy operational requirements, not merely whether their final answers look correct.
Notes from the work
Recent writing
ProofJudge: Can we align vibe-proving with human taste?
Towards measuring alignment with human taste in autoformalization with judge agents.
The Tests All Pass
METR’s SWE-bench analysis shows us taste isn’t verifiable.
Intro to GPUs For the Researcher
Getting comfortable with the hardware on a quest for more MFU.
Browse by thread