Evals Aren't Dead, But They're Low Res
Put down that ruler and go build a microscope.
Llms
No Autonomy Without Scalable Oversight
What to expect as we enter the Year of The Judge.
ProofJudge: Can we align vibe-proving with human taste?
Towards measuring alignment with human taste in autoformalization with judge agents.
The Tests All Pass
METR’s SWE-bench analysis shows us taste isn’t verifiable.
Intro to GPUs For the Researcher
Getting comfortable with the hardware on a quest for more MFU.
All Reduce Across the Atlantic: Bandwidth in Decentralized Training
The practical realities of devestatingly high communication cost in training.
Twenty Billion Tokens of What, Exactly?
Looking at the data and letting it look back at us.
Pretraining at home: 20B tokens from 222 hours to 12
Optimizing training a Llama 3.2 1B model so we can pretrain in a day without going broke.
Offsec Evals: Growing Up In The Dark Forest
If you contribute a public benchmark, are you giving free capability to your competitors?
DiLoCo: Data Parallelism for the Datacenter Poor
Distributed training sans datacenter.