Twenty Billion Tokens of What, Exactly?
Looking at the data and letting it look back at us.
Training
Pretraining at home: 20B tokens from 222 hours to 12
Optimizing training a Llama 3.2 1B model so we can pretrain in a day without going broke.
DiLoCo: Data Parallelism for the Datacenter Poor
Distributed training sans datacenter.