All Reduce Across the Atlantic: Bandwidth in Decentralized Training
The practical realities of devestatingly high communication cost in training.
Training
Twenty Billion Tokens of What, Exactly?
Looking at the data and letting it look back at us.
Pretraining at home: 20B tokens from 222 hours to 12
Optimizing training a Llama 3.2 1B model so we can pretrain in a day without going broke.