Training on Shane Caldwell https://hackbot.dad/categories/training/ Recent content in Training on Shane Caldwell Shane Caldwell https://hackbot.dad/ https://hackbot.dad/ Hugo -- 0.146.2 en-us Fri, 12 Dec 2025 00:00:00 +0000 All Reduce Across the Atlantic: Bandwidth in Decentralized Training https://hackbot.dad/writing/all-reduce-across-atlantic/ Fri, 12 Dec 2025 00:00:00 +0000 https://hackbot.dad/writing/all-reduce-across-atlantic/ The practical realities of devestatingly high communication cost in training. Twenty Billion Tokens of What, Exactly? https://hackbot.dad/writing/20b-tokens-of-what/ Mon, 01 Dec 2025 00:00:00 +0000 https://hackbot.dad/writing/20b-tokens-of-what/ Looking at the data and letting it look back at us. Pretraining at home: 20B tokens from 222 hours to 12 https://hackbot.dad/writing/pretraining-at-home/ Sun, 23 Nov 2025 00:00:00 +0000 https://hackbot.dad/writing/pretraining-at-home/ Optimizing training a Llama 3.2 1B model so we can pretrain in a day without going broke.