Today marks a really big achievement for Nous, but also potentially the AI Landscape.
We have begun a decentralized pretraining run of what is basically a dense Deepseek - 40B parameters, over 20T tokens, with MLA for long context efficiency.
All checkpoints, unannealed,
twitter.com/Teknium1/status/19...