xAI announced its AI training systems called Colossus, featured with 100,000 Nvidia H100 graphics cards, on September 2nd, 2024, which came online over the weekend. The equipped H100 is the most powerful AI processor powered with Transformer Engine module for running AI models based on Transformer neural network architecture.
Elon Musk claimed this is the most powerful AI training system in the world, which is even faster than the world’s fastest AI supercomputer, US Energy Department’s Aurora systems (10.6 exaflops top speed with 87% active hardware), according to Musk’s Monday post on X. Musk also mentioned that Colossus will double in size to 200,000 graphic cards with 50,000 H200 (a upgraded version of H200 with faster data swapping memory and doubled memory capacity of 141 gigabytes).
The newly developed Colossus would accelerate the development of new version of xAI’s LLM model Grok-2 to be released by the end of the year.
Related Resources:
Caisky Consulting 