AI's New Frontier: Training Trillion-Parameter Models With Much Fewer GPUs
ITProToday, Monday, January 8th, 2024
Researchers used just 8% of the world's most powerful supercomputer to train a model the size of ChatGPT.
Training a language model the size of OpenAI's ChatGPT would normally require a sizable supercomputer. But scientists working on the world's most powerful supercomputer discovered innovative techniques to train gigantic models using a lot less hardware.
In a new research paper, scientists from the famed Oak Ridge National Laboratory trained a one trillion parameter model using just a few thousand GPUs in their Frontier supercomputer, the most powerful non-distributed supercomputer in the world and one of only two exascale systems globally.
They used just 3,072 GPUs to train the giant large language model out of 37,888 AMD GPUs housed in Frontier. That means the researchers trained a model comparable to ChatGPT's rumored size of a trillion parameters on just 8% of Frontier's computing power.