Compute Is Easy, Memory Is Harder And Harder
TheNextPlatform, December 13th, 2022
What good is a floating point operation embodied in a vector or matrix unit if you can't get data into fast enough to actually use the compute engine to process it in some fashion in a clock cycle? The answer is obvious to all of us: Not much.
People have been talking about the imbalance between compute and memory bandwidth for decades, and every year the high performance computing industry has been forced to accept less and less memory bandwidth per floating point operation because increasing memory bandwidth is exceedingly difficult in a fashion that doesn't also end up being very pricey.