Investors who piled into SK Hynix’s $28 billion blockbuster Nasdaq debut on Friday should be aware: The business model on which the world’s leading memory chip makers are thriving is set to shift to one that requires a bit more strategic and financial gambling. So far, SK Hynix Inc. is making all the right moves but it’s unclear how well the field’s leading players will deal with that disruption.
The trigger is of course the artificial intelligence boom. More powerful AI systems have pushed memory chips to their limits in their job of moving data to and from the processor, the brain of a computer or server.
For most of computing history, calculations on the processor chip were the slow part. Now things have flipped, since today’s graphics processing units made by Nvidia Corp. can do math far more quickly, while the data being delivered back and forth from the memory chip is still slow in comparison. Imagine a genius savant who simply can’t write things down fast enough, with their notepad being the memory chip.
“AI machines are starved for memory bandwidth because these models are so big,” former Intel Corp. boss Pat Gelsinger told me in a recent interview.
That means AI chips aren’t achieving their full potential, and it also limits how much even the most sophisticated chatbots can remember. Anthropic’s Fable 5, for instance, can only hold about seven novels’ worth of text in its working memory. Ask it to consider more and it must start dropping what came earlier.
AI builders have been trying for years to get their models to remember far greater amounts, since that would let them carry out more sophisticated, continuous tasks in areas such as finance or back-office corporate work.
But they are constrained in part by physics. Servers, the computers in data centers that run AI systems, use a technique called high-bandwidth memory. The idea is to stack fundamental components of a memory chip on top of each other, shortening the distance that data has to travel to and from the processor (the GPU).
The technique has flaws, though. “HBM, as we know and love it today, is a lousy memory,” Gelsinger says. Heat gets trapped in the lower layers of the memory chip, and the rising temperatures force the GPU to suppress its speed.
One answer is for the world’s three big memory chip producers, SK Hynix, Samsung Electronics Co. and Micron Technology Inc., to build memory directly onto the processor itself. Instead of the genius savant having to write anything down, the notepad is effectively in her head.
That would make AI models many times more powerful, but it also means the three memory chip makers — whose aggregate market value has increased by $3 trillion in the past year thanks to rampant demand — must change the way they do business. An industry that has always made standardized chips that they could sell like a commodity must now make custom products.
