I didn't see cost. If cost is similar to flash drives then this could be massive. Every GPU ships with a lot of ram still but for AI inference and games you have 10TB of this stuff to stuff all your textures into and all your model weights in. For fine tuning models this would also be great if using lora or similar.
Wasn’t quite right before, but Optane was sold off, so it’s sad the buyer just sat on the IP.
Also the selling point was latency, but I suspect we’ll see bandwidth being the important metric with AI using deep pipelines to stream in the weights in a latency insensitive manner.
Intel and Micron sold off the fab making Optane, but I haven't heard anything about them selling off the design rights, not there's much they can do without the fab that could make them.
It's maybe more maddening when you realize it was part of Intel getting completely out of memory, so that stings a bit. On the other hand, Intel needed focus rather than the multi-pronged MBA-driven mediocrity they'd been pursuing.
So like.. conceptually kind of like memory mapped files on fast flash persistent storage, IIUC? Or maybe it's more like GPU-managed demand paging, caching and DMA? That could get you the capacity and better I/O characteristics.
I'm curious about how the unifying architecture is going to evolve between CPU/GPU having direct access to a singular pool of memory/storage also.
I also keep wondering when memristor technology might enter the ring, because as I understand it, it would be like moving compute into the memory, which would potentially remove the need to move the data in and out of storage as much also.
It feels like computing hardware infrastructure is fundamentally evolving.
That makes some sense. NAND Flash is massively parallel by its nature. That is rarely exposed outside the die though. You'll have that 8 bit double data rate bus and you'll learn to like it.
Now that model inference at scale is a thing though? Model weights, cached prefixes? There's a considerable demand for "slow writes, fast high bandwidth reads" memory. And every bit of storage you didn't have to use RAM for you can use for fatter KV cache instead.
I've been wondering when we'd get around to having the equivalent of "memory that runs at very GDDR6 speeds for reading, but is much slower for writing", which is exactly what you need when working with an AI model. Versus current HBM which has the same speeds writing as reading.
This feels like an exceptionally bad tradeoff. HBM is expensive due to the packaging cost and yield problems. Now you're trading the relatively inexpensive but high performance DRAM for Flash while splitting the addressable market into training and inference only hardware.
The low capacity of HBM isn't really a mistake. It's a design decision to keep the bandwidth to capacity ratio high. HBM systems with 96GB of memory tend to have around 3.5 TB/s which is a ratio of 35:1, meaning your theoretical maximum is 35 tokens of inference per second assuming you use the full storage just for parameters.
If you massively increase the capacity but keep the bandwidth the same, you just end up lowering this ratio. Your system is overall smaller, but it also has less performance.
This makes High-Bandwidth Flash an extremely niche product or the equivalent of industrially processing lampante olive oil and mixing into high quality olive oil. E.g you're spending an extreme amount of effort on making a worse product that is only marginally cheaper in absolute terms, but more expensive in terms of price to performance ratio.
They're reporting 1.6TB/s from a 512GB stack - even if they loaded just that 96GB model into it, effectively wasting most of the capacity, it appears they would arrive at a much cheaper system.
It certainly could in the long-term but the spec is still being finalized and realistically a few years away from being able to put a dent in the HBM memory market.
Imagine you are immensely intelligent beings who project into our space-time Continuum as mice. You need a lot of memory and processing but you don't need it quick. What do you do?
They did. Hard disk drive prices are up, even the used and officially re-certified drive prices are way up. Hard drive makers are reporting they have sold out of their yearly production, just like the memory manufacturers.
I have fond memories of Flash 5 in early 2000s. At the time I knew BASIC and Pascal. I could figure out how to draw things, how to animate them using keyframes and motion tweening, and how to make things interactive with ActionScript, all by poking around and reading the help manual (F1) on our underpowered offline PC.
Necessity being the mother of all invention. I also thought the compute-in-memory approach was interesting re: https://mythic.ai
I didn't see cost. If cost is similar to flash drives then this could be massive. Every GPU ships with a lot of ram still but for AI inference and games you have 10TB of this stuff to stuff all your textures into and all your model weights in. For fine tuning models this would also be great if using lora or similar.
I expect it to be closer to flash than to RAM, but still expensive. It'll require some special NAND flash dies, and "good" dies too.
Although they'll be able to do the usual thing, and "recycle" the bad dies by selling them off to be used in normal flash drives and SD cards.
Intel really missed out, when they discontinued the Optane line, right before the RAMpocalypse.
Wasn’t quite right before, but Optane was sold off, so it’s sad the buyer just sat on the IP.
Also the selling point was latency, but I suspect we’ll see bandwidth being the important metric with AI using deep pipelines to stream in the weights in a latency insensitive manner.
Optane latency was low enough that it could be drop-in RAM replacement for models. Bandwidth is just "stack more of them"
Intel and Micron sold off the fab making Optane, but I haven't heard anything about them selling off the design rights, not there's much they can do without the fab that could make them.
It's maybe more maddening when you realize it was part of Intel getting completely out of memory, so that stings a bit. On the other hand, Intel needed focus rather than the multi-pronged MBA-driven mediocrity they'd been pursuing.
So like.. conceptually kind of like memory mapped files on fast flash persistent storage, IIUC? Or maybe it's more like GPU-managed demand paging, caching and DMA? That could get you the capacity and better I/O characteristics.
I'm curious about how the unifying architecture is going to evolve between CPU/GPU having direct access to a singular pool of memory/storage also.
I also keep wondering when memristor technology might enter the ring, because as I understand it, it would be like moving compute into the memory, which would potentially remove the need to move the data in and out of storage as much also.
It feels like computing hardware infrastructure is fundamentally evolving.
AMD had an interesting take on this around a decade ago. Never went anywhere but very cool tech.
That makes some sense. NAND Flash is massively parallel by its nature. That is rarely exposed outside the die though. You'll have that 8 bit double data rate bus and you'll learn to like it.
Now that model inference at scale is a thing though? Model weights, cached prefixes? There's a considerable demand for "slow writes, fast high bandwidth reads" memory. And every bit of storage you didn't have to use RAM for you can use for fatter KV cache instead.
I've been wondering when we'd get around to having the equivalent of "memory that runs at very GDDR6 speeds for reading, but is much slower for writing", which is exactly what you need when working with an AI model. Versus current HBM which has the same speeds writing as reading.
This feels like an exceptionally bad tradeoff. HBM is expensive due to the packaging cost and yield problems. Now you're trading the relatively inexpensive but high performance DRAM for Flash while splitting the addressable market into training and inference only hardware.
The low capacity of HBM isn't really a mistake. It's a design decision to keep the bandwidth to capacity ratio high. HBM systems with 96GB of memory tend to have around 3.5 TB/s which is a ratio of 35:1, meaning your theoretical maximum is 35 tokens of inference per second assuming you use the full storage just for parameters.
If you massively increase the capacity but keep the bandwidth the same, you just end up lowering this ratio. Your system is overall smaller, but it also has less performance.
This makes High-Bandwidth Flash an extremely niche product or the equivalent of industrially processing lampante olive oil and mixing into high quality olive oil. E.g you're spending an extreme amount of effort on making a worse product that is only marginally cheaper in absolute terms, but more expensive in terms of price to performance ratio.
They're reporting 1.6TB/s from a 512GB stack - even if they loaded just that 96GB model into it, effectively wasting most of the capacity, it appears they would arrive at a much cheaper system.
Why doesn't someone bring back optane?
I hope it brings the RAM prices down!
It certainly could in the long-term but the spec is still being finalized and realistically a few years away from being able to put a dent in the HBM memory market.
It feels like the difference between short term and long term memory in a brain.
stick enough floppy's in parallel and you could do the same thing
Go ahead, take it further.
Imagine you are immensely intelligent beings who project into our space-time Continuum as mice. You need a lot of memory and processing but you don't need it quick. What do you do?
(With apologies to Douglas Adams)
Good use case for fast read, slow write.
[flagged]
we now only need AI companies to find a way to ruin hard disk storage and officially anything PC stops being affordable
They did. Hard disk drive prices are up, even the used and officially re-certified drive prices are way up. Hard drive makers are reporting they have sold out of their yearly production, just like the memory manufacturers.
I have fond memories of Flash 5 in early 2000s. At the time I knew BASIC and Pascal. I could figure out how to draw things, how to animate them using keyframes and motion tweening, and how to make things interactive with ActionScript, all by poking around and reading the help manual (F1) on our underpowered offline PC.
Wrong Flash.