Yeah, (Optane) 3D XPoint from Micron would have a decent density speed trade-off. I don't know whether it makes the margin they can get from RAM though. Or whether CiM for masked ROM with a 3mo wait for parameter update (and much lower power) is a better trade-off for inference.
"Memory cost and capacity are significant issues for AI accelerators.
Unlike game rendering, model inference can have a deterministic memory access pattern. You don’t need “random access memory” at all for model weights, and you could tolerate cold-start latencies in the multiple milliseconds, as long as continuous reads were delivered at the necessary bandwidth.
NAND flash is over 100 times cheaper per GB than HBM, so there should be opportunity there, even after giving a flash controller a 1024 bit interface with HBM bandwidth.
You could make a specialized pin protocol that just supported pipelined transfer of full 16KB+ pages from the flash to program-managed accelerator scratchpad memory and improve per-pin performance over HBM, but it might be more convenient to make it still look like a true random access memory with very fragile performance characteristics, where anything but sequential reads falls off a 1000x+ performance cliff.
That has the advantage of automatically using existing cache hierarchies, and providing a natural path to update the flash memory with new model weights. With the stream-to-scratch interface, code has to be completely rewritten before it works at all, while the ram-emulation interface will start off just extremely slow, and you can incrementally sort out the changes for full performance.
There may be cases where there isn’t enough scratchpad SRAM to hold the weights for a layer, which might force you to deploy the old optical drive optimization technique of duplicating data in multiple places on a sequential read to avoid seeking, but there would be capacity to burn.
It might be possible to do something like cuda graph capture to record a memory access trace and have everything magically remapped to a linear sequence, but deploying programmer / agent elbow grease to manage transfers and access in a scratch ram ring buffer would be lower risk.
A split memory system consisting of some channels of flash and some channels of HBM will probably be suboptimal compared to a uniform memory, but it could be much cheaper, and allow much larger models to be run.
I think th case is strong for inference, but you have to stretch more for training. You can still linearize all the weight memory accesses, both reads and writes, but flash memory would quickly wear out from the writes, even if they were all perfectly page aligned. Replacing low-latency HBM with massively parallel cheap(er) DRAM at high latency might still be a worthwhile cost savings."
Likely much worse write endurance than DRAM, since it’s still flash underneath. The saving grace is that model weights are mostly read-heavy, so endurance may matter less than it sounds.
Yeah, but how often is this? Certainly there are weird hobbyist edge cases that don't do well with this, that's true of anything, but a provider is loading weights once every few months
It all depends how many models you are serving from that flash and whether they all fit in there together. If you need to evict and load models, the flash will die a horrible death. This is one resource you should NEVER underprovision.
So far we have seen a lot of little things built into mainstream chips for the use of hyperscalers, things that don’t require big changes to the software stack. This proposal would require big changes to the software which may mean it remains a proposal.
This is almost entirely dominated by the read circuitry and the data path: it’s still taking 1/6 of a second to read the whole chip, which means that the flash cells aren’t working hard at all. (And that pitting the full weights of a dense model on these chips while using anywhere near all the capacity is a nonstarter if you intent to stream the weights as you run inference.)
I have a feeling Intel got rid of Octane a couple years too soon.
Yeah, (Optane) 3D XPoint from Micron would have a decent density speed trade-off. I don't know whether it makes the margin they can get from RAM though. Or whether CiM for masked ROM with a 3mo wait for parameter update (and much lower power) is a better trade-off for inference.
Is this the same idea John Carmack had? (https://x.com/ID_AA_Carmack/status/2074248758422864226?lang=...)
"Memory cost and capacity are significant issues for AI accelerators.
Unlike game rendering, model inference can have a deterministic memory access pattern. You don’t need “random access memory” at all for model weights, and you could tolerate cold-start latencies in the multiple milliseconds, as long as continuous reads were delivered at the necessary bandwidth.
NAND flash is over 100 times cheaper per GB than HBM, so there should be opportunity there, even after giving a flash controller a 1024 bit interface with HBM bandwidth.
You could make a specialized pin protocol that just supported pipelined transfer of full 16KB+ pages from the flash to program-managed accelerator scratchpad memory and improve per-pin performance over HBM, but it might be more convenient to make it still look like a true random access memory with very fragile performance characteristics, where anything but sequential reads falls off a 1000x+ performance cliff.
That has the advantage of automatically using existing cache hierarchies, and providing a natural path to update the flash memory with new model weights. With the stream-to-scratch interface, code has to be completely rewritten before it works at all, while the ram-emulation interface will start off just extremely slow, and you can incrementally sort out the changes for full performance.
There may be cases where there isn’t enough scratchpad SRAM to hold the weights for a layer, which might force you to deploy the old optical drive optimization technique of duplicating data in multiple places on a sequential read to avoid seeking, but there would be capacity to burn.
It might be possible to do something like cuda graph capture to record a memory access trace and have everything magically remapped to a linear sequence, but deploying programmer / agent elbow grease to manage transfers and access in a scratch ram ring buffer would be lower risk.
A split memory system consisting of some channels of flash and some channels of HBM will probably be suboptimal compared to a uniform memory, but it could be much cheaper, and allow much larger models to be run.
I think th case is strong for inference, but you have to stretch more for training. You can still linearize all the weight memory accesses, both reads and writes, but flash memory would quickly wear out from the writes, even if they were all perfectly page aligned. Replacing low-latency HBM with massively parallel cheap(er) DRAM at high latency might still be a worthwhile cost savings."
What will be the durability/lifetime properties of this technology in comparison to traditional DRAM?
Likely much worse write endurance than DRAM, since it’s still flash underneath. The saving grace is that model weights are mostly read-heavy, so endurance may matter less than it sounds.
True, but every time you load a new set of model weights, you are spending writes.
Yeah, but how often is this? Certainly there are weird hobbyist edge cases that don't do well with this, that's true of anything, but a provider is loading weights once every few months
It all depends how many models you are serving from that flash and whether they all fit in there together. If you need to evict and load models, the flash will die a horrible death. This is one resource you should NEVER underprovision.
So far we have seen a lot of little things built into mainstream chips for the use of hyperscalers, things that don’t require big changes to the software stack. This proposal would require big changes to the software which may mean it remains a proposal.
I had to double check those figures on Sk Hynix office web site [1], and it is not a typo or wrong capital "B".
It really is 3TB per second.
I literally paused for 5 min and thought how is this even possible.
[1] https://news.skhynix.com/en/hbf-at-fms-2026/
This is almost entirely dominated by the read circuitry and the data path: it’s still taking 1/6 of a second to read the whole chip, which means that the flash cells aren’t working hard at all. (And that pitting the full weights of a dense model on these chips while using anywhere near all the capacity is a nonstarter if you intent to stream the weights as you run inference.)
Half the battle is just knowing this exists
This sounds promising. I am surprised the capacity on the table ends at 512GiB. Is this per chip or per device?
HBF sounds like it has quite the future, as part of future computing's Memory Hierarchy:
https://en.wikipedia.org/wiki/Memory_hierarchy