For a lot of local workloads, sub-1 tok/s is useless in foreground and perfectly acceptable in background. If the choice is “this crashes” vs “this finishes overnight,” that’s still a meaningful capability jump.
the practical question is whether the read pattern is sequential enough to actually saturate nvme bandwidth or if the attention layer access pattern ends up being random enough to kill throughput. sequential reads on a decent nvme get you 5-7 GB/s, random reads drop to maybe 500 MB/s depending on queue depth.
for a 1T model youd need to stream something like 2TB of weights per forward pass at fp16. even at peak sequential thats 300+ seconds per token which is... not great for interactive use but maybe fine for batch inference where you dont care about latency.
still a cool proof of concept though. the gap between 'can run' and 'runs usefully' is where things get interesting.
This is a pretty cool project! Essentially this is like using Swap memory to extend your RAM, but in a 'smart' way so you don't overload the NVMe unnecessarily.
I do wonder in practice how the 'smarts' pan out, because putting a ton of stress on your NVMe during generation is probably not the best choice for it's longevity.
There needs to be something like this from Ollama. At the moment Ollama has a lot of flaws that prevent it from getting great performance. (My understanding is better GPU/CPU splits, etc). But Ollama is the only way to host an LLM and have it switch out on demand. Sigh.
You do not provide any comparison to llama.cpp with mmap.
You do not explain how any kind of predictor can work for MoE experts.
You do not explain how prediction can even be useful. I can predict the layers used in a dense model (all of them are used in order), but that doesn't help me much. It's still bottlenecked on bandwidth (hint: MoE doesn't change this).
OS paging would be significantly worse here. The kernel's page fault handler is reactive — it doesn't know
you're about to read layer 47's FFN weights, so it can't prefetch. You stall on every fault, wait for the
4KB/16KB page to load, then resume. With 80 layers of dense FFN streaming, that's thousands of cold faults per
token.
What makes this approach faster is that the model's access pattern is completely deterministic during
inference. You know exactly which tensors are needed next because transformer layers execute sequentially. So
you can issue large sequential reads and prefetch the next layer while the current one is computing on Metal.
The OS page cache can't do that — it has no concept of "layer N+1 comes after layer N."
For MoE it's even more stark. The OS would page in all 8 experts on the first token that routes to each one,
then evict them under memory pressure with LRU, which has no idea that expert 3 fires 10x more often than
expert 7. The neuron cache here is basically a domain-specific replacement policy.
Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
(github.com)143 points by tatef 3 hours ago | 61 comments
Comments
for a 1T model youd need to stream something like 2TB of weights per forward pass at fp16. even at peak sequential thats 300+ seconds per token which is... not great for interactive use but maybe fine for batch inference where you dont care about latency.
still a cool proof of concept though. the gap between 'can run' and 'runs usefully' is where things get interesting.
I do wonder in practice how the 'smarts' pan out, because putting a ton of stress on your NVMe during generation is probably not the best choice for it's longevity.
Come on, "Run" is not the right word. "Crawl" is.
Headlines like that are misleading.
You do not explain how any kind of predictor can work for MoE experts.
You do not explain how prediction can even be useful. I can predict the layers used in a dense model (all of them are used in order), but that doesn't help me much. It's still bottlenecked on bandwidth (hint: MoE doesn't change this).
[0] https://simonwillison.net/2026/Mar/18/llm-in-a-flash/