Only person who used their fucking brain and thought through this instead of relying on gross oversimplifications. This is why I will always listen to engineers over financial analysts when it comes to things like this. The only thing I’d add to this is that although the recurrent state is smaller the issue isn’t that you need less memory, we can naively assume that every time memory per user shrinks batch goes up to compensate. The issue is more that because you need to fetch, manipulate, write back multiple times for each linear layer and each user this makes flash and slower memory tiers less workable with faster memory like HBM being more necessary. The dumbest shit I read in this whole saga was Funda AI saying because recurrent state is small it’s fine for flash offload. That’s far from the truth because even if it’s small if you need to move it back and forth 3 times on the hot path the flash bandwidth becomes a bottleneck. To make it even worse Funda AI said that large KV was bad for flash because it was big. Again very naive and very simple thinking, if we had to move all 1M tokens back and forth every time for each user it wouldn’t matter if we put all the KV in HBM it wouldn’t still take forever. In reality the hot or recent KV stays resident in HBM while the rest is stored in flash and “scored” the scoring in effect allows us to only fetch the pages that are relevant and not move all of the KV. To draw an analogy the KV offload techniques are akin to searching through a library to grab a single page, while the delta net attention is like maintaining a notepad of a running calculation. But regardless great piece by GDP and I rarely ever give anyone credit for a good analysis, even the prefill vs. caching trade off he intuitively understood. Bravo! 👏
