HBM capacity never mattered only bandwidth did. This only makes sense when you model tokenomics/utilization @ rack scale b/c you have two levers to pull on which is parallelism & offload which both reduce flops & bw pressure. You can read the @0xBADB01E thread that proved it more thoroughly than I ever can. But this has been obviously proven true for inference and training (although a bit tougher argument). In that style of architecture you switch pressure from memory to the interconnect and the bottleneck is latency which means less switches. (Switching built into asic like the TPU already does as well as tensor torrent is a good example). There’s a balance though cause if you parallelize too much you need to orchestrate a huge scale up domain which is what TPU does with like over 9,600 chips in scale up. There probably is an optimal or physical limit here but too lazy to calculate probably as long as interconnect latency for all to all transaction is less than memory fetch you are good.
