Bubble’s Law: Every time you reduce context size you just increase batch size (until the GPU catches on fire). Fetching the context and manipulating it is still enough work that flash will have a place for a while. What most people don’t get is its not the bandwidth that dictates your memory tier its the access time. This will all make a lot more sense in 3 months.
