speedup generation

Introduction Autoregressive language models generate text one token at a time. Each new prediction requires a full forward pass through all transformer layers, leading to redundant computations. For example, generating the next token in: [What, is, in,] → [the] requires recomputing attention over [What, is, in,] even though these tokens haven’t changed. KV Caching solves this inefficiency by…