Subquadratic Attention with a Hierarchical Tree-Structured Indexer
AI Digest - ArXiv AI
Subquadratic Attention with a Hierarchical Tree-Structured Indexer
The quadratic scaling of attention is a fundamental bottleneck limiting the long-context capabilities of large language models (LLMs). A variety of approaches have attempted to circumvent this, but they either fail to match attention's expressiveness (e.g., sliding-window or linear attention variants with fixed-size state) or only achieve constant-factor speedups (e.g., sparse attention methods whose indexers themselves scale quadratically). In this paper, we derive an asymptotically subquadrati
Source: ArXiv AI