Subquadratic Attention with a Hierarchical Tree-Structured Indexer

Administrator 0 阅读

AI Digest - ArXiv AI

Subquadratic Attention with a Hierarchical Tree-Structured Indexer

The quadratic scaling of attention is a fundamental bottleneck limiting the long-context capabilities of large language models (LLMs). A variety of approaches have attempted to circumvent this, but they either fail to match attention's expressiveness (e.g., sliding-window or linear attention variants with fixed-size state) or only achieve constant-factor speedups (e.g., sparse attention methods whose indexers themselves scale quadratically). In this paper, we derive an asymptotically subquadrati


Source: ArXiv AI