Low-Bit N-Gram Embeddings for Language Models
AI Digest - ArXiv AI
Low-Bit N-Gram Embeddings for Language Models
Many recent open-weight language models use n-gram embedding tables to improve performance. These tables store learned representations of short token sequences, such as pairs or triples of consecutive tokens, which the model retrieves during processing. However, they can require substantial memory: DeepSeek-V4.1-Flash, for example, allocates 196 billion parameters to its Engram n-gram memory. We therefore investigate whether this memory cost can be reduced while preserving the performance improv
Source: ArXiv AI