Low-Bit N-Gram Embeddings for Language Models

Administrator 0 阅读

AI Digest - ArXiv AI

Low-Bit N-Gram Embeddings for Language Models

Many recent open-weight language models use n-gram embedding tables to improve performance. These tables store learned representations of short token sequences, such as pairs or triples of consecutive tokens, which the model retrieves during processing. However, they can require substantial memory: DeepSeek-V4.1-Flash, for example, allocates 196 billion parameters to its Engram n-gram memory. We therefore investigate whether this memory cost can be reduced while preserving the performance improv


Source: ArXiv AI