JustFit: 200K-Token LLM Serving on a 24 GiB Laptop with Just-in-Time State Management

AI Digest - ArXiv AI

JustFit: 200K-Token LLM Serving on a 24 GiB Laptop with Just-in-Time State Management

Capable open-weight models make local coding and reasoning attractive, but their context and execution state strain laptop memory. We present JustFit, an MLX-based inference runtime that combines KVExec for compressed KV execution, PhaseSwap for component residency, and StateTrans for state-preserving serving transitions. These mechanisms fuse reconstruction and coordinate just-in-time materialization and release, independently of model-weight quantization. In full-execution capacity tests on a

Source: ArXiv AI