Structure--Residual Inductive Biases for Language Model Pretraining
AI Digest - ArXiv AI
Structure--Residual Inductive Biases for Language Model Pretraining
Next-token prediction is relied upon to learn both predictive behavior and internal representations. We ask whether representation structure itself should be an explicit inductive bias. Motivated by the two-part coding view of Kolmogorov complexity, we study a structure--residual principle in which a representation separates structure from realization detail. We instantiate this principle in two complementary ways. Boundary Bottleneck Concept (BBC) routes information from token segments through
Source: ArXiv AI