Structure--Residual Inductive Biases for Language Model Pretraining

Administrator 0 阅读

AI Digest - ArXiv AI

Structure--Residual Inductive Biases for Language Model Pretraining

Next-token prediction is relied upon to learn both predictive behavior and internal representations. We ask whether representation structure itself should be an explicit inductive bias. Motivated by the two-part coding view of Kolmogorov complexity, we study a structure--residual principle in which a representation separates structure from realization detail. We instantiate this principle in two complementary ways. Boundary Bottleneck Concept (BBC) routes information from token segments through


Source: ArXiv AI