Canonical Color as a Lens into Concept Decodability in Vision Encoders and VLMs

AI Digest - ArXiv AI

Canonical Color as a Lens into Concept Decodability in Vision Encoders and VLMs

Visual encoders construct a representation of the image input for Vision-Language models. How much conceptual, as opposed to immediately visible, information does this representation contain? We use canonical color as a controlled test case to ask whether vision encoders make canonical-color information linearly accessible, even when color is removed from the input image. We construct a dataset of objects with canonical colors, and probe vision encoders for both color and object identity using c

Source: ArXiv AI