Pandora's AI Model Routing Box: Efficient Allocation with Costly Value Estimation

AI Digest - ArXiv AI

Pandora's AI Model Routing Box: Efficient Allocation with Costly Value Estimation

Heterogeneous AI systems composed of multiple models, architectures, harnesses, or inference-time settings can improve quality and efficiency by routing queries to the specialist who can answer most effectively at the lowest cost. Routing requires estimating each specialist's expected return, but this value estimation has a cost. Cheap estimators (e.g., embedding-based predictors) are fast but noisy, while accurate estimators (e.g., fine-tuned models with access to retrieval results or partial r

Source: ArXiv AI