Same Query, Different Answer: Run-to-Run Variance in Semantic Query Processing

Administrator 0 阅读

AI Digest - ArXiv AI

Same Query, Different Answer: Run-to-Run Variance in Semantic Query Processing

Semantic query engines ask a large language model (LLM) to judge individual rows, and benchmarks compare these engines on the quality of a few executions at temperature 0. We measure how much another execution changes. We re-execute the text and tabular workloads of SemBench, LRO-Bench, and UDA-Bench through their own harnesses, log every one of 1.2 million model calls, and reanalyze SemBench's published executions. At temperature 0, between 0.07% and 10.2% of rows change their operator output o


Source: ArXiv AI