Exact Bayes
Finite-sample theory and sharp Gaussian analyses isolate the transient, floor, and crossover.
The question
In-context examples help a predictor decide among the tasks it can consider. What happens when the true task is outside that fixed set?
Our benchmark separates a shrinking learning transient from a persistent support-mismatch floor.
See the theoretical resultMore context can refine the choice between A and B. It cannot add the missing task C.
Wrong and true restoration each add one task. The comparison tests relevance while matching library size.
01 / The theoretical benchmark
For a frozen prior over a finite task family, risk converges to the exact approximation floor.
Evidence concentrates on the KL-closest supported task. Competition among the available tasks vanishes exponentially.
The closest supported predictor may still differ from the true one. That discrepancy remains as the limiting approximation floor.
A finite frozen task family, positive KL-gap separation, componentwise exponential-moment control, and the theorem’s finite prediction moments.
The bound controls absolute deviation; finite-context risk need not decrease monotonically. Scalar and aligned Gaussian analyses sharpen the rate and reveal the boundary μ2 = 3μ1.
02 / From benchmark to behavior
We separate exact analysis, controlled predictors, explicit expert families, and language-model interventions.
Finite-sample theory and sharp Gaussian analyses isolate the transient, floor, and crossover.
A controlled Transformer follows the benchmark near the transition to floor-limited behavior.
Specified conditional experts make the support and posterior directly auditable.
Matched changes to exposed rules test behavioral signatures in Qwen and Llama.
03 / The decisive comparison
We compare true restoration with a wrong addition of the same size. Sequence-level scoring provides a complementary test beyond closed candidate-set mechanics.
Qwen2.5-7B-Instruct
1.22 [1.09, 1.36]
Wrong − True excess NLL at k = 32
Positive in all 12 context-seed meansLlama-3.1-8B-Instruct
1.60 [1.44, 1.76]
Wrong − True excess NLL at k = 32
A second model family preserves the orderingASSET-structured text-simplification scenario · full-sequence excess NLL relative to the oracle · higher Wrong − True contrasts favor true restoration. Qwen is pooled over three prompts and twelve context seeds; brackets denote paired 95% bootstrap confidence intervals. See Table 1 and Appendix M of the paper for the full protocol.
Subject-aligned multiple-choice controls, fixed nonce verbalizers, Latin-square mappings, calibration, ambiguous-context null comparisons, and a controlled near-tie sweep.
We analyze support mismatch in in-context prediction through a frozen-prior Bayes benchmark with finite discrete task support. The benchmark is diagnostic rather than literal: it does not identify a production language model with exact Bayesian updating. Under componentwise exponential-integrability of log-likelihood ratios, the predictor risk converges exponentially to the exact approximation floor; additional context removes posterior competition but cannot create a task omitted from frozen support. Scalar and aligned Gaussian location families characterize the exponent sharply and exhibit a regime change. We examine exact Bayes, controlled learned predictors, explicit conditional-expert families, and matched language-task interventions.
In the ASSET-structured Qwen2.5-7B-Instruct scenario, true restoration beats a wrong size-matched restoration at k = 32 by 1.22 [1.09, 1.36] oracle-relative full-sequence excess-NLL units across three prompts and twelve context seeds; four-gram SARI improves by 0.80 [0.58, 1.04], and Llama-3.1-8B-Instruct independently preserves the ordering. Twelve-seed subject-aligned multiple-choice controls, ambiguous-context null comparisons, nonce-label controls, and a controlled near-tie sweep support the same intervention-level picture while preserving the boundary that behavioral agreement does not recover an LM’s internal prior, posterior, or KL geometry.
Read the full paper04 / Explore the work
Theory, experiments, appendices, and checklist.
ZIP · 78.8 MBAnalysis code, configurations, documented records, and reproduction notes.
Analysis code is released under MIT. The package documents coverage and remaining limits on exact historical end-to-end reproduction.
@inproceedings{liu2026support,
author = {Jie Liu and Lanlan Liang and ZiXi Yan and Qinan Bao
and Linghao Meng and Wenbo Gong},
title = {Support Mismatch as a Benchmark Failure Mode
for {In-Context} Prediction},
booktitle = {Advances in Neural Information Processing Systems},
year = {2026},
note = {Accepted at NeurIPS 2026}
}