NeurIPS 2026 / Poster

Support Mismatch as a Benchmark Failure Mode for In-Context Prediction

Jie Liu1,*Lanlan Liang1,*,†ZiXi Yan2Qinan Bao3Linghao Meng4Wenbo Gong4
1 Information and Communication Engineering, Hainan University2 Computer Science and Technology, Hainan University3 Cyberspace Security, Hainan University4 Electrical and Information Engineering, Tianjin University

* Equal contribution · † Corresponding author

The question

More examples.
But the right task
is still missing.

In-context examples help a predictor decide among the tasks it can consider. What happens when the true task is outside that fixed set?

Our benchmark separates a shrinking learning transient from a persistent support-mismatch floor.

See the theoretical result
A fixed task libraryConceptual illustration
Supported task A
Supported task B
True task C is absent
Context examplesInfer within the library

More context can refine the choice between A and B. It cannot add the missing task C.

Wrong and true restoration each add one task. The comparison tests relevance while matching library size.

01 / The theoretical benchmark

From self-correction to an error floor.

For a frozen prior over a finite task family, risk converges to the exact approximation floor.

|Rkπ − εapprox2| ≤ C e−kρ
Transient: posterior competition Floor: support mismatch

What more context changes

Evidence concentrates on the KL-closest supported task. Competition among the available tasks vanishes exponentially.

What fixed support leaves behind

The closest supported predictor may still differ from the true one. That discrepancy remains as the limiting approximation floor.

When the result applies

A finite frozen task family, positive KL-gap separation, componentwise exponential-moment control, and the theorem’s finite prediction moments.

The bound controls absolute deviation; finite-context risk need not decrease monotonically. Scalar and aligned Gaussian analyses sharpen the rate and reveal the boundary μ2 = 3μ1.

02 / From benchmark to behavior

Four levels of evidence.
One carefully scoped question.

We separate exact analysis, controlled predictors, explicit expert families, and language-model interventions.

Controlled predictors track the benchmark. An 8-layer learned Transformer is compared with exact Bayes and unrestricted ridge regression across three mismatch regimes. Bands are over anchor tasks. This is a controlled learned setting, not a measurement of a production LM’s internal prior.
I

Exact Bayes

Finite-sample theory and sharp Gaussian analyses isolate the transient, floor, and crossover.

II

Learned predictors

A controlled Transformer follows the benchmark near the transition to floor-limited behavior.

III

Explicit expert families

Specified conditional experts make the support and posterior directly auditable.

IV

LM interventions

Matched changes to exposed rules test behavioral signatures in Qwen and Llama.

03 / The decisive comparison

Restore the right rule.
Control for the extra slot.

We compare true restoration with a wrong addition of the same size. Sequence-level scoring provides a complementary test beyond closed candidate-set mechanics.

Qwen2.5-7B-Instruct

1.22 [1.09, 1.36]

Wrong − True excess NLL at k = 32

Positive in all 12 context-seed means

Llama-3.1-8B-Instruct

1.60 [1.44, 1.76]

Wrong − True excess NLL at k = 32

A second model family preserves the ordering

ASSET-structured text-simplification scenario · full-sequence excess NLL relative to the oracle · higher Wrong − True contrasts favor true restoration. Qwen is pooled over three prompts and twelve context seeds; brackets denote paired 95% bootstrap confidence intervals. See Table 1 and Appendix M of the paper for the full protocol.

Additional checks

Subject-aligned multiple-choice controls, fixed nonce verbalizers, Latin-square mappings, calibration, ambiguous-context null comparisons, and a controlled near-tie sweep.

Abstract

We analyze support mismatch in in-context prediction through a frozen-prior Bayes benchmark with finite discrete task support. The benchmark is diagnostic rather than literal: it does not identify a production language model with exact Bayesian updating. Under componentwise exponential-integrability of log-likelihood ratios, the predictor risk converges exponentially to the exact approximation floor; additional context removes posterior competition but cannot create a task omitted from frozen support. Scalar and aligned Gaussian location families characterize the exponent sharply and exhibit a regime change. We examine exact Bayes, controlled learned predictors, explicit conditional-expert families, and matched language-task interventions.

In the ASSET-structured Qwen2.5-7B-Instruct scenario, true restoration beats a wrong size-matched restoration at k = 32 by 1.22 [1.09, 1.36] oracle-relative full-sequence excess-NLL units across three prompts and twelve context seeds; four-gram SARI improves by 0.80 [0.58, 1.04], and Llama-3.1-8B-Instruct independently preserves the ordering. Twelve-seed subject-aligned multiple-choice controls, ambiguous-context null comparisons, nonce-label controls, and a controlled near-tie sweep support the same intervention-level picture while preserving the boundary that behavioral agreement does not recover an LM’s internal prior, posterior, or KL geometry.

Read the full paper

04 / Explore the work

Paper, materials, and citation.

Analysis code is released under MIT. The package documents coverage and remaining limits on exact historical end-to-end reproduction.

Citation

Download .bib
@inproceedings{liu2026support,
  author    = {Jie Liu and Lanlan Liang and ZiXi Yan and Qinan Bao
               and Linghao Meng and Wenbo Gong},
  title     = {Support Mismatch as a Benchmark Failure Mode
               for {In-Context} Prediction},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026},
  note      = {Accepted at NeurIPS 2026}
}