Abstention Is Not Threshold Compliance: Stated Scoring Rules in Open Language Models

The authors report in this preprint: This paper investigates the proposal by Kalai et al. to use explicit probability thresholds in prompts as scoring rules, directing language models to answer only when their confidence exceeds threshold t. The authors probe 12 checkpoints across three post-training stages from four open 3-8B model recipes, alongside exploratory models up to 32B. The empirical tests reveal that while a strict abstention format line shifts mean abstention from 20% to 91% at t = 0.75, varying t from 0.5 to 0.9 within this format changes abstention by a mere -0.8 percentage points. Without formatting prompts, higher thresholds can paradoxically lower abstention, with up to 40% of items receiving decisions incompatible with threshold-independent beliefs. Furthermore, external thresholds on recalibrated P(True) showed brittle generalization across datasets like TriviaQA and PopQA, and models trained on threshold-conditioned targets failed to track unseen thresholds consistently. The study concludes that high abstention rates do not imply true rule compliance and outlines rigorous evaluation criteria for stated scoring rules.