Right, Wrong, or Unfinished: A Competing Risks Framework for Language-Model Reasoning under Token Budgets
Abstract
Assessing language-model performance under a token budget requires understanding both when a response will finish and whether it will be correct. We develop Competing Risks Analysis of Token Expenditure in Reasoning (CRATER), a statistically principled framework that treats correct and incorrect completion as distinct competing event types and responses unfinished at the generation limit as administratively right-censored. The fitted distribution provides probabilities of correct completion, incorrect completion, and remaining unfinished across token budgets, together with expected token consumption. We evaluate the proposed method using responses to MATH questions and demonstrate its value in two downstream applications. For routing before generation, CRATER guides the joint choice of a model configuration and token budget. Under hypothetical token prices, it reduces mean cost by $53.39 \%$ relative to a strong reference, with observed correctcompletion rates of $96.42 \%$ and $97.30 \%$, respectively. For repeated sampling, we combine CRATER’s single-attempt success probabilities with an existing beta-binomial calibration approach to predict pass@k. Across both applications, CRATER delivers predictions at any specified token budget up to the generation limit from a common model of completion time and correctness, eliminating the need for separate parameter estimation for each budget.