Book a 30-min call
← cd ../blogs
$ cat posts/your-retry-loop-voided-your-guarantee.mdx

Your retry loop voided your conformal guarantee

June 26, 2026 · ImmovableTech Team

  • Evaluation
  • Computer Vision
  • Production AI

The guarantee stopped being true when you shipped the retry

Conformal prediction is one of the few honest things you can put in front of a risk team. Calibrate a score on a held-out labelled set, pick an error level, and you get a distribution-free bound on how often the prediction set contains the truth — no assumption about the model, the architecture or the shape of the data. That is why it is being adopted for vision-language model outputs, where the alternative is a confidence threshold somebody picked by looking at a histogram.

The mathematics is not the problem. The problem is that the guarantee is a contract with conditions, and the most ordinary piece of production engineering there is — re-prompt when the score looks weak, keep the better-looking answer — breaches one of them. Retry-until-confident is a p-hacking loop with an SLA attached. If your deployed pipeline retries and your calibration did not, the coverage number on your model card is decoration.

We know this because we shipped it. A document-extraction pipeline of ours had a conformally calibrated abstain threshold on field-level outputs, agreed with the client as the basis for how much human review they staffed. Some weeks later a perfectly sensible change landed: low-confidence fields got re-read at a higher resolution and the better score was kept. Nobody thought of that as touching the statistics, because it wasn’t a model change. The threshold stayed where it was. The realised error rate on the next labelled audit sample sat above the level we had promised, and our first instinct — tighten alpha and move on — was wrong in a way that would have hidden the fault rather than fixed it.

What the guarantee actually says

Split conformal prediction is three steps. Fix a score function that measures how badly the model is doing on a labelled example. Compute it on n held-out calibration points. Take a specific empirical quantile of those scores, and at test time emit every label whose score falls under it. Angelopoulos and Bates state the resulting guarantee tightly: for an exchangeable calibration and test sample, the probability that the set contains the true label is at least 1 − alpha, and at most 1 − alpha + 1/(n+1).

Three things in that sentence do the work, and all three are routinely misread.

It is marginal. The probability is averaged over the draw of the calibration set and the draw of the test point. It is not a statement about the item in front of you, and it is not a statement about any subgroup you care about. Angelopoulos and Bates give the canonical illustration: a procedure that always covers the 90% majority group and never covers the 10% minority group has exactly 90% marginal coverage and is useless.

It assumes exchangeability. Calibration and test points must be drawn such that their joint distribution is invariant to reordering. Independent and identically distributed data qualifies. Production traffic six months after you froze the calibration set generally does not.

It certifies one fixed map from input to score. The theorem covers the score function you calibrated on, applied unchanged at test time. Any step that reads the score and then changes it is a different map, and inherits nothing.

Worth adding, because it is the most under-appreciated part: the coverage guarantee holds for any score function, including a bad one. Score with random noise and you still get valid marginal coverage, achieved by emitting enormous useless sets. Validity is free; usefulness is entirely the score’s job.

The retry loop is selection on the outcome

Here is the mechanism, stripped of vision or language specifics. Your calibrated rule accepts an output when its score clears a threshold. Your retry policy says: if the score is below the bar, try again — a second sample, a zoomed crop, a re-prompt — and keep the maximum. The deployed score is now the max over several attempts, which is pointwise greater than or equal to the score you calibrated. Against a fixed threshold, that accepts a strictly larger set of outputs than the one certified. The extra admissions are exactly the borderline cases, which is to say the ones most likely to be wrong. Nothing bounds their error rate.

This is not a hypothetical. Xu and colleagues measured it directly in a June 2026 arXiv paper, “Look Again Before You Abstain”. Their baseline is a conformal grounding filter that asserts a claim only if it looks supported by the image, with a distribution-free bound on the hallucination rate among asserted claims. The first finding is the price of that bound: to hold hallucination below 5% on balanced object-existence claims, the filter has to abstain on more than 80% of claims. A system that declines four questions in five is not much of a system, so the obvious move is to look again — crop, zoom, re-score — and rescue some of them.

The second finding is what happens if you do that naively. Bolting acquisition onto a filter whose threshold was calibrated pre-acquisition inflates realised risk badly. On their existence-claim setup, at a 10% target the 90th-percentile realised risk came out at 30%, and at a 20% target it came out at 41% — a two-fold violation, on the authors’ own description. Coverage looked wonderful the whole time: the naive variant asserted 74% of claims at the 20% target against the honest filter’s 39%. That is the shape of the failure. It does not look like a bug. It looks like the retry working.

The fix in the paper is the one that generalises: fold the entire acquisition policy into the score function and calibrate on post-acquisition scores, so calibration and deployment pass through one identical map. Their phrasing for it is the sentence we now quote in reviews — calibrate on the scores you will actually deploy.

Three other ways production voids the contract

Drift. Exchangeability between calibration and test is an empirical property with a shelf life. If the input distribution moves — new document templates, a new camera, a new customer segment — coverage is no longer guaranteed at the stated level, and nothing in the pipeline will tell you. Where the shift is purely in the inputs and you can estimate the likelihood ratio between old and new input distributions, weighted conformal prediction restores validity by reweighting calibration scores; Tibshirani and colleagues worked that out in 2019. Where you cannot estimate the ratio, you are recalibrating on fresh labels, and the only honest engineering answer is to budget for that labelling as a recurring cost.

Reusing the calibration set. If you use the same labelled data to pick the score function, tune the retry policy and set the threshold, the threshold is fitted to noise in that set and the bound is optimistic. The same paper quantifies the mild version: a variant that sets the threshold from the empirical calibration risk with no finite-sample correction overshot its targets at every level tested, landing at 8%, 13% and 24% realised risk against 5%, 10% and 20% targets. Small, consistent and entirely invisible without a held-out check. There are principled ways to select over a family of policies on calibration data — Learn then Test handles it as a multiple-testing problem — but you have to actually do the correction.

Subgroup coverage. Marginal coverage can hold exactly while some population you care about is covered far below the stated level, and this is not a fixable oversight. Barber, Candès, Ramdas and Tibshirani proved that exact distribution-free conditional coverage is impossible without assumptions. So the marginal number is not a floor for any subgroup, and the only thing to do is measure per-group coverage directly and report it alongside the headline.

Calibrate the pipeline, not the model

The reframing that fixed our thinking: the object being calibrated is the pipeline, retries and gates and fallbacks included, not the model inside it. Concretely, the score function passed to calibration has to be the function that runs in production.

import numpy as np
from scipy.stats import beta


def calibrate(pipeline_score, cal_items, alpha, delta, grid):
    """Threshold the score the pipeline actually deploys, retries included."""
    scores = np.array([pipeline_score(x) for x, _ in cal_items])
    labels = np.array([y for _, y in cal_items])
    for tau in np.sort(grid):  # grid fixed a priori, never read off the scores
        accepted = scores >= tau
        n, k = int(accepted.sum()), int((labels[accepted] == 0).sum())
        if n == 0 or k == n:
            continue
        # Clopper-Pearson upper bound, Bonferroni-corrected across the grid
        if beta.ppf(1 - delta / len(grid), k + 1, n - k) <= alpha:
            return tau
    return None  # nothing certifiable at this level: abstain on everything

Two details in there are load-bearing and both cost us time. The threshold grid must be fixed before you see the calibration scores — read it off the scores and you have reintroduced the selection problem you were trying to avoid. And the gate that decides which items get a retry must not depend on the calibration sample either. Defining the retry band as “scores between the 30th and 50th calibration percentiles” feels label-free and safe, and it is neither: every calibration item’s treatment then depends on the set containing it, and calibration and test stop sharing one map. Pre-fixed numeric bounds work; quantiles of your own calibration data do not.

The cost is real and worth stating. Folding retries into the score means every calibration item pays the retry compute, and recalibration is no longer a cheap afterthought you can run on a laptop while the deploy is in flight.

Watch realised coverage, not calibration-time coverage

Calibration tells you what was true on the calibration set. Only labelled production samples tell you what is true now. We keep a small continuous audit stream — a random sample of production items sent for human labelling — and track realised coverage against target as a monitored metric with an alert, the same as latency. It is the only signal that catches drift and it is the only signal that would have caught our retry regression on the day it shipped rather than at the next audit.

Report it stratified. Angelopoulos and Bates suggest two cuts and both earn their place: coverage by a feature you care about, and coverage by prediction-set size, which needs no prior decision about which groups matter. If your marginal coverage is 90% and your worst decile is 71%, the risk team needs the 71%.

Stratifying also exposes scores that are good on average and useless on a slice. In the vision-language work above, the global grounding score — how much a claim’s likelihood drops when you blank the image — separates true from hallucinated object-existence claims well, but sits near chance at 0.57 AUROC on left/right spatial relations, because blanking the image penalises “left of” and “right of” equally. A single horizontal flip, under which a correct directional claim becomes false, lifts that to 0.77. Much of the field’s evidence here comes from POPE, which polls yes/no questions about object existence and therefore says nothing about attributes or relations. Conformal prediction will faithfully wrap a chance-level score in a valid guarantee and hand you sets you cannot use.

What we’d do differently

We would have written the coverage contract down as an artefact with an owner, listing the calibration set, the score function, the retry policy in force and the date, and made any change to the inference path require a recalibration the way a schema change requires a migration. The failure was not statistical illiteracy; it was that the pipeline change and the statistical claim lived in different documents and different heads.

We would also have built the labelled audit stream before the calibrated threshold rather than after. We shipped the guarantee first and the means of checking it second, which is the same mistake as shipping a model before the eval set — a thing we have already written up in the context of verification and confidence routing and then went ahead and repeated in a different register.

And we would resist the pull towards tighter alphas. When realised risk exceeds target, lowering alpha makes the printed number look better while leaving the broken assumption in place. The assumption is the thing to fix.

References

We calibrate and monitor production ML pipelines as part of our AI & Machine Learning Engineering practice. Talk to us if you are quoting a statistical guarantee to a risk team and are not certain your inference path still earns it.