Occupancy-based Quantile Risk Control
OQRC uses finite occupancy to tighten finite-sample quantile-risk control, cutting MS COCO RiskGap by 78.64%.
Key Findings
Methodology
Occupancy-based Quantile Risk Control (OQRC) sorts calibration losses X(1)≤…≤X(n), partitions [L,U] into n+1 bins, and counts m test losses in each bin. The occupancy vector is uniformly distributed over Cm,n by the stars-and-bars result. OQRC replaces every loss in a bin by that bin’s maximum endpoint, yielding an upper risk functional Φ+. Its (1−δ)-quantile R+ is a valid upper confidence bound, and the calibrated parameter is λ*=inf{λ:R+(λ)≤α}.
Key Results
- Across Polyp, MS COCO, and Go Emotions, with n=200, m=500, α=0.4, and confidence 0.9, OQRC achieved coverage of roughly 0.904–0.959 while reducing conservatism. On MS COCO under VaR-Interval, it reduced RiskGap by 78.64% and AvgSize by 38.80% versus Two-sided BJ.
- Across model architectures on MS COCO, OQRC reduced mean RiskGap from 0.347 to 0.075 and AvgSize from 15.226 to 6.849. Across varying confidence levels, it achieved average reductions of 82.54% in RiskGap and 44.66% in AvgSize.
- Theorem 3.5 provides finite-sample validity, P(Rtest(λ*)≤α)≥1−δ. Under a density bounded away from zero and infinity, Theorem 3.6 gives |R+−M*|=Op(n−1/2), establishing convergence to the optimal risk bound.
Significance
The paper addresses a central deployment tension: distribution-free safety guarantees often produce unusably conservative predictions, whereas tighter L-statistic methods may lack finite-sample validity. OQRC offers a training-free, model-agnostic calibration layer for VaR, CVaR, and VaR-Interval. Its relevance spans medical vision, multilabel recognition, and language systems, where controlling tail errors—not merely average error—is operationally important. It therefore provides a practical bridge between formal uncertainty quantification and information-preserving deployment.
Technical Contribution
The key contribution is a finite-occupancy reformulation of quantile-risk control. Theorem 3.1 gives an exact discrete-uniform law for occupancy counts, avoiding direct estimation of the unknown loss distribution. The endpoint-based functional Φ+ converts these counts into a conservative risk envelope, whose quantile is a valid UCB. For scalability, stars-and-bars sampling generates uniform occupancy vectors, while the DKW inequality bounds Monte Carlo CDF error by 2e−2Bε². Tightness is proved at Op(n−1/2), and unconditional control follows from E[Rtest]≤E[Φ+].
Novelty
Relative to Snell et al.’s one-sided CDF construction, OQRC is substantially less conservative; relative to Chen et al.’s CDRC-L, it does not rely on asymptotic normality and remains valid in finite samples. The paper positions itself as the first quantile-risk method combining finite-sample validity with a proof of convergence to the optimal bound. Its conceptual innovation is replacing distribution estimation by exact combinatorial occupancy inference.
Limitations
- The main exposition assumes no ties among losses, although Appendix A.1 extends the theory to tied values. Discrete prediction scores or heavily quantized losses may still create numerical and calibration behavior that deserves dedicated evaluation.
- Exact enumeration scales poorly because |Cm,n|=C(m+n,n). Monte Carlo sampling improves feasibility but introduces sampling error and requires a principled choice of B alongside the statistical confidence budget.
- Experiments focus on FNP loss, n=200, m=500, and i.i.d. data; distribution shift, temporal dependence, extreme tails, and large-scale production latency remain insufficiently tested.
Future Work
Important directions include covariate and label shift, dependent or online calibration, adaptive Monte Carlo budgets, and faster dynamic-programming approximations. Future studies should also evaluate subgroup fairness, uncertainty explanations, deployment latency, and robustness under nonstationary medical, financial, and autonomous-driving streams.
AI Executive Summary
Machine-learning systems now influence medical diagnosis, finance, and autonomous driving, where average error is an inadequate safety measure. Conformal Risk Control (CRC) controls expected loss, while Quantile Risk Control (QRC) extends guarantees to VaR and CVaR. Yet one-sided CDF methods can be excessively conservative, and L-statistic approaches such as CDRC-L rely on asymptotic theory that may fail at realistic calibration sizes.
Shi and colleagues propose Occupancy-based Quantile Risk Control, or OQRC. It sorts calibration losses and uses them as boundaries for a finite set of bins. The method then counts how many test losses occupy each bin. A stars-and-bars argument shows that this occupancy vector has an exact discrete-uniform distribution, independent of the numerical loss values. By replacing each bin’s losses with its maximum endpoint, OQRC constructs an upper risk functional Φ+; its 1−δ quantile becomes a finite-sample upper confidence bound R+. The smallest λ satisfying R+≤α is selected, with Monte Carlo sampling and the DKW inequality used at scale.
On Polyp, MS COCO, and Go Emotions tasks, with n=200 and m=500, OQRC maintained approximately 0.904–0.959 coverage. On MS COCO VaR-Interval, it reduced RiskGap by 78.64% and AvgSize by 38.80% versus Two-sided BJ; across architectures, mean RiskGap fell from 0.347 to 0.075. The theory proves finite-sample validity and Op(n−1/2) convergence to the optimal bound. Remaining challenges include combinatorial computation, i.i.d. assumptions, distribution shift, and broader deployment validation.
Deep Analysis
Background
CRC extended conformal prediction from coverage to expected-loss control. QRC broadened the target to tail measures such as VaR and CVaR. Snell et al. used one-sided tests and lower CDF confidence bounds, but their procedure can overestimate risk. Chen et al.’s CDRC-L uses L-statistics for tighter control, yet its guarantees are asymptotic. OQRC targets the missing combination: tightness with finite-sample validity.
Core Problem
Given a black-box predictor h, monotone loss ℓλ∈[L,U], risk threshold α, and confidence 1−δ, select λ so that P(Rψ(FX;λ)≤α)≥1−δ. The difficulty is that quantile risk depends on the full loss distribution. A valid method must protect against unseen test losses without shrinking prediction sets so aggressively that useful information is discarded.
Innovation
First, OQRC converts continuous loss inference into a finite occupancy problem defined by calibration order statistics. Second, Theorem 3.1 supplies an exact distribution for occupancy counts, avoiding unknown-distribution estimation. Third, the endpoint functional Φ+ yields a valid risk envelope whose quantile converges to the optimal bound at Op(n−1/2). Fourth, the framework contains classical CRC as the m=1 expected-loss special case and supports unconditional quantile-risk control.
Methodology
- �� Input: calibration losses, test size m, weighting function ψ, target α, and confidence δ.
- �� Partition: set X(0)=L and X(n+1)=U; define I0=[L,X(1)] and Ij=(X(j),X(j+1)].
- �� Occupancy: Nj=Σt1{Xtestt∈Ij}; the vector sums to m and is uniform on Cm,n with probability m!n!/(m+n)! for each composition.
- �� Upper functional: for x, integrate ψ over cumulative occupancy intervals while assigning each bin its endpoint X(j+1), producing Φ+.
- �� Calibration: form F+(r), set R+=(F+)−1(1−δ), and choose λ*=inf{λ:R+(λ)≤α}.
- �� Scaling: sample uniform compositions through stars-and-bars; DKW bounds the empirical Monte Carlo CDF error by 2e−2Bε².
Experiments
The study uses pooled polyp-segmentation data from Kvasir, Hyper-Kvasir, CVC-ColonDB, CVC-ClinicDB, and ETIS-Larib (798 examples), 2,000 MS COCO images with 80 classes, and 5,427 Go Emotions comments with 28 emotion labels. Models include PraNet, ResNet50, BERT, plus EfficientNet, TResNet, and ConvNeXt. Calibration and test sizes are 200 and 500; experiments repeat 1,000 times. Baselines are OrderStats, one- and two-sided Berk–Jones methods, and CDRC-L. Metrics are coverage, RiskGap, and AvgSize.
Results
For CVaR, β=0.8; for VaR-Interval, [βmin,βmax]=[0.85,0.95]; α=0.4 and confidence is 0.9. OQRC coverage ranges from 0.925–0.959 for CVaR and 0.904–0.937 for VaR-Interval. On MS COCO VaR-Interval, it obtains RiskGap 0.066±0.041 and AvgSize 3.115±0.237, versus 0.309 and 5.090 for Two-sided BJ. CDRC-L coverage is only about 0.841–0.844 and fails the nominal requirement.
Applications
OQRC can calibrate tumor-segmentation systems to control tail false-negative pixel rates, constrain false negatives in multilabel image classification, and regulate rare-emotion errors in text classification. Deployment requires an independent calibration set that is approximately exchangeable with future data and a loss that is non-increasing and left-continuous in λ.
Limitations & Outlook
The guarantees rely on exchangeability and monotone, left-continuous losses; temporal dependence, covariate shift, and heavy tails can invalidate the stated protection. Exact occupancy enumeration is combinatorially expensive, while Monte Carlo trades computation for approximation error. The benchmark scope is also limited: it mainly uses FNP, fixed sample sizes, and offline evaluation. Future work should combine OQRC with shift detection, online recalibration, efficient sampling, and fairness constraints.
Plain Language Accessible to non-experts
Imagine a hospital triage machine that assigns every patient a danger score. The hospital wants a setting that keeps serious mistakes below a chosen limit, but it does not want to reject nearly everyone just to appear safe. First, the team uses patients with known outcomes to place score markers from low to high, creating a row of compartments. Every future patient falls into one compartment.
Instead of pretending to know the exact danger inside a compartment, the method treats everyone there as having the compartment’s worst score. This is cautious, but it prevents underestimating danger. It then considers all reasonable ways that future patients could be distributed across the compartments and finds a risk level that will not be exceeded in, say, 90% of repeated situations.
The system adjusts one control knob until this cautious risk is below the hospital’s target. Because it uses the number of patients in each compartment rather than guessing the entire score distribution, it can be both safe and less wasteful. In tests on polyp images, object-label images, and emotion text, it produced much smaller gaps from the target than older procedures. On MS COCO, the gap fell by 78.64% compared with a competing method.
ELI14 Explained like you're 14
Suppose you are designing a game character’s shield. A huge shield blocks almost everything but makes the character slow and boring; a tiny shield looks cool but lets dangerous attacks through. You want the smallest shield that still keeps the chance of a bad hit under a fixed limit.
OQRC first studies practice rounds where the real damage is known. It sorts the damage scores and draws boundaries, like levels in a game. New players’ scores land in these levels. The method does not assume everyone in a level is lucky: it counts them as if they received the worst damage allowed in that level.
Then it checks many possible ways players could be spread across the levels and finds a safe upper estimate. If that estimate is below the target, it chooses the setting. So the system stays safe without turning the shield into a giant wall around the whole character. Pretty clever, right?
Researchers tested it on tumor pictures, MS COCO images, and emotion comments. On MS COCO, its risk gap was 78.64% smaller than a competing approach, and it worked with ResNet, EfficientNet, TResNet, and ConvNeXt. It does not retrain the model; it adds a smart safety dial after training!
Glossary
Quantile Risk Measure
A risk measure based on selected parts of the loss distribution rather than only its mean. Formally, Rψ=∫0^1ψ(p)F−1(p)dp. It includes VaR, CVaR, and interval averages.
OQRC is designed for this general family.
Occupancy
The number of test observations falling into each interval defined by calibration order statistics. The resulting vector has an exact discrete-uniform law over all compositions of m.
Occupancy is the probabilistic foundation of OQRC.
Upper Confidence Bound
A random upper limit that exceeds the target risk with probability at least 1−δ. It enables safe parameter selection without estimating the full distribution.
OQRC uses R+=(F+)−1(1−δ).
CVaR
Conditional Value-at-Risk averages losses beyond a chosen quantile β, emphasizing the tail. In the paper, its weight is 1{p≥β}/(1−β).
Experiments set β=0.8.
RiskGap
The empirical average absolute difference between achieved and target risk. Among valid methods, a smaller value indicates less conservatism and tighter control.
It is a primary empirical comparison metric.
Op(n−1/2)
A probabilistic convergence rate indicating that error decreases approximately as the inverse square root of calibration size. It is an asymptotic tightness statement.
Theorem 3.6 gives this rate for OQRC’s bound.
Open Questions Unanswered questions from this research
- 1 The theory assumes exchangeability, but medical streams, user activity, and sensor data are dependent or drifting. A method that retains finite-sample guarantees under such conditions remains an open theoretical challenge.
- 2 The composition space grows rapidly with m and n. Adaptive selection of Monte Carlo size B, with simultaneous control of sampling error, calibration uncertainty, and computation, needs further study.
Applications
Immediate Applications
Medical image false-negative control
Hospitals can calibrate a trained PraNet-like segmenter on held-out cases and choose λ to control CVaR or VaR-Interval of missed tumor pixels. The approach requires future cases to be approximately exchangeable with calibration cases and provides an auditable finite-sample guarantee.
Safe multilabel predictions
Vision platforms can attach OQRC to ResNet50, ConvNeXt, or related classifiers to constrain false-negative risk while reducing prediction-set size. The MS COCO experiments indicate substantially smaller RiskGap and AvgSize than the tested baselines.
Long-term Vision
Auditable risk-aware AI layer
OQRC could become a post-training safety layer for healthcare, finance, and autonomous systems. Combined with drift detection, online recalibration, and subgroup constraints, it could update risk controls without repeatedly retraining the underlying black-box model.
Abstract
Conformal risk control is an emerging framework for the safe deployment of machine learning models with finite-sample guarantees. To accommodate a broader class of risk notions, quantile risk control extends this framework to quantile-based risk measures. However, existing methods either suffer from excessive conservatism or lack rigorous finite-sample guarantees. To address these limitations, we introduce Occupancy-based Quantile Risk Control (OQRC), a novel method that provides tight risk control bounds with finite-sample validity. Our key idea is to formulate risk control as a finite-occupancy problem by partitioning the loss space with the ordered calibration losses. Specifically, we estimate the distribution of test losses across the resulting bins and upper-bound the risk by the maximum loss attained within each bin. We then select the parameter $λ$ such that this upper bound does not exceed a predefined threshold $α$ with high probability $1-δ$. Theoretically, we establish a finite-sample guarantee showing that OQRC yields tight risk control bounds that converge to the optimal bounds at a provable rate of $\mathcal{O}_ p(n^{-1/2})$. Extensive experiments demonstrate the effectiveness of our method, reducing the risk gap by up to 78.64\% on common benchmarks.