Routing Without Training: Controllable-Ratio LLM Offloading via Reliability Gating
CARGO is a training-free routing framework using response agreement and Bayesian early stopping to decide when local models can answer reliably, supporting adjustable cloud offloading ratios.
Key Findings
Methodology
CARGO leverages prompt variation to generate diverse responses, then applies Bayesian posterior estimation of the mode mass (most frequent answer probability) to assess reliability. It employs Bayesian early stopping based on credible intervals to limit sampling, dynamically adjusting the number of responses needed. The system calibrates routing parameters at deployment to meet target offloading ratios, without additional training. This approach integrates multiple components—prompt variation, Bayesian mode estimation, early stopping, and calibration—forming an efficient, adaptable routing mechanism suitable across various tasks and models.
Key Results
- In mathematical reasoning and QA tasks, CARGO achieves 93.89% accuracy at a 30% collaboration ratio, outperforming unsupervised baselines (88.72%) and supervised routers. It reduces average sampling by 40%, maintaining high reliability. The Bayesian early stopping cuts response latency by 20%, demonstrating efficiency. Across multiple models, including Llama-3.2-1B and Qwen-7B, it consistently surpasses baselines, showing robustness and adaptability.
- CARGO’s response agreement signal correlates strongly with correctness, outperforming confidence scores and chain-of-thought steps, especially in resource-constrained settings. Its calibration supports flexible offloading ratios, making it suitable for dynamic deployment environments.
- Ablation studies confirm that prompt diversity and Bayesian early stopping are key to its performance, enabling high accuracy with fewer samples and better resource management.
Significance
This work addresses the critical challenge of resource-efficient, reliable deployment of large language models in edge and constrained environments. By eliminating the need for trained routers, it simplifies system design and enhances flexibility. The approach leverages intrinsic model behavior, paving the way for scalable, adaptive AI services that can dynamically balance local computation and cloud assistance, thus broadening the practical applicability of LLMs in real-world scenarios.
Technical Contribution
The paper introduces a novel combination of prompt-based diversity sampling with Bayesian posterior estimation of response agreement, providing a theoretically grounded, training-free reliability measure. The Bayesian early stopping mechanism ensures sample efficiency, while the calibration step supports deployment-specific collaboration ratios. This framework extends the concept of self-consistency into a practical, adaptive routing system, offering formal guarantees of convergence and reliability.
Novelty
This is the first work to utilize response agreement, estimated via Bayesian posterior, as a training-free, inference-time reliability signal for routing decisions in LLM deployment. It innovatively combines prompt variation with Bayesian uncertainty quantification, avoiding task-specific supervision and retraining, which sets it apart from prior learned or fine-tuned routing methods. The adaptive sampling and calibration strategies further contribute to its novelty.
Limitations
- The reliability estimation may be less accurate for highly ambiguous or adversarial inputs, leading to potential misrouting. The assumption of response independence may not hold perfectly in practice, affecting Bayesian estimates. Additionally, in extremely resource-constrained environments, even adaptive sampling incurs latency, which could limit real-time applicability. Future work should explore multi-modal signals and more robust uncertainty measures.
Future Work
Future research could incorporate multi-modal data to enhance reliability signals, develop more sophisticated adaptive sampling strategies, and explore multi-task joint calibration. Extending the framework to multi-turn dialogues and multi-modal inputs, as well as integrating reinforcement learning for dynamic parameter tuning, are promising directions to further improve robustness and efficiency.
AI Executive Summary
Deploying large language models (LLMs) on resource-constrained devices presents a fundamental challenge: balancing inference quality with efficiency. Existing solutions often rely on training dedicated routers or fine-tuning models to decide when to offload queries to cloud services. While effective, these methods lack flexibility, especially when deployment conditions change dynamically, such as varying latency budgets or cloud costs.
This paper introduces CARGO, a novel, training-free routing framework that leverages the model’s own response behavior to estimate reliability. The core idea is to generate multiple responses through prompt variation, then assess their agreement using a Bayesian posterior estimate of the mode mass—the probability that the model’s most frequent answer is correct. This intrinsic signal provides a robust measure of confidence without task-specific supervision.
To optimize efficiency, CARGO employs Bayesian early stopping, which halts sampling once the uncertainty around the agreement estimate falls below a threshold. This mechanism ensures minimal response generation while maintaining high reliability. Additionally, the system supports deployment-time calibration, allowing users to specify target collaboration ratios—such as offloading 10%, 30%, or 50% of queries—by adjusting a simple parameter.
Extensive experiments across diverse reasoning and question-answering datasets demonstrate that CARGO consistently outperforms unsupervised baselines and, in several cases, surpasses supervised learned routers. It achieves higher accuracy with fewer samples, reduces latency, and adapts seamlessly to different models and tasks. These results highlight that effective local-cloud collaboration can emerge directly from the local model’s intrinsic response behavior, eliminating the need for additional training or complex supervision.
The implications of this work are significant: it simplifies deployment pipelines, reduces costs, and enhances system flexibility. By grounding routing decisions in model-internal signals, CARGO paves the way for more scalable and adaptable AI services, especially in edge and resource-limited environments. Future directions include multi-modal integration, multi-task calibration, and reinforcement learning-based parameter tuning, promising further improvements in robustness and efficiency.
Deep Analysis
Background
随着大规模语言模型（LLM）在自然语言处理中的广泛应用，如何在有限资源环境中高效部署成为研究热点。传统方案依赖云端推理或微调模型，虽然效果显著，但成本高昂且缺乏灵活性。边缘设备对响应速度和能耗提出更高要求，推动边缘计算与云端协作的研究。早期方法多采用训练好的分类器或微调模型进行路由，存在适应性差、成本高的问题。近年来，模型的响应一致性（self-consistency）被提出作为无训练的可靠性指标，显示出在无需额外训练的情况下估算模型信心的潜力。本文基于此，提出一种全新的无训练路由方案，旨在利用模型内在行为特征实现高效、灵活的云边协作。
Core Problem
核心问题在于如何在无需训练额外路由器的情况下，利用模型自身的响应行为判断其在特定任务中的可靠性。现有方法依赖训练好的路由模型或微调，难以适应动态变化的部署环境。边缘设备对延迟和计算资源有限，要求路由机制既要高效又要灵活。如何在保证准确率的同时，减少响应采样次数，成为关键挑战。此外，还需支持不同合作比例的调节，以满足多样化应用场景。
Innovation
本文的创新主要包括：1）提出基于响应一致性的无训练信号，避免任务特定监督；2）结合贝叶斯后验估计，动态调节采样次数，提升样本效率；3）引入prompt变异机制，确保多样性采样而非随机噪声；4）设计贝叶斯早停策略，确保在保证可靠性前提下减少采样。这些创新实现了高效、灵活、无需训练的路由方案，适应多任务、多模型环境。
Methodology
- �� 采样多样性：通过设计一组语义等价的系统prompt，促使模型生成多样响应，避免随机噪声影响。• 可靠性估算：利用响应中最频繁答案的概率（mode mass）作为模型可靠性指标，采用Beta分布后验进行估计。• 贝叶斯早停：在采样过程中，计算后验置信区间，满足预设置信度和误差阈值后停止采样，提升效率。• 动态调节：通过校准参数λ，调整响应一致性阈值，实现目标合作比例。• 采样策略：逐步采样响应，直到置信区间满足条件或达到最大采样数。• 路由决策：根据估算的可靠性指标，利用sigmoid函数映射为云端或本地输出的概率，随机采样决定最终路径。
Experiments
采用多任务数据集（如MATH-lighteval、GSM8K、SVAMP、SQuAD）评估CARGO性能。比较基线包括随机、自信度阈值、链式推理（CoT）步数和监督路由。指标涵盖准确率、采样次数和延迟。模型包括Llama-3.2-1B、Qwen2.5-7B等，云端模型为DeepSeek-R1。调节合作比例，验证系统在不同资源限制下的适应性。进行消融实验，分析贝叶斯早停和prompt多样性的贡献。
Results
在合作比例为30%的条件下，CARGO实现93.89%的准确率，优于无路由（88.72%）和训练路由（88.88%）。平均节省40%的采样次数，响应延迟降低20%。多模型、多任务场景中，表现优异，超越部分监督路由，验证其高效性和适应性。响应一致性信号与正确率高度相关，优于传统信心指标，证明其作为可靠性指标的有效性。
Applications
适用于边缘设备、移动端和隐私敏感场景，无需微调或训练即可实现智能问答、推理和决策支持。支持动态调节合作比例，满足多样化需求。未来可结合多模态信息，提升复杂场景中的鲁棒性，为智能边缘系统提供新方案。
Limitations & Outlook
在极端复杂或模糊任务中，响应一致性判断可能不够准确，导致误判。贝叶斯估计假设响应独立性，实际中可能受模型偏差影响。采样过程仍存在延迟，极端资源受限环境下响应时间可能不理想。未来需优化采样策略和多模态融合，提升鲁棒性。
Plain Language Accessible to non-experts
想象你在厨房里准备一道菜，你可以用不同的调料和方法尝试多次，每次尝试后都品尝一下。若多次尝试后味道都差不多，说明这道菜已经做好了，可以端上桌；如果每次味道都不一样，说明还需要调整或请教厨师。CARGO就像你用不同的调料试味道，它让电脑模型多次生成答案，然后观察这些答案是否一致。如果答案都差不多，模型就可以自己回答问题；如果不一致，就让更强的模型帮忙。这种方法不用专门训练模型判断自己是否靠谱，只靠它自己生成的答案来决定，既快又灵活，就像你用不同的调料试味道一样。
ELI14 Explained like you're 14
想象你在学校的科学实验室里，面对一个难题，你可以用不同的方法尝试解决。每次尝试后，你会觉得这个方法是否靠谱，比如每次都得到差不多的答案，说明你对这个问题有信心；如果每次都答得不一样，说明还不确定，需要请老师帮忙。CARGO就像你用这个办法，它让电脑模型自己多次尝试回答问题，然后观察答案是否一致。如果答案都差不多，模型就可以自己解决问题；如果不一致，就让云端的更强模型帮忙。这样不用专门教模型怎么判断自己是否靠谱，只靠它自己生成的答案来决定。这种方法既快又灵活，能在不同任务和模型中都用得上，就像你用不同的尝试方法来判断问题的难度一样。
Abstract
Local-cloud collaboration is a practical way to deploy large language models under resource constraints, but existing methods often rely on trained routers or collaboration-aware finetuning that tie routing behavior to a particular operating regime. In this work, we show that such training may be unnecessary: the local model's own inference-time agreement across sampled responses already provides a strong signal for deciding when to trust local execution and when to offload to a stronger cloud model. We propose CARGO, a training-free routing framework that estimates this agreement through prompt-varied sampling, applies Bayesian early stopping for sample-efficient uncertainty control, and supports arbitrary target collaboration ratios through lightweight deployment-time calibration. Across diverse reasoning and question-answering tasks, multiple local LLM families and scales, and both pretrained and finetuned local models, CARGO consistently outperforms other training-free baselines and in several settings surpasses supervised learned routers. These results suggest that effective and adaptable local-cloud collaboration can emerge directly from the local model's intrinsic response behavior, without requiring an additional trained router.