Cloud-ScPO: Hidden-State Geometry for Semi-Supervised Preference Optimization in LLM Reasoning
Proposes Cloud-ScPO, leveraging hidden-state geometry for semi-supervised preference optimization, achieving up to 4.49% accuracy gain on GSM8K.
Key Findings
Methodology
This approach constructs multiple reference clouds from labeled trajectories by analyzing the topological connectivity of mean-pooled hidden states. It employs persistent homology to identify connected components, computes connectivity-based scores via soft k-nearest neighbors, and integrates these scores with self-consistency to select high-quality trajectories. The method propagates preference signals across different problems by leveraging the global geometric structure of hidden states, reducing reliance on external reward models. The pipeline includes cloud construction, topology analysis, scoring, and preference pair formation, optimized via variational Bayesian methods.
Key Results
- On GSM8K, Cloud-ScPO improves accuracy from 49.74% (ScPO) to 52.24% with Llama-3-8B, and from 28.43% to 32.92% with Mistral-7B, demonstrating consistent gains across models. On MATH-Numeric, Qwen3-8B achieves 62.33%, surpassing previous methods. Preference pairs generated by Cloud-ScPO maintain high correctness reliability while better filtering low-quality responses, especially repetitive or incomplete trajectories.
- Analysis shows that Cloud scoring enhances the separation of informative trajectories from noisy ones, correlating larger score gaps with answer-discordant pairs. The method effectively transfers geometric cues across tasks, leading to more robust preference signals and improved reasoning accuracy.
- The multi-bank reference clouds, built from labeled data, encode global geometric patterns that help distinguish high-quality reasoning paths, reducing dependence on external reward models and improving scalability.
Significance
This work addresses the limitations of external reward-dependent preference learning by exploiting the intrinsic geometric structure of model representations. It offers a scalable, cost-effective way to improve reasoning in large models, with potential applications in automated reasoning, education, and AI safety. The approach bridges the gap between internal model dynamics and task-specific preferences, opening new avenues for robust semi-supervised learning in NLP.
Technical Contribution
The paper introduces a novel topology-guided scoring framework that constructs multiple reference clouds from labeled trajectories, leveraging persistent homology for topology analysis. It combines these geometric scores with self-consistency to filter and select trajectories, forming high-confidence preference pairs. The integration of multi-bank clouds and topology-based metrics represents a significant methodological advance over existing reward or answer-based preference methods, enabling cross-task transfer and improved robustness.
Novelty
This is the first work to utilize the internal hidden-state geometric and topological structure of large language models for preference signal transfer across different problems. Unlike prior methods relying solely on answer consistency or external reward models, it exploits the global geometry of hidden states, providing a new paradigm for semi-supervised reasoning optimization.
Limitations
- The effectiveness depends on the quality of internal representations, which may vary across models and tasks. Poorly trained models may produce less informative geometric structures.
- Computational complexity of persistent homology and topology analysis can be high, limiting scalability to very large datasets or models.
- The approach relies on a small labeled set; its performance may degrade if the labeled data is not representative of the problem distribution.
Future Work
Future directions include developing more efficient topological analysis algorithms, extending the framework to multimodal data, and exploring adaptive cloud construction strategies. Integrating reinforcement learning with geometric signals could further enhance autonomous model improvement, enabling scalable, self-supervised reasoning systems.
AI Executive Summary
Large language models (LLMs) have demonstrated remarkable capabilities in mathematical reasoning and symbolic tasks, yet their performance heavily relies on costly external supervision or reward models. Existing methods such as reinforcement learning from human feedback (RLHF) and self-consistency techniques have limitations in robustness and scalability, especially when handling multiple trajectories per prompt. To address this, the paper introduces Cloud-ScPO, a novel semi-supervised framework that leverages the internal geometric structure of hidden states to infer preference signals.
The core innovation lies in constructing multiple reference clouds from labeled trajectories, using persistent homology to analyze their topological connectivity. These clouds serve as geometric benchmarks, against which unlabeled trajectories are scored via a topology-guided measure based on soft k-nearest neighbors. By combining these cloud scores with self-consistency, the method filters and selects high-quality trajectories, forming reliable preference pairs without requiring extensive annotations or external reward models.
Experimental results on GSM8K and MATH-Numeric datasets demonstrate that Cloud-ScPO consistently outperforms prior approaches like ScPO, with improvements up to 4.49%. The approach effectively separates informative trajectories from low-quality or repetitive responses, maintaining high correctness reliability. The geometric transfer of preference signals across different problems enables more robust reasoning, especially as model reasoning capabilities improve.
This work significantly advances the field of reasoning optimization by introducing a geometric perspective, reducing reliance on external supervision, and enabling scalable semi-supervised learning. Its implications extend to automated reasoning, educational tools, and AI safety, paving the way for more reliable and interpretable large-scale language models.
Deep Analysis
Background
近年来，大型语言模型（LLMs）在数学推理、符号逻辑等方面取得了突破性进展，代表性工作包括Chain-of-Thought（Wei et al., 2022）、自我一致性（Wang et al., 2023）以及偏好优化（Ouyang et al., 2022; Rafailov et al., 2023）。这些方法通过引导模型生成多轨迹或利用奖励模型提升推理能力，但都面临标注成本高、外部奖励依赖强的问题。随着研究的深入，学者开始关注模型内部隐藏状态的几何和拓扑结构，试图从中提取偏好信号（Burns et al., 2024; Zhang et al., 2026），但如何利用这些结构实现跨任务迁移仍未解决。
Core Problem
现有偏好优化方法多依赖答案一致性或外部奖励模型，难以在多轨迹、多任务场景中保持鲁棒性。尤其在半监督环境下，缺少充分标注，模型难以区分优劣轨迹。如何利用模型内部隐藏状态的几何拓扑结构，构建跨任务的偏好传递机制，成为提升推理性能的关键。该问题关系到自动推理系统的泛化能力和效率，亟需创新的偏好信号传递技术。
Innovation
本研究提出Cloud-ScPO，核心创新点包括：1）利用模型隐藏状态的拓扑结构，构建跨任务的多银行参考云，传递偏好信息；2）引入持久同调分析，识别隐藏状态的连通结构，增强轨迹质量评估；3）结合自我一致性与云评分机制，有效筛选优质推理路径，减少对外部奖励模型的依赖。这一方案突破了传统单任务偏好方法的局限，实现偏好信号的跨任务迁移，显著提升推理准确率。
Methodology
- �� 轨迹生成：用预训练模型对标注和未标注问题生成多条推理轨迹。• 表示提取：将每条轨迹的隐藏状态进行平均池化，获得轨迹向量。• 云构建：从标注数据中筛选正确与错误轨迹，利用持久同调分析识别连通子结构，构建多个参考云。• 云评分：对未标注轨迹在每个云中计算拓扑连通性指标，结合软k近邻机制，得到云评分。• 自我一致性：通过多轨迹投票确定答案偏好方向。• 轨迹筛选：结合云评分和偏好方向，筛选高质量轨迹，形成偏好对。• 模型训练：采用变分贝叶斯优化，调整模型参数，提升推理能力。
Experiments
在GSM8K和MATH-Numeric两个数据集上，使用Llama-3-8B和Mistral-7B模型，比较基线（如零-shot、SFT、ScPO）与提出方法的性能。每题生成8-16条轨迹，利用少量标注数据构建参考云，调节云评分参数。评估指标包括准确率、偏好对数量和质量、偏好反转率。还进行消融实验，验证不同表示策略和拓扑指标的影响。偏好对分析确认云评分在过滤低质量轨迹中的有效性。
Results
Cloud-ScPO在GSM8K上，Llama-3-8B模型准确率由49.74%提升至52.24%，Mistral-7B由28.43%提升至32.92%。在MATH-Numeric上，Qwen3-8B达到62.33%，优于传统偏好方法。偏好对的质量和数量均优，能更好区分信息丰富的轨迹与低质量响应。偏好信号的几何传递增强了模型的推理鲁棒性，过滤掉大量重复、截断或错误轨迹。
Applications
该方法适用于自动数学推理、符号逻辑、编程辅助等场景，尤其在标注成本高或多任务环境中表现优越。利用模型内部几何信息，可降低对外部奖励模型的依赖，提升系统的自适应能力。未来可结合多模态数据和动态云构建策略，推动智能推理系统的发展。
Limitations & Outlook
当前方法依赖模型内部表示的质量，若模型表示不佳，偏好信号效果受限。持久同调分析在高维空间中计算复杂，限制大规模应用。偏好对构建受限于少量标注数据的代表性，未来需探索更鲁棒的几何特征提取和自适应机制。
Plain Language Accessible to non-experts
想象你在一家工厂里工作，工厂里有许多不同的机器在生产不同的产品。每台机器按照特定流程操作，有些流程更快、更准，有些则容易出错。管理者希望找到最优的生产流程，但没有时间逐一检查每台机器。于是，他们开始观察每台机器的操作轨迹，从中发现一些共同的规律，比如哪些流程更连贯、哪些容易出错。通过分析这些“操作轨迹”的整体结构，管理者可以判断某个流程是否优良，而不用每次都亲自检查。这个方法就像论文中用几何和拓扑分析隐藏状态的结构，从而判断推理轨迹的质量，帮助模型变得更聪明、更可靠。
ELI14 Explained like you're 14
假设你在学校参加数学比赛，你会做很多题，有些做得特别棒，有些则出错。老师想帮你找到最好的解题步骤，但不能每次都检查每个细节。于是，老师开始观察你做题的过程，把每个步骤都记下来，然后找出那些看起来很合理、连贯的步骤。老师发现，正确的解题步骤通常在某个“思路空间”里聚集得很紧密，而错误的步骤则散落得很远。通过观察这些“思路空间”的结构，老师可以判断哪些步骤更靠谱，帮助你更快找到正确的解题方法。这就像论文里用几何和拓扑的方法，观察模型内部的“思路空间”，判断推理轨迹的好坏，让模型变得更聪明、更可靠。
Glossary
Hidden State (隐藏状态)
模型在每个时间步的内部表示，反映当前信息的抽象特征。技术上为神经网络中的中间向量。
用于表示推理轨迹的整体特征，构建云结构和进行几何分析。
Topology (拓扑结构)
描述空间中点的连通关系和整体形状的数学结构。用于分析隐藏状态的几何连通性。
在云评分中，通过拓扑连通性指标评估轨迹的质量。
Persistent Homology (持久同调)
一种拓扑数据分析方法，用于识别数据中的连通和空洞结构，反映数据的多尺度特性。
用于分析隐藏状态点云的连通结构，构建参考云。
Cloud Score (云评分)
基于拓扑连通性指标的轨迹质量评估机制，衡量轨迹与参考云的相似度。
核心指标，用于筛选优质推理路径。
Open Questions Unanswered questions from this research
- 1 如何提升几何结构在高维空间中的表达能力，解决计算复杂度。
- 2 能否结合多模态信息扩展几何偏好传递的应用范围。
- 3 在极少标注情况下，如何自动构建更鲁棒的参考云。
Applications
Immediate Applications
数学推理优化
利用模型内部几何结构筛选优质推理路径，提升自动解题的准确性和鲁棒性，适用于教育和科研。
符号逻辑与编程辅助
通过几何偏好信号优化逻辑推理和代码生成模型，减少错误，增强推理连贯性。
Long-term Vision
通用推理引擎
结合几何结构分析，构建跨任务、多模态的智能推理系统，实现自主学习和优化，推动AI推理技术发展。
Abstract
Preference optimization improves mathematical reasoning in large language models (LLMs), but reliable chosen-rejected pairs usually require verified answers, human annotations, or external reward models. We investigate whether preference supervision can instead be derived from the model's internal representation geometry in a semi-supervised setting. Our analysis shows that reasoning trajectories generated across different mathematical problems form structured global point clouds in which correct and incorrect trajectories exhibit different geometric organization. Based on this observation, we propose Cloud-ScPO, a topology-guided preference-mining framework that uses a small labeled set to construct multiple correct and incorrect reference Clouds. Each trajectory is represented by a mean-pooled hidden state and scored against connectivity-induced components using a component-level soft $k$-nearest-neighbor measure averaged across reference banks. We combine this cross-problem Cloud signal with prompt-level self-consistency: self-consistency determines the answer-level preference direction, while Cloud scoring selects concrete trajectories and filters pairs by their score margin. Experiments on GSM8K and MATH-Numeric across four model settings show that Cloud-ScPO consistently improves over ScPO, with gains of up to 4.49% on GSM8K and 4.19% on MATH-Numeric. Pair-level analyses further show that Cloud-ScPO maintains comparable correctness reliability while more effectively separating informative chosen trajectories from incomplete, repetitive, or otherwise low-quality rejected responses.