Invariant Reasoning Directions in Latent Trajectories of Language Models

Recent advances in deep learning have significantly improved the capabilities of large language models (LLMs) in reasoning tasks. Techniques like Chain-of-Thought prompting have demonstrated the importance of intermediate reasoning steps, but these often require explicit token generation and are sensitive to input variations. Latent reasoning models, which perform inference directly in hidden states, have emerged as a promising alternative, offering efficiency and flexibility. However, understanding the internal structure of their reasoning trajectories remains a challenge. Prior work has shown that these trajectories can be highly unstable, diverging under paraphrases or perturbations, and that existing refinement methods lack robustness. Theoretical insights into the geometric structure of these trajectories are limited, hindering targeted improvements. This study builds on the hypothesis that stable reasoning behavior is concentrated in a low-dimensional subspace, which can be exploited to enhance robustness and interpretability.

Despite the success of latent reasoning models, their internal trajectories often exhibit instability, sensitivity to paraphrases, checkpoint choices, and perturbations. Existing refinement methods, which apply contrastive signals uniformly across the latent space, tend to propagate unstable components, leading to inconsistent reasoning outcomes. This fragility limits the deployment of such models in real-world scenarios demanding high reliability. The core problem is how to identify and manipulate the stable, reasoning-related directions within the high-dimensional latent space without retraining or complex optimization. Achieving this would enable more robust, interpretable, and controllable reasoning processes, addressing a critical bottleneck in the development of trustworthy AI systems.

The key innovation lies in recognizing that the differences between strong and weak reasoning trajectories are concentrated in a low-rank subspace. The authors propose a novel, training-free method—TILR—that uses singular value decomposition (SVD) on contrastive differences to extract this invariant subspace. By constraining latent interventions within this subspace, the method filters out unstable, instance-specific noise. An adaptive gating mechanism dynamically adjusts intervention strength based on the alignment score, ensuring stability and effectiveness. Unlike prior approaches that operate in the full embedding space or require retraining, TILR leverages geometric properties for efficient, interpretable, and robust intervention, providing new insights into the internal structure of neural reasoning.

• �� Collect contrastive differences between strong and weak reasoning trajectories across a calibration set.
• �� Form a difference matrix and perform truncated SVD to identify the dominant low-rank invariant subspace.
• �� During inference, project the contrastive direction onto this subspace, filtering out noisy components.
• �� Compute an alignment score to assess the reliability of the projected direction.
• �� Use an adaptive gate to scale the intervention based on this score, ensuring stable updates.
• �� Combine residual blending with the constrained intervention to update latent states.
• �� Evaluate on six reasoning benchmarks, measuring accuracy, trajectory variance, and robustness under paraphrases.
• �� Conduct ablation studies to verify the contributions of subspace projection and gating mechanisms.

Experiments utilized GPT-2 models fine-tuned with the Coconut latent reasoning framework, tested on six datasets: GSM8K, MathQA, AQuA-RAT, SVAMP, GSM-Plus, and StrategyQA. Baselines included no reasoning, chain-of-thought prompting, unconstrained contrastive refinement, and adaptive anchor methods. Metrics encompassed accuracy, reference checkpoint sensitivity, answer consistency, and trajectory variance. Hyperparameters such as the rank threshold for SVD and gating parameters were tuned. Ablation studies compared full TILR against variants lacking subspace projection or gating, confirming their importance. The experiments demonstrated that TILR consistently reduced trajectory variance and improved reasoning stability across diverse tasks.

TILR achieved an average 10% increase in answer consistency and up to 50% reduction in latent trajectory variance across benchmarks. The low-rank structure was validated by an average effective rank of 12, with three datasets requiring only a single direction. Ablation results showed that removing subspace projection or gating significantly degraded performance, confirming their necessity. The method also proved robust to reference checkpoint variations, with less than 5% performance fluctuation, and maintained accuracy comparable to baseline models. These results substantiate the hypothesis that stable reasoning directions are concentrated in a low-dimensional invariant subspace, which can be exploited to enhance model robustness.

TILR can be integrated into existing language models for applications requiring high reasoning reliability, such as automated tutoring, legal document analysis, and medical diagnosis. Its training-free nature allows easy deployment without retraining, making it suitable for real-time systems. The approach enhances robustness against paraphrasing, input noise, and adversarial perturbations, improving user trust and system stability. Long-term, the geometric insights from TILR could inform the design of inherently stable neural architectures and interpretability tools, advancing AI safety and transparency in critical domains.

The low-rank assumption may not hold in highly complex or noisy tasks, limiting generalization. The effectiveness depends on the quality of contrastive models and calibration sets, which could introduce bias. The current validation is limited to smaller models like GPT-2; scalability to larger models such as GPT-3 or GPT-4 remains to be tested. Additionally, dynamic subspace adaptation and multi-modal reasoning are unexplored directions that could further improve robustness but pose implementation challenges. Future work should address these limitations to broaden applicability.