Intern-S2-Mobius: Foundation Model with Decoupled Knowledge and Reasoning
Mobius-v0 separates knowledge and reasoning, achieving nearly 4x inference speedup while maintaining accuracy.
Key Findings
Methodology
Mobius-v0 employs a globally shared Memory (FFN) and multiple Reasoners (Self-Attn). Hidden states act as cache and carrier, enabling Reasoners to repeatedly query the Memory for knowledge vectors and transmit knowledge back to reasoning operators. This architecture enhances knowledge compression and reasoning efficiency.
Key Results
- Result 1: The 7B model achieves comparable downstream performance to a Transformer 7B model using only 62.6% of the training data.
- Result 2: Intern-S2-Mobius, pretrained from Qwen3.5-35B, delivers nearly 4x inference speedup while maintaining performance.
- Result 3: On scientific tasks like Mol-Instructions, Intern-S2-Mobius scores 52.14, compared to Qwen3.5's 18.20.
Significance
Mobius addresses inefficiencies in Transformer architectures by decoupling knowledge storage and reasoning, providing a new pathway for efficient inference in large-scale models. Its impact spans academia and industry.
Technical Contribution
Mobius introduces backward residual connections and dynamic latent reasoning, significantly improving inference efficiency. It also achieves superior knowledge compression, reducing training data requirements.
Novelty
Mobius is the first architecture to fully decouple knowledge and reasoning, leveraging shared Memory and dynamic reasoning mechanisms to optimize efficiency, marking a clear departure from existing Transformer designs.
Limitations
- Limitation 1: The shared Memory may introduce memory access bottlenecks in large-scale models.
- Limitation 2: Efficiency improvements depend on specific tasks and may not generalize universally.
- Limitation 3: Performance on larger-scale models remains to be validated.
Future Work
Future directions include exploring Mobius's scalability in ultra-large models, optimizing Memory storage and access, and extending its applicability to more tasks.
AI Executive Summary
Mobius-v0 is a groundbreaking architecture designed to overcome inefficiencies in Transformer models related to knowledge storage and reasoning. By introducing a globally shared Memory (FFN) and multiple Reasoners (Self-Attn), Mobius achieves complete decoupling of knowledge and reasoning. Reasoners use hidden states to query the Memory repeatedly, retrieving necessary knowledge vectors and transmitting them back for efficient reasoning.
Experiments demonstrate Mobius's significant advantages. Its 7B model achieves comparable performance to a Transformer 7B model using only 62.6% of the training data. Additionally, the Intern-S2-Mobius version, pretrained from Qwen3.5-35B, delivers nearly 4x inference speedup while excelling in scientific tasks.
Key innovations include backward residual connections and dynamic latent reasoning, which enhance inference efficiency and reduce training data requirements. While challenges like memory access bottlenecks remain, Mobius offers a promising pathway for efficient reasoning in large-scale AI models and sets the stage for future advancements in the field.
Deep Analysis
Background
Transformer architectures dominate deep learning across domains like language and vision. However, inefficiencies in reasoning and redundant knowledge storage limit their scalability and practical deployment.
Core Problem
Transformer's quadratic self-attention complexity hinders long-context reasoning efficiency. Additionally, redundant knowledge storage across layers demands excessive training data, creating bottlenecks for real-world applications.
Innovation
Mobius addresses these issues through: 1) a globally shared Memory for knowledge compression, 2) backward residual connections for cross-layer information flow, and 3) dynamic latent reasoning for efficient inference.
Methodology
- �� Globally shared Memory: FFN stores knowledge vectors accessible to all Reasoners.
- �� Multiple Reasoners: Self-Attn modules query the Memory iteratively via hidden states.
- �� Dynamic latent reasoning: Reduces redundant reasoning chains and increases information density.
- �� Backward residual connections: Enable deep layers to access shallow-layer knowledge, enhancing synthesis.
Experiments
Experiments include training-from-scratch (TFS) and continual pretraining (CPT). The 7B model trained on 1TB tokens achieves comparable performance to Transformer using 62.6% data. Intern-S2-Mobius, pretrained from Qwen3.5-35B, excels in scientific tasks.
Results
Mobius achieves 62.6% data efficiency compared to Transformer on MMLU tasks. On scientific benchmarks, Intern-S2-Mobius scores 52.14 versus Qwen3.5's 18.20. Inference speed improves nearly 4x.
Applications
Mobius is ideal for scenarios requiring efficient reasoning, such as real-time QA, scientific computation, and code generation, where high information density and rapid response are critical.
Limitations & Outlook
While Mobius improves inference efficiency, its shared Memory may introduce memory bottlenecks. Task-specific performance variability and scalability to larger models require further validation.
Plain Language Accessible to non-experts
Imagine Mobius as a library where the Memory is the bookshelf and Reasoners are readers. Unlike traditional Transformers where each reader can only access books on their floor, Mobius lets all readers access the entire library simultaneously, speeding up information retrieval and reducing redundancy.
ELI14 Explained like you're 14
Think of Mobius like playing a game where you need clues to solve puzzles. Traditional models let you look at one room's clues at a time. Mobius is like having access to the whole map of clues instantly — faster and smarter, right? Cool stuff!
Glossary
FFN (Feed-Forward Network)
Module storing knowledge vectors, responsible for compression and transmission.
Used in Mobius to construct the shared Memory.
Self-Attn (Self-Attention)
Reasoning module responsible for synthesizing information.
Used in Mobius for iterative reasoning.
Backward Residual Connection
Mechanism allowing deep layers to access shallow-layer knowledge.
Enhances cross-layer information flow in Mobius.
Dynamic Latent Reasoning
Technique reducing redundant reasoning chains and optimizing inference.
Improves efficiency in Mobius.
Knowledge Compression
Technique reducing redundant storage to enhance model efficiency.
Achieved through shared Memory in Mobius.
Open Questions Unanswered questions from this research
- 1 How can shared Memory storage and access efficiency be optimized?
- 2 Can Mobius's scalability in ultra-large models be further validated?
Applications
Immediate Applications
Real-Time QA Systems
Enables rapid responses for customer support and search engines.
Scientific Computation
Accelerates complex problem-solving in biology and chemistry.
Long-term Vision
General AI
Paves the way for smarter models through efficient reasoning and knowledge storage.
Abstract
We introduce Mobius-v0, an architecture that comprises a globally shared Memory (FFN) that stores knowledge vectors and multiple Reasoners (Self-Attn) that iteratively achieve compositional reasoning. Using hidden states as cache and carrier, reasoners repeatedly query memory for required knowledge-vectors, while the knowledge is transmitted back to reasoning operators. Through this knowledge-reasoning-separation architecture, Mobius achieves better knowledge compression and reasoning efficiency. Built upon Mobius-v0 architecture: 1) Our 7B model trained-from-scratch achieves similar downstream score as a 7B Transformer baseline with 62.6% of baseline's training data. 2) Our Intern-S2-Mobius, continually-pretrained from Qwen3.5-35B, achieves similar downstream score while delivering nearly 4x end-to-end inference speedup.