Dead Weights, Live Signals: Feedforward Graphs of Frozen Language Models
We propose a feedforward graph architecture using frozen large language models as nodes, achieving 87.3% accuracy on ARC-Challenge.
Key Findings
Methodology
The study proposes a feedforward graph architecture using frozen large language models as computational nodes, communicating through a shared continuous latent space via learned linear projections. Projection matrices are optimized via backpropagation through residual stream injection hooks. Three small frozen models encode input into a shared latent space, injecting the aggregate signal into two larger frozen models, with a lightweight cross-attention output node making predictions.
Key Results
- Achieved 87.3% accuracy on ARC-Challenge, outperforming the best single model by 11.4 percentage points.
- Achieved 82.8% accuracy on OpenBookQA, exceeding the best single model by 6.2 percentage points.
- Achieved 67.2% accuracy on MMLU, surpassing the best single model by 1.2 percentage points.
Significance
This study demonstrates how frozen large language models can be used to achieve efficient multi-node communication, significantly improving performance on multiple benchmarks. It provides a new approach to enhancing model performance without increasing parameter count, especially important in resource-constrained environments.
Technical Contribution
Technical contributions include a novel feedforward graph architecture leveraging geometric compatibility of frozen models, achieving multi-node communication via learned linear projections. This method enables end-to-end training by optimizing projection matrices and output nodes without updating model weights.
Novelty
This study is the first to compose frozen language models into trainable multi-node graphs, using linear projections for cross-architecture activation translation, overcoming limitations of traditional model ensemble methods.
Limitations
- Layer 1 projection matrices fail to specialize, leading to representational diversity relying on model heterogeneity.
- Single training runs may be influenced by chance factors.
- Cosine annealing combined with checkpoint resumption causes oscillating validation accuracy.
Future Work
Future research directions include replacing fixed average pooling with learned attention-weighted pooling, scaling to deeper graphs, and exploring interpretable geometric structures in the shared latent space.
AI Executive Summary
Current language model performance improvements often rely on larger models and more data, but this is not always the most effective strategy. This paper proposes a novel feedforward graph architecture using frozen large language models as computational nodes, communicating through a shared latent space. This approach reduces training parameters while improving performance across multiple benchmarks.
The architecture uses learned linear projections to project hidden states from multiple frozen models into a shared latent space, then injects the aggregate signal into downstream nodes. Experimental results show significant performance improvements on ARC-Challenge, OpenBookQA, and MMLU.
Despite its strong performance, the method has limitations, such as the lack of specialization in layer 1 projection matrices. Future research could address these issues by introducing learned attention mechanisms and exploring deeper graph structures.
Deep Analysis
Background
In recent years, the size and complexity of language models have increased, but this is not always the best way to improve performance. Many small models excel in specific tasks, and models with different architectures exhibit different strengths at equivalent parameter counts. Effectively aggregating this information is a key research direction.
Core Problem
Current model ensemble methods often operate at the output level, unable to access the geometric structures within models. Achieving efficient communication between models without increasing parameter count is a pressing issue.
Innovation
This paper proposes a novel feedforward graph architecture using frozen large language models as nodes, communicating through learned linear projections in a shared latent space. This approach reduces training parameters while improving performance across multiple benchmarks.
Methodology
- �� Three small frozen models encode input, generating shared latent space signals.
- �� Signals are injected into two larger frozen models, with a lightweight cross-attention output node making predictions.
- �� Projection matrices are optimized via backpropagation through residual stream injection hooks.
Experiments
Experiments use ARC-Challenge, OpenBookQA, and MMLU datasets, comparing performance with single models and parameter-matched learned classifiers. Key hyperparameters include learning rate, weight decay, and batch size.
Results
Achieved 87.3% accuracy on ARC-Challenge, outperforming the best single model by 11.4 percentage points. Achieved 82.8% accuracy on OpenBookQA, exceeding the best single model by 6.2 percentage points. Achieved 67.2% accuracy on MMLU, surpassing the best single model by 1.2 percentage points.
Applications
This method can be used in scenarios requiring efficient model combination, such as running complex tasks on resource-constrained devices. It reduces training parameters while improving performance.
Limitations & Outlook
Layer 1 projection matrices fail to specialize, leading to representational diversity relying on model heterogeneity. Single training runs may be influenced by chance factors. Cosine annealing combined with checkpoint resumption causes oscillating validation accuracy.
Plain Language Accessible to non-experts
Imagine you're in a kitchen with various cooking tools and ingredients. Each tool represents a frozen language model, each excelling at different tasks. You need to combine these tools' functions to create a delicious dish. Our research is like a smart chef who learns how to best use these tools to enhance overall cooking performance without buying new tools.
ELI14 Explained like you're 14
Hey there! Imagine you're playing a super complex game with lots of characters, each having different skills. Our research is like a super smart player who knows how to combine these characters' skills to beat the big boss! It's like using different school subjects to solve a big problem. Isn't that cool?
Glossary
Frozen Model
A model whose weights are not updated during training.
Used as computational nodes in the feedforward graph architecture.
Linear Projection
A linear transformation that maps one vector space to another.
Used to transfer information between different models.
Latent Space
The representation space of data within a model.
A shared latent space is used to integrate information from multiple models.
Cross-Attention
An attention mechanism that allows information exchange between different inputs.
Used for prediction in the output node.
Residual Stream
A mechanism for transmitting information through residual connections.
Used to optimize projection matrices.
Open Questions Unanswered questions from this research
- 1 How to enhance model representational diversity without increasing parameters?
- 2 Can other mechanisms further optimize multi-node communication?
Applications
Immediate Applications
Complex Tasks on Mobile Devices
Run complex language tasks efficiently on resource-limited mobile devices.
Long-term Vision
Evolution of Smart Assistants
Achieve smarter personal assistants through efficient model combination.
Abstract
We present a feedforward graph architecture in which heterogeneous frozen large language models serve as computational nodes, communicating through a shared continuous latent space via learned linear projections. Building on recent work demonstrating geometric compatibility between independently trained LLM latent spaces~\cite{armstrong2026thinking}, we extend this finding from static two-model steering to end-to-end trainable multi-node graphs, where projection matrices are optimized jointly via backpropagation through residual stream injection hooks. Three small frozen models (Llama-3.2-1B, Qwen2.5-1.5B, Gemma-2-2B) encode the input into a shared latent space whose aggregate signal is injected into two larger frozen models (Phi-3-mini, Mistral-7B), whose representations feed a lightweight cross-attention output node. With only 17.6M trainable parameters against approximately 12B frozen, the architecture achieves 87.3\% on ARC-Challenge, 82.8\% on OpenBookQA, and 67.2\% on MMLU, outperforming the best single constituent model by 11.4, 6.2, and 1.2 percentage points respectively, and outperforming parameter-matched learned classifiers on frozen single models by 9.1, 5.2, and 6.7 points. Gradient flow through multiple frozen model boundaries is empirically verified to be tractable, and the output node develops selective routing behavior across layer-2 nodes without explicit supervision.