BrainJanus: A Unified Model for Understanding and Generation across Brain, Vision, and Language
BrainJanus uses a unified autoregressive model with a neural tokenizer to enable bidirectional brain, vision, and language understanding and generation.
Key Findings
Methodology
BrainJanus employs a novel Unified Brain Tokenizer based on VQ-VAE to discretize continuous neural signals into tokens aligned with visual and linguistic tokens in a shared Omni space. The core architecture is a single Transformer backbone trained with multi-task objectives, supporting bidirectional encoding and decoding across brain, vision, and language modalities. The model processes multimodal inputs by projecting them into the shared token space, then autoregressively predicts tokens to perform tasks like brain-to-image, brain-to-text, and their inverse. During training, the model leverages large-scale datasets (NSD) and synthetic detailed captions (Qwen) to enhance semantic alignment. The training involves joint optimization of multiple tasks, enabling seamless cross-modal translation and generation.
Key Results
- On brain-to-image reconstruction, BrainJanus achieves a PixCorr of 0.173 and CLIP semantic similarity of 94.4%, surpassing prior diffusion-based models like GIT-large (PixCorr 0.091). It maintains strong zero-shot generalization, with BLEU-4 scores of 22.45 on caption generation.
- In brain-to-text tasks, BLEU-4 reaches 22.45, showing improved semantic fidelity over task-specific models. The model also preserves interpretable cortical topography, validating neural representation quality.
- Across multiple benchmarks, BrainJanus demonstrates robust multi-modal translation, enabling flexible, arbitrary cross-modal generation without task-specific tuning, indicating high versatility and potential for real-world applications.
Significance
This work fundamentally advances the integration of neural signals with multimodal content, breaking the traditional siloed approach. By unifying brain, vision, and language within a single autoregressive framework, it opens new avenues for understanding neural representations, enhancing brain-computer interfaces, and developing intelligent systems capable of seamless multi-modal interaction. The ability to perform zero-shot cross-modal tasks addresses longstanding bottlenecks in neural decoding and generation, offering a scalable, interpretable, and biologically plausible model.
Technical Contribution
The key innovations include the neural Tokenizer based on VQ-VAE for discretizing neural signals, the shared Omni space for multimodal token alignment, and a single Transformer architecture supporting bidirectional, multi-task training. These contributions enable a unified, flexible framework that surpasses previous task-specific or uni-directional models, providing theoretical guarantees of cross-modal consistency and practical advantages in efficiency and interpretability.
Novelty
This is the first model to discretize neural signals into tokens and unify brain, vision, and language modalities within a single autoregressive Transformer. Unlike prior models limited to external sensory data, BrainJanus directly models internal neural activity, enabling full bidirectional translation across all three modalities, representing a significant leap forward in neuro-AI integration.
Limitations
- The model's performance diminishes in low signal-to-noise scenarios and requires extensive computational resources for training. Its generalization to unseen cognitive tasks or brain regions remains limited, necessitating further validation.
- Current neural Tokenizer relies on large annotated datasets and synthetic captions, which may introduce biases or limit applicability to diverse populations. The biological interpretability of internal representations still needs deeper investigation.
- Real-time deployment faces challenges due to high computational costs and data acquisition constraints, restricting immediate clinical or consumer applications.
Future Work
Future efforts will focus on improving robustness in noisy environments, reducing training costs, and enhancing biological interpretability. Integrating multi-region neural data and exploring unsupervised or semi-supervised learning could further extend the model’s capabilities. Additionally, translating these advances into real-time brain-computer interfaces and personalized neurotechnologies remains a key goal.
AI Executive Summary
BrainJanus marks a significant milestone in the quest to unify neural, visual, and linguistic understanding within a single AI framework. Traditional approaches in neuroscience and AI have largely treated brain encoding and decoding as separate, unidirectional tasks, often relying on external priors like CLIP or diffusion models. These methods, while effective in specific contexts, fail to capture the intrinsic multimodal nature of neural representations. Inspired by the biological reality that the brain processes information across multiple modalities simultaneously, the authors propose BrainJanus—a unified autoregressive model that leverages a novel neural Tokenizer to discretize brain signals into tokens aligned with visual and linguistic tokens in a shared Omni space.
This architecture is built upon a single Transformer backbone trained with multi-task objectives, enabling the model to perform any-to-any translation, including brain-to-image, brain-to-text, and their inverse. The model’s training incorporates large-scale datasets, synthetic detailed captions, and multi-task learning strategies, resulting in a system that not only achieves state-of-the-art performance on benchmarks like PixCorr and BLEU but also demonstrates remarkable zero-shot generalization capabilities.
Experimental results show that BrainJanus surpasses existing models in both reconstruction fidelity and semantic alignment, with PixCorr of 0.173 and CLIP similarity of 94.4%. Importantly, the model preserves biologically interpretable cortical topography, validating its neural plausibility. Its ability to seamlessly translate across modalities offers promising applications in brain-computer interfaces, neuroimaging, and multimodal AI, paving the way for more natural and efficient human-machine interactions.
Despite these advances, challenges remain, including computational costs, data requirements, and biological interpretability. Future research will aim to address these limitations, further integrating neural understanding with AI capabilities. Overall, BrainJanus provides a comprehensive, scalable framework that bridges the gap between neural signals and digital content, heralding a new era of neuro-AI synergy.
Deep Analysis
Background
The intersection of neuroscience and deep learning has evolved rapidly, with early models focusing on single-modality neural encoding and decoding, such as Huth et al. (2016). Recent advancements include transformer-based neural representation models like BrainLM and MindEye2, which explore semantic and visual alignment. However, these models often treat brain signals as isolated features, lacking a unified framework for multimodal integration. The emergence of large-scale neuroimaging datasets (e.g., NSD) and multimodal language models (e.g., Qwen) has created opportunities to develop comprehensive models capable of understanding and generating across multiple modalities. Despite progress, existing approaches are limited by task-specific pipelines and external priors, hindering full bidirectional understanding of brain signals in conjunction with vision and language.
Core Problem
Current brain encoding and decoding methods predominantly operate in unidirectional or isolated manners, relying heavily on external priors like CLIP or diffusion models. They lack a unified framework that can perform arbitrary cross-modal translation, especially involving internal neural signals. This fragmentation limits the understanding of neural representations and constrains applications such as brain-computer interfaces. The core challenge lies in discretizing continuous neural signals into a common representational space that can seamlessly interact with visual and linguistic modalities, enabling true multimodal understanding and generation. Addressing this gap requires innovative tokenization, shared representation, and a flexible autoregressive architecture capable of bidirectional tasks.
Innovation
The paper introduces several key innovations: 1) A neural Tokenizer based on VQ-VAE that discretizes neural signals into tokens aligned with visual and linguistic tokens, creating a unified neural-digital representation; 2) A shared Omni space where all modality tokens are embedded, facilitating cross-modal interactions; 3) A single Transformer backbone trained with multi-task objectives supporting bidirectional encoding and decoding across brain, vision, and language modalities; 4) Synthetic caption generation (Qwen) to enhance semantic richness and training data quality. These innovations collectively enable a flexible, scalable, and biologically plausible framework that surpasses prior task-specific models.
Methodology
- �� Data collection: Utilize NSD dataset with fMRI, natural images, and text annotations.
- �� Neural Tokenizer: Implement VQ-VAE to convert continuous neural signals into discrete tokens, trained with reconstruction and codebook loss.
- �� Multimodal Tokenization: Use pre-trained visual and text Tokenizers to convert images and texts into token sequences.
- �� Shared Omni space: Map all tokens into a common embedding space.
- �� Model architecture: Develop a Transformer-based autoregressive model that predicts the next token given previous tokens, supporting any-to-any cross-modal translation.
- �� Training: Conduct multi-task training on brain-image, brain-text, image-brain, and text-brain tasks, leveraging large datasets and synthetic captions.
- �� Fine-tuning: Use LoRA for parameter-efficient adaptation, optimizing for specific tasks while freezing other components.
Experiments
Experiments are conducted on the NSD dataset, involving 4 subjects with 9,000 training and 1,000 test images. The model is evaluated on brain-to-image reconstruction, brain-to-text captioning, and cross-modal translation tasks. Metrics include PixCorr, SSIM, CLIP similarity, BLEU, and others. Comparisons with SOTA diffusion models and specialized neural models demonstrate significant performance gains. Ablation studies analyze the impact of neural Tokenizer, shared space, and multi-task training. Zero-shot tests assess generalization to unseen data and tasks, confirming the robustness and versatility of BrainJanus.
Results
BrainJanus achieves a PixCorr of 0.173 in brain-to-image tasks, outperforming diffusion-based models. It maintains a CLIP semantic similarity of 94.4%, with BLEU-4 scores of 22.45 in captioning. The model generalizes well to unseen data, showing strong zero-shot capabilities. It preserves cortical topography, validating biological plausibility. The results demonstrate that unified modeling enhances both understanding and generation, surpassing task-specific baselines across multiple benchmarks.
Applications
Potential applications include brain-computer interfaces for communication and control, neuroimaging analysis, and multimodal AI systems. The framework can be used for real-time neural decoding, assistive technologies, and cognitive research. Its ability to translate neural signals into visual and textual content opens new avenues for understanding brain function and developing intelligent neurotechnologies. Long-term, it could enable personalized neurofeedback, neural prosthetics, and advanced human-AI collaboration.
Limitations & Outlook
Current limitations involve high computational costs, reliance on large annotated datasets, and challenges in modeling complex neural dynamics. The biological interpretability of internal representations remains partial, and generalization to diverse populations or cognitive states needs further validation. Future work should focus on improving efficiency, robustness, and biological grounding, as well as expanding to other neural modalities and real-time applications.
Plain Language Accessible to non-experts
想象你在一家超级复杂的工厂里，里面有很多不同的机器：一些负责处理原材料（脑信号），一些负责制造成品（图片和文字）。过去，这些机器各自工作，几乎不交流。现在，有个聪明的调度员（BrainJanus），他能把所有机器的工作内容翻译成一种通用的语言（Token），让它们可以互相理解和合作。这个调度员不仅能理解工厂的运作，还能根据需要，把脑里的想法变成图片或文字，反过来也能把图片或文字变成脑里的想法。就像一个超级翻译官，把复杂的工厂流程变成一套统一的语言，让所有机器都能协作，生产出更丰富、更智能的产品。这让工厂变得更高效，也让我们更容易理解大脑的秘密。
ELI14 Explained like you're 14
想象你在学校里，有很多不同的老师：有讲故事的、画画的、讲数学的。以前，他们各自教自己的内容，几乎不交流。现在，有个超级老师（BrainJanus），他能听懂所有老师的话，把故事、画画和数学都用一种特别的语言表达出来。这个超级老师还能根据你说的话，画出图片，写出故事，甚至猜出你脑袋里在想什么。它就像一个神奇的翻译机，把不同老师的内容变成一样的语言，让他们可以合作，教你更多有趣的东西。这样，你学到的东西就更丰富，也更容易理解。
Abstract
Modeling the bidirectional correspondence between external sensory stimuli and internal neural activity has emerged as a critical frontier in neuroscience. However, existing approaches predominantly treat brain encoding and decoding as isolated tasks, relying heavily on unimodal alignment and external priors while overlooking the brain's intrinsic nature as a multimodal integration system. To address these limitations, we propose BrainJanus, the first unified brain model that integrates brain, vision, and language within a single framework. Specifically, we introduce a Unified Brain Tokenizer to quantize continuous neural dynamics into discrete tokens aligned with visual and linguistic representations in a shared Omni space. Building on this, we utilize an All-in-One autoregressive architecture that leverages next-token prediction to enable seamless any-to-any generation, which encompasses image-to-brain and text-to-brain encoding, and brain-to-image and brain-to-text decoding. Extensive experiments demonstrate that BrainJanus achieves superior performance across diverse benchmarks. Furthermore, our framework exhibits zero-shot generalization and preserves interpretable biological topography, highlighting its potential as a general-purpose brain modeling paradigm. The code is available at \href{https://github.com/HaitaoWuTJU/BrainJanus}{GitHub}.