Do Unified Multimodal Models Think in One Space? A Lens Through Cross-Branch Steering

The evolution of multimodal models has transitioned from early fusion techniques like VisualBERT and LXMERT to more unified architectures such as Janus-Pro and UniPic-1. While these models have achieved impressive task performance, their internal semantic representations remain underexplored. Prior work primarily focused on performance metrics, with limited analysis of whether understanding and generation components encode compatible semantic information. The challenge lies in the heterogeneity of representations: textual tokens versus visual latents, different training objectives, and hierarchical semantics. As AI applications demand more controllable and interpretable systems, understanding the internal semantic alignment becomes crucial. Recent advances in interpretability, such as probing and intervention techniques, have provided insights into language models but less so in multimodal settings. This paper builds on these foundations, aiming to diagnose the semantic coherence across model branches.

Despite architectural unification, it remains unclear whether understanding and generation branches in multimodal models share a common semantic space. This gap hampers controllability and interpretability, as misaligned representations can lead to unpredictable outputs. The core difficulty is that these branches operate over different token types, training objectives, and levels of semantic abstraction. Direct comparison of internal representations is challenging due to heterogeneity. The question is whether semantic directions learned in one branch can causally influence the other, serving as a diagnostic for shared semantics. Addressing this requires developing intervention-based tools capable of probing cross-modal semantic alignment without relying solely on performance metrics.

The key innovation is the introduction of cross-branch semantic steering, a diagnostic framework that extracts semantic direction vectors from one branch and injects them into the other during inference. This approach leverages contrastive concept pairs to learn interpretable semantic directions using algorithms like CAA, RepE, and ITI. The authors also construct UMMSteer, a comprehensive dataset covering diverse semantic categories, enabling systematic evaluation. Unlike prior work limited to single-modality interpretability, this method probes the internal semantic structure of multimodal models, revealing asymmetries in transferability. The framework is architecture-agnostic, validated across models like Janus-Pro, UniPic-1, and Show-o2, providing a new lens for understanding and improving semantic alignment in multimodal AI.

• �� Construct UMMSteer dataset with 51 concept pairs spanning color, count, spatial, style, appearance, and persona categories.
• �� Generate contrastive question-answer pairs for understanding and text prompts for generation, differing only in target semantic attributes.
• �� Use the understanding branch to process these pairs, extracting residual stream activations at relevant tokens.
• �� Apply algorithms (CAA, RepE, ITI) to derive semantic direction vectors from activation differences.
• �� During inference, encode input prompts into text prefix KV caches.
• �� Inject learned semantic vectors into VAE token representations at each transformer layer by adding bias terms, controlling the output attributes.
• �� Evaluate transfer effectiveness via success rates in controlled image generation and QA tasks, using automated and human assessments.
• �� Quantify semantic alignment with PSP metrics, analyzing the representational match across branches.

The experiments utilize the UMMSteer dataset to evaluate cross-branch transfer. For understanding to generation, semantic vectors are extracted and injected into models like Stable Diffusion-based architectures, with success measured by attribute control accuracy (>85%). For reverse transfer, vectors from the generation branch are tested on understanding tasks, showing negligible influence. Multiple models, including Janus-Pro and Show-o2, are tested to verify robustness. The evaluation employs automatic metrics and human judgments, with additional analysis via PSP to quantify semantic subspace alignment. Ablation studies vary the strength of injection (λ) and layer-wise effects, confirming the asymmetry in transferability and the influence of architecture on semantic coherence.

The primary result is that understanding-derived vectors reliably control image attributes with success rates exceeding 85%, demonstrating shared semantic subspace. Conversely, generation-derived vectors fail to influence understanding tasks, with success rates below 20%. The PSP metric confirms high semantic alignment for understanding-to-generation transfer, but poor alignment in the reverse. These findings are consistent across multiple architectures, indicating a fundamental representational mismatch. Qualitative examples show that understanding vectors can modify object attributes like color and position effectively, while generation vectors primarily affect superficial appearance features. The results suggest that internal representations differ in hierarchy and content, impacting transferability.

The framework enables diagnostic analysis and controllable content generation in multimodal systems, useful for virtual assistants, content creation, and AI interpretability. It can guide architecture design to improve semantic coherence, enhance model transparency, and facilitate targeted interventions. Long-term, this approach could lead to models with inherently aligned semantic spaces, improving robustness and user trust in AI systems across diverse applications.

The method depends on predefined semantic categories, limiting scope for abstract or complex concepts. Transfer effectiveness varies with model architecture, with pure autoregressive models showing weaker results. Static analysis may overlook dynamic or contextual factors influencing semantics. Extending to more complex reasoning, multi-turn interactions, and richer semantic hierarchies remains challenging. Computational costs for multi-layer interventions could hinder scalability, necessitating more efficient algorithms and broader datasets for comprehensive evaluation.