COMPASS: Grounding Composition-Intent Guidance in Unified Multimodal Models
COMPASS uses shared token τc to connect composition recognition and generation, trained on 389,031 images and about 1.2M VQA pairs.
Key Findings
Methodology
COMPASS adds Composition-specific Mixture-of-Experts (C-MoE) and a learnable expert token τc to a unified multimodal model. The perception branch freezes the original MoE backbone and trains a parallel composition branch; τc is optimized with weighted multi-label binary cross-entropy over 11 composition classes. For generation, the reference is grayscale-pixelized, a customized causal attention mask suppresses content leakage, and τc is injected as a global condition into the diffusion denoiser to transfer layout while preserving text semantics.
Key Results
- The paper reports substantial gains in category-level composition understanding and more composition-consistent, prompt-faithful generation than strong baselines. However, the supplied text contains no full accuracy, F1, CLIP, or human-evaluation table, so an exact percentage improvement cannot be verified.
- Comp-11 combines AVA, TAD66K, CADB, and KU-PCP into 389,031 images, covering 11 classes including Rule of Thirds, Center, Horizontal, Symmetric, Diagonal, Curved, Vertical, Triangle, Pattern, Leading Lines, and Fill the Frame.
- The design evidence favors grayscale pixelization over Canny edges and Gaussian blur for reducing semantic leakage. The attention mask lets τc read reference-image tokens while blocking ordinary text and query tokens from directly accessing them.
Significance
COMPASS reframes composition from a descriptive attribute in aesthetic critique into an explicit, reusable, and controllable structural intent. It addresses a persistent weakness of unified multimodal models: they can discuss an image but often fail at fine-grained layout recognition and cannot reliably execute that layout during synthesis. Comp-11 supplies a unified taxonomy and instruction-following supervision. Industrially, the ability to transfer arrangement without copying reference objects is valuable for photography assistance, advertising, visual ideation, and content production.
Technical Contribution
The central engineering contribution is an expert-anchor design. C-MoE adds a parallel composition branch beside the original MoE-FFN, with FFN(h)=(1-s)MoEbase(h)+sMoEcomp(h), so only new routers and expert MLPs are trained. τc is inserted after the answer prefix; its hidden state hc produces logits o=Wϕ(hc) for 11-class multi-label prediction. During generation, structural bottlenecking, masked information flow, cross-attention refinement, and diffusion conditioning form a perception-to-control pathway.
Novelty
The authors present COMPASS as the first unified multimodal framework that couples composition perception and composition-guided generation through one shared intent token. Unlike LoRA adaptation or low-level edge, depth, and bounding-box controls, it models composition as a high-level rule-based intent and uses self-supervised structural bottlenecking rather than requiring paired images with identical layouts and different semantics.
Limitations
- The supplied manuscript excerpt does not report complete experimental tables, metric definitions, or statistical tests. Therefore, the magnitude and significance of the claimed improvements cannot be independently checked from the provided material.
- Grayscale pixelization reduces recognizable appearance but can also remove thin lines, perspective cues, and small-object geometry. The 11-class taxonomy may not represent hybrid, atypical, or continuously varying photographic layouts.
- Freezing the backbone limits adaptation, while adding MoE branches, queries, and a diffusion denoiser increases computation. Robustness across domains, cultures, video, and heavily occluded scenes remains under-tested.
Future Work
Future work should release complete evaluation protocols and report macro-F1, composition consistency, text faithfulness, human judgments, and confidence intervals. More expressive hierarchical or continuous layout representations could handle rule coexistence and ambiguous samples. Promising directions include stronger semantic isolation, video and 3D transfer, editable τc controls, parameter-efficient adaptation, and fairness studies across photographic cultures and domains.
AI Executive Summary
Visual composition governs subject placement, spatial balance, and viewing paths—the grammar of photography. Yet unified multimodal models usually bury composition inside broad aesthetic judgments. They may describe an image as balanced without reliably identifying horizontal, diagonal, or Rule-of-Thirds structure, and text alone is a low-bandwidth channel for precise geometry. Reference images offer a richer spatial blueprint, but existing methods often copy their semantic content.
COMPASS addresses this gap with a shared expert token, τc, that anchors composition intent across understanding and generation. Its Composition-specific Mixture-of-Experts (C-MoE) adds specialized experts while freezing the original backbone. A weighted multi-label classifier supervises τc over 11 composition categories. For synthesis, the reference is converted into a grayscale pixelated image, reducing recognizable appearance. A customized causal mask prevents ordinary text and query tokens from reading reference-image tokens; τc alone summarizes layout cues and conditions the diffusion denoiser.
The authors construct Comp-11 from AVA, TAD66K, CADB, and KU-PCP: 389,031 images become roughly 1.2 million reasoning-augmented VQA pairs. They report better composition understanding, layout consistency, and prompt faithfulness than strong baselines, while the provided text does not include the numerical result tables needed to quantify the gains. The conceptual advance is a shared bridge from “recognizing composition” to “executing composition.” Future evaluation must test continuous layouts, cross-domain robustness, and complete reproducibility.
Deep Analysis
Background
Composition research progressed from regression to rule classification and active cropping. CADB supports composition-aware assessment; KU-PCP classifies rules and localizes geometric elements; SACD, UGCrop5K, and FCDB support crop optimization, while GenCrop uses diffusion augmentation. Unified models such as NExT-GPT, SEED-X, Chameleon, Transfusion, Show-o, Emu3, Janus, and BAGEL integrate modalities, but mainly optimize alignment, scale, or image quality. Composition has rarely been treated as an explicit controllable structure.
Core Problem
The task has two linked goals: predict an 11-dimensional multi-hot composition vector from an image and instruction, then generate a new image satisfying Comp(Igen)≈Comp(Iref) and Sem(Igen)≈t. The bottlenecks are scarce systematic labels, long-tailed categories, catastrophic interference from full fine-tuning, and entanglement between reference semantics and layout. Paired images with identical composition but different content are rarely available.
Innovation
- �� Comp-11 unifies 11 composition classes and adds rationale, multiple-choice, and judgment VQA.
- �� C-MoE adds a parallel expert branch while preserving the pretrained backbone.
- �� τc compresses composition cues into a supervised, reusable intent representation.
- �� Grayscale pixelization destroys appearance detail while retaining coarse spatial massing.
- �� Masked attention and diffusion conditioning turn passive recognition into layout control, unlike edge, depth, or box-based low-level guidance.
Methodology
- �� Perception input: image I, instruction, and response sequence enter the unified LMM.
- �� Routing: the backbone uses Top-K MoE routing; task flag s selects MoEcomp or MoEbase.
- �� Token supervision: τc is inserted before the answer; hc yields o=Wϕ(hc), trained with weighted multi-label BCE.
- �� Structural proxy: Iref becomes grayscale and pixelated, weakening semantic shortcuts.
- �� Information control: causal masking blocks text and queries from reference tokens; τc alone reads them for layout summarization.
- �� Generation: text cross-attention and diffusion loss produce Igen that follows the prompt while inheriting composition.
Experiments
Comp-11 aggregates AVA, TAD66K, CADB, and KU-PCP, totaling 389,031 images, and expands expert-verified labels into approximately 1.2 million VQA pairs using a strong LMM. Evaluation covers multi-label recognition, open-ended rationale, multiple-choice discrimination, judgment, and reference-guided generation. Baselines include unified multimodal and aesthetic models. Ablations compare pixelization, Canny edges, blur, attention masking, and related generation components.
Results
COMPASS is reported to outperform strong baselines in composition understanding and reference-guided synthesis, producing images that better preserve reference layout while following new text. Grayscale pixelization is favored over Canny and Gaussian blur, suggesting that disrupting contour identity is more effective than merely smoothing texture. Because the supplied excerpt omits accuracy, F1, similarity, and human-rating tables, the numerical size of these gains cannot be independently established.
Applications
The system can support composition-aware photography tutors, advertising layout transfer, storyboard generation, and concept-art tools. A designer can provide a reference arrangement while replacing its people, products, or environment with prompt-specified content. Deployment requires a reliable vision encoder and diffusion generator, curated domain labels, copyright-compliant references, and evaluation protocols separating layout fidelity from semantic copying.
Limitations & Outlook
The method assumes grayscale pixelization sufficiently separates structure from semantics, but complex contours, occlusion, and small objects may still leak identity or lose important geometry. Eleven discrete labels cannot fully encode continuous spatial relations or conflicting rules, and annotations may reflect cultural and annotator bias. Backbone freezing reduces interference but limits deep adaptation; additional MoE and diffusion modules increase cost. Future work should test video, 3D, cross-domain transfer, continuous representations, and publish full reproducible metrics.
Plain Language Accessible to non-experts
Imagine a design studio making posters. An ordinary employee looks at a photograph and says, “It feels nice.” COMPASS instead asks a composition specialist to inspect the arrangement and write a small card, τc: “the main subject is centered,” “the road leads from the corner,” or “the subject sits near a third-line intersection.” The card describes how the picture is organized, not what exact objects appear in it.
When a client asks for a train scene, the studio does not copy the reference photograph’s trees, mountains, or train. First, the reference is turned into a gray, blocky sketch—like keeping only the large furniture in a room. Only the specialist may read that sketch and summarize the arrangement; the other workers cannot secretly copy its details. The artist therefore learns where the train should sit and where the tracks should point, while the client’s words decide what the scene contains.
Comp-11 is the studio’s training library: 389,031 images, 11 common arrangement styles, and about 1.2 million question-and-answer examples. The reported result is a system that can both recognize arrangements and reuse them for new scenes. It is not perfect: a blocky sketch may erase fine clues, and 11 styles cannot describe every photograph. The supplied paper text also lacks complete score tables, so the exact size of the improvement remains unknown.
ELI14 Explained like you're 14
Think of COMPASS as a “same formation, new players” game. You show it a soccer formation and ask for a basketball scene. A normal system might copy the original players and stadium, or understand “basketball” but place everyone randomly. COMPASS tries to keep the formation while changing the whole story.
Its special note, τc, records ideas such as “the main object is in the middle,” “lines point toward the center,” or “both sides match.” Before using a reference picture, the system turns it into a gray pixel mosaic. That is like blurring a game screenshot until you can see where the walls are but not which character is standing there. Then only the composition specialist can inspect it, so the generator is less tempted to copy.
The authors also build Comp-11, a huge practice book with 389,031 images and about 1.2 million Q&A examples. It teaches 11 styles, including center, symmetry, diagonal lines, patterns, and Rule of Thirds. The model answers questions, explains reasons, and chooses between options. After training, it reportedly gets better at recognizing layouts and generating new pictures that keep them.
But there is no magic button! Pixel mosaics can hide tiny clues, and real photos often mix several styles. Also, the provided text does not show the full accuracy and F1 tables, so we cannot say exactly how much better it is. Future versions should handle video, unusual layouts, and more editable controls.
Glossary
C-MoE (Composition-specific Mixture-of-Experts)
A dedicated expert branch for composition reasoning placed alongside the original MoE. It is activated by a task flag so specialized training does not overwrite general multimodal competence.
The main perception-side adaptation mechanism in COMPASS.
τc (expert token)
A learnable special token that concentrates composition intent into one representation. Its hidden state is directly supervised for 11-class multi-label prediction.
It links perception to diffusion-based generation control.
Structural bottleneck
An input transformation that preserves coarse structure while suppressing appearance and identity cues. COMPASS uses a grayscale, pixelated reference.
It reduces semantic contamination during layout transfer.
Weighted multi-label BCE
A binary cross-entropy objective allowing one image to have several composition labels. Class-specific positive and negative weights compensate for long-tailed label frequencies.
It trains the classifier attached to τc.
Customized attention mask
A rule specifying which token positions may attend to which others. COMPASS blocks ordinary text and query tokens from reference-image tokens while allowing τc to read them.
It suppresses reference-content leakage during generation.
Comp-11
A composition-focused instruction dataset with 11 photographic composition categories. It contains 389,031 images and approximately 1.2 million reasoning-augmented VQA pairs.
It supports training and evaluation of actionable composition understanding.
Open Questions Unanswered questions from this research
- 1 The excerpt lacks complete result tables, so the relative gains over each baseline remain unverifiable. Future reports should include macro-F1, layout consistency, prompt faithfulness, human ratings, and confidence intervals.
- 2 Discrete 11-class labels cannot fully express continuous layouts or conflicts among multiple rules. Spatial relations, keypoints, or hierarchical representations are needed.
- 3 The robustness of pixelized references in video, 3D, severe occlusion, and culturally diverse photography is unknown and requires dedicated benchmarks.
Applications
Immediate Applications
Reference-guided advertising
A designer can provide a photograph with an ideal arrangement and request a new product, person, or environment. COMPASS aims to preserve subject placement, symmetry, or leading lines while reducing direct copying of the reference’s semantic content.
Composition tutoring
A photography platform can identify Rule of Thirds, center, diagonal, or leading-line structures and explain the decision. Students can then ask for the same subject to be regenerated under different composition rules and compare the results.
Long-term Vision
Editable visual-intent interfaces
A future system could expose τc-like controls for subject position, visual balance, gaze direction, and perspective flow. Such interfaces could transform image and video production, game-scene design, storyboarding, and eventually spatial planning for embodied agents.
Abstract
Composition is a high-level visual intent that governs where subjects are placed and how a scene is organized, yet current unified multimodal models remain unreliable at fine-grained composition recognition and struggle to turn such intent into controllable generation. We present COMPASS, the first unified multimodal framework that grounds composition-intent control in a single system spanning both composition perception and composition-guided generation, with a shared expert token $τ_c$ as the central intent anchor. On the perception side, COMPASS injects composition expertise into an MoE backbone in a minimally invasive manner and distills the inferred intent into $τ_c$. On the generation side, COMPASS reuses $τ_c$ as a global conditioning signal that steers the denoising trajectory, effectively converting passive composition analysis into explicit layout control. To support systematic instruction-following composition learning and evaluation at scale, we construct Comp-11, a large-scale dataset with an 11-class taxonomy and reasoning-augmented annotations. Extensive experiments show that COMPASS substantially improves category-level composition understanding and delivers more composition-consistent, prompt-faithful generation than strong baselines.