SAMTok: Representing Any Mask with Two Words

The evolution of multimodal AI has seen significant breakthroughs with models like CLIP, ALIGN, and Florence, which align visual and textual representations at a global level. These models excel in tasks such as image retrieval and captioning but fall short in pixel-level understanding, which is critical for detailed scene analysis. Traditional solutions involve complex region-based encoders like Mask R-CNN or Detectron2, which segment images into regions and then process each region separately. While effective, these approaches are computationally intensive and difficult to scale for large models. Recent advances like SAM (Segment Anything Model) have provided a universal framework for mask generation, but integrating pixel-level understanding into large models remains challenging due to the complexity of encoding and decoding detailed masks efficiently. The gap persists in achieving scalable, high-fidelity pixel-wise perception within the flexible, end-to-end training paradigm of large language models.

The core challenge is how to encode arbitrary region masks into a form that large models can understand and generate, without relying on heavy, specialized encoders or decoders. Existing methods are limited by their complexity, which hampers scalability and generalization across diverse tasks. Moreover, the fidelity of mask reconstruction and the ability to learn pixel-level semantics through standard training procedures remain unresolved. This bottleneck restricts the deployment of multimodal models in real-world applications requiring detailed scene understanding, such as autonomous driving, medical diagnostics, and interactive robots. The problem is to develop a simple, scalable, and effective method to represent masks as tokens that can be seamlessly integrated into existing language modeling frameworks.

The key innovation is the discretization of any region mask into just two tokens using a residual vector quantizer, transforming masks into a form akin to language tokens. This approach leverages SAM2's mask encoder to extract features, which are then quantized into two discrete symbols, capturing rich information with minimal complexity. These tokens are integrated into the language modeling pipeline, enabling the model to learn pixel-level tasks through standard autoregressive prediction and reinforcement learning, without architectural modifications. This method simplifies the entire pipeline, reduces computational costs, and enhances scalability. It also introduces a new paradigm where visual regions are represented as symbolic tokens, bridging the gap between pixel-level perception and language understanding.

• �� Input: Arbitrary region masks; • Feature extraction: SAM2 mask encoder processes masks to produce feature vectors; • Quantization: Residual Vector Quantizer compresses features into two discrete tokens; • Token integration: Tokens are treated as new words in the language model vocabulary; • Training: Use standard autoregressive prediction (e.g., Transformer-based models) to predict tokens sequentially; • Reinforcement learning: Introduce a textual answer-matching reward to optimize mask generation quality; • Data: 209 million diverse masks used for training, covering various scenes and objects. The entire process avoids complex architecture changes, relying on large-scale data and simple token prediction to learn pixel-wise understanding.

The training dataset comprises 209 million masks from diverse sources, enabling robust learning across multiple tasks. The model, based on QwenVL architecture, undergoes fine-tuning with SAMTok-formatted data. Evaluation includes region captioning (BLEU, CIDEr), VQA accuracy, grounded segmentation mIoU, and scene graph parsing metrics. Ablation studies compare discrete token encoding versus continuous representations, demonstrating superior robustness and efficiency. Reinforcement learning further enhances performance, validated on benchmarks like GRES and GCG. The experiments confirm that the discrete token approach maintains high fidelity and generalizes well across tasks, with training efficiency gains over traditional methods.

QwenVL-SAMTok achieves BLEU scores of 45.2 and CIDEr of 125.4 in region captioning, surpassing previous models by 3-5 points. VQA accuracy improves by 3.5%, and mIoU in referring segmentation reaches 78.2%. Reinforcement learning yields an additional 4.5% performance boost on GRES and 3.8% on GCG benchmarks. Ablation results show that token discretization reduces error rates in complex scenes by approximately 20%, with faster convergence during training. The model demonstrates strong cross-task generalization, validating the effectiveness of the discrete mask tokens.

This approach enables high-precision scene understanding in autonomous vehicles, facilitating accurate detection and segmentation of pedestrians, vehicles, and traffic signs. In medical imaging, it allows precise delineation of anatomical structures, aiding diagnosis. In robotics, it supports environment perception for navigation and manipulation. The method's simplicity and scalability make it suitable for deployment in resource-constrained settings, provided large-scale mask datasets are available. It also opens opportunities for interactive AI systems that require detailed visual reasoning.

Current limitations include occasional inaccuracies in highly textured or intricate masks, especially in boundary regions. The reliance on large annotated datasets poses challenges for domain-specific adaptation. The discretization process may lead to information loss in extremely complex scenes, affecting fine-grained tasks. Computational costs during training remain high due to the large dataset requirements. Future work should focus on improving quantization fidelity, reducing data dependency, and extending the approach to dynamic scenes with temporal coherence.