TextCaps: a Dataset for Image Captioning with Reading Comprehension

The evolution of image captioning has seen significant milestones, from early template-based methods to deep neural models like Show and Tell, and attention-based models such as Show, Attend and Tell. Large datasets like MS COCO and Flickr30k have driven performance improvements, but primarily focus on visual objects. OCR technology has matured, enabling scene text recognition, yet integrating scene text into descriptive models remains underexplored. Recent datasets like TextVQA and ST-VQA have advanced question-answering with text, but lack the capacity for generating detailed long descriptions that incorporate scene text. This gap motivates the creation of TextCaps, aiming to combine OCR recognition with natural language generation for comprehensive scene understanding.

Current image captioning models struggle to incorporate scene text, limiting their understanding of environments where textual information is crucial. OCR recognition alone does not suffice, as models lack the ability to reason about the semantic and spatial relationships between text and objects. Moreover, existing datasets do not emphasize the need for long, descriptive sentences that include scene text, especially with the challenge of zero-shot OCR tokens—texts unseen during training. This bottleneck hampers applications requiring detailed scene comprehension, such as navigation aids for the visually impaired and complex scene analysis in autonomous systems.

Key innovations include: 1) The construction of TextCaps, a large-scale dataset with diverse, scene-text-rich images and captions requiring reading comprehension; 2) Development of a multimodal model (M4C) that integrates visual, spatial, and semantic features, supporting dynamic switching between vocabulary and OCR tokens; 3) Introduction of zero-shot OCR recognition capabilities, enabling the model to handle unseen text; 4) Extensive analysis of the importance of different feature types and their contribution to caption quality. These innovations collectively push the boundary of scene understanding by enabling models to read, reason, and describe complex environments more accurately.

• �� Data collection: Selected 28,408 images from Open Images v3, verified text presence via OCR and human annotation.
• �� Caption annotation: Human annotators wrote sentences requiring reading scene text, with quality control via voting.
• �� Feature extraction: Used Faster R-CNN for object features, FastText and PHOC for OCR semantic features, and spatial coordinates.
• �� Model architecture: Employed a multimodal transformer (M4C) with attention modules to fuse features.
• �� Relationship modeling: Captured spatial and semantic relations among OCR tokens and objects.
• �� Training: End-to-end optimization with loss functions targeting caption quality and OCR accuracy.
• �� Inference: Dynamic switching between vocabulary and OCR tokens during caption generation, leveraging attention and pointer mechanisms.

Models trained on COCO and TextCaps datasets, with hyperparameters tuned for CIDEr and BLEU scores. Baselines included BUTD and AoANet, while the proposed M4C model incorporated multimodal features. Evaluation involved automatic metrics and human judgment, with ablation studies removing feature types to assess their impact. Zero-shot OCR recognition was tested on unseen texts. Results showed significant improvements when OCR features were integrated, with CIDEr scores reaching 89.6, and human scores averaging 125.5, indicating the model's capacity to generate contextually rich descriptions. The experiments validated the importance of multimodal fusion and feature diversity.

The M4C model trained on TextCaps achieved a CIDEr score of 89.6, far exceeding traditional captioning models. Incorporating semantic, spatial, and visual OCR features led to substantial performance gains, with ablation studies confirming their importance. Zero-shot OCR recognition remained challenging, with recognition rates around 39.5%, but the model effectively used OCR tokens for accurate descriptions. Human evaluation scores highlighted the gap between machine and human performance, emphasizing the complexity of scene text understanding and reasoning. These results demonstrate the dataset's effectiveness in pushing the boundaries of image captioning.

This technology can enhance assistive tools for visually impaired users by providing detailed scene descriptions that include textual information. It also benefits autonomous vehicles, robotics, and surveillance systems requiring detailed scene comprehension. The ability to read and interpret scene text in real-time can improve navigation, safety, and contextual awareness in complex environments. Additionally, it can aid in automated content moderation and multimedia indexing, where understanding scene text is crucial.

Despite progress, zero-shot OCR recognition remains imperfect, especially for highly cluttered or stylized text. The model's complexity and computational demands limit real-time deployment. Handling densely packed or ambiguous scene text poses ongoing challenges. Future work must focus on improving zero-shot capabilities, reducing model size, and extending multilingual support to broaden applicability.