Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
Up-Down attention combines Faster R-CNN region features with task-conditioned attention, reaching CIDEr 117.9 and winning the 2017 VQA Challenge.
Key Findings
Methodology
The Bottom-Up and Top-Down Attention framework first uses Faster R-CNN with a ResNet-101 backbone to propose salient object regions, producing typically 2,048-dimensional features. A task-conditioned attention module then assigns weights to these regions. The captioning model uses two LSTMs: one attends using language context and the other generates words. The VQA model uses a GRU question encoder, gated-tanh fusion, and multilabel answer prediction.
Key Results
- On the MSCOCO Karpathy test split, the single-model Up-Down system optimized for CIDEr achieved BLEU-4 36.3, METEOR 27.7, CIDEr 120.1, and SPICE 21.4. Relative to the ResNet baseline, CIDEr increased from 111.1 to 120.1, with 3–8% relative gains across the reported metrics.
- On the online MSCOCO server, a four-model ensemble achieved BLEU-4 36.9, CIDEr 117.9, and SPICE 21.5, exceeding published and unpublished submissions at the time. On VQA v2.0 validation, Up-Down reached 63.2% overall accuracy versus 59.4% for the best ResNet 7×7 baseline.
- The SPICE breakdown shows semantic improvements beyond object recognition. Under CIDEr optimization, Up-Down scored 39.1 on objects, 10.0 on attributes, and 6.5 on relations, compared with 37.0, 9.2, and 6.1 for the ResNet model.
Significance
The paper changes the basic unit of visual attention from regular CNN grid cells to objects and salient regions. This addresses misalignment, object fragmentation, and the difficulty of binding attributes or relations to the same entity. Academically, it established a highly influential baseline for region-based visual-language modeling. Practically, the same precomputed region representation supports captioning, VQA, retrieval, accessibility tools, and visual search. Its importance also lies in showing that relatively simple attention and recurrent architectures can outperform more elaborate grid-based systems when the visual representation is better structured.
Technical Contribution
The central contribution is a clean separation between hard region selection and soft task attention. Faster R-CNN selects a compact set of candidate regions, while an LSTM or GRU assigns continuous, task-dependent weights. Captioning uses a two-layer LSTM and Self-Critical Sequence Training (SCST) for CIDEr optimization. VQA uses gated-tanh transformations to fuse the question and attended visual feature, followed by multilabel classification. The design is modular, transferable across tasks, and supported by controlled ResNet-versus-Up-Down comparisons.
Novelty
Compared with prior methods that attend over 10×10 or 14×14 CNN grids, this work systematically establishes Faster R-CNN proposals as a general bottom-up attention substrate for visual-language tasks. Earlier region-based captioning methods used Selective Search, Edge Boxes, or spatial transformers; this paper instead exploits detector pretraining on Visual Genome. Its fundamental innovation is not a more complicated attention stack, but the division of labor between content-driven region discovery and task-driven weighting.
Limitations
- The model depends on Faster R-CNN and Visual Genome supervision. Small, occluded, novel, or background objects may be missed, and hard confidence filtering means downstream attention cannot recover a region that the detector never proposes.
- VQA predicts within a fixed vocabulary of 3,129 frequent answers, limiting rare, compositional, and explanatory responses. The one-pass attention mechanism also does not explicitly model long multi-step reasoning or uncertainty.
Future Work
Promising directions include end-to-end training of region proposals and language objectives, open-vocabulary region representations, explicit relation graphs, iterative reasoning, and uncertainty-aware attention. Future systems should reduce the memory and inference cost of precomputed Faster R-CNN features and evaluate robustness under occlusion, long-tail classes, domain shift, and interactive real-world use.
AI Executive Summary
Image captioning and visual question answering require a system to identify what is present and determine which visual evidence matters for the current linguistic task. Earlier attention models generally operated on regular CNN grids. Such grids are computationally convenient but visually arbitrary: a single object may span several cells, while one cell may mix unrelated objects. The paper argues that objects and salient regions are a more natural basis for attention.
The proposed Up-Down architecture combines two mechanisms. Bottom-Up attention uses Faster R-CNN with ResNet-101 to detect candidate regions and learn 2,048-dimensional features from Visual Genome, including object and attribute supervision. Top-Down attention then weights those regions according to the partial caption or question. The captioner uses two LSTMs and Self-Critical Sequence Training for CIDEr optimization; the VQA model uses a GRU question encoder, gated-tanh fusion, and multilabel answer prediction.
The results were substantial. On the MSCOCO Karpathy split, the CIDEr-optimized single model achieved CIDEr 120.1, SPICE 21.4, and BLEU-4 36.3, versus CIDEr 111.1 for the ResNet baseline. On the online MSCOCO server, a four-model ensemble reached CIDEr 117.9, SPICE 21.5, and BLEU-4 36.9. On VQA v2.0 validation, it achieved 63.2% overall accuracy, compared with 59.4% for the strongest ResNet grid baseline. The approach became influential because it offered a simple, reusable visual interface, although it remained dependent on detector quality, precomputed features, and a fixed answer vocabulary.
Deep Analysis
Background
Captioning and VQA had increasingly adopted Top-Down attention, conditioning spatial CNN features on a partial caption or question. Methods such as Review Net, Adaptive Attention, and SCST improved generation, but commonly attended over regular grids. Grid resolution creates a coarse-versus-fine trade-off and does not respect object boundaries. This paper imports a human-inspired distinction: Bottom-Up processing proposes visually salient regions, while Top-Down control selects evidence relevant to the task.
Core Problem
The key problem is joint selection of candidate visual evidence and task-relevant evidence. A fixed 10×10 or 14×14 grid cannot adapt its regions to image content, encouraging object fragmentation, attribute misbinding, and weak relation modeling. A useful solution must also provide reusable visual features so that captioning can focus on the next word while VQA can focus on the question.
Innovation
- ��Faster R-CNN replaces hand-designed proposals and fixed grids with learned object regions.
- ��A ResNet-101 detector is pretrained on Visual Genome with 1,600 object classes and 400 attribute classes.
- ��Additive Top-Down attention softly weights the proposed regions using language or question context.
- ��The same region representation supports both captioning and VQA, demonstrating cross-task generality.
Methodology
- ��Region extraction: the RPN predicts objectness and box refinements; RoI pooling produces region features. Per-class NMS and confidence filtering yield V={v1,…,vk}, with vi∈R2048.
- ��Captioning: the attention LSTM receives the previous language state, mean image feature, and previous word. It computes ai,t=w_a^T tanh(W_va vi+W_ha h_t^1), αt=softmax(a_t), and v̂t=Σiαi,tvi. A second LSTM predicts words; training uses cross-entropy and SCST.
- ��VQA: a GRU encodes the question as q; ai=w_a^T f_a([vi,q]) produces region scores. Gated-tanh functions fuse q and v̂, followed by multilabel prediction over 3,129 candidate answers.
Experiments
Captioning uses MSCOCO 2014, including the 113,287-image Karpathy training split, 5K validation images, and 5K test images; server training uses 123K images. VQA uses VQA v2.0, with 1.1M questions and 11.1M answers, augmented by approximately 485K Visual Genome questions. ResNet grid baselines use 1×1, 7×7, or 14×14 features. Metrics include BLEU, METEOR, ROUGE-L, CIDEr, SPICE, and standard VQA accuracy.
Results
On MSCOCO, Up-Down improved cross-entropy CIDEr from 105.4 to 113.5 and BLEU-4 from 33.4 to 36.2; with CIDEr optimization, scores were 120.1 versus 111.1 and 36.3 versus 34.0. On VQA validation, Up-Down reached 63.2%, versus 56.3% for ResNet 1×1, 59.4% for 7×7, and 57.9% for 14×14. Gains occurred across Yes/No, Number, and Other questions.
Applications
The framework directly supports automatic captioning, accessible image reading, visual retrieval, and question answering over product or media images. A practical deployment can precompute Faster R-CNN features and reuse them across language tasks. It requires adequate detector coverage for the target domain and, in VQA, an answer vocabulary that captures expected responses.
Limitations & Outlook
Hard region selection is a bottleneck: missed objects cannot be recovered by Top-Down attention, while precomputed Faster R-CNN features add storage and latency. Visual Genome annotations contain noise, overlapping labels, and long-tail imbalance. Fixed-vocabulary VQA limits open-ended responses, and one-pass attention is insufficient for complex multi-step reasoning. Future work should pursue end-to-end region learning, open-vocabulary answers, relational reasoning, and efficient detectors.
Plain Language Accessible to non-experts
Imagine a careful tour guide explaining a photograph. A traditional system cuts the picture into equal squares and asks the guide to inspect those squares. That is convenient, but a person may be split across several squares, while one square may contain a person, a ball, and part of the sky. The Up-Down method first gives the guide an assistant whose job is to circle likely meaningful things: people, vehicles, balls, trees, and so on. This is the bottom-up stage.
A second guide listens to the assignment. If the task is to describe “a child playing soccer,” it gives more attention to the child and ball. If the question asks about the ball’s color, it shifts attention toward the ball and nearby evidence. The system combines the important circles and turns them into words or an answer. In this way, one part decides what might matter, while another decides what matters now.
The experiments show why the division helps. Compared with looking at regular squares, the model produced more accurate captions and answered questions more reliably. It still has a weakness: if the first assistant fails to circle an object, the second guide has little chance to recover it. The method is therefore powerful, but only as good as its visual inventory.
ELI14 Explained like you're 14
Think of a picture-question game. If you chop every image into identical squares, you might cut a dog in half or mix a skateboard with the pavement. The paper’s system works more like a detective team. First, Faster R-CNN circles likely important things—people, balls, cars, trees, and their properties. That is the “bottom-up” part: look at the image and find possible clues!
Next comes the “top-down” part. The question or sentence tells the system what clue matters right now. To write “a boy is playing soccer,” it checks the boy and ball. To answer “What color is the ball?”, it focuses much more strongly on the ball. So the system does not stare equally at every part of the picture. It chooses evidence based on the mission.
For captions, two LSTMs remember the words already written and decide what to say next. For VQA, a GRU reads the question and combines it with the selected image regions. Then the model chooses an answer from its answer list. It is a bit like gaming: first locate possible items on the map, then use the mission objective to select the useful one!
The scores were impressive: MSCOCO captioning reached CIDEr 117.9 and BLEU-4 36.9 on the online server, while VQA validation accuracy reached 63.2%. But there is a catch. If the detector never notices a tiny or hidden object, later steps cannot easily use it. Also, the VQA system mainly picks from 3,129 common answers, so it cannot freely answer every unusual question.
Glossary
Bottom-Up Attention
A mechanism that first proposes visually salient regions, such as objects, before task-specific reasoning. Technically, it provides a compact set of detector-derived feature vectors.
Faster R-CNN supplies the regions and 2,048-dimensional features in this paper.
Top-Down Attention
Task-conditioned weighting of visual features using linguistic context. It determines which candidate regions are relevant to the current word or question.
An LSTM drives it for captioning, while a GRU question representation drives it for VQA.
Faster R-CNN
A two-stage object detector whose Region Proposal Network suggests boxes before classification and box refinement. It aligns visual features with candidate objects.
The paper uses a ResNet-101 Faster R-CNN pretrained on Visual Genome.
Self-Critical Sequence Training
A policy-gradient method that uses the model’s greedy output as a baseline and increases the probability of sampled sequences with higher reward. Here the reward is commonly CIDEr.
It optimizes captioning beyond token-level cross-entropy.
SPICE
A caption metric that compares scene-graph structures, emphasizing objects, attributes, and relations. It measures semantic content more directly than simple n-gram overlap.
The Up-Down ensemble achieved SPICE 21.5 on the MSCOCO server.
Open Questions Unanswered questions from this research
- 1 It remains unclear whether region proposal and language objectives can be trained end to end without sacrificing detector quality; hard filtering currently propagates missed detections downstream.
- 2 The fixed VQA answer vocabulary restricts open-ended reasoning. Integrating region relations, external knowledge, and iterative evidence gathering remains unresolved.
- 3 Robustness under occlusion, small objects, long-tail categories, and cross-dataset domain shift requires broader evaluation.
Applications
Immediate Applications
Accessible image description
Accessibility tools can precompute Faster R-CNN region features and use Up-Down captioning to describe people, objects, attributes, and relations in photos. Deployment requires a detector suited to the target domain and a reliable language model.
Retail and media visual QA
Users can ask about product color, count, or scene content. Region attention can suppress irrelevant backgrounds, but systems need domain adaptation, sufficient detector coverage, and an answer vocabulary containing likely responses.
Long-term Vision
Evidence-grounded visual agents
Future agents could repeatedly search, verify, and reason over regions, relation graphs, and external knowledge, producing answers linked to visual evidence. Main obstacles include open-vocabulary detection, latency, uncertainty, and reliable multi-step reasoning.
Abstract
Top-down visual attention mechanisms have been used extensively in image captioning and visual question answering (VQA) to enable deeper image understanding through fine-grained analysis and even multiple steps of reasoning. In this work, we propose a combined bottom-up and top-down attention mechanism that enables attention to be calculated at the level of objects and other salient image regions. This is the natural basis for attention to be considered. Within our approach, the bottom-up mechanism (based on Faster R-CNN) proposes image regions, each with an associated feature vector, while the top-down mechanism determines feature weightings. Applying this approach to image captioning, our results on the MSCOCO test server establish a new state-of-the-art for the task, achieving CIDEr / SPICE / BLEU-4 scores of 117.9, 21.5 and 36.9, respectively. Demonstrating the broad applicability of the method, applying the same approach to VQA we obtain first place in the 2017 VQA Challenge.