Learning to Think Like a Cartoon Captionist: Incongruity-Resolution Supervision for Multimodal Humor Understanding

Humor understanding is a significant aspect of human intelligence, involving the integration of visual perception, cultural knowledge, and creative reasoning. In recent years, with the advancement of multimodal learning and natural language processing technologies, researchers have begun to explore how computers can understand and generate humor. However, most existing approaches treat humor understanding as a black-box prediction task, overlooking the structured reasoning processes underlying humor comprehension. This approach has limitations in handling complex humor tasks, as humor is not merely about selecting or ranking captions but involves identifying and resolving incongruities.

The New Yorker Cartoon Caption Contest (NYCC) is a key resource for studying multimodal humor, aligning visual input, linguistic creativity, expert judgment, and crowd preferences. While prior work primarily treats NYCC as a classification or ranking benchmark, it also offers insight into the reasoning processes underlying visual humor. To improve humor understanding, researchers proposed the Incongruity-Resolution Supervision (IRS) framework, which decomposes humor understanding into three components: incongruity modeling, resolution modeling, and preference alignment.

The core problem of humor understanding lies in how to identify incongruities in visual scenes and transform them into coherent and humorous interpretations. This process involves identifying the mismatch between expectation and observation and resolving it in a coherent yet surprising way. Although existing multimodal language models exhibit some capability in humor understanding tasks, they still have significant gaps in identifying and resolving incongruities. To address this gap, researchers proposed the Incongruity-Resolution Supervision (IRS) framework, which supervises intermediate reasoning processes through structured traces.

The core innovations of the IRS framework lie in its decomposition of humor understanding into three explicit, learnable stages: incongruity modeling, resolution modeling, and preference alignment.

• �� Incongruity Modeling: Identifies mismatches in the visual scene, helping the model recognize the difference between expectation and observation.

• �� Resolution Modeling: Constructs coherent reinterpretations of these mismatches, enabling the model to transform incongruities into coherent and humorous interpretations.

• �� Preference Alignment: Evaluates candidate interpretations under human judgments, ensuring that the model-generated interpretations align with human humor preferences. This approach is not only innovative in the field of humor understanding but also offers new insights for other complex reasoning tasks.

The IRS framework achieves humor understanding through the following steps:

• �� Incongruity Modeling: Identifies mismatches in the visual scene through domain-adaptive pretraining. Using a curated corpus of captionist discussions, editorial analyses, and caption-writing guides, the model's representations are biased toward humor-relevant concepts.

• �� Resolution Modeling: Supervises resolution modeling through captionist reasoning traces, teaching the model how incongruities are reinterpreted into coherent humorous readings. The generated traces are verified under human supervision to ensure consistency with expert reasoning patterns.

• �� Preference Alignment: Reinforcement learning with humor-specific rewards optimizes the reasoning process. Using GRPO, the reasoning process is directly optimized without a value network, ensuring that model-generated interpretations align with human humor preferences.

The experimental design includes evaluating the performance of the IRS framework on NYCC using 7B, 32B, and 72B models for matching and ranking tasks. Experiments also include zero-shot transfer to external humor benchmarks such as YesBut and DeepEval to test IRS's generalization capabilities. Key parameters used in the experiments include model scale, pretraining corpus, and the generation and verification of reasoning traces. Ablation studies are conducted to evaluate the contributions of incongruity modeling, resolution modeling, and preference alignment to performance.

Experimental results show that the IRS framework performs exceptionally well on NYCC, particularly approaching expert-level performance in ranking tasks. The 72B model achieves a ranking accuracy of 76.10%, surpassing all baseline models, including the closed model o3. Ablation studies show that resolution modeling (RM) is the main source of improvement, especially when combined with incongruity modeling (IM), providing additional gains in more challenging ranking tasks. Additionally, IRS exhibits good generalization capabilities when zero-shot transferred to external humor benchmarks, indicating that IRS learns generalizable reasoning patterns.

The IRS framework has potential value in multiple application scenarios, including creative assistance tools, educational systems, and human-computer interaction research. In creative assistance tools, IRS can help generate content that aligns better with human humor preferences. In educational systems, IRS can be used to develop more interpretable and stylistically aware teaching tools. In human-computer interaction research, IRS can enhance the transparency and explainability of models when dealing with complex, culturally dependent phenomena such as humor.

IRS may have limitations in handling cultural differences, as humor is subjective and culturally dependent. Models trained on specific humor traditions may not generalize uniformly across cultures or communities. Additionally, the computational cost of IRS is high, particularly when training on large-scale models, which may limit its application in resource-constrained environments. Future research directions include extending the IRS framework to handle a wider range of humor types and cultural contexts, and applying IRS's structured reasoning approach to other complex reasoning tasks.