InterMT: Multi-Turn Interleaved Preference Alignment with Human Feedback
InterMT aligns multi-turn multimodal preferences with human feedback, featuring 15.6k prompts and 52.6k dialogue instances.
Key Findings
Methodology
InterMT employs tool-augmented MLLMs to construct multi-turn QA instances, incorporating expert annotations to capture human preferences. The dataset includes 15.6k prompts, 52.6k multi-turn dialogue instances, and 32.4k preference pairs.
Key Results
- InterMT dataset excels in multimodal understanding and generation tasks, significantly enhancing model dialogue consistency and preference alignment.
- Experiments show models evaluated with InterMT-Bench perform well in multi-turn tasks, especially in judge moderation.
- The multi-turn scaling law reveals preference transfer phenomena in multi-turn interactions.
Significance
InterMT addresses the gap in complex interactive capabilities of current large models by introducing a multi-turn multimodal preference dataset, advancing research in human-computer interaction.
Technical Contribution
InterMT pioneers multi-turn multimodal preference alignment, providing new datasets and benchmarks to support complex multimodal interaction tasks.
Novelty
InterMT is the first dataset focused on multi-turn multimodal understanding and generation, combining human feedback and tool-augmented workflows, pioneering new research directions.
Limitations
- Current models still struggle with consistency in long-term dialogues, especially in complex scenarios.
- The quality and accuracy of multimodal generation remain suboptimal in certain tasks.
Future Work
Future research can explore more efficient preference modeling methods to further enhance alignment in multi-turn multimodal interactions.
AI Executive Summary
Multimodal large models (MLLMs) have advanced in complex tasks but still lack multi-turn, multimodal interaction capabilities. InterMT introduces the first dataset for multi-turn multimodal preference alignment, incorporating human feedback and expert annotations to fill this gap. The dataset includes 15.6k prompts, 52.6k multi-turn dialogue instances, and 32.4k preference pairs, supporting complex multimodal interaction tasks.
InterMT's innovation lies in its tool-augmented workflow, using powerful MLLMs to construct high-quality multi-turn QA instances. Experimental results show InterMT significantly enhances model performance in multi-turn tasks, particularly in judge moderation.
Despite significant progress in multi-turn multimodal preference alignment, models still struggle with consistency in long-term dialogues. Future research can explore more efficient preference modeling methods to further enhance alignment in multi-turn multimodal interactions.
Deep Analysis
Background
Multimodal large models (MLLMs) have made significant progress in understanding and generating various modalities. However, existing models still lack multi-turn interaction capabilities, particularly in understanding and generating complex multimodal contexts.
Core Problem
Current multimodal large models lack support for multi-turn, multimodal interactions, making it difficult to achieve consistency and coherence in complex dialogue scenarios.
Innovation
InterMT introduces a multi-turn multimodal preference dataset, combining human feedback and tool-augmented workflows to achieve multi-turn multimodal preference alignment, advancing research in MLLMs.
Methodology
- �� Use tool-augmented MLLMs to construct multi-turn QA instances.
- �� Incorporate expert annotations to capture human preferences.
- �� Introduce InterMT-Bench benchmark to evaluate model performance in multi-turn tasks.
Experiments
The experimental design includes using InterMT-Bench to evaluate model performance in multi-turn multimodal tasks, focusing on model performance in judge moderation.
Results
Experimental results show InterMT significantly enhances model performance in multi-turn tasks, particularly in judge moderation.
Applications
InterMT can be used to evaluate and improve MLLMs' performance in complex interaction tasks, especially in multi-turn dialogue scenarios.
Limitations & Outlook
Despite significant progress in multi-turn multimodal preference alignment, models still struggle with consistency in long-term dialogues.
Plain Language Accessible to non-experts
Imagine a person chatting with a robot, where the robot needs to understand what the human is saying and respond appropriately. InterMT acts like a training guide for the robot, helping it better understand and respond to various human needs. Through continuous dialogue practice, the robot can learn to communicate naturally in different scenarios.
ELI14 Explained like you're 14
Imagine chatting with a super-smart robot. This robot can not only understand what you say but also see the pictures you send. InterMT is like a training manual that helps the robot get smarter in conversations. It can understand what you want and give the best response.
Glossary
Multimodal Large Models (MLLMs)
Models capable of processing multiple modalities such as text, images, and audio.
Used to achieve multi-turn multimodal interactions.
Preference Alignment
Adjusting model outputs to better match human preferences and expectations.
Achieved through human feedback.
Tool-Augmented
Enhancing model capabilities using external tools like image editing and generation.
Used to construct multi-turn QA instances.
Multi-turn Dialogue
Dialogue involving multiple turns, requiring the model to maintain consistency.
InterMT evaluates model performance in multi-turn tasks.
Human Feedback
Improving model performance through human evaluations and suggestions.
Used to capture human preferences.
Open Questions Unanswered questions from this research
- 1 How to improve model consistency and coherence in long-term dialogues?
- 2 Why do existing methods struggle to achieve high-quality generation in complex multimodal tasks?
Applications
Immediate Applications
Multimodal Dialogue Systems
Can be used to develop smarter dialogue systems, enhancing user experience.
Long-term Vision
Future of Human-Computer Interaction
Advances human-computer interaction technology, enabling more natural communication.
Abstract
As multimodal large models (MLLMs) continue to advance across challenging tasks, a key question emerges: What essential capabilities are still missing? A critical aspect of human learning is continuous interaction with the environment -- not limited to language, but also involving multimodal understanding and generation. To move closer to human-level intelligence, models must similarly support multi-turn, multimodal interaction. In particular, they should comprehend interleaved multimodal contexts and respond coherently in ongoing exchanges. In this work, we present an initial exploration through the InterMT -- the first preference dataset for multi-turn multimodal interaction, grounded in real human feedback. In this exploration, we particularly emphasize the importance of human oversight, introducing expert annotations to guide the process, motivated by the fact that current MLLMs lack such complex interactive capabilities. InterMT captures human preferences at both global and local levels into nine sub-dimensions, consists of 15.6k prompts, 52.6k multi-turn dialogue instances, and 32.4k human-labeled preference pairs. To compensate for the lack of capability for multi-modal understanding and generation, we introduce an agentic workflow that leverages tool-augmented MLLMs to construct multi-turn QA instances. To further this goal, we introduce InterMT-Bench to assess the ability of MLLMs in assisting judges with multi-turn, multimodal tasks. We demonstrate the utility of \InterMT through applications such as judge moderation and further reveal the multi-turn scaling law of judge model. We hope the open-source of our data can help facilitate further research on aligning current MLLMs to the next step. Our project website can be found at https://pku-intermt.github.io .