GptGet

DaMo: Data Mixing Optimizer in Fine-tuning Multimodal LLMs for Mobile Phone Agents

TL;DR

DaMo employs neural networks to predict optimal data mixing ratios in multimodal LLM fine-tuning, achieving 3.38% performance gain on PhoneAgentBench.

cs.CV 🔴 Advanced 2025-10-22 42 views
Kai Shi Jun Yang Ni Yang Binqiang Pan Qingsong Xie Chao Zhang Zhenyu Yang Tianhuang Su Haonan Lu
AI Reader Arxiv Page Download PDF
multimodal large language models data mixing optimization fine-tuning benchmark mobile agents

Key Findings

Methodology

DaMo utilizes a trainable multi-layer perceptron (MLP) to forecast downstream task performance based on data mixture proportions. By sampling a limited set of mixture ratios from the fixed data space, training models at checkpoints, and evaluating their performance, it constructs a performance trajectory function. The MLP models this nonlinear relationship, capturing complex interactions between datasets. During inference, DaMo predicts performance across the entire data mixture space, guiding the selection of optimal ratios for fine-tuning. This approach significantly reduces the need for exhaustive empirical testing, enabling efficient optimization of data composition for multitask multimodal models.

Key Results

  • On PhoneAgentBench, DaMo improved average performance by 3.38% (from 54.83% to 68.18%), surpassing baseline strategies like uniform and natural mixtures. It also outperformed DML methods by 2.57% on average across benchmarks, with a 12.47% boost on BFCL-v3. Transferability tests on Qwen and InternVL models showed prediction correlations above 0.75, which improved to over 0.9 after linear calibration. The method effectively predicts performance trends, enabling targeted data mixture selection, thus saving training resources and enhancing model capabilities in real-world scenarios.
  • Results demonstrate that DaMo accurately forecasts performance in complex multi-task, multimodal settings, facilitating rapid and cost-effective model fine-tuning. Its ability to generalize across models and tasks underscores its potential for industrial deployment, especially in resource-constrained mobile environments. The approach addresses key bottlenecks in data-driven model optimization, offering a scalable solution for future multimodal AI systems.
  • The experimental outcomes confirm that neural network-based performance modeling can capture nonlinear performance trajectories better than traditional exponential or power-law fits. This breakthrough provides a new theoretical and practical framework for data mixture optimization in multitask learning, enabling models to adaptively select training data ratios that maximize downstream performance, thus advancing the state-of-the-art in efficient multimodal model fine-tuning.
  • This work’s novelty lies in directly modeling the performance landscape with neural networks, a first in the context of data mixture optimization for large multimodal models. Unlike prior heuristic or rule-based methods, DaMo learns complex nonlinear relationships, offering a data-driven, scalable, and model-agnostic solution. Its successful application across multiple benchmarks and models demonstrates its broad applicability and potential to revolutionize multitask multimodal AI training workflows.

Significance

This research addresses a critical bottleneck in deploying multimodal large language models in practical mobile scenarios. By enabling precise, data-driven optimization of training data proportions, DaMo reduces the reliance on manual trial-and-error, cutting costs and accelerating deployment cycles. Its ability to predict performance without full retraining makes it highly suitable for industrial settings where training resources are limited. The comprehensive benchmark PhoneAgentBench fills a vital gap, providing a standardized platform for evaluating mobile multimodal AI. Overall, this work paves the way for more efficient, scalable, and adaptable AI systems capable of handling complex real-world tasks, fostering broader adoption of intelligent mobile agents in industry and daily life.

Technical Contribution

DaMo introduces a neural network-based performance predictor that models the nonlinear relationship between data mixture ratios and downstream task performance. It leverages a limited set of sampled data mixtures and checkpoints to train an MLP, which then extrapolates performance across the entire mixture space. This approach overcomes the limitations of traditional exponential or power-law models, providing a flexible, accurate, and scalable solution. Additionally, DaMo constructs a comprehensive mobile benchmark, PhoneAgentBench, to evaluate models across multiple capabilities. Its transferability and model-agnostic design enable application to various architectures with minimal calibration, significantly advancing the methodology of data-driven model fine-tuning.

Novelty

This is the first work to employ neural networks explicitly for predicting performance trajectories in multi-task, multimodal model fine-tuning. Unlike prior methods relying on heuristic or static models, DaMo learns complex nonlinear performance patterns, enabling precise data mixture optimization. Its integration with a new comprehensive benchmark tailored for mobile scenarios further distinguishes it from existing approaches, marking a significant step forward in automating and improving model fine-tuning processes.

Limitations

  • The method relies on a limited number of sampled data mixtures, which may not fully capture the performance landscape in highly diverse or unseen scenarios, potentially affecting accuracy.
  • Transferability across models requires a small calibration step, which, although minimal, still introduces some overhead.
  • Scalability to extremely large models or highly dynamic environments remains to be validated, as performance prediction may degrade with increasing complexity.

Future Work

Future directions include integrating reinforcement learning to adaptively refine data mixture predictions, expanding the benchmark to cover more real-world scenarios, and developing adaptive sampling strategies for better performance landscape exploration. Additionally, exploring unsupervised or semi-supervised approaches to reduce calibration efforts and extending the framework to continual learning settings will further enhance its practicality and robustness in industrial deployments.

AI Executive Summary

Deep Dive

Plain Language Accessible to non-experts

想象你在厨房准备一道菜，要用不同的食材比例。以前厨师们只能靠经验反复试验，才能找到味道最好的搭配。这很费时间，也不一定每次都成功。现在，有个聪明的助手（DaMo），它可以根据少量试验，预测不同食材比例的味道表现，帮你提前选出最优搭配。这样，你不用反复试错，就能快速做出美味佳肴。这就像在训练AI模型一样，合理选择训练数据比例能大大提升模型表现，节省时间和成本。

ELI14 Explained like you're 14

假设你在学校准备演讲，想知道用不同的资料（比如图片、视频、文章）搭配，哪个组合能让你的演讲更精彩。以前，你只能自己反复试，花很多时间。现在，有个聪明的朋友（DaMo），它可以根据你试过的几次，预测不同资料组合的效果，帮你提前选出最棒的搭配。这样，你不用每次都试错，就能更快准备好精彩的演讲。这就像训练AI一样，合理选择训练数据能让模型更厉害，效率更高。

Abstract

Mobile Phone Agents (MPAs) have emerged as a promising research direction due to their broad applicability across diverse scenarios. While Multimodal Large Language Models (MLLMs) serve as the foundation for MPAs, their effectiveness in handling multiple mobile phone tasks simultaneously remains limited. Although multitask supervised fine-tuning (SFT) is widely adopted for multitask learning, existing approaches struggle to determine optimal training data compositions for peak performance. To address this challenge, we propose DaMo (Data Mixture Optimizer) - a novel solution employing a trainable network that predicts optimal data mixtures by forecasting downstream task performance for any given dataset ratio. To support comprehensive evaluation, we introduce PhoneAgentBench, the first specialized benchmark to evaluate MLLMs on multimodal mobile phone tasks, comprising 1235 QA pairs spanning diverse real-world industrial mobile application scenarios. Demonstrating strong predictive capability (R^2=0.81) in small-scale pilot experiments, DaMo efficiently extrapolates optimal data mixing configurations. Our results show DaMo achieves a 3.38% performance improvement on PhoneAgentBench compared to alternative methods. Furthermore, extensive experiments across established benchmarks including BFCL-v3, MME-Reasoning, MME-Perception, and OCRBench reveal DaMo's superior generalization, outperforming other approaches by 2.57% in terms of average score. When used solely for MLLM optimization on the BFCL-v3 task, DaMo improves the metrics by 12.47% than other methods. Notably, DaMo maintains robust scalability, preserving its effectiveness when applied to other model architectures. The code and dataset are available at https://github.com/OPPO-Mente-Lab/DaMo.git

cs.CV