Learning Next Action Predictors from Human-Computer Interaction
LongNAP combines parametric and retrieval mechanisms, trained with policy gradients on 360K actions over 1,800 hours, outperforming baselines significantly.
Key Findings
Methodology
This paper introduces LongNAP, a model that integrates parametric learning with retrieval mechanisms, trained via policy gradient algorithms on a large-scale dataset of 360,000 user actions collected over a month, totaling 1,800 hours of screen time. The approach involves a two-stage process: first, the model reasons to generate a query (zretrieve) based on current context, then retrieves relevant past trajectories (D) from a memory bank using BM25. In the second stage, it reasons to predict (zpredict) by integrating retrieved information with current context, generating future action trajectories (Et+1:t+h). The training employs a reward based on a large language model (LLM) judged semantic similarity between predicted and actual behaviors, enabling end-to-end optimization. Data collection is facilitated by the open-source NAPsack pipeline, which passively records and annotates user interactions with vision-language models, ensuring large-scale, naturalistic, privacy-preserving datasets. The model's performance is evaluated on both single-user and cross-user data, demonstrating significant improvements over supervised fine-tuning and prompting baselines, with a prediction accuracy of 17.1% for trajectories aligned with actual user actions, rising to 26% for high-confidence predictions.
Key Results
- LongNAP achieves a 79% improvement over supervised fine-tuning and a 39% improvement over prompt-based baselines in predicting user actions, as measured by LLM-judge similarity scores, on held-out user data. It generalizes well across users, maintaining performance when trained on multiple individuals.
- On a dataset of 360K actions and 1,800 hours of screen time, the model effectively captures long-term behavioral patterns, with an average trajectory similarity score of 0.52, indicating substantial semantic alignment. The high-confidence prediction rate reaches 26%, demonstrating practical reliability.
- The retrieval mechanism significantly enhances prediction quality by leveraging relevant past trajectories, especially in complex, multimodal scenarios. Data compression strategies (event-driven) reduce storage by 75% without compromising accuracy, showing efficiency gains. The model's ability to predict diverse future actions across thousands of possibilities underscores its robustness.
Significance
This work advances the frontier of long-term, multimodal user behavior modeling, addressing fundamental challenges in AI proactive support systems. By leveraging passive data collection and sophisticated retrieval-augmented reasoning, it enables AI to anticipate user needs more accurately and naturally, fostering more intuitive human-computer interactions. The approach offers scalable, privacy-conscious solutions for deploying personalized AI assistants across devices. Its implications extend to personalized health monitoring, adaptive interfaces, and intelligent automation, promising a future where AI seamlessly integrates into daily life, understanding users at a deeper level. The combination of large-scale passive data, innovative training methods, and retrieval-augmented reasoning marks a significant step toward truly proactive AI systems.
Technical Contribution
The paper introduces LongNAP, a novel architecture that combines parametric models with a retrieval mechanism, trained via policy gradient methods, to handle long, multimodal interaction sequences. It innovatively employs a two-phase reasoning process: first generating a reasoning trace (zretrieve) to query relevant history, then integrating retrieved trajectories with current context (zpredict) for future action prediction. The use of LLMs as an end-to-end evaluator for semantic similarity enables direct optimization of behavior prediction quality. The open-source NAPsack pipeline automates large-scale passive data collection and annotation, reducing reliance on manual labeling. This integration of retrieval, reasoning, and reinforcement learning constitutes a significant technical advance in long-term behavior modeling, enabling models to adapt dynamically and generalize across users.
Novelty
This is the first work to combine retrieval-augmented reasoning with policy gradient training for long-sequence user behavior prediction, leveraging large-scale passive data annotation. Unlike traditional fine-tuning or prompt-based methods, LongNAP dynamically retrieves relevant past behaviors to inform predictions, addressing the limitations of fixed context windows. The innovative use of LLM-based semantic similarity as a reward function for end-to-end training enables more accurate and contextually aligned predictions. This approach effectively captures complex, long-term, multimodal user behaviors, setting a new benchmark in proactive AI modeling.
Limitations
- The model relies heavily on passive behavior data, raising privacy concerns and requiring secure data handling mechanisms. Its performance may degrade in scenarios involving highly novel or sparse behaviors, where historical data offers limited guidance.
- Computational complexity and inference latency remain challenges, especially for real-time applications. The retrieval and reasoning processes are resource-intensive, necessitating further optimization for deployment.
- The model's ability to handle extreme outliers or abrupt behavioral shifts is limited, and its robustness in unpredictable situations needs enhancement. Future work should focus on improving adaptability and interpretability, as well as reducing computational costs.
Future Work
Future research will explore integrating multi-modal signals (e.g., audio, sensor data) for richer context understanding, employing federated learning to enhance privacy, and developing more efficient retrieval and reasoning algorithms. Additionally, extending the framework to support real-time prediction in embedded devices and incorporating user feedback for continual learning are promising directions. Further, investigating explainability and robustness in diverse, real-world settings will be crucial for practical deployment. These efforts aim to create more adaptive, privacy-preserving, and scalable proactive AI systems that seamlessly support users in their daily activities.
AI Executive Summary
In the rapidly evolving landscape of artificial intelligence, the ability for systems to proactively anticipate user needs has become a critical frontier. Traditional language models and behavior prediction techniques largely operate within narrow contexts, focusing on immediate prompts or short-term signals. These approaches often fall short in capturing the rich, long-term, multimodal behaviors that characterize real-world human-device interactions. Recognizing this gap, the present study introduces LongNAP, a novel model designed to predict usersβ next actions by reasoning over extensive, diverse interaction histories.
The core challenge addressed by this work is how to effectively model and utilize long sequences of user behaviors, which are inherently sparse, noisy, and multimodal. To tackle this, the researchers developed NAPsack, an open-source passive data collection pipeline that passively records and annotates behavior traces from smartphones, leveraging vision-language models for automatic labeling. Over a month, data from 20 users yielded 360,000 annotated actions spanning 1,800 hours of screen time, providing a rich dataset for training and evaluation.
Building on this data foundation, the authors designed LongNAP, a two-stage reasoning architecture that combines parametric learning with retrieval mechanisms. In the first stage, the model generates a reasoning trace (zretrieve) based on recent context, which it uses to query a memory bank of past behaviors via BM25 retrieval. In the second stage, it integrates retrieved traces with current reasoning (zpredict) to generate future action trajectories. The entire system is trained end-to-end using policy gradients, with a large language model (LLM) serving as a semantic similarity judge, enabling direct optimization of prediction quality.
Experimental results demonstrate that LongNAP significantly outperforms baseline methods, including supervised fine-tuning and prompt-based models. On held-out data, it achieves a 79% improvement over fine-tuning and a 39% improvement over prompts, with 17.1% of predicted trajectories closely aligned with actual user actions. When trained across users, the model generalizes effectively, maintaining strong performance. The retrieval mechanism notably enhances prediction accuracy, especially in complex, multimodal scenarios, and data compression strategies reduce storage requirements by 75% without sacrificing quality.
This research marks a substantial step toward truly proactive AI systems capable of understanding and predicting human behavior over extended periods. Its implications span personalized digital assistants, health monitoring, adaptive interfaces, and beyond. By combining passive large-scale data collection, innovative training strategies, and retrieval-augmented reasoning, the work addresses longstanding challenges in long-term behavior modeling. Future directions include improving real-time inference, enhancing privacy protections, and expanding multimodal integration, paving the way for AI that seamlessly supports users in their daily lives with unprecedented accuracy and personalization.
Deep Dive
Abstract
Truly proactive AI systems must anticipate what we will do next. This foresight demands far richer information than the sparse signals we type into our prompts -- it demands reasoning over the entire context of what we see and do. We formalize this as next action prediction (NAP): given a sequence of a user's multimodal interactions with a computer (screenshots, clicks, sensor data), predict that user's next action. Progress on this task requires both new data and modeling approaches. To scale data, we annotate longitudinal, naturalistic computer use with vision-language models. We release an open-source pipeline for performing this labeling on private infrastructure, and label over 360K actions across one month of continuous phone usage from 20 users, amounting to 1,800 hours of screen time. We then introduce LongNAP, a user model that combines parametric and in-context learning to reason over long interaction histories. LongNAP is trained via policy gradient methods to generate user-specific reasoning traces given some context; retrieve relevant traces from a library of past traces; and then apply retrieved traces in-context to predict future actions. Using an LLM-as-judge evaluation metric (0-1 similarity to ground truth), LongNAP significantly outperforms supervised finetuning and prompted baselines on held-out data (by 79% and 39% respectively). Additionally, LongNAP generalizes to held out users when trained across individuals. The space of next actions a user might take at any moment is unbounded, spanning thousands of possible outcomes. Despite this, 17.1% of LongNAP's predicted trajectories are well-aligned with what a user does next (LLM-judge score $\geq$ 0.5). This rises to 26% when we filter to highly confident predictions. In sum, we argue that learning from the full context of user behavior to anticipate user needs is now a viable task with substantial opportunity.
References (20)
Okapi at TREC-3
S. Robertson, S. Walker, S. Jones et al.
Screenomics: A Framework to Capture and Analyze Personal Life Experiences and the Ways that Technology Shapes Them
Byron Reeves, Nilam Ram, Thomas N. Robinson et al.
Training language models to follow instructions with human feedback
Long Ouyang, Jeff Wu, Xu Jiang et al.
MemGPT: Towards LLMs as Operating Systems
Charles Packer, Vivian Fang, Shishir G. Patil et al.
Constitutional AI: Harmlessness from AI Feedback
Yuntao Bai, Saurav Kadavath, Sandipan Kundu et al.
Learning Universal Policies via Text-Guided Video Generation
Yilun Du, Mengjiao Yang, Bo Dai et al.
Generative Agents: Interactive Simulacra of Human Behavior
J. Park, Joseph O'Brien, Carrie J. Cai et al.
QLoRA: Efficient Finetuning of Quantized LLMs
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman et al.
Judging LLM-as-a-judge with MT-Bench and Chatbot Arena
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng et al.
Understanding Social Reasoning in Language Models with Language Models
Kanishk Gandhi, Jan-Philipp Franken, Tobias Gerstenberg et al.
Learning Interactive Real-World Simulators
Mengjiao Yang, Yilun Du, Kamyar Ghasemipour et al.
AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback
Yann Dubois, Xuechen Li, Rohan Taori et al.
On Second Thought, Letβs Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning
Omar Shaikh, Hongxin Zhang, William B. Held et al.
Training Verifiers to Solve Math Word Problems
K. Cobbe, Vineet Kosaraju, Mo Bavarian et al.
Transformers generalize differently from information stored in context vs in weights
Stephanie C. Y. Chan, Ishita Dasgupta, Junkyung Kim et al.
Out of One, Many: Using Language Models to Simulate Human Samples
Lisa P. Argyle, E. Busby, Nancy Fulda et al.
Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos
Bowen Baker, Ilge Akkaya, P. Zhokhov et al.
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Tri Dao, Daniel Y. Fu, S. Ermon et al.
STaR: Bootstrapping Reasoning With Reasoning
E. Zelikman, Yuhuai Wu, Noah D. Goodman
Lost in the Middle: How Language Models Use Long Contexts
Nelson F. Liu, Kevin Lin, John Hewitt et al.
Cited By (4)
On the Fragility of Self-Improving Agents: Variance, Task Order, and Underspecification
Learning from 53.6K Real-World Developer Edits of AI-Generated Code
A Benchmark and Framework for Evaluating Next Action Predictions in Spreadsheets
Offloading Score: Measuring AI Reliance Through Counterfactual Workflows