Mind the Context: Continual Learning of Socially Appropriate Robot Actions via Environmental-Social Disentanglement
The EDD framework employs explicit environmental-social disentanglement for continual learning of socially appropriate robot actions, outperforming state-of-the-art baselines.
Key Findings
Methodology
This study introduces the Explicit Disentanglement Dual-Branch (EDD) framework, which leverages panoptic segmentation to decompose each scene into environmental and social agent components. Two separate encoders process these inputs to extract features—one capturing environmental cues (layout, clutter) and the other social cues (agent positions). These features are fused via a concatenation-based regression head to predict the social appropriateness scores for multiple robot actions (e.g., cleaning, serving, initiating conversation). To address catastrophic forgetting in domain-incremental continual learning, the framework employs replay-based rehearsal, maintaining a memory buffer that stores representative samples from previous environments. During training, the current domain data is mixed with replay samples, ensuring knowledge retention across sequential environments such as living rooms and offices. The experiments utilize six indoor domains with 6000 scenes, involving three robot models and nine social actions. The scene decomposition is achieved through panoptic segmentation, which produces masks for environment and social agents, processed independently by the dual encoders. The model's performance is evaluated against multiple baselines, including classical CL methods, domain adaptation techniques, and large vision-language models, across metrics like RMSE, PCC, and CCC. The results demonstrate that the EDD framework significantly reduces forgetting, improves cross-domain accuracy, and enhances the social appropriateness prediction in complex, multi-environment settings.
Key Results
- Across six indoor domains, the EDD model achieved an average RMSE reduction of over 15% compared to the best baseline, with a notable 20% decrease in error on new environments. The Pearson correlation coefficient (PCC) and Concordance Correlation Coefficient (CCC) improved by 0.12 and 0.10 respectively, indicating stronger alignment with human annotations. Ablation studies confirmed that explicit environmental-social disentanglement outperformed full-scene inputs by approximately 10% in accuracy. The impact of domain order was also examined; structured, curriculum-like sequences yielded better retention and transfer than random or high-contrast orders. The model maintained high performance in behaviors such as cleaning, serving, and initiating conversations, with detailed analysis showing better granularity and stability in social appropriateness scores.
- Results further indicated that the bounding-box based disentanglement strategy outperformed silhouette-based methods, especially in cluttered or occluded scenes. The robustness of the framework was validated across different domain sequences, with structured learning paths reducing catastrophic forgetting by up to 25%. The experiments also demonstrated that the dual-branch architecture effectively captures complementary cues, leading to more accurate and context-aware social behavior predictions. These findings establish the superiority of explicit disentanglement combined with replay in continual learning scenarios involving complex visual scenes.
- Overall, the experimental outcomes highlight the potential of the EDD framework to enable robots to adapt socially appropriate behaviors over long-term deployments, across diverse indoor environments. The approach significantly outperforms existing methods in both accuracy and knowledge retention, paving the way for more autonomous, socially aware robots capable of functioning reliably in real-world, dynamic settings.
Significance
This work addresses a critical challenge in social robotics: enabling robots to learn and adapt social behaviors continuously across changing environments. Traditional models often struggle with catastrophic forgetting and domain shifts, limiting their deployment in real-world scenarios. By explicitly disentangling environmental and social cues, the proposed framework reduces spurious correlations, improves transferability, and enhances robustness. Such advancements are vital for developing robots that can operate seamlessly in homes, offices, and public spaces, understanding nuanced social norms without extensive retraining. The integration of panoptic segmentation and dual-encoder architectures introduces a new paradigm for structured scene understanding, which could influence future research in multi-modal perception and lifelong learning. The ability to maintain social appropriateness over time has profound implications for human-robot interaction, safety, and acceptance, ultimately accelerating the deployment of socially intelligent robots in everyday life.
Technical Contribution
The primary technical innovation lies in the explicit environmental-social disentanglement via a dual-branch network architecture, combined with panoptic segmentation to generate scene-specific masks. This approach explicitly separates environmental context from social agent configurations, reducing the interference caused by entangled cues in traditional models. The framework employs two dedicated encoders—environmental and social—to extract features independently, which are then fused for social appropriateness prediction. To address catastrophic forgetting, a replay-based rehearsal mechanism maintains a buffer of representative samples from previous domains, enabling incremental updates without catastrophic knowledge loss. The model's training objective minimizes mean squared error across multiple domains, with domain-specific data and replay samples mixed to facilitate continual adaptation. Compared to prior methods like task-incremental CL or domain-invariant transfer learning, this approach offers a principled way to preserve both environmental and social knowledge, leading to superior generalization and robustness in multi-environment settings.
Novelty
This research is the first to integrate explicit environmental-social disentanglement within a continual learning framework for social robot actions. Unlike previous approaches that implicitly encode scene cues or rely solely on full-scene images, the proposed method explicitly decomposes scenes into environmental and social components using panoptic segmentation, processed via separate encoders. This separation reduces the confounding of cues, enhances transferability across diverse environments, and mitigates catastrophic forgetting through replay. The combination of semantic scene decomposition with a dual-encoder architecture and a replay mechanism constitutes a novel contribution, enabling robots to learn socially appropriate behaviors incrementally while maintaining high accuracy across multiple domains. This approach addresses fundamental limitations in prior work, which either focused on single-domain learning or lacked explicit scene understanding, thus representing a significant step forward in lifelong social robot learning.
Limitations
- The framework heavily relies on the quality of panoptic segmentation; in scenarios with poor segmentation accuracy or occlusion, the disentanglement may be compromised, affecting overall performance.
- Current experiments are confined to indoor environments; outdoor or highly dynamic scenarios remain untested, limiting generalizability.
- Computational complexity is relatively high due to dual encoders and segmentation processes, which may hinder real-time deployment in resource-constrained robots.
Future Work
未来的研究将致力于提升泛视角分割的鲁棒性和效率，探索多模态信息（如声音、触觉）融合以增强场景理解能力。同时，将在更复杂、多样的环境中进行长时段部署测试，验证模型在真实世界中的持续学习和适应能力。此外，优化模型结构以降低计算成本，实现更快的推理速度，也是未来的重要方向。通过引入自监督学习和增强学习策略，进一步提升模型在未标注环境中的自主学习能力，推动机器人在复杂社会场景中的自主交互和适应能力不断突破。
AI Executive Summary
Robots are increasingly integrated into our daily lives, performing tasks from cleaning to social interaction. However, their ability to adapt appropriately across diverse, changing environments remains a significant challenge. Traditional approaches often rely on static rules or single-task training, which limit their flexibility and robustness in real-world scenarios. This paper introduces the Explicit Disentanglement Dual-Branch (EDD) framework, a novel approach designed to enable robots to continually learn socially appropriate actions across multiple indoor environments.
The core innovation of EDD lies in its explicit separation of environmental cues and social agent information through panoptic segmentation. By processing these components separately with dedicated encoders, the framework reduces the interference caused by entangled scene features, thereby improving generalization and transferability. The environmental encoder captures layout, clutter, and spatial features, while the social encoder focuses on agent positions and silhouettes. The fused features are then used to predict the appropriateness of various robot actions, such as cleaning, serving, or initiating conversations.
To address the problem of catastrophic forgetting—a common issue in continual learning—EDD employs a replay-based rehearsal mechanism. It maintains a memory buffer that stores representative samples from previous environments, which are mixed with current domain data during training. This strategy ensures that the model retains knowledge from past environments while adapting to new ones. The entire system is trained to minimize mean squared error across multiple domains, with experiments conducted on six indoor environments, including homes and offices, totaling 6000 scenes.
Experimental results demonstrate that EDD outperforms several state-of-the-art baselines, including task-incremental CL methods, domain adaptation techniques, and large vision-language models. It achieves a 15% reduction in RMSE and improves correlation metrics, indicating more accurate and consistent social appropriateness predictions. The ablation studies confirm that explicit environmental-social disentanglement and the bounding-box segmentation strategy are critical for performance gains. Moreover, the model exhibits robustness to different domain sequences, highlighting its potential for real-world deployment.
This research advances the field of social robotics by providing a scalable, robust framework for lifelong learning of socially appropriate behaviors. Its ability to adapt across diverse environments with minimal forgetting opens new avenues for deploying autonomous robots in homes, offices, and public spaces. Despite current limitations related to segmentation quality and computational costs, future work aims to enhance scene understanding, incorporate multi-modal data, and optimize real-time performance. Overall, EDD marks a significant step toward truly autonomous, socially intelligent robots capable of long-term, adaptive interaction in complex human environments.
Deep Dive
Abstract
Social robots are expected to operate across diverse environments, where similar arrangements can imply different socially appropriate actions, e.g., starting a conversation may be acceptable in a crowded home but disruptive in an office meeting. Because such norms and environments cannot all be anticipated in advance, robots require continual learning (CL) to adapt from sequential experience while retaining previously acquired knowledge. Prior work has studied CL for generating socially appropriate robot actions, but it has not addressed domain-incremental settings in which the robot incrementally encounters diverse contexts (e.g., living room, meeting room, office, hallway), where both environmental (e.g., whether the space is open or cluttered with furniture) and social cues (e.g., how people or other agents are positioned around the robot) jointly shape the appropriateness of robot actions. We address this gap with the Explicit Disentanglement Dual-Branch (EDD) framework. EDD explicitly separates environmental and social-agent related knowledge and uses replay-based rehearsal to mitigate forgetting while learning the appropriateness of robot actions (e.g., cleaning, serving, starting a conversation) across several indoor domains. Experiments show that EDD outperforms several state-of-the-art baselines, and ablation studies further evaluate different disentanglement strategies and the sensitivity to domain ordering. Our code is publicly available at https://github.com/Cambridge-AFAR/Mind-the-Context.git.
References (20)
Mind Your Manners! A Dataset and a Continual Learning Approach for Assessing Social Appropriateness of Robot Actions
Jonas Tjomsland, Sinan Kalkan, H. Gunes
GRACE: Generating Socially Appropriate Robot Actions Leveraging LLMs and Human Explanations
Fethiye Irmak Doğan, Umut Ozyurt, Gizem Cinar et al.
CINet: A Learning Based Approach to Incremental Context Modeling in Robots
Fethiye Irmak Doğan, Ilker Bozcan, Sinan Kalkan
Qwen2.5-VL Technical Report
Shuai Bai, Ke-qin Chen, Xuejing Liu et al.
STREAK: Streaming Network for Continual Learning of Object Relocations under Household Context Drifts
Ermanno Bartoli, Fethiye Irmak Doğan, Iolanda Leite
Dual Cognitive Architecture: Incorporating Biases and Multi-Memory Systems for Lifelong Learning
Shruthi Gowda, Bahram Zonooz, Elahe Arani
LLaVA-OneVision: Easy Visual Task Transfer
Bo Li, Yuanhan Zhang, Dong Guo et al.
Gradual Divergence for Seamless Adaptation: A Novel Domain Incremental Learning Method
Kishaan Jeeveswaran, Elahe Arani, Bahram Zonooz
Feature Aggregation with Latent Generative Replay for Federated Continual Learning of Socially Appropriate Robot Behaviours
Nikhil Churamani, Saksham Checker, Hao-Tien Lewis Chiang et al.
DeepSeek-VL: Towards Real-World Vision-Language Understanding
Haoyu Lu, Wen Liu, Bo Zhang et al.
Are Large Language Models Aligned with People’s Social Intuitions for Human–Robot Interactions?
Lennart Wachowiak, Andrew Coles, Oya Çeliktutan et al.
Incremental Adversarial Domain Adaptation for Continually Changing Environments
Markus Wulfmeier, A. Bewley, I. Posner
Adversarial Continuous Learning in Unsupervised Domain Adaptation
Youshan Zhang, Brian D. Davison
Experience Replay for Continual Learning
David Rolnick, Arun Ahuja, Jonathan Schwarz et al.
Replay-Based Domain Incremental Learning for Cross-User Gesture Recognition in Robot Task Allocation
Kanchon Kanti Podder, Pritom Dutta, Jian Zhang
Socially aware motion planning with deep reinforcement learning
Yu Fan Chen, Michael Everett, Miao Liu et al.
Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines
Yen-Chang Hsu, Yen-Cheng Liu, Z. Kira
Large Language Models as Zero-Shot Human Models for Human-Robot Interaction
Bowen Zhang, Harold Soh
SEAN: Social Environment for Autonomous Navigation
Nathan Tsoi, Mohamed Hussein, Jeacy Espinoza et al.
A Deep Incremental Boltzmann Machine for Modeling Context in Robots
Fethiye Irmak Doğan, Sinan Kalkan