Feature Aggregation with Latent Generative Replay for Federated Continual Learning of Socially Appropriate Robot Behaviours

Robots operating in social environments require adaptive learning systems capable of understanding norms and behaviors. Early approaches like deep reinforcement learning and imitation learning achieved task-specific autonomy but faced limitations in multi-task, privacy-sensitive settings. Federated learning (FL) emerged as a solution to distributed data privacy, successfully applied in domains like healthcare and finance, yet its application in robotics, especially for social behaviors, remains limited. Continual learning (CL) techniques such as Elastic Weight Consolidation (EWC) and Memory Aware Synapses (MAS) address catastrophic forgetting but are resource-intensive and less scalable in multi-robot systems. Recent advances combine FL and CL to enable lifelong learning, but challenges persist in balancing resource efficiency, privacy, and performance. This paper advances the field by proposing a feature separation strategy combined with generative replay, tailored for social robot behavior learning, filling a critical gap in scalable, privacy-preserving continual learning frameworks.

The core challenge lies in enabling multiple robots to learn social behaviors continuously without compromising privacy or incurring prohibitive resource costs. Traditional federated models aggregate entire neural networks, risking data leakage and high communication overhead. Furthermore, environments are dynamic, causing models to forget previously learned behaviors—a phenomenon known as catastrophic forgetting. Achieving efficient, privacy-preserving, and scalable continual learning in social robotics remains unresolved, especially under real-world constraints where data is non-i.i.d. and environments are highly variable. Addressing these issues requires innovative model architectures and learning strategies that can adapt in real-time, retain past knowledge, and operate within limited computational budgets.

The primary innovation is the FedRoot architecture, which isolates feature extraction layers for federated aggregation, reducing communication costs and enhancing privacy. This is complemented by FedLGR, integrating Variational Autoencoders for pseudo-rehearsal, which allows the system to generate synthetic representations of past experiences, mitigating forgetting. Unlike conventional methods that aggregate entire models, this approach selectively shares feature representations, enabling efficient multi-task continual learning. The framework is validated in simulated social environments, demonstrating superior resource efficiency and performance. This combination of feature separation and generative replay is novel in federated social robotics, offering a scalable solution for lifelong learning that balances privacy, efficiency, and adaptability.

• �� Model split into Root (feature extractor) and Top (task predictor); only Root aggregated across clients. • FedLGR adds a Variational Autoencoder (VAE) as Generator for pseudo-rehearsal of features. • Clients update Root and Top locally with different learning rates, ensuring stability. • Pseudo-samples of previous tasks are generated via VAE and interleaved with current data for training. • Regularization and pseudo-rehearsal balance new and old knowledge. • Periodic federated aggregation of Root parameters, with local Top and Generator remaining private. • Experiments conducted on MANNERS-DB, simulating social interactions, comparing resource use and accuracy across multiple clients and tasks.

The dataset comprises Unity-generated scenes of a Pepper robot in a simulated living room, with social actions rated on a Likert scale. Experiments involved 2 and 10 clients, predicting social appropriateness of 8 actions using MobileNetV2 features. Models trained with Adam optimizer, evaluated via RMSE and PCC. Baselines included FedAvg, FedProx, and variants with resource monitoring. Data augmentation tested robustness. Multi-round federated aggregation and continual learning scenarios assessed model stability, forgetting, and resource consumption. Hyperparameters tuned via grid search, with regularization coefficients optimized for best performance.

FedRoot reduced CPU usage from 19.06s to 2.55s (86.6%) and GPU from 0.63% to 0.24% (61.9%), with comparable prediction accuracy. FedLGR further improved stability, lowering RMSE to 0.198 and PCC to 0.563, outperforming non-rehearsal methods. The approach maintained high social behavior prediction quality while significantly decreasing resource demands, demonstrating scalability and robustness across social scenarios. Ablation studies confirmed the importance of feature separation and generative replay in balancing resource efficiency and continual learning performance.

This framework enables social robots in service, healthcare, and assistive domains to learn and adapt continuously while preserving user privacy. It supports multi-robot collaboration in dynamic environments, reducing computational costs and data leakage risks. The approach is suitable for deployment on resource-constrained platforms, facilitating real-time social behavior adaptation. Long-term, it can underpin autonomous systems capable of lifelong learning, improving human-robot interaction quality and acceptance in diverse settings.

The assumption of static feature distributions may limit adaptation in highly volatile environments. Pseudo-rehearsal quality depends on VAE performance, which may degrade in complex social scenarios. Scaling to larger models or real-time applications requires further optimization. Future work should explore multi-modal data fusion, online adaptation, and deployment on physical robots to address these limitations.