PRISM: Personalized Robotic Dataset Generation via Image-based Scene and Motion Synthesis
PRISM generates personalized robotic datasets from a single image and instruction, constructing semantically aligned digital scenes with instance diversity.
Key Findings
Methodology
PRISM employs an end-to-end pipeline: scene generation via VLM-based reasoning, asset retrieval, and scene assembly; demonstration synthesis through instruction parsing and task-motion planning (TAMP); and dataset construction with visual randomization. It extracts objects from a single RGB-D image, retrieves matching 3D assets, constructs semantically and geometrically aligned scenes, and synthesizes executable trajectories. Motion-aware grasp ranking improves demonstration quality, while trajectory-preserving visual randomization enhances data diversity and robustness.
Key Results
- Policies trained on PRISM datasets outperform baselines on LIBERO and LIBERO-Plus, achieving up to 100% success rate. In real-world tasks, success exceeds 80%, demonstrating strong transferability. The approach maintains performance across environment variations, confirming its robustness.
- Comparison between digital twin and cousin scenes shows the latter balances fidelity and diversity, leading to consistent success rates. Motion-aware grasping improves imitation, raising success by ~20%. Visual randomization boosts generalization to appearance changes.
- Cross-scenario tests reveal PRISM’s data enhances policy robustness, with significant improvements in unseen environments, validating its practical value for real-world deployment.
Significance
This work addresses the critical challenge of personalized data generation for robot adaptation. By combining semantic-geometry aligned scene synthesis with instance diversity, PRISM significantly improves transfer learning efficiency. It reduces reliance on costly teleoperation and digital twin reconstruction, offering scalable, target-specific data generation. The approach paves the way for robots to autonomously adapt to diverse, real-world environments, accelerating deployment in industry, service, and domestic applications.
Technical Contribution
PRISM introduces a comprehensive pipeline integrating VLM reasoning, large-scale asset retrieval, task and motion planning, and trajectory-preserving visual randomization. It innovatively synthesizes environment-specific datasets from a single image, balancing environment fidelity and instance diversity. The motion-aware grasp ranking enhances demonstration quality, while the randomization scheme improves data efficiency and policy robustness, setting a new standard for personalized robotic dataset generation.
Novelty
This is the first method to generate semantically and geometrically aligned digital scenes from a single RGB-D image, combined with automated demonstration synthesis. It uniquely integrates scene reconstruction, asset retrieval, and task planning with motion-aware grasping and trajectory-preserving randomization, enabling scalable, target-specific dataset creation without human teleoperation. This approach bridges the gap between target environment fidelity and instance diversity, advancing the state-of-the-art in simulation-based robot training.
Limitations
- Currently limited to rigid objects and Franka arms; extension to deformable objects or different robot types remains challenging.
- Scene reconstruction from a single image can be inaccurate under occlusion, affecting demonstration quality.
- Computational costs are high, especially for large-scale data synthesis, requiring further optimization for real-time applications.
Future Work
Future efforts will focus on supporting soft and deformable objects, multi-robot scenarios, and multi-view scene reconstruction. Improving scene understanding under occlusion, reducing computational overhead, and integrating multi-modal inputs like videos will further enhance the robustness and scalability of PRISM, broadening its application scope.
AI Executive Summary
Robotic manipulation in complex, real-world environments demands scalable, personalized training data. Traditional methods like teleoperation are precise but costly and hard to scale, while simulation approaches often lack target environment fidelity. PRISM introduces a novel solution: from a single RGB-D image and natural language instruction, it automatically constructs a semantically aligned digital scene that preserves the environment’s structure while introducing instance diversity.
This process leverages vision-language models (VLMs) for scene understanding, large-scale asset retrieval for scene assembly, and task and motion planning (TAMP) to synthesize executable demonstrations. To improve demonstration quality, PRISM incorporates motion-aware grasp ranking, selecting grasps aligned with natural approach directions, and employs trajectory-preserving visual randomization, which varies environmental appearance without altering robot motions. These innovations significantly enhance data efficiency and policy robustness.
Extensive experiments demonstrate that policies trained on PRISM-generated datasets outperform baseline methods, achieving success rates up to 100% in simulation and over 80% in real-world tasks. The approach maintains performance across environment variations, confirming its robustness and transferability. By constructing target-specific yet diverse datasets, PRISM effectively bridges the gap between simulation and reality, enabling scalable, personalized robot training.
This work has profound implications for deploying robots in industrial, service, and domestic settings, reducing data collection costs and improving adaptation to new environments. Future directions include extending support to soft objects, multi-robot systems, and multi-view scene reconstruction, aiming to further democratize autonomous robot learning in complex, dynamic environments.
Deep Analysis
Background
机器人自主学习经历了深度学习、迁移学习和模拟技术的快速融合。代表性工作如OpenVLA、Octo模型和数字孪生技术，推动了场景理解和策略迁移。尽管如此，这些方法多依赖大量人工标注或高成本遥控，场景多样性不足，难以满足个性化需求。模拟环境虽能扩展数据，但难以与实际环境保持一致，影响迁移性能。近年来，场景合成、数字场景重建和示范生成成为研究热点，但仍存在目标环境对齐和实例多样性难以兼得的问题。
Core Problem
核心问题在于如何在保持环境结构一致的基础上，生成丰富多样的训练数据，以提升策略的泛化能力。遥控操作成本高，难以规模化；纯模拟场景缺乏真实感和多样性，影响迁移效果。单图像重建受遮挡影响大，可能导致重建误差，影响演示质量。这些限制阻碍了机器人在复杂环境中的自主适应能力，亟需一种高效、目标导向的数据生成方法。
Innovation
PRISM的创新包括：1）利用单一RGB-D图像构建语义几何对齐的数字场景，结合资产检索实现环境多样性；2）引入任务规划自动合成可执行轨迹，避免人工遥控；3）设计运动感知的抓取策略，提升示范的可模仿性；4）采用轨迹保持的视觉随机化，增强数据多样性和模型鲁棒性。这些创新共同解决了环境目标一致性与实例多样性难以兼得的问题，显著提升迁移性能。
Methodology
- �� 场景生成：提取对象信息，检索匹配资产，构建语义几何一致的虚拟场景；
- �� 演示合成：将自然指令转化为动作序列，随机扰动场景，利用任务规划生成轨迹；
- �� 抓取策略：根据运动感知排名抓取姿势，优化轨迹平滑性；
- �� 数据构建：在保持轨迹不变的基础上进行视觉随机化，采集多样化训练样本，提升策略泛化能力。
Experiments
采用LIBERO和LIBERO-Plus数据集，生成400条轨迹，训练策略并在不同环境中测试。对比数字孪生和场景表亲效果，验证环境多样性对迁移的影响。真实环境测试中，成功率最高达100%，优于基线。还进行了消融实验，验证运动感知和随机化策略的贡献。结果显示PRISM在迁移性能和鲁棒性方面表现优异。
Results
PRISM训练的策略在LIBERO和LIBERO-Plus中成功率显著提升，最高达100%，在真实操控任务中表现优越，成功率超过80%。环境多样性增强模型的泛化能力，跨模拟和真实环境迁移效果优异。运动感知抓取策略和轨迹保持随机化显著改善示范质量和数据效率，验证了方法的有效性。
Applications
适用于工业自动化、家庭服务和个性化机器人开发。只需提供单一环境图像和自然指令，即可快速生成训练数据，降低成本。未来可扩展到软体物体、多机器人系统，推动机器人自主学习在复杂环境中的应用。
Limitations & Outlook
目前仅支持刚性物体和Franka机械臂，软体和变形物体的场景重建仍具挑战。单图像重建对遮挡敏感，可能影响演示质量。计算成本较高，尤其在大规模数据合成时，仍需优化效率。未来需解决多模态融合和多机器人协作等难题。
Plain Language Accessible to non-experts
想象你在厨房准备一道菜。你只有一张厨房的照片，但想让机器人帮你做饭。PRISM就像用这张照片，自动画出厨房的布局和物品位置，然后模拟出机器人可以操作的场景。它还会根据你的指令，设计机器人怎么拿起锅、放菜，甚至模拟不同的厨房布置，让机器人学会在各种厨房里都能做菜。这样，不用每次都教机器人具体操作，它就能自己学会应对不同的厨房环境，帮你做饭变得更快更智能。
ELI14 Explained like you're 14
想象你要让机器人帮你整理房间，但你只有一张房间的照片。PRISM就像用这张照片，帮你画出房间的布局，然后模拟出机器人怎么搬东西。它还能让房间看起来不一样，比如换个灯光或家具颜色，让机器人学会在不同的房间都能工作。这样，你不用每次都教它怎么做，只要给一张照片和一句话指令，它就能自己学会怎么帮你整理。这个方法让机器人变得更聪明，也更容易在真实世界中帮忙。
Glossary
Vision-Language Model (视觉-语言模型)
一种结合视觉和语言信息理解的深度学习模型，用于场景理解和指令解析。
在场景生成和指令解析中，用于提取对象和理解任务。
任务与运动规划 (Task and Motion Planning, TAMP)
一种规划方法，结合任务级动作序列和机械臂运动轨迹，生成可执行的机器人动作。
用于合成示范轨迹，确保动作的物理可行性。
数字场景 (Digital Scene)
虚拟环境中的场景模型，反映真实环境的结构和对象布局。
用于模拟和训练机器人，避免实际操作成本。
资产检索 (Asset Retrieval)
从大规模资产库中找到与场景对象匹配的3D模型。
实现场景的多样化和语义一致性。
视觉随机化 (Visual Randomization)
在保持动作轨迹不变的情况下，改变环境的视觉外观以增强模型鲁棒性。
提升策略对外观变化的适应能力。
Open Questions Unanswered questions from this research
- 1 如何进一步提升场景重建在遮挡复杂环境中的准确性？
- 2 多模态输入（如视频、多角度图像）对场景理解的影响？
- 3 软体物体和变形物的模拟与合成技术如何突破？
Applications
Immediate Applications
个性化机器人训练
为不同用户快速生成适应其环境的训练数据，降低定制成本，提升迁移效果。
工业场景模拟
在工厂或仓库中快速构建虚拟场景，用于机器人调试和任务规划，节省时间和成本。
Long-term Vision
自主适应复杂环境
未来机器人能在未知或变化环境中自主生成训练数据，实现高效自主学习。
Abstract
Recent advances in large-scale pretrained vision-language-action models have improved robot policy learning, but directly deploying such policies in user-specific environments remains challenging due to limited generalization, which inevitably requires collecting a dataset tailored to the target environment. Teleoperation yields well-aligned data but is costly and difficult to scale, whereas simulation scales easily but struggles to resemble the target environment and generate task-specific trajectories. To meet both simultaneously, we propose PRISM, an end-to-end pipeline that generates personalized robotic datasets from a single image and a natural-language instruction. PRISM constructs digital cousin scenes that are semantically and geometrically aligned with the user environment yet diverse at the instance level, and synthesizes executable demonstrations without human teleoperation. Extensive experiments show that policies trained on PRISM-generated datasets outperform those trained on baseline-generated datasets on LIBERO and LIBERO-Plus, achieve up to 100\% success rate on three real-world manipulation tasks, and maintain stronger performance when evaluated in environments that differ from those seen during training.