LabDex: A Hierarchical Benchmark for Dexterous Manipulation in Laboratories
LabDex evaluates dexterous lab manipulation through hierarchical tasks, validating its effectiveness in real and simulated environments.
Key Findings
Methodology
LabDex unifies real and simulated platforms, providing standardized task definitions, demonstrations, and evaluation protocols. Tasks are divided into three levels: Atomic Skills, Compositional Skills, and Long-Horizon Laboratory Workflows, supporting cross-level evaluation.
Key Results
- For Atomic Skills, the π0.5 model achieved an average success rate of 0.57, significantly outperforming ACT at 0.34 and DP at 0.02.
- In Compositional Skills, π0.5 averaged 0.73 completed atomic skills, compared to 0.52 for ACT, while DP completed none.
- For Long-Horizon Workflows, π0.5 completed 0.62 atomic skills on average, ACT 0.40, and DP none.
Significance
LabDex provides a standardized evaluation framework for dexterous manipulation in chemical labs, addressing gaps in existing benchmarks for multi-finger dexterity and lab-specific tasks, advancing autonomous lab robot research.
Technical Contribution
LabDex is the first to unify real and simulated platforms under a single framework, offering systematic evaluation methods for lab operations and supporting training and evaluation of existing robotic policies.
Novelty
LabDex is the first hierarchical benchmark for chemical lab dexterous manipulation, supporting multi-finger dexterity and lab-specific tasks, unlike existing benchmarks.
Limitations
- LabDex's success rate in long-horizon workflows remains limited, indicating challenges for existing methods in complex tasks.
- Current models perform poorly in insertion stages, limiting continuous task execution.
Future Work
Future work could explore improving model stability in long-horizon tasks, enhancing support for multi-finger dexterity, and expanding to other lab domains.
AI Executive Summary
Laboratory automation is crucial for accelerating scientific discovery. Existing benchmarks fail to capture dexterous hand use, real lab interactions, and multi-stage procedures. LabDex introduces a hierarchical task framework, unifying real and simulated platforms, providing standardized task definitions, demonstrations, and evaluation protocols. Experimental results show LabDex effectively supports training and evaluation of existing robotic policies across different levels of lab dexterous manipulation tasks, validating its design and demonstration data. However, the success rate in long-horizon workflows remains limited, indicating challenges for existing methods in complex tasks. Future work could explore improving model stability in long-horizon tasks, enhancing support for multi-finger dexterity, and expanding to other lab domains.
Deep Analysis
Background
Laboratory automation integrates machine learning, robotics, and modular platforms to improve experimental efficiency and accelerate scientific discovery. Recent work increasingly integrates cognitive reasoning with embodied AI to enhance system autonomy and adaptivity. However, existing systems often depend on predefined protocols, bespoke hardware, fixed interfaces, and task-specific workstations, limiting their flexibility, generalizability, and scalability.
Core Problem
Existing benchmarks fail to jointly capture dexterous hand use, real-world laboratory interactions, and multi-stage experimental procedures, limiting systematic training and evaluation. Chemical laboratory operations naturally exhibit a clear hierarchical structure, where complete experimental procedures can typically be decomposed into fundamental atomic operations.
Innovation
LabDex unifies real and simulated platforms under a common framework, providing systematic evaluation methods for lab operations. Its hierarchical design supports the evaluation of end-task performance and enables the analysis of how fundamental dexterous skills compose and influence more complex laboratory operations.
Methodology
- �� LabDex unifies real and simulated platforms, providing standardized task definitions, demonstrations, and evaluation protocols. • Tasks are divided into three levels: Atomic Skills, Compositional Skills, and Long-Horizon Laboratory Workflows. • Experimental results validate the effectiveness of LabDex's task design and demonstration data.
Experiments
Experiments were conducted in real and simulated environments using a Franka Research 3 robot arm and XHand dexterous hand. Representative robot learning methods, including ACT, Diffusion Policy, and π0.5, were evaluated. Each task was evaluated over 50 trials, with manipulated object positions randomized.
Results
Experimental results show that π0.5 achieved an average success rate of 0.57 for Atomic Skills, significantly outperforming ACT at 0.34 and DP at 0.02. In Compositional Skills, π0.5 averaged 0.73 completed atomic skills, compared to 0.52 for ACT, while DP completed none. For Long-Horizon Workflows, π0.5 completed 0.62 atomic skills on average, ACT 0.40, and DP none.
Applications
LabDex provides a standardized evaluation framework for dexterous manipulation in chemical labs, supporting training and evaluation of existing robotic policies and advancing autonomous lab robot research.
Limitations & Outlook
LabDex's success rate in long-horizon workflows remains limited, indicating challenges for existing methods in complex tasks. Current models perform poorly in insertion stages, limiting continuous task execution. Future work could explore improving model stability in long-horizon tasks, enhancing support for multi-finger dexterity.
Plain Language Accessible to non-experts
Imagine a kitchen where a robot needs to complete a series of complex cooking tasks. First, it needs to master basic skills like chopping, stirring, and pouring. These skills are like the Atomic Skills in LabDex. Next, the robot needs to combine these basic skills to complete small tasks, like making a salad or cooking a soup, which corresponds to the Compositional Skills in LabDex. Finally, the robot needs to complete an entire cooking process, from preparing ingredients to serving dishes, which is the Long-Horizon Laboratory Workflows in LabDex. LabDex helps robots complete complex tasks in labs through this hierarchical task structure.
ELI14 Explained like you're 14
Imagine you're playing a super complex game with lots of levels, and each level has different tasks. LabDex is like this game, helping robots complete various tasks in a lab. First, the robot learns basic skills like picking up and placing things, just like the basic levels in a game. Then, it combines these skills to complete small tasks, like the intermediate levels. Finally, the robot completes a whole set of experiments, like the ultimate level in the game. LabDex makes robots smarter and able to do more in the lab!
Glossary
LabDex
LabDex is a hierarchical benchmark for evaluating dexterous manipulation in chemical labs, supporting multi-finger dexterity and lab-specific tasks.
LabDex unifies real and simulated platforms, providing standardized task definitions, demonstrations, and evaluation protocols.
Atomic Skills
Atomic Skills are fundamental operations like grasping, placing, and inserting, which can be independently evaluated and reused across tasks.
LabDex organizes lab operations into Atomic Skills, Compositional Skills, and Long-Horizon Workflows.
Compositional Skills
Compositional Skills consist of multiple Atomic Skills executed sequentially according to lab procedure logic, completing specific experimental functions.
LabDex's task structure supports evaluation of Compositional Skills.
Long-Horizon Laboratory Workflows
Long-Horizon Workflows consist of multiple Compositional Skills connected according to experimental protocols, representing complete or relatively complete experimental procedures.
LabDex evaluates the robot's ability to complete complex experimental procedures.
Franka Research 3
Franka Research 3 is a robot arm equipped with an XHand dexterous hand, used for executing lab manipulation tasks.
LabDex's real-world platform uses the Franka Research 3 robot arm and XHand dexterous hand.
Open Questions Unanswered questions from this research
- 1 How to improve robot stability and success rate in long-horizon workflows? Current methods perform poorly in complex tasks, requiring improvement.
- 2 How to enhance support for multi-finger dexterity? Existing benchmarks provide limited evaluation in this area.
Applications
Immediate Applications
Chemical Lab Automation
LabDex can be used to evaluate and improve automation in chemical labs, helping robots better complete experimental tasks.
Long-term Vision
Cross-Domain Lab Automation
LabDex's framework can be extended to other lab domains, enhancing automation levels and driving scientific discovery.
Abstract
Autonomous laboratories hold great promise for accelerating scientific discovery. To achieve this vision, robots are supposed to dexterously manipulate diverse labware and instruments and execute long-horizon, state-dependent experimental procedures. Yet existing benchmarks do not jointly capture dexterous hand use, real-world laboratory interactions, and multi-stage experimental procedures, limiting systematic training and evaluation. To bridge this gap, we introduce LabDex, a large-scale real-world dataset and benchmark for dexterous manipulation in chemistry laboratories, organized around a hierarchical task taxonomy spanning atomic skills, compositional tasks, and long-horizon experiments. First, LabDex is cross-platform and, for the first time, unifies real-world and simulation platforms under a common framework, providing standardized task definitions, demonstrations, and evaluation protocols. Second, LabDex is large-scale and systematically organizes chemistry laboratory operations into three interconnected levels: Atomic Skills, which characterize fundamental dexterous manipulation capabilities; Compositional Skills; and Long-Horizon Laboratory Workflows. This hierarchical design not only supports the evaluation of end-task performance, but also enables the analysis of how fundamental dexterous skills compose and influence more complex laboratory operations. We conduct cross-level evaluations of representative robot learning methods in both real-world and simulation environments. The experimental results validate the effectiveness of the LabDex task design and demonstration data, and show that the benchmark supports the training and systematic evaluation of existing robotic policies across laboratory dexterous manipulation tasks at different levels, providing a foundation for further research and development of autonomous laboratory robots.