Grasp-MPC: Closed-Loop Visual Grasping via Value-Guided Model Predictive Control
Grasp-MPC uses a learned value function inside MPPI control, reaching 67.2% simulated and 74.4% overall real-world grasp success.
Key Findings
Methodology
Grasp-MPC combines M2T2 grasp prediction, CuRobo motion planning, and MPPI control. A PointNet++-MLP value model is trained on 2,105,296 successful and failed trajectories involving 8,515 Objaverse objects and about 115 million states. Online control minimizes CuRobo collision and smoothness costs plus a learned grasp cost, allowing feedback-based correction of noisy targets.
Key Results
- On 5,400 FetchBench trials with annotated grasp poses, Grasp-MPC achieved 74.9% success, close to the open-loop OSC oracle at 79.0%, and above IQL at 64.4% and the other learned closed-loop baselines.
- With ±2 cm translation and ±18° orientation perturbations, success was 60.3%, a 14% relative drop according to the paper, whereas open-loop OSC lost about 40%. With M2T2-predicted poses, Grasp-MPC reached 67.2%, versus 63.6% for OSC and 36.5% for imitation-learning baselines.
- In real experiments, success rates were 86.7% on an empty table, 67.7% on a cluttered table, and 70.0% near a cluttered shelf, yielding 74.4% overall versus 41.1% for CuRobo-GraspAPI.
Significance
The work connects the generalization of large-scale visual learning with the feedback, constraints, and safety of MPC. It addresses a persistent robotics gap: a grasp can be geometrically plausible yet fail because the object moves, perception is noisy, or the arm collides with clutter. Rather than extracting a policy directly from offline data, the method learns a value landscape and lets online optimization choose actions. This design is relevant to warehouse picking, recycling, shelf replenishment, and household manipulation, where objects are novel and execution conditions are not controlled.
Technical Contribution
The central contribution is a visual cost-to-go model embedded in MPPI. Training uses yt=ct+γVϕ′(xt+1), loss (yt−Vϕ(xt))², γ=0.99, and an exponential-moving-average target network. Online optimization uses CGrasp-MPC=CCuRobo+ωCgrasp with ω=1000. The value network fuses PointNet++ point-cloud features with an MLP for proprioception, uses full and randomly rendered partial clouds, and is noise-augmented for deployment. Successful and failed trajectories provide a broad supervision signal without requiring direct policy extraction.
Novelty
Relative to distance-based grasp costs, few-demonstration CV-MPC, and closed-loop policies restricted to clean tabletops, Grasp-MPC uses more than two million visual trajectories spanning 8,515 Objaverse objects to learn a success-oriented cost. Its fundamental novelty is architectural: an imperfect open-loop grasp predictor supplies initialization, while a learned value function converts that prediction into a soft, feedback-correctable objective. The paper therefore combines data-driven semantics with model-based safety rather than replacing one with the other.
Limitations
- Training covers motions from a pre-grasp pose only 15 cm from the grasp. Consequently, the moving-object experiment achieved 60% success, and performance for long-range approach, fast motion, or major occlusion remains unknown.
- Trajectories were generated with CuRobo and not individually physics-validated to accelerate collection. The value model can therefore face out-of-distribution states; ensemble-based pessimistic uncertainty bounds were not beneficial in the reported experiments.
Future Work
The authors propose training on larger grasping motions to improve dynamic-object performance. Natural extensions include real-world failure-data fine-tuning, uncertainty-aware value ensembles, calibrated out-of-distribution detection, better segmentation under occlusion, and systematic studies of control frequency, latency, gripper morphology, and tactile feedback. These additions could turn a strong reactive grasp controller into a more general manipulation policy.
AI Executive Summary
Robotic grasping becomes difficult when objects are novel, scenes are cluttered, and the target moves during execution. Open-loop systems predict a grasp and execute it, so a small pose error can cause failure. Closed-loop reinforcement- or imitation-learning systems can react, but prior methods often rely on small datasets, clean tabletops, or unsafe behavior near obstacles.
Grasp-MPC combines an M2T2 grasp predictor, CuRobo motion planning, and MPPI model predictive control. Its key component is a PointNet++-MLP value function trained on 2,105,296 synthetic trajectories across 8,515 Objaverse objects. The trajectories include both successes and failures. At deployment, the value estimates the cost-to-go from the segmented point cloud and end-effector pose; MPPI then optimizes this task cost together with collision avoidance and smooth execution.
The results show substantial robustness. In 5,400 FetchBench trials, success with annotated grasp poses was 74.9%, near OSC’s 79.0% oracle. Under ±2 cm and ±18° perturbations, success remained 60.3%; with M2T2-predicted poses it reached 67.2%. In the real world, performance was 86.7% on an empty table, 67.7% on a cluttered table, and 70.0% at a cluttered shelf, for 74.4% overall versus 41.1% for CuRobo-GraspAPI. Yet the method mainly learns the final 15 cm of approach, depends on accurate segmentation, and still lacks calibrated uncertainty for out-of-distribution states.
Deep Analysis
Background
Open-loop systems such as M2T2 and ACRONYM-style grasping generalize to novel objects but cannot correct execution online. RL, imitation learning, FetchBench’s Transformer policy, Diffusion Policy, and IQL add feedback but suffer from limited data, domain mismatch, or unsafe collision behavior. MPC offers real-time constrained optimization, but grasping requires a task cost richer than distance to a predicted pose.
Core Problem
Given a segmented object cloud, a rough grasp target, and robot state, the system must perform 6-DoF grasping under unknown object dynamics. The main bottlenecks are prediction error, object motion after pre-grasping, clutter-induced collisions, partial visibility, and the mismatch between geometric proximity and actual grasp success. A useful controller must therefore combine semantic success information with hard motion constraints.
Innovation
- �� Learn a visual cost-to-go from over two million success and failure trajectories rather than only imitate actions.
- �� Fuse PointNet++ geometry features with proprioceptive MLP features.
- �� Insert the learned value directly into MPPI alongside CuRobo collision and smoothness costs.
- �� Train on full and partial point clouds with Gaussian noise, improving transfer from Isaac Sim to RealSense observations.
- �� Preserve an open-loop predictor for initialization while making the final approach reactive and constraint-aware.
Methodology
- �� Data generation: for 8,515 Objaverse objects, GraspGen poses are shifted 15 cm to create pre-grasps, then perturbed by U(-0.04,0.04) cm translation and U(-0.04π,0.04π) rotation; CuRobo plans trajectories.
- �� Supervision: feasible grasp terminal and near-terminal states receive cost 0; other states receive cost 1. The value is trained with yt=ct+γVϕ′(xt+1), γ=0.99.
- �� Perception: the deployment system uses M2T2 candidates, SAM-Track segmentation, and NVBlox signed-distance fields; real-world pre-grasp offset is 10 cm.
- �� Control: CuRobo’s MPPI samples joint-acceleration sequences, uses Euler integration, predicts a horizon, minimizes CCuRobo+1000Σγt′−tV(xt′), executes the first action, and replans from new observations.
Experiments
Simulation uses UR10, Robotiq 2F-140, Isaac Sim, and FetchBench: 90 scenes with 60 problems each, totaling 5,400 cases. Success means lifting an object at least 1 cm. Baselines are OSC, a Transformer policy, Diffusion Policy, and IQL. Tests cover annotated poses, ±2 cm/±18° noise, and M2T2 predictions. Real trials use two RealSense L515 cameras, SAM-Track, and NVBlox across empty-table, cluttered-table, and cluttered-shelf scenes, 30 trials per scene. Training used batch size 1,536, learning rate 10^-4, and one RTX 4090 for six days.
Results
With annotated poses, Grasp-MPC reached 74.9%, versus 79.0% for OSC and 64.4% for IQL. Under pose noise it achieved 60.3%, while OSC dropped by about 40%. With M2T2 predictions, it achieved 67.2%, ahead of OSC at 63.6% and imitation-learning methods at 36.5%. Real-world success was 86.7%, 67.7%, and 70.0% across the three scenes, versus 50.0%, 43.3%, and 30.0% for CuRobo-GraspAPI. Moving-object trials achieved 60% over 30 attempts.
Applications
The approach fits warehouse picking, recycling, shelf replenishment, and household object retrieval, especially when object identities are unseen and obstacles surround the target. Deployment requires depth sensing, target segmentation, environment signed-distance modeling, a calibrated gripper, and a sufficiently fast GPU control stack. Its strongest immediate value is safer recovery from imperfect grasp predictions rather than replacing perception or planning entirely.
Limitations & Outlook
The method assumes usable segmentation, candidate grasps, and obstacle geometry; severe occlusion or perception errors can corrupt the value input. Synthetic trajectories were generated without per-trajectory physics validation, and the learned model may be unreliable outside its data distribution. Training covers only the final 15 cm, while MPPI and point-cloud inference impose computation. The paper does not report broad ablations of camera failure, latency, gripper type, or uncertainty calibration. Longer motions, real failure replay, tactile sensing, and conservative ensembles are natural next steps.
Plain Language Accessible to non-experts
Imagine a warehouse worker picking an unfamiliar package from a crowded shelf. An open-loop robot is like someone who looks once, decides exactly where the hand should go, closes their eyes, and reaches. If the package shifts or another box blocks the path, the plan fails. Grasp-MPC behaves more like a careful worker who keeps looking while reaching.
Its experience comes from a huge computer practice room. Researchers gave a virtual hand 8,515 kinds of objects and recorded more than two million attempts. The system learned which hand positions usually lead to a successful pickup and which ones tend to fail. It does not memorize one route; it learns a map of promising situations.
Before moving, the robot receives a rough suggestion about where to grasp. A planner finds a safe route to the object, while the controller repeatedly tests possible next movements in its imagination. It chooses a movement that is both likely to succeed and unlikely to hit nearby obstacles, then looks again and repeats. This is why it can recover from a shifted target.
The trade-off is that it needs good cameras, object outlines, computing power, and training that matches reality. Its practice mostly covers the final 15 centimetres, so long approaches and very fast moving objects remain difficult.
ELI14 Explained like you're 14
Imagine playing a game where you control a robotic hand trying to grab a new item in a messy room. A normal automatic move is like pressing “go” and never checking the screen again. If the item slides a little, the hand misses. Grasp-MPC is more like a smart player who keeps checking the screen and changes the move every moment.
How does it learn? Researchers created a virtual world with 8,515 different objects and made a simulated robot try more than two million times. The robot recorded which attempts worked and which failed. From this, it learned a kind of success map: given the object’s shape and the hand’s current position, it estimates whether the next move is promising.
First, M2T2 gives a rough guess for the grasp. CuRobo plans a route that avoids obstacles. Then MPPI imagines many possible short futures, scores them, and chooses one that balances “likely to grab” with “don’t crash.” After moving a little, it observes the scene again and repeats. That feedback is the big difference from a one-shot plan!
The scores are impressive: 67.2% success with predicted grasps in simulation, and 74.4% overall in three real settings. But it is not unbeatable. If the camera cannot separate the target, if the object moves extremely fast, or if the hand starts far away, it may struggle. The next level is giving it longer practice routes, better uncertainty awareness, and perhaps touch sensors.
Glossary
Model Predictive Control (MPC)
A controller that predicts several future steps, executes the first action, then replans after new observations. Technically, it solves a constrained receding-horizon optimization problem.
Grasp-MPC uses MPPI within CuRobo to repeatedly optimize the grasp approach.
Value Function
A function estimating expected future cost from a state; lower values indicate states more likely to lead to success. Here it maps point-cloud geometry and relative end-effector pose to grasp cost.
The learned value is inserted into MPC as the task objective.
MPPI
Model Predictive Path Integral control is a sampling-based algorithm that evaluates perturbed control sequences and weights them by predicted cost.
CuRobo uses MPPI to optimize joint-acceleration sequences online.
PointNet++
A hierarchical neural architecture for unordered 3D point sets. It learns local geometric features and aggregates them into object-level representations.
It encodes segmented object clouds before fusion with proprioceptive features.
FetchBench
A benchmark for robotic grasping in cluttered environments, containing procedurally generated and ACRONYM objects.
The paper evaluates 5,400 simulated cases in an Isaac Sim adaptation of FetchBench.
Signed Distance Field
A spatial representation storing the distance to the nearest obstacle, with sign indicating inside or outside. It supports collision checking and clearance-aware control.
NVBlox supplies the environment field used by CuRobo planning and MPC.
Open Questions Unanswered questions from this research
- 1 Reliability outside the training distribution remains unresolved. The field needs calibrated uncertainty, conservative value ensembles, and tests linking prediction confidence to collision and grasp failure.
- 2 The dataset emphasizes the final 15 cm, leaving long-range approach, fast target motion, persistent occlusion, and contact-rich manipulation largely untested.
- 3 The effects of segmentation errors, camera placement, control latency, gripper morphology, and tactile feedback lack systematic ablations.
Applications
Immediate Applications
Cluttered shelf picking
Warehouse robots can combine depth cameras, SAM-Track, NVBlox, and CuRobo with Grasp-MPC to correct imperfect grasps between shelf obstacles. The reported cluttered-shelf success rate was 70.0%, compared with 30.0% for the open-loop baseline.
Novel-object sorting
Recycling and e-commerce systems can use M2T2 to propose candidates and the learned value controller to recover from pose errors. Required prerequisites include reliable segmentation, calibrated camera extrinsics, a compatible gripper, and real-time GPU inference.
Long-term Vision
General-purpose household manipulation
With longer-motion training, language-specified targets, and tactile feedback, the method could support kitchen and home robots handling unfamiliar objects in changing scenes. Safety certification, severe occlusion, and the cost of real failure data remain major barriers.
Abstract
Grasping of diverse objects in unstructured environments remains a significant challenge. Open-loop grasping methods, effective in controlled settings, struggle in cluttered environments. Grasp prediction errors and object pose changes during grasping are the main causes of failure. In contrast, closed-loop methods address these challenges in simplified settings (e.g., single object on a table) on a limited set of objects, with no path to generalization. We propose Grasp-MPC, a closed-loop 6-DoF vision-based grasping policy designed for robust and reactive grasping of novel objects in cluttered environments. Grasp-MPC incorporates a value function, trained on visual observations from a large-scale synthetic dataset of 2 million grasp trajectories that include successful and failed attempts. We deploy this learned value function in an MPC framework in combination with other cost terms that encourage collision avoidance and smooth execution. We evaluate Grasp-MPC on FetchBench and real-world settings across diverse environments. Grasp-MPC improves grasp success rates by up to 32.6% in simulation and 33.3% in real-world noisy conditions, outperforming open-loop, diffusion policy, transformer policy, and IQL approaches. Videos and more at http://grasp-mpc.github.io.