Fourier Features Let Agents Learn High Precision Policies with Imitation Learning
Fourier features improve point-cloud imitation, raising RoboCasa success from 13% to 34% and real-world normalized score from 14.8% to 40.2%.
Key Findings
Methodology
The paper inserts NeRF-style Fourier encoding before point-cloud encoders in an EDM-based diffusion imitation policy. Each point p=(x,y,z) is mapped with 16 logarithmically spaced bands from 4.0 m to 2 cm, producing 96 sinusoidal features. The representation is tested with PointPatch, PointPatch-attn, PCM, DP3, and PointTransformer. A decoder-only Transformer predicts action chunks, trained by score matching and sampled with DDIM; VariableJitter improves robustness.
Key Results
- On RoboCasa, PointPatch improves CloseDrawer from 34% to 72% and TurnOffSinkFaucet from 28% to 63%; mean success across the evaluated suite rises from 13% to 34%, with gains up to 20 percentage points.
- On ManiSkill3, Fourier features provide improvements of up to 7 percentage points across four grasping and tool-use tasks. On four real-robot tasks, normalized score increases from 14.8% to 40.2%, with smoother and more precise motions.
- Benefits persist across encoder architectures, benchmarks, and hyperparameter choices. Gains are strongest when point clouds contain rich geometric detail, but improvements also appear with limited detail, possibly through better optimization dynamics.
Significance
The work reframes unstable point-cloud policy performance as partly a representation problem. MLP-based encoders learn slowly varying functions first, whereas insertion, alignment, and contact require sharp geometric decision boundaries. A lightweight input transformation therefore improves data-efficient, high-precision imitation without requiring a new policy backbone. For robotics, this offers a practical bridge between explicit 3D geometry and the strong multimodal action modeling of diffusion policies.
Technical Contribution
The central contribution is a systematic transfer of NeRF-style Fourier positional encoding to point-cloud diffusion imitation learning. The authors hold the EDM decoder-only Transformer fixed while comparing multiple message-passing and aggregation encoders, isolating the effect of input frequency content. They combine fixed multiscale wavelengths, bounded coordinates, and VariableJitter augmentation, showing that the intervention needs little architectural change or task-specific regularization and can function as a general plug-in for point-cloud policies.
Novelty
Unlike PointNet, PointPatch, DP3, and PointTransformer variants that primarily process slowly changing Cartesian coordinates, this paper evaluates a shared Fourier representation across architectures, simulation suites, and a real robot. Adapt3R used Fourier features mainly for unseen-viewpoint generalization; this study targets diffusion-based imitation and high-precision manipulation, providing broader empirical evidence for the spectral-bias hypothesis.
Limitations
- The evaluation depends on depth-derived point clouds and removes color in simulation, so robustness to reflective materials, severe occlusion, depth artifacts, and semantic target identification is not established. Real-world validation covers only four tasks.
- Periodic encodings require suitable spatial bounds and wavelengths. Very short bands can overfit, long bands may not remove spectral bias, and unconstrained coordinates can create aliasing; the paper does not fully characterize compute scaling or frequency selection.
- Real tasks use only 75–102 human demonstrations each, leaving large-scale multi-robot transfer and deployment economics unresolved.
Future Work
Promising directions include learned or task-adaptive frequencies, rotation- and scale-aware encodings, and fusion with RGB foundation models, point maps, or tactile sensing. Larger real-world studies should test dynamic objects, severe occlusions, cross-robot transfer, sample efficiency, inference cost, and safety. A useful theoretical direction is to connect frequency choices with task geometry and diffusion score complexity.
AI Executive Summary
High-precision manipulation often hinges on millimetres: whether a peg is aligned, a drawer has entered its rail, or a grasped object should be repositioned. RGB policies must infer 3D structure from a 2D projection and are vulnerable to viewpoint, lighting, and scale. Point clouds provide explicit geometry, yet their performance remains inconsistent.
Gyenes and colleagues propose Fourier Features for Point Cloud Imitation Learning. Before PointPatch, PCM, DP3, or PointTransformer processing, each 3D coordinate is expanded into 96 multiscale sine-cosine features. An EDM diffusion policy, implemented with a decoder-only Transformer, then denoises action chunks using DDIM. The intuition is that MLPs have spectral bias: they learn smooth, low-frequency functions more readily than the sharp boundaries needed for precise contact.
The simple input change yields broad gains. On RoboCasa, average success rises from 13% to 34%; CloseDrawer improves from 34% to 72%. ManiSkill3 gains reach 7 percentage points, while four real-world tasks improve in normalized score from 14.8% to 40.2%. The study suggests that better frequency access can unlock existing 3D architectures rather than replacing them. Its evidence is still limited by four real tasks, depth quality, and wavelength choices, but Fourier encoding is a credible general-purpose component for precise point-cloud imitation.
Deep Analysis
Background
Diffusion Policy and EDM have advanced imitation learning by modeling multimodal action distributions, while PointNet, PointPatch, and PointTransformer provide explicit 3D geometry. RGB remains semantically powerful but suffers depth and viewpoint ambiguity. Most point-cloud encoders use MLPs on XYZ coordinates, creating a mismatch between smooth input processing and the fine contact geometry required by manipulation.
Core Problem
A policy must distinguish neighboring states that look almost identical but demand opposite actions—for example, insert a peg versus reposition it. Cartesian coordinates vary slowly locally, and neural networks preferentially learn low-frequency functions. Consequently, a policy may observe the correct 3D scene yet fail to represent the sharp score function or decision boundary needed for reliable alignment and contact.
Innovation
The paper replaces or augments Cartesian coordinates with multiscale Fourier features. It differs from architecture-specific prior uses by testing the same principle across PointPatch, PointPatch-attn, PCM, DP3, and PointTransformer, under a common diffusion backbone. Fixed wavelengths from 4 m to 2 cm expose both global and voxel-level structure, while VariableJitter reduces sensitivity to unhelpful frequencies without elaborate task-specific regularization.
Methodology
- �� Unproject depth using X_i=D_iK^{-1}(i,j,1)^T, transform each camera cloud to world coordinates, and concatenate views.
- �� Encode p=(x,y,z) with γ_k(x)=[sin(2πx/λ_k), cos(2πx/λ_k)], using L=16 logarithmic bands; three axes yield 96 features.
- �� Build local KNN neighborhoods around FPS-selected centers for PointPatch-family tokenization; use attention or max pooling where required by each encoder.
- �� Feed observation tokens, a frozen CLIP RN-50 language-goal token, a diffusion-noise token, and noisy action tokens to a decoder-only Transformer.
- �� Train the EDM score network with L_SM=E[α(σ)||Dθ(a+ε,o,g,σ)-a||²], then sample actions with a DDIM solver.
- �� Apply VariableJitter with σmax up to 5 mm in ManiSkill3, 2 mm in RoboCasa, and 1 mm in real experiments.
Experiments
The study evaluates 16 RoboCasa atomic tasks with 50 human demonstrations per task, four ManiSkill3 grasping/tool-use tasks with 500 expert demonstrations each, and four real tasks—Drawer, Cup-Stacking, Arranging, and Folding—with 75–102 human demonstrations each. Models use language goals and five random seeds, with three checkpoints per seed. Success is measured over 50 RoboCasa or 100 ManiSkill3 rollouts; bootstrap interquartile means and 95% confidence intervals are reported. Clouds are downsampled to 32,768 points and voxelized at 1 cm.
Results
Fourier features improve multiple encoders rather than one isolated architecture. RoboCasa PointPatch reaches 72% versus 34% on CloseDrawer and 63% versus 28% on TurnOffSinkFaucet; overall performance rises 13% to 34%. ManiSkill3 gains reach 7 percentage points. Real-world normalized score rises 14.8% to 40.2%. The consistent cross-encoder pattern supports the claim that frequency-aware input representation, not merely aggregation design, is the main driver.
Applications
The method is directly relevant to insertion, assembly, grasp correction, drawer operation, stacking, and tool use using calibrated depth cameras. Engineers can prepend Fourier encoding to an existing point-cloud encoder while retaining the EDM policy and language conditioning. It is especially attractive when demonstrations are limited, contact geometry is critical, and RGB-only depth inference is unreliable.
Limitations & Outlook
The method assumes usable depth, calibrated coordinates, and a meaningful spatial bound. Periodicity can cause aliasing outside that bound; frequency ranges also trade off resolution and overfitting. The paper does not fully quantify memory and inference costs, scaling with point count, or robustness to dynamic scenes and severe occlusion. Only four real tasks are tested, and simulation intentionally omits color. Future work should add adaptive frequencies, geometric invariances, RGB/tactile fusion, and broader safety-focused deployment studies.
Plain Language Accessible to non-experts
Imagine a factory worker assembling tiny parts. A normal ruler may tell her that two pieces are almost aligned, but not whether the gap is one millimetre to the left or right. A robot using ordinary 3D coordinates faces the same problem: nearby positions look too similar, so it may push ahead when it should adjust.
Fourier features give the worker many rulers at once. One ruler measures the whole table, another measures the work area, and another measures tiny gaps. Each location receives a distinctive pattern of marks, making nearby places easier to tell apart. The robot then watches the marked 3D scene and gradually chooses a smooth action through its action-generation system.
The researchers tested this idea in kitchen simulations, manipulation simulations, and on a real robot. RoboCasa average success rose from 13% to 34%; CloseDrawer rose from 34% to 72%; real-task normalized score rose from 14.8% to 40.2%. The lesson is simple: sometimes the robot already has the right information, but its internal ruler is too coarse.
ELI14 Explained like you're 14
Think of a game where you must plug a USB cable into a tiny port. The screen shows both objects, but a different camera angle can make the distance hard to judge. Robots have the same issue. A photo hides depth, and even a 3D scan may make two almost-identical positions feel like the same place.
This paper adds a clever map called Fourier features. Instead of writing only “this point is here,” the map gives every position several wave-like labels. Big waves describe the overall scene; tiny waves describe small gaps and edges. It is like viewing a school map with country, city, street, and house-number zoom levels at once. Nearby points get different signatures, so the robot can notice tiny geometric changes.
The researchers plug this map into several point-cloud networks and train a diffusion policy from demonstrations. The policy starts with a messy guess and cleans it step by step until it gets a useful action. On RoboCasa, average success jumps from 13% to 34%, and CloseDrawer goes from 34% to 72%.
Is it perfect? Nope! Bad depth scans, hidden objects, or poorly chosen wave sizes can still confuse it. But the idea is exciting because it improves existing robot brains without rebuilding them. With better cameras, touch sensors, and more real examples, this could help robots assemble parts, stack cups, and tidy objects much more accurately!
Glossary
Fourier features
A coordinate representation built from sine and cosine functions at multiple frequencies. Technically, it maps low-dimensional positions into a high-dimensional space where fine spatial variation is easier for neural networks to model.
The paper maps each XYZ point into 96 features before point-cloud encoding.
Spectral bias
The tendency of neural networks to learn slowly varying patterns before rapidly changing ones. In this paper, it explains why MLP encoders may miss sharp manipulation boundaries.
It motivates replacing raw Cartesian inputs with Fourier inputs.
EDM
Elucidated Diffusion Models learn a data distribution by adding noise and reversing that process. Here, the generated data are conditional action chunks.
It is the policy-training framework.
PointPatch
A point-cloud architecture that groups nearby points into local patches and converts them into tokens. KNN neighborhoods and FPS centers organize the unordered set.
It is a primary encoder family evaluated with and without Fourier features.
DDIM
A fast diffusion sampling method that performs reverse denoising in relatively few steps. It converts noise into an action sequence during policy execution.
Used for efficient action generation.
VariableJitter
An augmentation method that samples a different noise amplitude for each point cloud. This broadens training variation while avoiding a fixed-jitter tuning problem.
Used to improve robustness to irrelevant Fourier frequencies and geometric noise.
Open Questions Unanswered questions from this research
- 1 It remains unclear how well the method handles severe occlusion, moving objects, reflective surfaces, and unreliable depth. Large real-world benchmarks with synchronized RGB, depth, and tactile data are needed.
- 2 Fixed frequency bands are robust in these experiments but not necessarily optimal. Future work should determine whether learned frequencies improve sample efficiency without causing aliasing or overfitting.
Applications
Immediate Applications
Precision assembly
Industrial robot teams can add Fourier encoding before an existing point-cloud policy for peg insertion, connector alignment, and small-part assembly. Requirements include calibrated depth, bounded coordinates, and demonstrations; the expected benefit is fewer failed contacts and corrective motions.
Household manipulation
Service robots can use the method for closing drawers, stacking cups, arranging objects, and correcting grasps. Background cropping, voxelization, and language goals remain necessary, but the change is compatible with existing EDM-style policies.
Long-term Vision
General 3D manipulation foundation models
Fourier representations could become a standard input layer for large robot models combining depth, RGB, and touch. Major obstacles include sensor-coordinate consistency, computational scaling, dynamic scenes, cross-robot transfer, and safety certification.
Abstract
High-precision robotic manipulation requires fine-grained spatial reasoning that is often difficult to achieve with RGB-only policies due to depth ambiguity and perspective scale issues. Policies that leverage 3D information directly, such as those based on point clouds, offer a stronger geometric prior over purely image-based ones, yet their performance remains highly task-dependent. We hypothesize that this discrepancy may be due to the spectral bias of neural networks towards learning low frequency functions, which especially affects architectures conditioned on slow-moving Cartesian features. We thus propose to map point clouds from Cartesian space into high-dimensional Fourier space, effectively equipping the point cloud encoder with direct access to high-frequency features. We experimentally validate the use of Fourier features on challenging manipulation tasks from the RoboCasa and ManiSkill3 benchmarks and on a real robot setup. Despite their simplicity, we find that Fourier features provide significant benefits across diverse encoder architectures and benchmarks and are robust across hyperparameters. Our results indicate that Fourier features let policies leverage geometric details more effectively than Cartesian features, showing their potential as a general-purpose tool for point cloud-based imitation learning. We provide source code and videos on our project page: https://fourier-il.github.io/fourier-il