Mimic Intent, Not Just Trajectories

Recent advances in robot imitation learning, including Behavior Cloning, inverse reinforcement learning, and transformer-based vision-language models, have achieved notable success in controlled environments. However, these methods predominantly focus on surface-level trajectory replication, which limits their ability to generalize across diverse tasks and environmental variations. Action tokenization techniques, such as VQ-VAE, have been introduced to encode actions into discrete semantic units, but often lack explicit constraints linking tokens to meaningful behaviors. Multi-scale hierarchical models attempt to capture high-level intent and low-level details, yet they rely on heuristic or time-domain reconstruction, which may not effectively disentangle semantics. The introduction of spectral analysis, particularly frequency domain decomposition, offers a promising direction for explicitly modeling behavioral intent as low-frequency components, providing a more robust and interpretable representation for manipulation tasks.

Despite progress, current imitation learning models struggle with generalization and skill transfer due to their reliance on surface trajectory imitation. They lack explicit modeling of the underlying behavioral intent, leading to overfitting and poor adaptation in new environments. Existing action tokenization approaches often focus on compression rather than semantics, and hierarchical models do not enforce spectral disentanglement. This results in models that cannot efficiently transfer skills via one-shot demonstrations or adapt to environmental disturbances. The core challenge is to develop a representation that captures the high-level intent while maintaining detailed execution, enabling flexible planning, robust transfer, and interpretability in complex manipulation tasks.

The key innovation lies in integrating frequency domain spectral analysis with multi-scale residual quantization to explicitly disentangle behavior intent from execution details. The SDAT framework transforms actions into frequency spectra, enforcing spectral reconstruction constraints at multiple scales. Low-frequency components encode global intent, while high-frequency residuals capture fine details. This spectral separation provides a semantically meaningful latent space. The multi-scale autoregressive policy predicts tokens progressively, guided by intent, enabling long-horizon planning and one-shot transfer. Unlike prior methods, this approach offers explicit interpretability, robustness, and transferability, bridging the gap between low-level kinematics and high-level semantics.

• �� Segment demonstration trajectories into overlapping chunks using sliding windows.
• �� Apply Discrete Cosine Transform (DCT) to convert each chunk into frequency domain.
• �� Use a multi-scale Variational Autoencoder (VAE) to encode residual spectral features at increasing resolutions.
• �� Impose spectral reconstruction loss at each scale, ensuring low-frequency components (S1) capture global intent, while finer scales model residual details.
• �� During training, reconstruct actions progressively from coarse to fine, enforcing spectral disentanglement.
• �� Construct a multi-scale autoregressive policy that predicts tokens at each scale conditioned on coarser scales, using hybrid attention mechanisms.
• �� During inference, inject an intent token from a demonstration to enable one-shot skill transfer, and generate detailed actions through autoregression.
• �� Decode predicted tokens into continuous trajectories with a learned decoder, ensuring actions are executable and semantically meaningful.

Experiments involve evaluating MINT on LIBERO, MetaWorld, CALVIN, and real robot platforms. The training uses spectral reconstruction, codebook, and auxiliary losses, with hyperparameters like number of scales K and frequency thresholds. Success rates, robustness under environmental disturbances, and transfer performance are measured, comparing against baselines such as Diffusion Policy, WorldVLA, and UniVLA. Ablation studies analyze the impact of multi-scale spectral constraints versus single-scale or time-domain methods. The models are tested in both simulation and real-world settings, with success rates exceeding 98%, and real robot transfer success surpassing 29% over baselines. The experiments demonstrate the effectiveness of spectral disentanglement in improving generalization and transferability.

MINT achieves success rates over 98% across LIBERO, MetaWorld, and CALVIN, outperforming existing methods by 3-5%. In LIBERO-Plus, success improves by 15% under disturbances. In real robot tests, only 20 demonstrations enable effective zero-shot transfer, with a 29% higher success rate than traditional models. Ablation confirms that multi-scale spectral reconstruction significantly enhances behavior representation, especially in long-horizon tasks. The approach maintains robustness against environmental variations, validating the spectral disentanglement's role in improving generalization and transfer.

This framework is suitable for industrial automation, service robots, and autonomous systems requiring skill transfer and adaptation. Minimal demonstrations suffice for new tasks, reducing data collection costs. Its robustness makes it ideal for unstructured environments, enabling robots to learn complex behaviors efficiently. Future integration with reinforcement learning could further enhance adaptability, supporting autonomous operation in dynamic, real-world scenarios.

The reliance on linear DCT may limit performance in highly nonlinear behaviors, especially in complex environments. Training complexity and computational demands are high due to multi-scale spectral reconstruction. Handling rapid, unpredictable actions remains challenging, and further research into nonlinear spectral methods is needed. Real-time deployment requires optimization to reduce inference latency and resource consumption.