Hearing Hands: Generating Sounds from Physical Interactions in 3D Scenes

3D scene reconstruction has advanced rapidly through neural rendering and implicit scene representations, with methods such as NeRF making scenes look visually plausible. Yet most reconstructions remain silent, because they model geometry and appearance rather than interaction consequences. In parallel, audio generation has progressed from autoregressive models like WaveNet to diffusion-based systems, but these methods typically lack grounding in physical action. Hearing Hands sits at the intersection of these two trends: it treats hand interaction as a missing modality for scene understanding and asks whether sound can be reconstructed from motion alone. The result is a view of 3D scenes as potentially interactive, not merely visual.

The core problem is to predict the sound produced when a human hand physically interacts with objects in a 3D scene. This is difficult because the mapping from motion to sound is many-to-many: force, contact location, material, and motion pattern all matter. The input is sequential and high-dimensional, while the output is a waveform with subtle temporal structure. The challenge is not just to generate plausible audio, but to tie it to the correct action semantics so that the sound matches the implied physical event.

• �� New task formulation: the paper defines sound prediction directly from 3D hand trajectories, rather than from video dubbing or static scene audio modeling. This focuses the problem on interaction, which is the true source of the sound.

• �� Motion-conditioned generation: 3D hand poses are used as conditioning signals, giving the model a temporally aligned representation of how the hand moves through space. This is more informative than a single image or a categorical action label.

• �� Rectified flow as the generator: instead of hand-crafted Foley rules or retrieval-and-splice systems, the model learns a continuous generative flow from noise to audio, conditioned on motion. This is well-suited to subtle, graded acoustic variation.

• �� Interactive scene view: the paper reframes a 3D reconstruction as a scene that can be queried acoustically. Sound becomes part of the scene’s semantics, not just a post-processing layer.

• �� Data collection: record a person manipulating objects inside a 3D scene, and capture synchronized audio. The paired data provide the supervision needed to connect motion and sound.

• �� Motion extraction: convert the video into 3D hand trajectories/poses. These trajectories serve as the main condition because they encode what the hand does over time.

• �� Generative modeling: train a rectified flow model that maps noise to audio along a learned continuous vector field, guided by the hand-motion condition. Intuitively, the model learns how to “push” random noise toward the correct sound.

• �� Query at test time: given a new sequence of hand poses, the user can ask the model to predict the corresponding interaction sound, enabling what-if auditory simulation.

• �� Learning objective: by optimizing on action-sound pairs, the model captures latent factors such as material response, contact intensity, and interaction type, rather than memorizing fixed sound templates.

The experiments focus on whether generated sounds preserve the perceptual cues of real physical interaction. The authors evaluate synthesized audio against real recordings and report human-observer judgments of realism and distinguishability. The qualitative finding is that the sounds are often hard to tell apart from real ones, while still reflecting the correct action and material properties. The provided text does not state the dataset name, sample count, or numerical scores, but it is clear that the evaluation is centered on paired interaction clips and subjective listening tests.

The main result is that the model produces audio that listeners often judge as realistic enough to be confused with the original recording. This is especially important because it suggests the model captures not only surface acoustics but also the hidden structure of contact events. A second result is controllability: new hand-motion sequences can be queried at test time, showing that the system generalizes beyond the training clips rather than simply memorizing them. Together, these results indicate that 3D hand trajectories are a strong conditioning signal for interaction sound generation.

The most immediate application is in VR/AR, where virtual objects could emit plausible sounds when users move their hands near or onto them. Another use case is digital content production, where designers and animators can prototype interaction audio from motion without recording every sound manually. In robotics and embodied AI, such a system could also serve as a perceptual prior for understanding object properties from contact events, complementing vision-based scene interpretation.

The excerpt does not provide full benchmark numbers or detailed failure analysis, so the method’s robustness under heavy occlusion, long-horizon action chains, or crowded multi-object scenes remains uncertain. Its dependence on accurate 3D hand trajectories and clean paired audio also suggests sensitivity to sensing noise and synchronization errors. Future work should address multi-person interactions, richer acoustics, and stronger physical grounding to improve generalization and control.