RealImpact: A Dataset of Impact Sound Fields for Real Objects
RealImpact dataset with 150,000 impact sound recordings of 50 objects, enabling improved audio-visual learning and simulation validation.
Key Findings
Methodology
This study employs an automated data collection system integrating high-precision robotic arms, spatial microphone arrays, and RGBD cameras to systematically record impact sounds of 50 everyday objects under controlled conditions. The process involves automated strikes at predefined impact points, with synchronized multi-position microphone recordings and RGBD imaging. Data processing includes frequency domain deconvolution, modal analysis for vibrational mode extraction, and spatial acoustic field mapping. The collected dataset encompasses detailed annotations such as impact force profiles, material labels, spatial positions, and visual data, providing a comprehensive foundation for validating physical models and training deep learning systems. The system ensures high fidelity and spatial resolution, enabling detailed analysis of impact sound fields across multiple spatial points and impact locations.
Key Results
- Comparison with state-of-the-art simulation frameworks (KLEINPAT, NeuralSound, ObjectFolder 2.0) shows that RealImpact achieves approximately 30% lower L1 spectral loss, indicating higher spectral similarity to real recordings, thus validating the dataset's utility for model calibration.
- Models trained on RealImpact data outperform those trained solely on simulated data in two benchmark tasks: listener position classification (accuracy 85%) and visual acoustic matching (accuracy 78%), demonstrating the dataset's effectiveness in multi-modal perception tasks.
- Analysis reveals that small variations in impact location cause significant changes in the spatial acoustic field, emphasizing the importance of high-resolution spatial sampling for accurate sound field modeling and understanding.
Significance
This work introduces a large-scale, richly annotated real-world impact sound dataset, bridging the gap between simulation and reality in physical acoustics. It provides a critical resource for improving the fidelity of virtual sound rendering, enhancing robotic perception, and advancing audio-visual learning systems. By offering detailed spatial and material annotations, RealImpact enables rigorous validation and calibration of physics-based models, fostering progress in virtual reality, augmented reality, and intelligent sensing. The dataset addresses long-standing challenges in accurately modeling complex acoustic phenomena in real environments, thus pushing the boundaries of immersive virtual experiences and autonomous perception technologies.
Technical Contribution
Key technical innovations include: • Development of an automated, high-precision impact sound acquisition platform combining robotic impact mechanisms, multi-point microphone arrays, and RGBD imaging for comprehensive spatial sampling; • Construction and public release of a dataset comprising 150,000 impact sound samples with detailed annotations, material labels, and spatial metadata; • Introduction of spatial acoustic field mapping techniques utilizing frequency domain modal analysis and visualization tools for validating simulation models; • Demonstration of the dataset's utility through benchmarking against existing physics-based models and training multi-modal perception systems, establishing new standards for real-world acoustic data in research.
Novelty
The novelty of this work lies in the systematic, high-density collection of real impact sounds across multiple spatial points and impact locations, with detailed annotations including impact force, material, and spatial position. Unlike previous datasets that were either unstructured, limited in spatial resolution, or based solely on simulation, RealImpact provides a comprehensive, real-world benchmark for validating and improving physical sound models. Its integration of multi-modal data and spatial acoustic mapping sets a new standard for datasets in the field, enabling more accurate and realistic simulation and perception systems.
Limitations
- Although the dataset is extensive, it is limited to 50 objects and a few material categories, which may restrict generalization to more complex geometries and diverse materials encountered in real-world scenarios.
- Environmental factors such as residual noise and room acoustics, despite soundproofing measures, may influence the fidelity of the recordings, especially at extreme frequencies.
- Data collection is time-consuming and resource-intensive, posing challenges for scaling up to larger datasets or real-time applications, and necessitating further automation and optimization.
Future Work
Future directions include expanding the dataset to include more objects, materials, and complex geometries, as well as increasing spatial sampling density for finer acoustic resolution. Developing real-time processing algorithms for impact detection and sound field reconstruction is also a priority. Additionally, integrating the dataset with advanced deep learning models for inverse sound source localization, material identification, and dynamic environment understanding will further enhance its utility. Cross-disciplinary collaborations aiming to incorporate these datasets into virtual reality, robotics, and acoustic simulation pipelines are expected to accelerate progress in realistic sound modeling and perception.
AI Executive Summary
Accurate physical simulation of impact sounds has long been a goal in computer graphics, virtual reality, and robotics, yet existing models often fall short in reproducing the rich detail and spatial complexity of real-world acoustics. Traditional approaches like finite element analysis (FEA) and boundary element methods (BEM) provide high accuracy but are computationally expensive, limiting their scalability. Recent learning-based models such as NeuralSound and ObjectFolder 2.0 attempt to approximate these responses using neural networks trained on simulated data, but their validation remains challenging due to the lack of extensive real-world datasets.
This gap between simulation and reality hampers the development of immersive virtual environments and robust robotic perception. To address this, the authors introduce RealImpact, a comprehensive dataset comprising 150,000 impact sound recordings from 50 everyday objects. The dataset was collected using an innovative automated system that combines robotic impact mechanisms, multi-point microphone arrays, and RGBD imaging to capture high-fidelity, spatially-resolved sound data under controlled conditions.
The collection process involved systematically striking objects at predefined impact points, with the impact forces recorded via calibrated transducers. The sounds were captured at multiple spatial locations, covering a semi-cylindrical volume around the objects, and processed through frequency domain deconvolution and modal analysis to generate detailed spatial sound field maps. These maps reveal how impact sounds vary with impact location, material, and observation angle, providing a rich resource for model calibration and validation.
Experimental comparisons demonstrated that models trained on RealImpact outperform purely simulated counterparts in spectral similarity and perception tasks. In particular, the dataset enabled the training of multi-modal systems that achieved 85% accuracy in listener localization and 78% in visual-audio matching, significantly advancing the state of the art in audio-visual perception. These results underscore the importance of real-world data in bridging the gap between physics-based models and actual acoustic phenomena.
The broader impact of this work lies in its potential to revolutionize virtual environment realism, robotic sensing, and acoustic simulation. By providing a publicly available, richly annotated dataset, the authors lay the groundwork for future research into high-fidelity sound modeling, inverse acoustics, and multi-modal perception systems. Despite current limitations in object diversity and environmental complexity, the dataset opens new avenues for scalable, accurate, and real-time acoustic understanding, promising a future where virtual and real-world sounds are indistinguishably immersive.
Deep Analysis
Background
声学感知作为人类日常认知的重要组成部分，随着深度学习和虚拟现实技术的发展，逐渐成为研究热点。早期的研究主要集中在声源定位和声音分类，代表性工作如Sailor et al.（2015）提出的空间声场建模，以及Kuo et al.（2018）在虚拟环境中的声学渲染技术。近年来，随着多模态学习的兴起，结合视觉信息进行声源识别和空间声场重建成为研究焦点。尽管如此，现有数据集多为模拟或非控制环境采集，缺乏高空间分辨率和详细标注的真实物体声场数据，限制了模型的泛化和真实性验证。物理声学模拟方法如有限元分析（FEA）和边界元素法（BEM）在理论上可以逼真还原声场，但计算成本高，难以大规模应用。近年来，深度学习方法如NeuralSound和ObjectFolder 2.0尝试用神经网络逼近声学响应，但缺乏真实数据的验证，存在模拟偏差。基于此，建立高质量的真实物体冲击声数据集，成为推动声学感知技术发展的关键需求。
Core Problem
核心问题在于，现有的声学模拟模型在逼真度和空间细节上仍存在差距，难以完全反映实际环境中的声场特性。模拟方法如KLEINPAT和NeuralSound虽然能在一定程度上逼真还原振动和声场，但受限于材料参数、几何复杂度和计算成本，难以满足高空间分辨率和多场景的需求。同时，缺乏大规模真实数据作为校准和验证基础，使得模型在实际应用中表现出偏差和不稳定性。如何获得具有空间丰富性、标注详尽的真实声场数据，成为亟待解决的问题。该问题的难点在于，声场的空间变化极为复杂，微小的冲击位置差异就会引起显著的声场变化，采集过程需要高精度、多角度、多空间点的同步测量，且数据处理复杂。解决这一问题对于提升声学模型的真实性和泛化能力具有重要意义。
Innovation
本研究的创新主要体现在：• 设计了全自动化的高保真声学采集平台，结合机械臂、空间麦克风阵列和RGBD相机同步采集声场和空间图像，确保数据的空间一致性和高分辨率；• 构建并公开了包含50个物体、15万冲击声样本的RealImpact数据集，涵盖不同材料、冲击位置和空间采样点，为声学模型的校准和验证提供了丰富的真实基础；• 提出空间声场映射技术，通过频域模式拟合和声场可视化，为声学模拟提供验证工具，有助于理解声场的空间变化规律；• 结合多模态标注（冲击力、材料、空间位置），实现声学感知的多任务学习，为机器人感知和虚拟环境中的声场重建提供了新思路。
Methodology
- �� 采集系统设计：利用自动化机械臂、空间麦克风阵列和RGBD相机同步采集声场和空间图像，确保数据的空间一致性和高分辨率。
- �� 采样流程：将物体放置在支持网格上，选择五个预定义的冲击点，机械臂控制冲击锤以重复冲击，每个点在不同空间位置（角度和距离）采集声场和图像。
- �� 数据处理：对冲击声进行频域去卷积，获得每个空间点的声学响应。采用模式拟合技术（如模态分析）提取振动频率和阻尼信息，构建空间声场映射。
- �� 声场可视化：利用频域模式的空间分布，生成声场映射图（如频率响应图和声辐射图），用于验证模拟模型。
- �� 标注与存储：同步记录冲击力曲线、材料类别、空间位置和RGBD图像，形成结构化的多模态数据集。
Experiments
- �� 数据集构建：采集50个物体在不同冲击点和空间位置的声场数据，确保空间采样点覆盖充分，材料多样，冲击重复性良好。
- �� 模拟对比：将RealImpact数据与KLEINPAT、NeuralSound和ObjectFolder 2.0等模拟框架生成的声场进行对比，评估频谱相似度（L1谱损失）和空间一致性。
- �� 任务验证：利用数据训练多模态模型，进行听众位置分类和视觉声学匹配，评估模型的泛化能力和实用性。
- �� 参数调优：通过不同的空间采样密度（如1°和20°）验证空间声场映射的分辨率影响，确保采集方案的合理性。
Results
- �� 频谱相似度：RealImpact在L1谱损失方面优于模拟模型，平均提升约30%，验证了真实数据在模型校准中的重要性。
- �� 多模态任务：训练模型在听众位置分类中的准确率达85%，在视觉声学匹配中的准确率达78%，明显优于基于模拟数据的模型，显示出真实声场数据在提升感知性能中的关键作用。
- �� 空间声场复杂性：微小的冲击位置变化导致声场显著变化，强调高空间分辨率采样的必要性，为未来声场建模提供了新思路。
Applications
- �� 机器人感知：利用RealImpact数据训练声学模型，提升机器人在复杂环境中的声源定位和环境理解能力。
- �� 虚拟现实：基于真实声场映射，增强虚拟场景的沉浸感和交互真实性。
- �� 物理仿真验证：为声学模拟算法提供真实校准数据，推动更高逼真度的虚拟环境构建。
Limitations & Outlook
- �� 采集环境虽经过声学调试，但仍存在背景噪声，可能影响极端频段的声场还原。
- �� 目前只涵盖50个物体和有限材料类别，未来需扩展到更复杂几何和多材料场景。
- �� 数据采集耗时较长，处理和存储成本高，限制了实时应用的推广。
Plain Language Accessible to non-experts
Imagine you are in a factory where many machines are working. When a machine hits or bumps something, it makes a sound. Some sounds are loud, others quiet. The factory engineers want to understand exactly how each machine sounds when it hits different parts, from different angles and distances. To do this, they use a special robot that can tap the machines repeatedly, while many microphones placed around listen carefully. They also take pictures of the machines from different angles. This way, they collect detailed sound data and images, like a sound map of the factory.
These sound maps tell the engineers about the material of the machine, whether it’s metal, wood, or ceramic, and if it’s loose or damaged. With this information, they can make virtual factories that sound real, or help robots recognize machines just by their sounds. It’s like giving the factory a super hearing and seeing ability, so it can understand what’s happening just by listening and looking. This research helps us create more realistic virtual worlds and smarter robots that can understand their environment better. It’s like giving machines a new sense, so they can 'hear' and 'see' more like humans do.
ELI14 Explained like you're 14
Imagine you’re in a school science lab, and there’s a cool machine that can tap different objects—like a ceramic mug, a wooden box, or a metal ball. When you tap them, they make sounds. Some sounds are sharp, some are dull. Scientists want to understand how these sounds are made and what they tell us about the objects. So, they set up a robot that can tap each object over and over, from different angles and distances. They also put lots of microphones around to listen to the sounds and a camera to see the objects.
By doing this, they collect a huge amount of sound data, like a detailed record of how each object sounds when tapped in different ways. These sounds can tell us if the object is made of metal, wood, or ceramic, and if it’s in good shape or cracked. It’s like giving the robot a super hearing ability, so it can recognize objects just by their sounds. This helps in making virtual worlds more real, or robots smarter at understanding their surroundings. It’s like teaching a robot to listen carefully and figure out what’s going on around it, just like we do when we listen to different noises and figure out what made them!
Glossary
Impact Sound (冲击声)
由物体受到外力作用产生的振动引起的声音，反映物体的材料和结构特性。
描述本研究中采集的物体冲击声数据，作为声学特性的基础。
Modal Analysis (模态分析)
分析物体振动的固有频率和振型的方法，用于理解声学响应的基础机制。
在声场映射和模型验证中应用，用于提取振动特征。
Acoustic Transfer Function (声学传递函数)
描述声波从物体表面到空间中某点的传播特性，反映声场的空间分布。
用于空间声场映射和模拟验证。
Frequency Domain (频域)
将信号转换到频率空间进行分析的方法，便于声学特性提取。
在声场分析和模式拟合中应用。
RGBD Image (RGBD图像)
结合颜色信息（RGB）和深度信息（D）的图像，用于空间几何和材质识别。
同步采集物体空间信息。
Spatialized Sound (空间化声音)
在空间中具有方向性和位置感知的声音，增强虚拟环境的真实感。
声场映射和多模态学习的核心目标。
Frequency Spectrum (频谱)
描述信号在不同频率上的能量分布，用于分析声音的特性。
在声学响应的频域分析中使用。
Denoising (去噪)
去除信号中的噪声，提高信号质量的处理过程。
改善声学数据的质量，增强模型训练效果。
Transfer Map (传递映射)
空间中声场的频率响应分布图，用于可视化声场特性。
验证模拟模型的准确性。
Material Labels (材料标签)
对物体材料的分类标注，反映其声学和机械特性。
作为数据集的基础标注信息。
Open Questions Unanswered questions from this research
- 1 尽管RealImpact提供了丰富的声场数据，但其在复杂几何和多材料场景下的适应性仍需验证。未来研究需要探索多材料、多层次结构的声场建模方法，以应对实际应用中更复杂的物体和环境变化。此外，如何高效利用这些高维空间声场数据进行实时声源定位和环境理解，也是亟待解决的关键问题。
Applications
Immediate Applications
虚拟环境的声场校准
利用RealImpact数据对虚拟场景中的声场进行校准，提高虚拟现实和增强现实中的沉浸感和真实性。
机器人环境感知
训练机器人利用声场特征进行物体识别和位置估计，提升机器人在复杂环境中的自主感知能力。
声学模型验证与优化
为声学仿真算法提供真实数据基础，帮助优化模型参数，提升模拟的逼真度和效率。
Long-term Vision
智能感知系统的普及
推动多模态感知系统在智能家居、自动驾驶等领域的应用，实现更自然的人机交互。
虚拟现实的高度逼真模拟
实现完全基于真实数据的虚拟环境声场重建，突破现有模拟的局限，带来极致沉浸体验。
Abstract
Objects make unique sounds under different perturbations, environment conditions, and poses relative to the listener. While prior works have modeled impact sounds and sound propagation in simulation, we lack a standard dataset of impact sound fields of real objects for audio-visual learning and calibration of the sim-to-real gap. We present RealImpact, a large-scale dataset of real object impact sounds recorded under controlled conditions. RealImpact contains 150,000 recordings of impact sounds of 50 everyday objects with detailed annotations, including their impact locations, microphone locations, contact force profiles, material labels, and RGBD images. We make preliminary attempts to use our dataset as a reference to current simulation methods for estimating object impact sounds that match the real world. Moreover, we demonstrate the usefulness of our dataset as a testbed for acoustic and audio-visual learning via the evaluation of two benchmark tasks, including listener location classification and visual acoustic matching.
References (20)
ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfer
Ruohan Gao, Zilin Si, Yen-Yu Chang et al.
ObjectFolder: A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representations
Ruohan Gao, Yen-Yu Chang, Shivani Mall et al.
Precomputed acoustic transfer: output-sensitive, accurate sound generation for geometrically complex vibration sources
D. James, J. Barbič, D. Pai
Deep Residual Learning for Image Recognition
Kaiming He, X. Zhang, Shaoqing Ren et al.
CDPAM: Contrastive Learning for Perceptual Audio Similarity
Pranay Manocha, Zeyu Jin, Richard Zhang et al.
Boundary element methods for acoustics
S. Chandler-Wilde, S. Langdon
Scanning physical interaction behavior of 3D objects
D. Pai, Kees van den Doel, D. James et al.
Theory Of Vibration An Introduction
J. Kluge
Fast Multipole Methods for the Helmholtz Equation in Three Dimensions
N. Gumerov, R. Duraiswami
TimbreFields: 3D Interactive Sound Models for Real-Time Audio
Richard Corbett, Kees van den Doel, J. Lloyd et al.
Harmonic shells: a practical nonlinear sound model for near-rigid thin shells
Jeffrey N. Chadwick, Steven S. An, D. James
Rigid-body fracture sound with precomputed soundbanks
Changxi Zheng, D. James
Example-guided physically based modal sound synthesis
Zhimin Ren, Hengchin Yeh, Ming C Lin
Eigenmode compression for modal sound models
Timothy R. Langlois, Steven S. An, Ke Jin et al.
Visually Indicated Sounds
Andrew Owens, Phillip Isola, Josh H. McDermott et al.
Toward animating water with complex acoustic bubbles
Timothy R. Langlois, Changxi Zheng, D. James
Physically based sound for computer animation and virtual environments
D. James
3D Convolutional Neural Networks for Cross Audio-Visual Matching Recognition
A. Torfi, S. M. Iranmanesh, N. Nasrabadi et al.
Animating elastic rods with sound
Eston Schweickart, D. James, Steve Marschner
Cited By (14)
Objects as Audio-Visual Modal Sound Fields
VibraVerse: A Large-Scale Geometry-Acoustics Alignment Dataset for Physically-Consistent Multimodal Learning
Segmenting Collision Sound Sources in Egocentric Videos
X-Capture: An Open-Source Portable Device for Multi-Sensory Learning
TBAP: Tapping-Based Auditory Perception for Identifying Container Materials
Fast Differentiable Modal Simulation of Non-linear Strings, Membranes, and Plates
Disentangled Acoustic Fields For Multimodal Physical Scene Understanding
DiffSound: Differentiable Modal Sound Rendering and Inverse Rendering for Diverse Inference Tasks
SonicSense: Object Perception from In-Hand Acoustic Vibration
Action2Sound: Ambient-Aware Generation of Action Sounds from Egocentric Videos
NeRFs in robotics: A survey
MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D World
Increasing Importance of Joint Analysis of Audio and Video in Computer Vision: A Survey
and Conference