Automated Weld Seam Recognition and 3D Mapping for Robotic Post Processing Using Photogrammetry and Semantic Segmentation
Combining semantic segmentation and photogrammetry, the method localizes weld seams with ~4.2mm RMSE using smartphone images.
Key Findings
Methodology
The approach captures multiple overlapping images of the workpiece from different viewpoints, then employs the SegFormer deep learning model for semantic segmentation to identify weld regions. These masks undergo contour filtering and centerline extraction to produce weld splines in 2D. Simultaneously, RealityScan software performs photogrammetric reconstruction, generating a 3D point cloud and camera parameters. Using depth maps and camera poses, the 2D weld splines are projected into 3D space. Multi-view fusion consolidates these points into a continuous weld path in the reconstructed model, enabling spatial localization of weld seams for robotic applications.
Key Results
- Using 68 high-resolution images, the 3D model achieved a geometric accuracy of approximately 3mm. The weld detection accuracy was 85%, with the projected weld path exhibiting an RMSE of 4.2mm, demonstrating millimeter-level localization suitable for robotic path planning.
- The method effectively reduced data acquisition effort compared to traditional laser scanning, with a robust detection rate across varying lighting conditions. The fusion of multi-view data minimized noise, producing a reliable weld path for subsequent automation tasks.
- Experimental validation confirmed the feasibility of combining deep learning-based detection with photogrammetric reconstruction, providing a low-cost, efficient alternative for large workpiece weld inspection.
Significance
This work addresses the high cost and time-consuming nature of traditional laser-based weld inspection, proposing a low-cost, flexible solution using common smartphone cameras. By integrating deep learning and photogrammetry, it offers a scalable approach for industrial environments, enabling rapid, coarse weld localization that guides high-precision measurements. The method reduces reliance on expensive equipment, facilitating broader adoption of automated welding quality control and robotic post-processing, thus advancing Industry 4.0 initiatives.
Technical Contribution
The key innovation lies in the seamless integration of semantic segmentation with multi-view photogrammetric reconstruction, enabling the extraction of 3D weld paths from simple images. The use of SegFormer enhances detection robustness, while depth-assisted projection ensures accurate spatial placement. This approach differs from prior point-cloud-based methods by leveraging low-cost imaging and deep learning, providing a practical, scalable pipeline for large-scale industrial applications. It bridges the gap between 2D detection and 3D localization, offering a new paradigm for robotic welding workflows.
Novelty
This is the first work to combine deep learning-based semantic segmentation with photogrammetric 3D reconstruction specifically for large workpieces in industrial settings. Unlike existing methods relying solely on laser scanners or structured light, this approach uses standard smartphone images, significantly reducing costs. The multi-view fusion and projection strategy enable millimeter-level weld localization without expensive equipment, representing a novel integration of AI and photogrammetry for industrial automation.
Limitations
- Model performance is sensitive to lighting and surface reflections, which can cause missed detections or false positives, especially in reflective or poorly lit environments.
- Photogrammetric reconstruction accuracy is limited by surface texture and image quality, leading to potential deviations in weld path localization.
- The current workflow is optimized for simple, continuous welds; complex or irregular weld geometries may require more advanced segmentation and path extraction algorithms.
Future Work
Future efforts will focus on expanding training datasets to improve robustness across diverse conditions, integrating surface normal and tool orientation estimation for smoother path planning, and developing real-time processing capabilities. Additionally, combining multiple sensing modalities, such as laser and visual data, could further enhance accuracy and reliability, paving the way for fully autonomous robotic welding systems in industrial environments.
AI Executive Summary
In modern manufacturing, ensuring weld quality is critical for structural integrity. Traditional inspection methods rely on laser scanners or structured light sensors, which, while accurate, are costly and slow, especially for large workpieces. This bottleneck limits the efficiency of automated welding and post-processing workflows. To address this, researchers have explored low-cost, flexible alternatives leveraging everyday devices like smartphones.
This study introduces an innovative pipeline that combines multi-view smartphone photography, deep learning-based semantic segmentation, and photogrammetric 3D reconstruction to localize weld seams rapidly and accurately. The core idea is to capture multiple overlapping images of the workpiece, then use a pre-trained SegFormer model to identify weld regions in each image. These 2D masks undergo contour filtering and centerline extraction to produce weld splines. Simultaneously, RealityScan software reconstructs the workpiece into a 3D model, providing geometric context.
By projecting the 2D weld splines into 3D space using depth maps and camera poses, the method generates a spatial weld path. Multi-view fusion consolidates these points, resulting in a continuous weld trajectory with an RMSE of 4.2mm. The experiments demonstrate that this approach achieves millimeter-level localization accuracy, sufficient for guiding robotic grinding, inspection, or finishing tasks. The process is low-cost, scalable, and adaptable to various industrial scenarios.
Looking ahead, the research aims to enhance model robustness under challenging lighting and surface conditions, incorporate surface normal and tool orientation estimation, and develop real-time processing capabilities. The ultimate goal is to enable fully autonomous robotic weld inspection and post-processing, significantly improving efficiency and reducing costs in manufacturing. This work paves the way for more accessible, intelligent automation solutions in industry, bridging the gap between low-cost imaging and high-precision robotic operations.
Deep Analysis
Background
随着工业自动化的推进，焊缝检测成为确保焊接质量的核心环节。传统方法主要依赖激光扫描或结构光设备，虽然能提供高精度的几何信息，但成本高昂且操作繁琐，难以在大面积场景中高效应用。近年来，深度学习与摄影测量技术的结合为大规模工件的快速检测提供了新思路。已有研究如Ruiz de Oña等的Photogrammetric Suite WeldMap，主要应用于小型工件，难以应对复杂场景。激光点云虽精确，但在大面积场景中效率低下。本文旨在结合普通手机图像、多视角摄影测量与深度学习模型，突破现有技术瓶颈，实现大工件焊缝的快速粗略定位。
Core Problem
大工件焊缝检测面临多重挑战：高精度扫描成本高、耗时长，且在复杂表面反光或纹理不足时识别困难。传统激光扫描设备虽能提供精确几何信息，但在大面积场景中效率低下，且设备成本高昂。如何利用低成本设备实现快速、准确的焊缝粗略定位，成为工业自动化中的关键难题。现有方法多依赖点云或结构光，受环境影响大，难以满足现场快速检测需求。本文提出利用普通手机图像结合深度学习与摄影测量，解决这一瓶颈。
Innovation
本研究的创新点在于：1）引入SegFormer深度学习模型进行焊缝语义分割，提升检测鲁棒性；2）结合摄影测量技术实现多视角三维重建，降低成本；3）将二维焊缝轮廓投影到三维模型中，获得空间路径。不同于传统依赖激光点云的方案，该方法利用普通手机图像，极大降低设备成本和操作复杂度。多视角融合技术确保焊缝路径的空间连续性，为机器人路径规划提供了可靠基础。这一创新组合实现了从二维检测到三维定位的无缝转换，推动工业自动化向低成本、智能化迈进。
Methodology
- �� 图像采集：用手机从多个角度拍摄工件，确保焊缝在不同视角下都被覆盖。• 语义分割：利用预训练的SegFormer模型对每张图像进行焊缝掩码预测，区分焊缝与背景。• 轮廓提取：对掩码进行滤波，连接焊缝区域，利用主成分分析估算焊缝方向，沿方向切片提取中心点，形成焊缝样条线。• 三维投影：结合摄影测量生成的深度图和相机参数，将焊缝样条线投影到三维空间。• 多视角融合：将不同视角的焊缝点合并，去除重复点和异常值，得到连续的三维焊缝路径。• 最终输出：生成空间焊缝模型，为机器人路径规划提供基础。
Experiments
实验使用一块40cm×10cm×5cm钢管焊接件，配备AprilTag标记以实现尺度校准。采集68张高分辨率图像，覆盖所有可视面。利用RealityScan软件进行摄影测量重建，获得点云模型，几何误差约3mm。焊缝检测采用SegFormer模型，训练数据包括388张公开图像，识别准确率达85%。将焊缝轮廓投影到三维模型，误差（RMSE）为4.2mm。结果验证了多视角图像结合摄影测量的可行性，整体流程实现了毫米级空间定位，为后续机器人操作提供了基础。
Results
重建模型几何误差约3mm，验证了流程的精度。焊缝识别准确率达85%，在不同光照条件下表现稳定。焊缝空间定位误差为4.2mm，满足工业路径规划需求。多视角融合有效减少了噪声和误差，整体流程实现了快速、低成本的焊缝粗略定位，为机器人后续精测和处理提供了可靠基础。这些结果表明，结合深度学习与摄影测量的方案具有实际应用潜力。
Applications
该技术适用于工业现场焊缝快速检测、机器人路径规划、自动打磨和检验。只需普通手机和少量标记，即可实现大面积工件的焊缝粗略定位。未来可结合实时处理技术，提升现场应用效率，推动工业自动化升级。长远来看，该方法有望实现全自动化焊缝检测与机器人作业，降低人工成本，提升生产效率。
Limitations & Outlook
模型对光照和反光敏感，识别在强反光或低光环境下效果下降。摄影测量在纹理不足或表面反光强烈区域存在误差，影响焊缝路径精度。流程主要适用于规则焊缝，复杂焊缝形态可能需要更复杂的后处理算法。未来需增强模型鲁棒性，提升在多变环境中的适应能力，并实现实时处理。
Plain Language Accessible to non-experts
想象你在厨房准备一道菜，你需要找到锅里的调料瓶。传统的方法可能用手去摸，逐个检查每个瓶子，既慢又不方便。现在，你用手机拍摄厨房的全景，然后用软件自动识别出所有调料瓶的位置和轮廓。接着，利用这些图片，软件可以构建出厨房的三维模型，显示每个调料瓶的具体位置。这样，你就可以快速找到需要的调料，不用逐个摸索。这就像用相机和智能算法帮你快速找到焊缝的位置，节省时间，提高效率。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的拼图游戏，拼图块很多，要找到一条特定的线很难。可是，如果你用手机拍下拼图的不同角度，然后用特殊的软件帮你分析，识别出那条线在哪里，事情就变得简单多了。软件会把这些线的图片拼在一起，形成一个3D模型，就像你用积木搭建一个小房子一样。这样，你就可以很快知道那条线在房子上的具体位置。这个方法就像用手机和智能程序帮你找到焊缝的位置，不用费劲地用肉眼一块块找，节省了很多时间，也更准确！
Glossary
Semantic Segmentation (语义分割)
一种图像处理技术，将图像中的每个像素分类到不同类别中，帮助识别特定区域。
用来识别焊缝区域与背景的区别。
Photogrammetry (摄影测量)
利用多张图片通过几何关系重建物体三维模型的方法。
实现工件的三维重建。
SegFormer
一种基于Transformer的深度学习模型，用于高效的语义分割任务。
用于焊缝区域的自动识别。
RMSE (Root Mean Square Error)
衡量预测值与真实值偏差的指标，数值越小代表误差越低。
用于评估焊缝空间定位的精度。
RealityScan
一款基于摄影测量的三维重建软件。
用来生成工件的三维模型。
Open Questions Unanswered questions from this research
- 1 模型在极端光照和复杂焊缝形态下表现不足，未来需增强鲁棒性。
- 2 如何实现实时焊缝检测与路径规划，仍是技术难点。
- 3 多模态信息融合（如激光与图像结合）仍需深入研究。
Applications
Immediate Applications
工业焊缝粗略检测
利用手机图像快速识别大工件焊缝位置，指导机器人进行后续高精度测量和处理。
机器人路径规划预处理
为机器人自动打磨或检验提供空间焊缝路径，减少人工干预。
Long-term Vision
全自动焊缝检测与机器人作业
实现现场实时焊缝识别、路径规划与自动作业，推动智能制造全面升级。
Abstract
Accurate identification of weld seam geometries is essential for automated robotic post processing operations such as grinding, finishing, and inspection. For large workpieces, complete surface scanning using high precision laser scanners or structured light sensors can be time consuming and often generates substantial amount of data that are not relevant. This paper presents an experimental vision based pipeline for the approximate localization of weld seams. This serves as a preliminary stage before high precision measurement. The proposed approach aims to reduce the overall scanning effort and data acquisition efficiency. The proposed method includes capturing images of the workpiece from multiple viewpoints, identifying weld seams from the images using semantic segmentation, reconstructing the workpiece using photogrammetry, and projection of identified weld seams into the reconstructed model.