GIFT: Bootstrapping Image-to-CAD Program Synthesis via Geometric Feedback

The evolution of CAD from rule-based systems to deep generative models has revolutionized engineering design. Early methods relied on explicit rules and templates, limiting flexibility. Recent advances include point cloud and image-to-3D models like PointNet and ShapeNet, but these often suffer from limited diversity and robustness. Symbolic programs as intermediate representations gained popularity due to their compactness and editability, exemplified by systems like CadQuery and OpenCASCADE. However, training data scarcity and verification challenges hinder their widespread adoption. Existing datasets like GenCAD provide some progress, but the diversity and verification quality remain insufficient for robust real-world applications.

Current image-to-program methods face significant bottlenecks: limited training data, poor generalization to complex geometries, and high inference costs. Supervised fine-tuning (SFT) relies on scarce, low-diversity datasets, leading to brittle models. Reinforcement learning (RL) improves alignment but is computationally expensive and unstable due to CAD kernel reliance. Moreover, existing models struggle with diverse design variations and complex geometries, especially in out-of-distribution scenarios. Overcoming these issues requires scalable data augmentation strategies that can leverage test-time feedback without additional annotation burdens.

GIFT introduces a geometric verification-based self-augmentation framework, transforming inference search into training data. It combines two novel mechanisms: Soft-Rejection Sampling (SRS), which captures multiple valid programs beyond the ground truth, and Failure-Driven Augmentation (FDA), which converts near-miss failures into synthetic input-output pairs. This dual approach significantly enhances data diversity and robustness, enabling the model to learn from its own geometric feedback. Unlike traditional methods, GIFT does not depend on extra labels or complex architectures, instead leveraging deterministic geometric kernels for high-quality supervision. This paradigm shift allows scalable, data-efficient training that improves generalization and reduces inference costs.

• �� Generate multiple candidate programs per image using a pretrained model. • Verify each candidate with a CAD kernel (OpenCASCADE), calculating IoU with ground truth. • Filter candidates based on IoU thresholds: high-fidelity (≥0.9), near-misses (0.5-0.9). • Use SRS to retain diverse high-quality programs, expanding the output distribution. • Render near-miss programs into synthetic images, pairing them with ground-truth programs for input augmentation. • Incorporate verified high-quality and challenging samples into training, optimizing model parameters. • Repeat across multiple compute budgets (8-128) to balance exploration and precision, ensuring diverse, robust training data.

The evaluation used the GenCAD dataset with 163k training pairs and 8k test pairs, plus 400 out-of-distribution samples. Models trained with standard SFT, GIFT with SRS, and GIFT with both SRS and FDA were compared. The training set expanded from 160k to 370k samples through geometric verification. Performance metrics included mean IoU, problem-solving ratio, and robustness on complex geometries. Ablation studies confirmed the effectiveness of each augmentation component. The experiments demonstrated that GIFT consistently outperformed baseline models, especially on challenging and OOD samples, validating the robustness and scalability of the approach.

GIFT achieved a mean IoU of 0.781, surpassing the supervised baseline by 12%, while reducing inference computation by 80%. The data augmentation strategy expanded training data by over 100%, leading to improved robustness, especially on complex geometries and out-of-distribution samples. The dual mechanisms effectively mitigated model collapse and increased diversity, solving 53% more problems than baseline models. These results highlight the potential of geometric feedback for scalable, high-quality image-to-program synthesis, setting new benchmarks in the field.

This approach can be directly applied to automated engineering design, CAD generation from images, and structural optimization. It requires only existing datasets and geometric kernels, making it suitable for industrial automation, rapid prototyping, and design validation. Long-term, GIFT could enable fully autonomous design systems, reducing manual effort and accelerating innovation. Its scalability and robustness make it promising for integration into CAD software, robotics, and manufacturing workflows, fostering smarter, faster engineering processes.

Dependence on geometric kernels like OpenCASCADE introduces computational overhead, limiting real-time deployment. Performance may decline with highly complex or ambiguous inputs where verification becomes unreliable. The current focus on 2D images and parametric programs restricts extension to 3D scenes and more intricate geometries. Further research is needed to optimize verification speed, handle diverse data modalities, and improve robustness in extreme cases. Addressing these limitations will be crucial for industrial-scale adoption and broader application.