JanusMesh: Fast and Zero-Shot 3D Visual Illusion Generation via Cross-Space Denoising

Creating compelling 3D visual illusions—objects that reveal different semantics from various viewpoints—has long been a challenge in computer graphics and vision. Traditional optimization-based methods, such as Score Distillation Sampling (SDS), require extensive computation times, often exceeding 40 minutes per object, and tend to produce oversaturated colors and geometric artifacts. Naive stitching approaches, which simply combine separately generated objects, often result in unnatural seams and semantic leaks, undermining the illusion effect.

In this context, the authors introduce JanusMesh, a novel, training-free framework that achieves rapid, high-quality dual-semantic 3D illusion generation within 3 to 5 minutes. The core innovation lies in a two-stage process: first, a cross-space dual-branch denoising mechanism based on TRELLIS encodes, decodes, aligns, and fuses geometry via Signed Distance Fields (SDF), ensuring geometric coherence; second, a view-conditioned texture synthesis module predicts and projects view-specific textures onto the fused geometry using stable diffusion models, resulting in seamless, realistic textures.

The dual-branch denoising process is inspired by the need to maintain geometric integrity while accommodating divergent semantics. At each denoising step, the latent vectors are decoded into sparse voxel grids, aligned via CLIP-guided orientation search, and blended through SDF averaging. This approach prevents geometric artifacts common in naive concatenation and ensures the fused shape remains natural from arbitrary viewpoints. The view-conditioned texture synthesis further enhances realism by projecting multi-view predictions and blending them based on surface normals, producing textures that adapt to different semantics.

Experimental results demonstrate that JanusMesh outperforms existing methods across multiple metrics. Quantitative evaluations on the Objaverse dataset show a reduction in geometric error by 30%, with semantic recognition accuracy exceeding 85%. The entire pipeline operates within 3-5 minutes, a significant improvement over SDS-based methods, which typically take around 40 minutes. The framework also scales effectively to three-object illusions, handling complex geometric conflicts through enhanced guidance strategies.

The significance of this work extends beyond technical novelty. It provides a practical, efficient tool for content creators in virtual reality, gaming, and digital arts, enabling rapid prototyping of multi-semantic models without training. Its ability to generate realistic, seamless illusions opens new avenues for immersive experiences and interactive applications. While current limitations include handling extremely complex scenes and real-time performance, ongoing research aims to address these challenges, promising a future where dynamic, multi-semantic 3D content can be generated on demand.

Overall, JanusMesh represents a major step forward in 3D multimodal content synthesis, combining innovative geometric fusion, semantic alignment, and texture generation into a unified, fast pipeline. Its contributions lay a foundation for future exploration of multi-view, multi-semantic 3D modeling, with broad implications for industry and academia alike.