GINA-3D: Learning to Generate Implicit Neural Assets in the Wild
GINA-3D generates tri-plane implicit assets from Waymo sensor data, achieving FID 59.5 on WOD-Vehicle.
Key Findings
Methodology
GINA-3D is a two-stage 3D-aware generative framework. Stage 1 uses a Vision Transformer and 2D-to-3D cross-attention to encode images into 16×16×3 tri-plane latents with channel dimension 32. A 2,048-entry vector-quantized codebook discretizes them; a Style-based generator and NeRF volume renderer reconstruct visible object pixels. Stage 2 uses a bidirectional MaskGIT Transformer to iteratively sample masked discrete tokens, enabling class-, scale-, semantic-, and time-of-day-conditioned asset generation.
Key Results
- On WOD-Vehicle, the complete model reaches FID 59.5, compared with 137.6 for EG3D and 105.3 for GIRAFFE. It obtains geometry FOU 1.80, geometry COV 11.42, and geometry consistency 0.98, indicating stronger realism, shape coverage, and 3D consistency.
- Adding tri-plane latents reduces FID from 147.9 to 79.0; adding scaled-box conditioning and LiDAR supervision further reduces it to 60.5 and 59.5. Geometry consistency improves from 1.55 to 0.98, showing complementary benefits from structured representation and sensor supervision.
- The benchmark contains 901K WOD-Vehicle images, 321K WOD-Pedestrian images, and 80K Longtail-Vehicle images. GINA-3D generates common and rare assets, including construction equipment, garbage trucks, cable cars, and school buses, with leading image and geometry diversity.
Significance
The work turns real autonomous-driving observations into generative, editable 3D assets, reducing dependence on artists and curated image collections. It directly addresses occlusion, illumination variation, incomplete viewpoints, and long-tail object distributions that are central to road data. Academically, it contributes a large object-centric benchmark and a sensor-grounded generative formulation. Industrially, it offers a route toward scalable simulation assets for perception testing, rare-event augmentation, and camera-LiDAR validation.
Technical Contribution
The central contribution is to decouple representation learning from generative modeling. An encoder-decoder first learns a reconstructive discrete tri-plane representation; MaskGIT then models its distribution. Occlusion-aware composition uses Panoptic-DeepLab pseudo-labels to supervise visible target pixels, while alpha regularization encourages object density and suppresses sky or road density. Unlike GIRAFFE's scene-level disentanglement and EG3D's continuous latent prior, GINA-3D's spatial codebook supports conditional sampling, local editing, LiDAR depth supervision, and extensions with DINO semantic fields.
Novelty
The novelty is the system-level integration of discrete tri-plane latents, occlusion-aware autoencoding, and MaskGIT sampling for real multi-sensor driving data. It differs fundamentally from methods trained on clean images or synthetic scenes, and from single-instance completion systems: GINA-3D aims to sample previously unseen assets rather than merely reconstruct observed vehicles. The paper therefore connects scalable real-world data mining with generative 3D asset creation.
Limitations
- Training depends on 3D boxes, camera-LiDAR registration, and Panoptic-DeepLab pseudo-labels. Heavy occlusion, segmentation errors, and unseen vehicle backs can therefore produce hallucinated or contaminated geometry.
- Experiments use 256² inputs, 128² rendering, and mostly static object-centric crops; robustness in full scenes, dynamic interactions, high-resolution rendering, and closed-loop simulation is not established.
- The data are concentrated in San Francisco and Phoenix. Although 80K long-tail images are included, cross-city, cross-weather, and cross-cultural distribution coverage remains limited.
Future Work
Future research should combine denser LiDAR, multi-view temporal observations, radar, and explicit uncertainty modeling to improve unseen geometry. Stronger diffusion or autoregressive priors could control weather, materials, articulation, and behavior. Important evaluations include cross-city generalization, sensor-transfer robustness, closed-loop driving performance, and sim-to-real benefit. Engineering work should also reduce tri-plane decoding cost and measure throughput, memory, and asset-library maintenance at fleet scale.
AI Executive Summary
Autonomous-driving simulation requires enormous libraries of realistic vehicles and pedestrians, yet manual modeling is expensive, slow to update, and poorly suited to rare objects such as garbage trucks or construction equipment. Existing 3D generative systems often rely on curated images or hand-built synthetic scenes. Real road data are more valuable but substantially harder: objects are occluded, illumination changes, and a single vehicle is rarely observed from all sides.
GINA-3D addresses this gap with a two-stage pipeline. First, a Vision Transformer and cross-attention encoder convert object images into structured tri-plane latents, which are discretized by a 2,048-entry codebook. A Transformer, Style-based generator, and NeRF renderer reconstruct visible pixels under an occlusion-aware loss. Second, MaskGIT learns the distribution of discrete tokens and iteratively samples new assets, optionally conditioned on class, scale, semantic features, or time of day.
The authors build a benchmark from Waymo Open Dataset: 901K vehicle images, 321K pedestrian images, and 80K long-tail vehicle images. On WOD-Vehicle, complete GINA-3D achieves FID 59.5, versus 137.6 for EG3D and 105.3 for GIRAFFE; geometry consistency reaches 0.98 and geometry COV 11.42. Ablations show that tri-plane structure, scaled boxes, and LiDAR supervision progressively improve quality and geometry. The approach offers a scalable path from fleet data to simulation assets, but remains limited by pseudo-label errors, partial observation, resolution, and geographic bias.
Deep Analysis
Background
3D-aware generation evolved from GRAF and GIRAFFE's NeRF-based object or scene models to EG3D's efficient tri-plane representation. Shape-generation research also covers voxels, point clouds, meshes, and implicit functions. However, many systems assume clean images, synthetic renderings, or complete multi-view silhouettes. Autonomous-driving data are abundant, current, and task-relevant, but contain occlusions, motion, rolling-shutter effects, lighting changes, and long-tail categories. A real-world object-centric benchmark and a sensor-aware generative method are therefore needed.
Core Problem
The goal is to learn a distribution of vehicle and pedestrian 3D assets from real road images, while generating novel, renderable, and editable objects. The problem is difficult because foregrounds mix with occluders and backgrounds, single views expose only partial geometry, illumination and scale vary, and accurate 3D supervision is scarce. Directly applying GIRAFFE can cause disentanglement failures, while unmasked EG3D training may collapse without explicit foreground-background modeling.
Innovation
First, GINA-3D represents 3D structure as spatially organized discrete tri-plane tokens rather than one global continuous code. Second, its 2D-to-3D encoder and 3D-to-2D neural-rendering decoder separate representation learning from prior modeling. Third, Panoptic-DeepLab pseudo-labels support occlusion-aware reconstruction: visible object pixels are supervised, while sky and road density are suppressed. Fourth, MaskGIT reuses the codebook for conditional synthesis and local editing. Scale conditioning, DINO semantic fields, and LiDAR depth can be added without redesigning the core representation.
Methodology
- �� Input: 256² object crops, object boxes, camera rays, and pseudo-masks; training renders at 128².
- �� Encoding: ViT converts 16×16 image patches into tokens; tri-plane positional queries and cross-attention produce 16²×3×32 continuous embeddings.
- �� Quantization: each embedding is replaced by the nearest vector in a 2,048-entry codebook, yielding a discrete sequence.
- �� Decoding: a token Transformer and four Style-based convolution blocks upsample tri-plane features; an MLP predicts density σ and view-independent color c, followed by NeRF volume rendering.
- �� Optimization: L2, LPIPS, GAN, VQ commitment, and alpha losses are jointly optimized. Object pixels are encouraged toward alpha 1, while sky and road pixels toward alpha 0.
- �� Sampling: MaskGIT masks random tokens during training and predicts them with negative log-likelihood. At inference, high-confidence tokens are retained and the remainder repeatedly refined before codebook lookup and decoding.
Experiments
The benchmark uses 1,150 Waymo scenes and contains WOD-Vehicle: 901K images and 23.6K instances; WOD-Pedestrian: 321K images and 8.1K instances; and Longtail-Vehicle: 80K images and 3.7K instances. Baselines are GIRAFFE and EG3D, trained at 128². Metrics include FID, Mask FOU, geometry FOU, COV, MMD, and semantic consistency. GINA-3D uses an 8×Tesla V100 setup, Adam, 150K Stage-1 steps, and 80K Stage-2 steps. Ablations test tri-plane latents, scaled boxes, and LiDAR supervision.
Results
Complete GINA-3D obtains FID 59.5 on WOD-Vehicle, compared with EG3D 137.6 and GIRAFFE 105.3. It achieves geometry consistency 0.98, geometry COV 11.42, and mesh FOU 4.57. Without tri-plane structure, FID is 147.9; adding tri-plane latents reduces it to 79.0, scaled-box conditioning to 60.5, and LiDAR supervision to 59.5. The ablation indicates that structured discrete latents drive the largest quality gain, while scale and depth cues further improve geometric fidelity and coverage.
Applications
Generated tri-plane assets can be inserted into background neural fields or converted into meshes for conventional rendering and geometry-aware compositing. MaskGIT supports class-, scale-, time-of-day-, and semantic-conditioned synthesis, making the system useful for rare-event generation, dataset augmentation, detector and tracker testing, and camera-LiDAR fusion evaluation. Deployment requires calibrated sensor models, collision-ready geometry, and a temporal scene or behavior layer; the paper primarily demonstrates static object-level generation.
Limitations & Outlook
The method inherits errors from boxes, registration, and pseudo-segmentation; contaminated masks may bake occluders or background into the asset. Single-view evidence and sparse LiDAR cannot reliably recover backsides, transparent parts, or fine details. Evaluation is low-resolution and object-centric, with limited evidence for full-scene composition, physical interaction, dynamic behavior, or cross-city transfer. Future work should use temporal multi-view data, stronger priors, uncertainty estimates, broader weather and geography, and report decoding speed, memory, and closed-loop simulation gains.
Plain Language Accessible to non-experts
Think of GINA-3D as a factory that makes 3D toy vehicles from street photographs. The factory first looks at a photo and identifies which pixels belong to the target car, even when a pole or another car blocks part of it. It then breaks the visual information into three crossing sheets, like three transparent maps showing the car from different directions. Each small square on those maps is replaced by a reusable part from a numbered catalog.
A second production line learns which parts usually fit together. It covers most squares, guesses the most certain missing ones, and repeats the process until a complete object appears. This is the role of MaskGIT. The factory can make a new car that was never photographed, or follow instructions such as “make it larger,” “use a garbage truck,” or “show a nighttime version.” A rendering machine then photographs the invisible object from any chosen viewpoint.
The factory learned from Waymo data: more than 1.2 million vehicle and pedestrian images plus 80,000 long-tail vehicle images. Its FID is 59.5, much lower than EG3D's 137.6. But it still guesses when evidence is missing. Severe occlusion, wrong labels, and limited regional data can lead to inaccurate shapes.
ELI14 Explained like you're 14
Imagine you are building an autonomous-driving video game. Reusing the same three cars would make the world boring, but hand-making every bus, truck, construction vehicle, and cable car takes forever. GINA-3D learns to create new 3D objects from real street photos collected by Waymo.
Street photos are messy: another car may cover half the target, sunlight changes its appearance, and the camera may never see the back. GINA-3D turns each image into a kind of three-layer spatial puzzle and stores each little piece as a code from a catalog. Think of it as taking a complicated LEGO model apart while remembering where every brick belongs.
Then MaskGIT plays a smart guessing game. It hides many LEGO pieces, predicts the ones it is most confident about, keeps those, and repeats. Eventually it can build a vehicle that was not in the original photos. You can also ask for a certain class, size, semantic style, or day/night condition. A virtual camera can view the result from new angles—pretty useful for game worlds and robot testing!
The results are strong: FID is 59.5 for the full model, compared with 137.6 for EG3D. Still, it is not magic. If the photo is too blocked or the pixel labels are wrong, the model may invent a strange shape. More viewpoints, LiDAR, weather, and motion data could make the generated world much more trustworthy!
Glossary
Implicit Neural Asset
A 3D object represented by a neural function rather than a fixed mesh. Querying a spatial point returns quantities such as density and color for rendering.
GINA-3D generates vehicles and pedestrians in this form.
Tri-plane
Three orthogonal 2D feature planes used to approximate a 3D representation. A spatial query interpolates features from the planes, saving memory relative to dense volumetric grids.
GINA-3D uses tri-planes for both latent structure and decoded features.
NeRF
A Neural Radiance Field predicts density and color from position and viewing direction, then integrates values along camera rays. It enables differentiable novel-view rendering.
The decoder uses NeRF-style volume rendering.
MaskGIT
A bidirectional Transformer generator that repeatedly predicts randomly masked discrete tokens. Confidence-based iterative refinement makes generation parallel and controllable.
It samples Stage-2 tri-plane code sequences.
Vector Quantization
A continuous feature is replaced by the nearest entry in a learned finite codebook. This produces a compact discrete representation that can be modeled token by token.
A 2,048-entry codebook discretizes GINA-3D latents.
FID
The Fréchet Inception Distance compares feature distributions of generated and real images; lower values generally indicate closer visual statistics. It is a distribution-level, not a complete geometric, measure.
The full model reaches FID 59.5 on WOD-Vehicle.
Open Questions Unanswered questions from this research
- 1 How reliably can a single partially occluded image recover unseen backsides and fine geometry? The paper does not provide calibrated uncertainty, so temporal multi-view observations and denser LiDAR are needed.
- 2 Whether generated assets improve real autonomous-driving safety is not directly measured. Closed-loop simulation, sim-to-real transfer, and long-tail driving evaluations remain necessary.
Applications
Immediate Applications
Long-tail scenario augmentation
Simulation teams can train the conditional sampler on Longtail-Vehicle data to generate garbage trucks, construction equipment, and cable cars, then place them into background neural fields. Required checks include sensor calibration, segmentation quality, collision geometry, and camera-LiDAR consistency.
Multi-sensor regression testing
Engineers can generate assets by class, scale, and time of day, then render synchronized camera and LiDAR observations for detector, tracker, and fusion tests. Integration requires a calibrated renderer and validation of depth, occlusion, and object-coordinate alignment.
Long-term Vision
Self-updating road digital twins
Continuous fleet data could automatically add new vehicle models, regional styles, and rare objects to a simulation library with little artist labor. Combining GINA-3D with temporal behavior, physical attributes, weather, and cross-city data could support scalable closed-loop validation.
Abstract
Modeling the 3D world from sensor data for simulation is a scalable way of developing testing and validation environments for robotic learning problems such as autonomous driving. However, manually creating or re-creating real-world-like environments is difficult, expensive, and not scalable. Recent generative model techniques have shown promising progress to address such challenges by learning 3D assets using only plentiful 2D images -- but still suffer limitations as they leverage either human-curated image datasets or renderings from manually-created synthetic 3D environments. In this paper, we introduce GINA-3D, a generative model that uses real-world driving data from camera and LiDAR sensors to create realistic 3D implicit neural assets of diverse vehicles and pedestrians. Compared to the existing image datasets, the real-world driving setting poses new challenges due to occlusions, lighting-variations and long-tail distributions. GINA-3D tackles these challenges by decoupling representation learning and generative modeling into two stages with a learned tri-plane latent structure, inspired by recent advances in generative modeling of images. To evaluate our approach, we construct a large-scale object-centric dataset containing over 1.2M images of vehicles and pedestrians from the Waymo Open Dataset, and a new set of 80K images of long-tail instances such as construction equipment, garbage trucks, and cable cars. We compare our model with existing approaches and demonstrate that it achieves state-of-the-art performance in quality and diversity for both generated images and geometries.