Generative Models as Distributions of Functions
GASP learns continuous function distributions, reaching FID 7.42/19.16 on CelebAHQ while enabling cross-resolution generation.
Key Findings
Methodology
The paper introduces Generative Adversarial Stochastic Process (GASP). A latent variable z is mapped by a hypernetwork gφ to the weights of an implicit function fθ, which maps coordinates x to features y. Random Fourier Features, γ(x)=(cos(2πBx),sin(2πBx)), capture high frequencies. A PointConv discriminator operates on unordered coordinate-feature point clouds while respecting spatial distances; training uses the non-saturating GAN objective plus an input-feature R1 penalty.
Key Results
- On CelebAHQ, GASP obtains FID 7.42 at 64×64 and 19.16 at 128×128. The set-discriminator baseline scores 236.82 at 64×64, while the auto-decoder scores 117.80; the 128×128 baseline values were not reported. These results highlight the importance of spatially aware PointConv discrimination.
- A function trained only on 64×64 images can be evaluated on a 256×256 grid to produce convincing 4×-resolution samples. For ShapeNet chairs, GASP trains from only 4,096 sampled points although each original 32³ voxel model contains 32,768 points.
- The same core architecture is used for images, 3D shapes, and ERA5 climate data, changing only function input/output dimensions and Fourier-feature settings. The ShapeNet experiment contains 6,778 chair models and supports rendering at 16³, 32³, 64³, and 128³ resolutions.
Significance
GASP reframes generative modeling from fixed-size arrays to distributions over functions, reducing dependence on grid resolution. It offers one continuous framework for images, voxels, irregular point sets, and manifold data, while enabling random coordinate subsampling during training. Academically, it connects implicit neural representations, hypernetworks, and adversarial learning. Practically, it provides a memory-efficient route to high-resolution 3D generation, scientific fields, and irregular sensor data, although it does not eliminate all computational or tuning costs.
Technical Contribution
The method combines three elements: a hypernetwork gφ defining a weight distribution p(θ); Random Fourier Features for high-frequency fitting; and PointConv instead of grid-bound CNN or purely set-based PointNet/DeepSets discriminators. PointConv parameterizes kernels with an MLP W:Rd→Rcout×cin and computes fout(x)=Σxi∈NxW(xi−x)fi. Neighbor selection uses coordinate-space distances, yielding translation equivariance and permutation invariance. The paper also adapts R1 regularization to penalize discriminator gradients with respect to point-cloud features.
Novelty
Unlike GRAF, GIRAFFE, and pi-GAN, which use conventional image convolutional discriminators, GASP keeps the generator, discriminator, and loss continuous and directly defined on coordinate-feature point clouds. To the authors’ knowledge, it is the first demonstration of one such framework across images, 3D shapes, and manifold climate data, while producing sharp samples on complex CelebAHQ rather than only on MNIST.
Limitations
- At 128×128, CelebAHQ samples still contain artifacts and occasional failures; FID 19.16 remains worse than the specialized convolutional GAN’s 5.74. Continuous parameterization therefore does not automatically deliver state-of-the-art image fidelity.
- Function capacity, Fourier frequencies, and PointConv neighborhoods require tuning. Fixed MLP structures may limit complex signals, while neighborhood search and hypernetwork-generated weights add computational overhead.
Future Work
Future work should explore stronger function architectures, adaptive or learnable frequencies, conditional generation, and more stable adversarial objectives. The framework could be extended to higher-dimensional manifolds, dynamic spatiotemporal fields, and genuinely irregular observations. A unified evaluation of cross-resolution quality, functional smoothness, sampling efficiency, and memory scaling would clarify trade-offs and help close the gap with image-specialized GANs.
AI Executive Summary
Most generative models treat images, voxel objects, and waveforms as fixed arrays. Increasing image resolution therefore increases cost with area, while 3D voxel cost grows with volume. Dupont, Teh, and Doucet propose GASP, which treats each example as a continuous function from coordinates to features and learns a distribution over such functions.
A latent vector drives a hypernetwork gφ that emits the weights of an implicit MLP fθ. Random Fourier Features improve high-frequency detail. Instead of a grid-dependent CNN discriminator, GASP uses PointConv on coordinate-feature point clouds, trained with a non-saturating GAN loss and a feature-level R1 penalty. Real examples are represented as sets of pairs and can be randomly subsampled.
Experiments cover CelebAHQ, ShapeNet chairs, and ERA5 climate data. CelebAHQ FID is 7.42 at 64×64 and 19.16 at 128×128; GASP also generates plausible 256×256 images from functions trained at 64×64. ShapeNet contains 6,778 chairs, yet training uses 4,096 points rather than 32,768 voxels per object. The framework is flexible and resolution-agnostic, but image quality still trails specialized convolutional GANs, and computational tuning remains important.
Deep Analysis
Background
Implicit neural representations encode an image as f:R²→R³ or a 3D shape as a coordinate-to-occupancy function. Occupancy Networks, DeepSDF, and NeRF-related models established the value of continuous representations for geometry and rendering. However, many generative methods still use grids, auto-decoders, or CNN discriminators, limiting resolution independence and modality generality.
Core Problem
The target is a distribution pdata(s) over sets s={(xi,yi)} of coordinate-feature pairs. The model must learn without direct access to ground-truth functions; the discriminator must be permutation invariant yet sensitive to spatial locality; ordinary ReLU MLPs struggle with high frequencies; and computation should not scale with the full 2D or 3D grid.
Innovation
- ��A hypernetwork maps z to function weights instead of producing a fixed pixel array. •RFF maps coordinates to sinusoidal features for high-frequency detail. •PointConv defines continuous, distance-aware kernels and preserves translation equivariance and permutation invariance. •Random point subsets decouple training memory from the complete grid and make irregular or manifold data possible.
Methodology
- ��Representation: fit minθΣi||fθ(xi)−yi||². •Encoding: γ(x)=[cos(2πBx),sin(2πBx)], with B commonly sampled from N(0,σ²). •Generation: sample z, compute θ=gφ(z), and output yi=fθ(γ(xi)). •Discrimination: PointConv uses an MLP kernel W(xi−x) over neighbors Nx. •Optimization: use the non-saturating GAN objective plus R1(s)=1/2Σyi||∇yiD(s)||². •Training: randomly select K coordinate-feature pairs rather than processing the entire grid.
Experiments
The study uses CelebAHQ at 64×64 and 128×128, ShapeNet Chairs, and ERA5 climate data. The function MLP has three hidden layers of width 128; the hypernetwork has two hidden layers of widths 256 and 512. Training uses one 2080Ti GPU with 11GB RAM. Image baselines include auto-decoder, set-discriminator, ConvNP, and a convolutional GAN; FID is the main metric. 3D comparisons include Occupancy Network VAE and DeepSDF with a set discriminator.
Results
GASP achieves FID 7.42 and 19.16 on CelebAHQ 64×64 and 128×128, versus 236.82 for the set discriminator and 117.80 for the auto-decoder at 64×64; the convolutional GAN reaches 4.00 and 5.74. A 64×64-trained function generates plausible 256×256 images. On ShapeNet, training drops from 32,768 voxels to 4,096 points per chair and still supports smooth high-resolution meshes. PointNet/DeepSets failed on complex image distributions, supporting spatially aware discrimination.
Applications
Potential uses include high-resolution image synthesis, 3D shape sampling, sparse sensor-field completion, and climate modeling on spheres or other manifolds. Users need a coordinate system, feature definition, and appropriate distance metric; high-frequency signals also require suitable RFF settings. Design, scientific simulation, and remote sensing can benefit from subset training and arbitrary-resolution querying.
Limitations & Outlook
GASP does not outperform image-specialized convolutional GANs, and 128×128 samples retain artifacts. Fixed function capacity, Fourier hyperparameters, and PointConv neighborhoods may limit complexity; neighborhood search and hypernetwork weight generation add cost. Evaluation remains limited for very high resolutions, dynamic data, conditional generation, and systematic ablations. Physical consistency for climate applications is also not guaranteed.
Plain Language Accessible to non-experts
Imagine a restaurant that serves maps rather than meals. A conventional system prints every square of a menu in advance. If the menu grows from 64×64 to 128×128, it needs four times as many printed squares; a 3D menu becomes even more expensive. GASP instead trains a chef who can answer a location query: tell the chef a coordinate, and the chef returns the color, material, or weather value there.
A hypernetwork acts like a head chef who creates a new specialist from a random recipe. Each specialist represents one possible picture, chair, or climate map. Random Fourier Features are a box of fine brushes, helping the specialist draw hair, edges, and other tiny details. PointConv is the reviewer: it checks not only each location but also whether nearby locations fit together.
Because the specialist can be questioned anywhere, training can inspect only a sample of locations and later render a denser version. The same kitchen can handle pictures, 3D objects, and unevenly sampled scientific data. The trade-off is that the chefs and reviewer are more complicated to train, and the results are flexible but not yet as polished as the best image-specific systems.
ELI14 Explained like you're 14
Think of a game-world generator. A normal generator draws every square of the map beforehand. Going from 64×64 to 128×128 means four times more squares; in 3D, the explosion is even bigger. This paper trains a rule-based “map robot” instead. Give it a coordinate, and it tells you the color, object, or weather value at that spot.
A small robot called a hypernetwork receives random numbers and builds another robot. That second robot creates one particular face, chair, or climate map. Random Fourier Features work like a box of tiny drawing pens, helping it reproduce hair, sharp borders, and fine texture instead of only blurry shapes.
The quality checker is unusual too. It does not demand a neat square grid; it looks at nearby points and asks whether they make sense together. During training, it can inspect only some points rather than the whole map. That saves memory and lets the system handle 3D clouds or data on curved surfaces.
The results are impressive: trained on 64×64 faces, it can query the same function on a 256×256 grid. For ShapeNet chairs, it uses 4,096 points instead of 32,768 voxels. But it still makes occasional strange faces, and its FID scores, 7.42 and 19.16, trail specialized convolutional GANs. So it is more adaptable, but it still needs practice to become a perfect artist!
Glossary
Implicit Neural Representation
A neural network represents a continuous mapping from coordinates to signal values instead of storing a fixed array. Resolution is chosen at query time.
GASP uses an MLP fθ for images, occupancy fields, and climate fields.
Hypernetwork
A network that generates the parameters of another network. Here it converts a latent vector into the weights of a sampled function.
gφ(z) produces the weights θ of GASP’s implicit function.
Random Fourier Features
A sinusoidal coordinate embedding that makes neural networks better at fitting high-frequency variation. Its frequencies are controlled by B and its variance.
GASP uses γ(x) to recover image texture and sharp boundaries.
PointConv
A convolution defined over continuous point locations, with kernels generated from coordinate differences. It uses neighborhoods while remaining permutation invariant.
PointConv forms the resolution-agnostic discriminator.
R1 penalty
A regularizer that penalizes the discriminator’s input-gradient norm to stabilize GAN training. In this paper, gradients are taken with respect to point-cloud features.
The penalty is R1(s)=1/2Σyi||∇yiD(s)||².
FID
A metric comparing feature distributions of real and generated images; lower values generally indicate better similarity and quality.
GASP obtains 7.42 and 19.16 on CelebAHQ 64×64 and 128×128.
Open Questions Unanswered questions from this research
- 1 How to retain continuous, resolution-independent generation while matching convolutional GAN image quality remains open. Stronger continuous discriminators, conditioning, and more stable objectives are needed.
- 2 The theoretical relationship among function complexity, Fourier frequencies, sample count K, error, memory, and compute is not established.
- 3 Stable generation of dynamic spatiotemporal fields and high-dimensional manifold functions remains largely untested.
Applications
Immediate Applications
3D shape generation
Design systems can generate occupancy functions for chairs and other objects using sampled voxel points rather than complete grids. The function can later be queried at higher resolution to produce smoother meshes while reducing GPU memory requirements.
Climate-field modeling
Climate researchers can represent ERA5-like data as distributions mapping locations to atmospheric variables. This supports irregular sampling and sparse-observation completion, although physical constraints and calibration must be added for reliable scientific deployment.
Long-term Vision
Unified scientific-field generator
A continuous generator could eventually cover images, medical volumes, ocean fields, and planetary surfaces in one framework. Combined with physical constraints and conditioning, it may support cross-resolution forecasting, simulation, and design optimization.
Abstract
Generative models are typically trained on grid-like data such as images. As a result, the size of these models usually scales directly with the underlying grid resolution. In this paper, we abandon discretized grids and instead parameterize individual data points by continuous functions. We then build generative models by learning distributions over such functions. By treating data points as functions, we can abstract away from the specific type of data we train on and construct models that are agnostic to discretization. To train our model, we use an adversarial approach with a discriminator that acts on continuous signals. Through experiments on a wide variety of data modalities including images, 3D shapes and climate data, we demonstrate that our model can learn rich distributions of functions independently of data type and resolution.