Neural Field Tokenizations with Hierarchy and Spatial Locality Priors
LH-NeF adds hierarchical locality to neural-field tokens, cutting memory 42× and enabling 133× larger batches.
Key Findings
Methodology
LH-NeF embeds coordinate–value observations, sorts them with a space-filling locality key, and applies Hierarchical Perceiver grouped attention to form multiscale local tokens. Its renderer selects k nearest groups, applies learnable Gaussian weights, reads each group with Cross-Attention, and uses FiLM modulation based on normalized relative coordinates. End-to-end training minimizes L1 field reconstruction loss.
Key Results
- On CIFAR-10, CelebA-HQ, and ImageNet1k, LH-NeF reaches 44.4, 36.1, and 28.3 dB PSNR. On CelebA-HQ it uses only 657K parameters, versus ENF's 34.6 dB and 3.2M parameters.
- On ShapeNet16 it obtains 93.1% IoU; on ERA5 its temperature MSE is 3.57E-05. Frozen tokens yield 91.2% CIFAR-10 accuracy, 96.8% ShapeNet16 accuracy, and CelebA-HQ FID 9.7.
- At 64×64 training resolution, LH-NeF uses approximately 42× less memory and supports 133× larger batches than the strongest modality-agnostic baseline. Replacing locality-aware grouping with HiP reduces CelebA PSNR to 29.9 dB.
Significance
The work addresses a central trade-off in neural-field learning: structured representations usually require modality-specific encoders, while modality-agnostic methods rely on expensive per-instance optimization. LH-NeF shows that geometry-aware locality and hierarchy can be injected through a generic coordinate–value interface. This makes high-resolution, multi-instance neural-field learning substantially more practical for vision, 3D data, and scientific fields.
Technical Contribution
The key engineering contribution is to impose locality through input ordering rather than through modality-specific convolutions or grids. Morton, Hilbert, or spherical keys create compact group supports; hierarchical grouped attention builds multiscale tokens; centroid and extent metadata enable Gaussian soft routing. Cross-Attention plus relative-coordinate FiLM reconstructs local detail. Query complexity becomes O(kKLD), rather than scanning all groups.
Novelty
Unlike Functa and ENF, which infer latents through MAML or related optimization, LH-NeF produces structured tokens in one forward pass. It also differs from standard HiP: HiP groups a flattened sequence and can violate metric geometry, whereas LH-NeF uses space-filling permutations so group supports remain spatially compact and input-adaptive. The novelty is the integration of locality, hierarchy, and queryable decoding in a modality-agnostic neural-field framework.
Limitations
- The method assumes a meaningful coordinate metric and a usable locality key. Inputs with obscure geometry, non-metric relations, or complex topology may not admit an ordering that preserves useful neighborhoods.
- Space-filling sorting, nearest-group search, metadata maintenance, and multilevel attention add preprocessing and inference costs. Very sparse, noisy, or highly nonuniform observations may yield groups whose centroids and extents poorly summarize fine structure.
- Evaluation covers images, voxel shapes, and ERA5, but not sufficiently dynamic video, high-dimensional PDEs, or challenging out-of-distribution geometries.
Future Work
Promising directions include learned locality keys, adaptive group counts, input-dependent routing and bandwidths, and extensions to spatiotemporal fields, video, PDEs, and complex manifolds. Further work should study token compression, diffusion compatibility, robustness under sparse or noisy sampling, out-of-distribution queries, and distributed scaling.
AI Executive Summary
Neural fields represent images, shapes, and physical measurements as continuous functions from coordinates to values. They can be queried at arbitrary resolution, but general-purpose methods such as Functa typically infer each instance's latent code through MAML-style inner-loop optimization. That process stores long computation graphs and second-order gradients. Conversely, CNN, grid, and point-cloud encoders are efficient but impose modality-specific assumptions.
Urbano and colleagues introduce LH-NeF, a locality-preserving hierarchical tokenization framework. Coordinate–value observations are first ordered with a space-filling key such as Morton order. Hierarchical grouped attention then merges neighboring observations into multiscale spatial tokens. At query time, the renderer selects nearby groups, combines them with learnable Gaussian weights, extracts information through Cross-Attention, and applies FiLM modulation using the query's normalized position within its nearest group.
Across CIFAR-10, CelebA-HQ, ImageNet1k, ShapeNet16, and ERA5, LH-NeF reaches 44.4, 36.1, and 28.3 dB PSNR on the three image benchmarks, 93.1% IoU on ShapeNet16, and 3.57E-05 temperature MSE on ERA5. Frozen representations achieve 91.2% CIFAR-10 accuracy, 96.8% ShapeNet accuracy, and FID 9.7 on CelebA-HQ. Training memory falls about 42× while batch capacity rises 133×. The approach is compelling, but still depends on reliable geometry and needs validation on dynamic and high-dimensional fields.
Deep Analysis
Background
Neural fields parameterize signals with coordinate-based networks; SIREN and NeRF established high-quality continuous fitting, while DeepSDF introduced conditional fields with optimized latent codes. Functa extended this idea across modalities, but its optimization-based inference is expensive. CNNs, ViTs, Occupancy Networks, LIIF, and related encoders exploit locality efficiently, yet usually assume images, grids, or 3D geometry rather than a general coordinate–value interface.
Core Problem
The goal is to infer a structured representation from sampled pairs {(xi,vi)} and reconstruct fθ(x) at arbitrary coordinates in one forward pass. The challenge is simultaneously preserving modality agnosticism, respecting geometry, and scaling training. Naive sequence grouping can place distant coordinates in one receptive field, while MAML must retain K inner-loop computation graphs, sharply limiting memory and batch size.
Innovation
First, LH-NeF uses Morton, Hilbert, or spherical indexing to make sequence adjacency reflect metric proximity. Second, it modifies Hierarchical Perceiver so grouped attention creates compact, progressively coarser spatial supports and retains multiple final groups. Third, centroid μg and extent λg become explicit routing metadata. Fourth, Gaussian soft routing, group-wise Cross-Attention, and normalized relative-coordinate FiLM provide smooth, local decoding without modality-specific grids or convolutions.
Methodology
- �� Input: embed values and add sinusoidal positional encoding to coordinate–value observations.
- �� Ordering: sort by locality key κ(x); Euclidean domains use quantized Morton codes, while spherical domains use S2 indices.
- �� Encoding: process contiguous chunks through L Grouped Attention blocks, producing Y(L), centroids μg, and extents λg.
- �� Routing: select k nearest groups and compute wg∝exp(-dX(x,μg)^2/2σθ²).
- �� Aggregation: form q(x)=MLPq(PE(x)); apply Cross-Attention to each selected group and take a weighted sum.
- �� Modulation: encode normalized relative coordinate ˜x with FiLM, h'=(1+γ)⊙h+β, then decode fθ(x) with an output MLP. Training uses L1 reconstruction loss.
Experiments
The evaluation uses CIFAR-10 32², CelebA-HQ 64², ImageNet1k 256², ShapeNet16 32³, and ERA5 climate fields. Baselines are Functa, Spatial Functa, and ENF; generation additionally compares GEM, GASP, and DPF. Metrics include PSNR, IoU, temperature MSE, FID, classification accuracy, and parameter count. Ablations replace locality-aware ordering with standard HiP, alter weighting, use k=1 hard routing, or remove FiLM.
Results
LH-NeF obtains 44.4 dB on CIFAR-10, 36.1 dB on CelebA-HQ, 28.3 dB on ImageNet1k, and 93.1% IoU on ShapeNet16. CelebA-HQ uses 657K parameters. Downstream results include FID 9.7, 91.2% CIFAR-10 accuracy, and 96.8% ShapeNet accuracy. Replacing LH-NeF grouping with HiP lowers CelebA PSNR by 6.4 dB; hard routing reduces ShapeNet IoU to 84.8%, confirming the value of geometric locality and smooth multi-group interpolation.
Applications
LH-NeF can support continuous image compression and reconstruction, cross-resolution rendering, 3D shape recognition and generation, and climate-field forecasting. A practitioner needs coordinate–value observations and a suitable metric or locality key. After pretraining, the tokenizer can be frozen and its tokens supplied to classifiers, diffusion models, or predictors, avoiding per-instance latent optimization.
Limitations & Outlook
The framework is not assumption-free: it requires meaningful geometry, and a space-filling key may distort neighborhoods on difficult manifolds or irregular samples. k-nearest routing and hierarchical attention still incur computational overhead. The reported benchmarks do not establish robustness for dynamic video, severe noise, extreme sparsity, high-dimensional PDEs, or out-of-distribution geometries. Adaptive keys, group sizes, and bandwidths are natural next steps.
Plain Language Accessible to non-experts
Imagine a huge warehouse containing objects with both a location and a description. A traditional system asks a worker to walk through much of the warehouse again whenever a customer asks about one object. That is similar to per-example optimization: accurate, but slow and memory-hungry.
LH-NeF first organizes nearby objects into nearby shelves, then summarizes small shelves into larger zones. Each zone has an information card and records where it is located. When a customer asks about a point, the system consults only the closest few zones. Nearby zones receive more influence, so the answer changes smoothly while the customer moves between shelves.
The system also checks where the requested point lies inside the nearest zone—near its center or near an edge—so it can recover fine details. The same warehouse procedure works whether the objects describe image colors, 3D occupancy, or global temperature. Experiments show that this organized approach uses far less memory than repeatedly exploring every warehouse. Its catch is that the warehouse must have a meaningful map; without reliable distances, “nearby” shelves may not be useful.
ELI14 Explained like you're 14
Think of an open-world game where the map is not just a picture. You can click any location and ask what is there, even between the pixels you originally stored. A neural field is a clever mathematical way to build that kind of continuously searchable world.
Older methods are like sending a helper to re-explore each new map from scratch. The helper may become accurate, but doing it for every image or 3D object consumes lots of time and computer memory. LH-NeF acts more like a smart map-maker: it groups nearby places into neighborhoods, then combines neighborhoods into larger regions.
When you click somewhere, the computer checks the closest few neighborhoods instead of the whole map. It lets nearby neighborhoods vote, with closer ones counting more. That prevents a weird jump when your cursor crosses a boundary. It also notices whether the point is near the middle or edge of a neighborhood, helping it draw small details.
Researchers tested it on pictures, 3D shapes, and climate maps. It reached 93.1% shape-overlap quality on ShapeNet16 and 91.2% classification accuracy on CIFAR-10, while using about 42 times less training memory. Pretty impressive! But the trick works best when the map has a trustworthy idea of distance, so moving videos and very complicated spaces still need more research.
Glossary
Neural field
A continuous function that maps coordinates to values such as color, occupancy, or temperature. It can be queried at locations that were not explicitly sampled.
LH-NeF learns a conditional neural field fθ(x).
Locality-preserving ordering
An ordering designed to keep spatially nearby points close in a sequence. Morton codes achieve this by interleaving quantized coordinate bits.
The ordering determines which observations enter each attention group.
Hierarchical Perceiver
An attention architecture that processes groups and merges them across layers, creating coarse-to-fine latent structure. It provides a general interface for variable input sequences.
LH-NeF adapts it while preserving coordinate-space locality.
Soft group routing
A query is connected to several nearby groups rather than exactly one, and each contribution is distance-weighted. Gaussian weighting makes the decoded field smoother across group boundaries.
It is the renderer's first stage.
FiLM
Feature-wise Linear Modulation applies condition-dependent scale and bias to an intermediate representation. LH-NeF uses h'=(1+γ)⊙h+β.
The modulation encodes the query's position inside its nearest group.
Open Questions Unanswered questions from this research
- 1 How should locality keys be learned for dynamic video, spatiotemporal PDEs, or high-dimensional manifolds? The paper mainly evaluates static images, voxel shapes, and ERA5.
- 2 Are fixed k, group counts, and Gaussian bandwidths optimal across sampling densities? Adaptive routing and token budgets remain open design problems.
- 3 Robustness under out-of-distribution geometry, noisy observations, and extreme sparsity is not systematically established and needs dedicated benchmarks.
Applications
Immediate Applications
Continuous image reconstruction
Image systems can feed coordinate–pixel observations to LH-NeF and query arbitrary resolutions. This supports compression, zooming, and super-resolution without a fixed output grid, provided training images have consistent spatial coordinates.
3D shape encoding
Point or voxel occupancy observations can be converted into structured tokens and passed to recognition or generation models. The reported ShapeNet16 results—93.1% IoU and 96.8% classification accuracy—indicate practical value for shape pipelines.
Long-term Vision
Unified spatiotemporal scientific models
A future system could represent climate, fluid, and video data through one coordinate–value interface, enabling prediction, simulation, and generation with compact tokens. Major obstacles are time-dependent geometry, complex manifolds, and distributed scaling.
Abstract
Neural fields parameterize data as functions from coordinates to values, providing a unified framework for representation learning across modalities. Existing approaches are dominated by per-sample meta-learning, which scales poorly due to memory-intensive inner-loop optimization. The natural alternative -- feed-forward encoding -- typically introduces modality-specific assumptions, sacrificing the generality that makes learning with neural fields attractive. We argue that locality and hierarchy are useful priors for learning field representations that can be injected without compromising modality-agnosticism. We propose LH-NeF, a framework to learn general-purpose tokenized representations of continuous signals. A locality-preserving hierarchical encoder maps raw coordinate-value field observations to structured tokens, from which the field is reconstructed during training. By replacing meta-learning's inner loop with a single forward pass, LH-NeF uses 42x less memory and supports 133x larger batches than the strongest modality-agnostic baseline. Across images, 3D shapes, and climate fields, our learned representations match or exceed performance of modality-agnostic, modality-specific, and specialized generative neural field baselines on both reconstruction and downstream tasks.