Social Ways: Learning Multi-Modal Distributions of Pedestrian Trajectories with GANs
Social Ways uses Info-GAN to preserve multimodal pedestrian futures, reaching 0.39/0.64 ADE/FDE on ETH.
Key Findings
Methodology
Social Ways is a conditional GAN that predicts one target pedestrian from the historical trajectories of all agents. LSTM-E encodes temporal states; an attention-pooling module aggregates neighbors using distance, bearing angle, and distance of closest approach; LSTM-D combines the target history, pooled social representation, and noise z to generate futures. Info-GAN adds a continuous latent code c and a Q network, maximizing mutual information so distinct codes correspond to distinct motion modes.
Key Results
- On ETH, Social Ways obtains ADE/FDE=0.39/0.64, outperforming Social-GAN at 0.68/1.26, SoPhie at 0.70/1.43, and the Linear baseline at 0.59/1.22. On Hotel it reaches 0.39/0.66, also improving over the principal stochastic baselines.
- Results on Univ, ZARA01, and ZARA02 are 0.55/1.31, 0.44/0.64, and 0.51/0.92. ZARA scenes have narrower pathways and lower trajectory variance, so the model’s diversity advantage does not consistently produce the lowest pointwise error.
- On the synthetic three-mode benchmark, Info-GAN preserves nearly all modes, whereas Vanilla-GAN, L2-GAN, and S-GAN-V20 collapse modes. L2 accelerates convergence but harms coverage; 1-NN and EMD likewise favor Info-GAN and Unrolled-GAN.
Significance
Pedestrian motion has no single inevitable future: identical histories may lead to avoidance, following, turning, or continuation. A model that outputs only an average path can misrepresent risk for autonomous vehicles. Social Ways reframes forecasting as sampling a conditional distribution, providing downstream planners with alternative futures. Improvements on ETH and Hotel suggest value for nonlinear motion, open spaces, and group behavior. The synthetic benchmark is equally important because real ETH/UCY data contain few clearly multimodal conditional cases, making ordinary accuracy metrics insufficient for testing mode preservation.
Technical Contribution
The paper combines conditional trajectory GANs with Info-GAN and deliberately removes the generator’s L2 loss. Its LSTM encoder-decoder is paired with structured, learnable social attention rather than unstructured pooling. Three geometric priors—Euclidean distance, relative bearing, and constant-velocity closest approach—are embedded and weighted through scaled dot-product softmax attention. The discriminator encodes observations and futures separately, while Q reconstructs c. This training objective targets both realism and distributional coverage, addressing a failure mode that minimum-error objectives often conceal.
Novelty
The fundamental novelty is not merely injecting random noise, but enforcing an information link between a latent code and generated trajectories. Compared with Social-GAN, SoPhie, and Variety-loss approaches, this directly attacks mode dropping through the objective function. The authors also introduce a six-condition synthetic dataset with three modes per condition, then evaluate generated-versus-real distributions using 1-nearest-neighbor accuracy and Earth Mover’s Distance rather than relying only on qualitative plots or best-of-K error.
Limitations
- Clearly multimodal conditional examples are rare in the real ETH/UCY data, so the strongest evidence for mode preservation comes from a hand-designed synthetic benchmark whose realism is limited.
- The sampler generates a target pedestrian independently; neighbors influence it through encoded histories, but their futures are not jointly generated, so cross-agent consistency is not guaranteed.
- The study gives limited analysis of computation, calibration, hyperparameter sensitivity, and modern alternatives such as graph networks or diffusion models.
Future Work
Future work should jointly generate all agents, incorporate maps, obstacles, goals, visual semantics, and intent, and evaluate calibration and collision risk on larger and more diverse datasets. Combining Info-GAN with graph neural networks or diffusion samplers could improve long-horizon consistency. Deployment studies should report sampling latency, uncertainty quality, collision rate, and planner-level safety rather than ADE/FDE alone.
AI Executive Summary
Predicting where pedestrians will move next is central to autonomous driving, mobile robots, and urban safety. The challenge is not simply noise: one observed history can plausibly lead to several futures, such as continuing forward, avoiding a person, or changing direction. Constant-velocity and Social-Force models provide strong priors but limited adaptability. Social-LSTM captures interactions with recurrent networks, while Social-GAN and SoPhie generate samples; nevertheless, regression can average incompatible futures and ordinary GANs can collapse to a few common modes.
Amirian, Hayet, and Pettre introduce Social Ways, a conditional trajectory GAN. LSTM-E encodes the target and neighboring histories. Attention pooling learns social influence from distance, bearing angle, and distance of closest approach. LSTM-D then combines the target representation, pooled neighbor features, and noise to generate 12 future frames. The central design is Info-GAN: a latent code c is fed to the generator, and Q is trained to recover it from the generated trajectory, maximizing a mutual-information lower bound. The authors also remove the L2 generator loss because it accelerates convergence at the cost of diversity.
Using roughly 2.8 seconds of observations and forecasting 4.8 seconds, the model achieves ADE/FDE=0.39/0.64 on ETH and 0.39/0.66 on Hotel. On ETH, this beats Social-GAN’s 0.68/1.26 and SoPhie’s 0.70/1.43. Performance is less dominant in low-variance ZARA scenes. A dedicated synthetic benchmark with three modes per condition shows that Info-GAN preserves nearly all modes after about 90,000 iterations, while Vanilla-GAN, L2-GAN, and Variety-loss models collapse. The work’s broader contribution is therefore distributional: it asks not only for a close trajectory, but for a faithful map of plausible futures. Joint multi-agent consistency and real-world safety validation remain open challenges.
Deep Analysis
Background
Pedestrian prediction has evolved from constant-velocity, computational-geometry, and Social-Force models to data-driven Gaussian processes and recurrent neural networks. Social-LSTM introduced social pooling; Social-GAN and SoPhie addressed sampling and attention. These methods improved realism, but point estimates can average distinct intentions, while GANs are vulnerable to mode collapse. ETH and UCY provide real trajectories, yet contain few unambiguous multimodal conditional examples.
Core Problem
For N pedestrians, each state is xi_t=[position, velocity]. Given τ+1 observed frames, the system must predict the target’s next T states conditioned on all agents’ histories. The desired output is p(xi_1:T|Xi−τ:0), not one deterministic path. The problem is difficult because social interaction is coupled with unknown neighbor intentions, long-horizon uncertainty, sparse multimodal evidence, and GAN instability.
Innovation
- �� Info-GAN introduces latent code c and Q, maximizing mutual information to separate behavioral modes.
- �� Removing generator L2 avoids pulling all samples toward a conditional mean.
- �� Attention pooling uses three interpretable geometric features, while the network learns their combination and neighbor weights.
- �� A six-condition, three-mode synthetic dataset directly tests mode retention with 1-NN and EMD, exposing failures hidden by best-of-K accuracy.
Methodology
- �� Input: target and neighbor histories, with τ=7 and four-dimensional position/velocity states.
- �� Encoding: LSTM-E recursively computes hi_t=λe(hi_{t−1},μ(xi_t)).
- �� Social representation: embed δij=[distance, bearing, closest approach] as fij=φ(δij); obtain aij through scaled dot-product softmax attention.
- �� Generation: LSTM-D receives [hi_t,Σj≠iaijhj_t,z] and predicts two-dimensional future positions through 64–32–2 fully connected layers.
- �� Discrimination: separate LSTMs encode observed and future portions; D predicts real/fake and Q reconstructs c.
- �� Objective: adversarial value plus the Info-GAN term −λE[log Q(c|G)], with no L2 term.
Experiments
Experiments use ETH, Hotel, ZARA01, ZARA02, and Univ, with leave-one-subset-out evaluation across five scenes. Baselines include Linear, S-Force, Social-LSTM, S-GAN, S-GAN-P, and SoPhie. The model observes about 2.8 seconds and predicts 4.8 seconds; stochastic methods generate K=20 samples and use the closest sample for ADE/FDE. PyTorch implementation uses batch size 64, learning rates 0.001 for G and 0.0001 for D, momentum 0.9, and 20,000 training epochs. Synthetic ablations compare Vanilla-GAN, L2-GAN, S-GAN-V20, and Unrolled10.
Results
Social Ways reaches 0.39/0.64 on ETH versus 0.68/1.26 for S-GAN, and 0.39/0.66 on Hotel versus 0.44/0.89 for S-GAN-P. Scores on Univ, ZARA01, and ZARA02 are 0.55/1.31, 0.44/0.64, and 0.51/0.92. Its advantage is strongest where paths are open and nonlinear. On the synthetic benchmark, Info-GAN retains almost all modes after roughly 90,000 iterations. Adding L2 produces faster convergence but worse 1-NN and EMD distribution matching; Variety loss shows no clear advantage over Vanilla-GAN.
Applications
Autonomous vehicles can sample alternative pedestrian futures and evaluate collision risk under each, instead of trusting an averaged path. Service robots can select among continuing, yielding, following, and detouring behaviors in hospitals or malls. Practical deployment requires reliable tracking, coordinate normalization, sufficient history, real-time sampling, and additional map or obstacle constraints. ETH and Hotel suggest particular value in open environments with nonlinear interaction.
Limitations & Outlook
The model samples one target at a time, so independently plausible predictions may be mutually inconsistent. Its interaction features rely partly on constant-velocity geometry and omit maps, obstacles, goals, appearance, and explicit intent. Real-data evidence for multimodality is weak, making synthetic results central but not definitive. The paper does not report collision rates, probabilistic calibration, inference latency, computational scaling, or comparisons with later diffusion and graph-based predictors.
Plain Language Accessible to non-experts
Imagine a route adviser standing at a busy intersection. It watches one person’s recent steps and also observes nearby people: who is close, who is approaching from the side, and whether two walkers would nearly bump into each other if they kept going. Instead of announcing one supposedly correct route, the adviser writes several believable scenarios—straight ahead, stepping around someone, or following a companion.
Training uses a critic and a route designer. The critic tries to identify routes that do not look human, while the designer improves them. A basic designer may discover one popular route and repeat it forever; this is mode collapse. Social Ways gives the designer a hidden “style card” and requires the card to be recoverable from the finished route. Different cards therefore encourage genuinely different possibilities.
The authors remove another rule that forces every generated route to hug one recorded answer. That rule can make all routes converge toward an average. On ETH, Social Ways reaches 0.39 average displacement error and 0.64 final displacement error. In a special three-route test, it preserves almost every branch. It is therefore less like guessing one answer and more like drawing a useful map of what might happen next.
ELI14 Explained like you're 14
Suppose you are watching a student walk outside school. After a few seconds, where will they go? Maybe the classroom, the cafeteria, or around a crowd. If you predict only one path, you miss the fact that several answers can be reasonable!
Social Ways works like a game that writes multiple future storylines. It watches the student’s recent movement and the people nearby. It checks who is close, which direction they are coming from, and whether their paths might get dangerously close. Then it uses different hidden “plot choices” to create many possible futures.
There is also a referee. The referee sees whether a route looks like something a real person would walk, while the route-maker tries to fool it. Info-GAN adds a clever rule: each plot choice should leave a recognizable signature in the route. So the system cannot simply copy the same future again and again—bye-bye, mode collapse!
The researchers built a mini world with three different route branches to test this. Social Ways kept almost all of them. On the ETH dataset, its error numbers were 0.39 and 0.64, better than Social-GAN’s 0.68 and 1.26. It is not magic, though: it mainly predicts one person, not everyone together. Add maps, traffic lights, and goals, and future robots could make much safer decisions!
Glossary
GAN (Generative Adversarial Network)
A generator creates samples while a discriminator tries to distinguish them from real data. Adversarial training pushes generated samples toward the data distribution.
Social Ways uses a conditional GAN to generate futures from observed trajectories.
Info-GAN
Info-GAN augments a GAN with a latent code and maximizes mutual information between that code and generated outputs. This encourages interpretable, diverse variations.
It is the main mechanism for reducing mode collapse and mode dropping.
Mode collapse
A generator produces only a narrow set of similar outputs instead of covering the multiple modes of the target distribution. In forecasting, this hides meaningful alternative futures.
The synthetic benchmark is designed specifically to expose this failure.
Attention pooling
Attention pooling assigns learned weights to neighboring representations and combines them into one social feature. Softmax usually normalizes the weights.
Social Ways uses it to learn which pedestrians influence the target.
ADE/FDE
Average Displacement Error averages position errors across predicted times; Final Displacement Error measures the final-position error. Lower values indicate closer predictions.
They are the main accuracy metrics on ETH and UCY.
Earth Mover’s Distance
EMD measures the minimum cost of transporting mass from one sample distribution to another. Here, trajectory ADE is used as the ground distance.
It evaluates whether generated and real trajectory sets have similar distributions.
Open Questions Unanswered questions from this research
- 1 Real datasets rarely provide many identical histories with clearly different futures, so it remains uncertain how well the synthetic three-mode benchmark predicts real intent distributions.
Applications
Immediate Applications
Autonomous-driving risk estimation
A vehicle can sample multiple pedestrian futures and calculate collision risk for each, rather than relying on one averaged path. Required components include robust tracking, scene coordinates, history buffering, and sufficiently fast sampling.
Social navigation for robots
Hospital or mall robots can choose whether to continue, yield, follow, or detour using diverse forecasts and learned neighbor influence. Deployment should add maps, obstacle constraints, and explicit safety margins.
Long-term Vision
Joint probabilistic crowd-world models
Future systems could generate mutually consistent futures for every pedestrian while incorporating goals, maps, visual semantics, and intent. Such models could become probabilistic world models for autonomous planning, subject to safety and calibration validation.
Abstract
This paper proposes a novel approach for predicting the motion of pedestrians interacting with others. It uses a Generative Adversarial Network (GAN) to sample plausible predictions for any agent in the scene. As GANs are very susceptible to mode collapsing and dropping, we show that the recently proposed Info-GAN allows dramatic improvements in multi-modal pedestrian trajectory prediction to avoid these issues. We also left out L2-loss in training the generator, unlike some previous works, because it causes serious mode collapsing though faster convergence. We show through experiments on real and synthetic data that the proposed method leads to generate more diverse samples and to preserve the modes of the predictive distribution. In particular, to prove this claim, we have designed a toy example dataset of trajectories that can be used to assess the performance of different methods in preserving the predictive distribution modes.