Perturbed gradient descent efficiently escapes saddle points, converging to approximate second-order stationary points with nearly dimension-free complexity.

Chi Jin, Rong Ge, Praneeth Netrapalli et al.

Proves SGD learns conjugate kernel functions in deep networks within polynomial time, covering depths up to log(n).

Amit Daniely

Proposes deep energy-based soft Q-learning for continuous spaces, improving exploration and skill transfer with a Boltzmann policy approximation.

Tuomas Haarnoja, Haoran Tang, Pieter Abbeel et al.

This paper proves that for a convolutional neural network with ReLU activations and Gaussian inputs, gradient descent converges to the global optimum in polynomial time.

Alon Brutzkus, Amir Globerson

This paper proves the optimal regret in combinatorial bandits grows as \(\widetilde{\Theta}(k^{3/2}\sqrt{dT})\), refuting prior conjectures.

Alon Cohen, Tamir Hazan, Tomer Koren

Introducing SiLU/dSiLU activations combined with on-policy TD(λ)/Sarsa(λ), surpassing DQN in Atari and Tetris benchmarks.

Stefan Elfwing, Eiji Uchibe, Kenji Doya

Introduces sparsely-gated mixture-of-experts (MoE) layer achieving over 1000x capacity increase with 137 billion parameters, outperforming SOTA in language modeling and translation.

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz et al.

PixelCNN++ uses discretized Logistic mixture likelihood, improving training speed and image quality, achieving state-of-the-art on CIFAR-10.

Tim Salimans, Andrej Karpathy, Xi Chen et al.

Proposed a Thompson Sampling extension for stochastic decisions with graph feedback, significantly reducing Bayesian regret.

Aristide C. Y. Tossou, Christos Dimitrakakis, Devdatt Dubhashi

EWC uses Fisher-weighted constraints to reduce forgetting; it works on permuted MNIST and sequential Atari learning.

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz et al.

Proposed TGAN with dual generators and Singular Value Clipping for stable, high-quality unsupervised video synthesis.

Masaki Saito, Eiichi Matsumoto, Shunta Saito

UNREAL combines A3C with multi-task auxiliary learning, boosting Atari and Labyrinth scores to 87%, with 10× faster learning.

Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki et al.

Learned dexterous manipulation policies using trajectory optimization and deep learning; nearest neighbors achieved highest performance.

Vikash Kumar, Abhishek Gupta, Emanuel Todorov et al.

Using random label/noise experiments, reveals deep networks' capacity to memorize, challenging classical generalization theories.

Chiyuan Zhang, Samy Bengio, Moritz Hardt et al.

Proposes Equation Learner (EQL), a neural network that learns analytical expressions and excels in extrapolation beyond training data.

Georg Martius, Christoph H. Lampert

Deep ReLU networks more efficiently approximate smooth functions in Sobolev spaces than shallow networks.

Dmitry Yarotsky

Combining deep video prediction with MPC enables robots to manipulate unseen objects without calibration or physical models.

Chelsea Finn, Sergey Levine

SeqGAN combines GAN with policy gradient reinforcement learning to generate high-quality discrete sequences, overcoming gradient issues.

Lantao Yu, Weinan Zhang, Jun Wang et al.

PGA and SGLD poison MovieLens factorization recommenders; at β=0.6, SGLD reaches detection-test p-values above 0.7.

Bo Li, Yining Wang, Aarti Singh et al.

Using the Polyak-Lojasiewicz (PL) inequality, this paper proves linear convergence of gradient and proximal gradient methods for non-strong convex problems.

Hamed Karimi, Julie Nutini, Mark Schmidt