LINE efficiently embeds large-scale networks by optimizing first- and second-order proximities with edge sampling, handling millions of nodes and billions of edges.

Jian Tang, Meng Qu, Mingzhe Wang et al.

Exp3.G classifies feedback graphs: strongly observable gives ~√(αT), weakly observable ~δ^(1/3)T^(2/3), and unobservable Θ(T).

Noga Alon, Nicolò Cesa-Bianchi, Ofer Dekel et al.

Proposes a meta-algorithm (SAOL) transforming low-regret algorithms into strongly adaptive ones, ensuring near-optimal performance on every interval with \( O(\log T) \) overhead.

Amit Daniely, Alon Gonen, Shai Shalev-Shwartz

TRPO (Trust Region Policy Optimization) guarantees monotonic policy improvement using KL constraints, excelling in large neural network policy training for robotics and Atari games.

John Schulman, Sergey Levine, Philipp Moritz et al.

Proposes a multi-layer LSTM encoder-decoder framework for unsupervised video representation learning, improving action recognition especially with limited labeled data.

Nitish Srivastava, Elman Mansimov, Ruslan Salakhutdinov

MADE uses masked autoencoders to enforce autoregressive constraints, boosting distribution estimation efficiency.

Mathieu Germain, Karol Gregor, Iain Murray et al.

Proposes Batch Normalization (BN) to standardize layer inputs, accelerating training and reducing internal covariate shift, achieving 4.9% top-5 error on ImageNet.

Sergey Ioffe, Christian Szegedy

Introduced a visual attention-based image captioning model achieving state-of-the-art performance on MS COCO.

Kelvin Xu, Jimmy Ba, Ryan Kiros et al.

Deep Fried ConvNets replace fully connected layers with adaptive Fastfood transforms, reducing parameters by over 90% without accuracy loss.

Zichao Yang, Marcin Moczulski, Misha Denil et al.

Adam combines first and second moment estimates for adaptive learning rates, accelerating large-scale stochastic optimization.

Diederik P. Kingma, Jimmy Ba

Elastic Averaging SGD (EASGD) leverages elastic force to enable efficient distributed deep learning, reducing communication overhead while enhancing exploration, leading to faster convergence and better accuracy.

Sixin Zhang, Anna Choromanska, Yann LeCun

Implicit regularization, not network size, controls capacity in deep learning.

Behnam Neyshabur, Ryota Tomioka, Nathan Srebro

By adding a cross-covariance penalty to autoencoders, hidden factors of variation in deep networks are discovered.

Brian Cheung, Jesse A. Livezey, Arjun K. Bansal et al.

A spherical spin-glass mapping explains why SGD in large networks reaches low-energy, high-quality local minima.

Anna Choromanska, Mikael Henaff, Michael Mathieu et al.

This paper analyzes the computational complexity of training neural networks, showing over-parameterized networks are easier to optimize and proposing polynomial activation-based algorithms for depth-2 and depth-3 networks.

Roi Livni, Shai Shalev-Shwartz, Ohad Shamir

Volumetric spanners use at most 12d support points to enable low-variance exploration and efficient near-optimal BLO over general convex sets.

Elad Hazan, Zohar Karnin, Raghu Mehka

Proposes spectral and local clustering graph networks with input-size-independent parameters, effective on low-dimensional graphs.

Joan Bruna, Wojciech Zaremba, Arthur Szlam et al.

Deep Mixture of Experts excels on MNIST, learning location and class-specific experts.

David Eigen, Marc'Aurelio Ranzato, Ilya Sutskever

Four gradient estimators for stochastic binary neurons enable efficient conditional computation and sparsity in deep networks.

Yoshua Bengio, Nicholas Léonard, Aaron Courville

Introduces Exp3-DOM algorithm leveraging dominating sets to bound regret in directed, dynamic observation graphs (under 30 words)

Noga Alon, Nicolò Cesa-Bianchi, Claudio Gentile et al.