Proposes a multi-layer LSTM encoder-decoder framework for unsupervised video representation learning, improving action recognition especially with limited labeled data.

Nitish Srivastava, Elman Mansimov, Ruslan Salakhutdinov

MADE uses masked autoencoders to enforce autoregressive constraints, boosting distribution estimation efficiency.

Mathieu Germain, Karol Gregor, Iain Murray et al.

Proposes Batch Normalization (BN) to standardize layer inputs, accelerating training and reducing internal covariate shift, achieving 4.9% top-5 error on ImageNet.

Sergey Ioffe, Christian Szegedy

Introduced a visual attention-based image captioning model achieving state-of-the-art performance on MS COCO.

Kelvin Xu, Jimmy Ba, Ryan Kiros et al.

Deep Fried ConvNets replace fully connected layers with adaptive Fastfood transforms, reducing parameters by over 90% without accuracy loss.

Zichao Yang, Marcin Moczulski, Misha Denil et al.

Adam combines first and second moment estimates for adaptive learning rates, accelerating large-scale stochastic optimization.

Diederik P. Kingma, Jimmy Ba

Elastic Averaging SGD (EASGD) leverages elastic force to enable efficient distributed deep learning, reducing communication overhead while enhancing exploration, leading to faster convergence and better accuracy.

Sixin Zhang, Anna Choromanska, Yann LeCun

A spherical spin-glass mapping explains why SGD in large networks reaches low-energy, high-quality local minima.

Anna Choromanska, Mikael Henaff, Michael Mathieu et al.

This paper analyzes the computational complexity of training neural networks, showing over-parameterized networks are easier to optimize and proposing polynomial activation-based algorithms for depth-2 and depth-3 networks.

Roi Livni, Shai Shalev-Shwartz, Ohad Shamir

Volumetric spanners use at most 12d support points to enable low-variance exploration and efficient near-optimal BLO over general convex sets.

Elad Hazan, Zohar Karnin, Raghu Mehka

Proposes spectral and local clustering graph networks with input-size-independent parameters, effective on low-dimensional graphs.

Joan Bruna, Wojciech Zaremba, Arthur Szlam et al.

Deep Mixture of Experts excels on MNIST, learning location and class-specific experts.

David Eigen, Marc'Aurelio Ranzato, Ilya Sutskever

Four gradient estimators for stochastic binary neurons enable efficient conditional computation and sparsity in deep networks.

Yoshua Bengio, Nicholas Léonard, Aaron Courville

Introduces Exp3-DOM algorithm leveraging dominating sets to bound regret in directed, dynamic observation graphs (under 30 words)

Noga Alon, Nicolò Cesa-Bianchi, Claudio Gentile et al.

Proposes BOLD and QPM-D algorithms for online learning with delayed feedback; theoretical bounds show multiplicative regret in adversarial and additive in stochastic settings.

Pooria Joulani, András György, Csaba Szepesvári

Proposes GP-UCB-PE, combining UCB and pure exploration for parallel Gaussian process optimization, with bounds outperforming sequential methods.

Emile Contal, David Buffoni, Alexandre Robicquet et al.

Establishes Ω(T^{2/3}) regret lower bound for online learning with switching costs under bandit feedback, highlighting the difficulty compared to full-information settings.

Nicolo Cesa-Bianchi, Ofer Dekel, Ohad Shamir

Introduces a weighted averaging scheme in projected stochastic subgradient method achieving O(1/t) convergence, with simple implementation and proof.

Simon Lacoste-Julien, Mark Schmidt, Francis Bach

Regularized reconstruction error enables auto-encoders to learn local features of data distribution, estimating gradients and Hessian matrices.

Guillaume Alain, Yoshua Bengio

Vovk extends inductive conformal prediction to conditional guarantees and proves that exact object-conditional validity is generally inefficient.

Vladimir Vovk