Online Learning under Delayed Feedback

Online learning has become a cornerstone of adaptive systems, with foundational work like Auer et al. (2002) on UCB algorithms setting the stage. As applications expanded into distributed and web-based environments, feedback delays emerged as a critical obstacle. Early studies, such as Weinberger & Ordentlich (2002), analyzed fixed delays, establishing that delays inflate regret multiplicatively. Later, works like Dudik et al. (2011) and Neu et al. (2010) extended analysis to stochastic and bandit settings, but lacked a unified framework. The complexity of real-world delays—stochastic, unbounded, and correlated—necessitated new algorithms and bounds. This paper synthesizes prior insights, proposing general algorithms with tight regret guarantees, thus pushing the frontier of delayed online learning.

The main challenge is that feedback delays distort the learning process, causing the regret to grow beyond classical bounds. In adversarial settings, delays lead to multiplicative regret inflation, severely impairing performance. In stochastic environments, the impact is more manageable but still significant, adding an additive bias. Existing algorithms either assume fixed delays or lack theoretical guarantees under variable, unbounded delays. The core problem is designing algorithms that adaptively handle arbitrary delay distributions, maintain low regret, and are computationally feasible, addressing a gap between theory and practice in asynchronous systems.

The paper introduces two key innovations. First, BOLD employs multiple instances of a base algorithm, scheduling them based on delay estimates, ensuring regret scales with the maximum delay G*. Second, QPM-D buffers feedback in dedicated queues for each action, enabling the use of standard algorithms as if feedback were immediate. These methods incorporate delay estimates into regret bounds, leveraging concentration inequalities like Bernstein bounds to control the impact of delays. The algorithms are designed to be scalable and adaptable, providing near-optimal guarantees in both adversarial and stochastic regimes, a significant step beyond prior fixed-delay or heuristic approaches.

• �� BOLD: For each time step, select a free instance of the base algorithm or instantiate a new one; run instances in parallel, each handling a subset of the feedback, with the number of instances proportional to the maximum delay G*.
• �� QPM-D: Maintain a FIFO queue for each action; upon prediction, buffer incoming feedback; predict using the current action; update the base algorithm when feedback arrives.
• �� Both algorithms estimate G* using concentration inequalities, ensuring bounds hold with high probability.
• �� Theoretical analysis derives regret bounds by relating the number of instances and buffered feedback to the delay distribution, using properties of the base algorithms and probabilistic inequalities.
• �� Proofs establish that in adversarial environments, regret scales multiplicatively with delay, while in stochastic settings, it adds a term proportional to the expected delay, matching lower bounds in theory.

Experiments involved synthetic multi-armed bandit simulations and real-world click data from web advertising. The algorithms were compared against classical UCB and delayed variants, measuring cumulative regret over varying delay levels. Hyperparameters such as exploration constants and delay bounds were tuned for fairness. Results showed that BOLD and QPM-D maintained low regret even at high delays, outperforming benchmarks by 30-50%. Ablation studies confirmed the importance of G* estimation and queue management. The experiments validated theoretical bounds and demonstrated scalability, with runtime and memory overheads comparable to standard algorithms.

In adversarial scenarios, the regret bound for BOLD is (τ+1)f(n/(τ+1)), confirming the multiplicative inflation. For stochastic environments, the regret increases by O(E[τ]) plus a √E[τ] log n term, closely matching theoretical lower bounds. Empirical results showed that at delays up to τ=50, the proposed algorithms reduced regret by over 40% compared to naive delayed algorithms. The bounds adapt well to different delay distributions, including unbounded and heavy-tailed cases, demonstrating robustness and practical relevance.

These algorithms are applicable in distributed recommendation systems, online advertising, asynchronous reinforcement learning, and sensor networks, where feedback inherently experiences delays. They enable systems to maintain near-optimal performance without requiring synchronization or delay assumptions, thus improving user experience, system robustness, and scalability. The methods are particularly suited for large-scale, real-time environments where feedback latency varies unpredictably.

The algorithms depend on accurate estimation of G*, which may be challenging in highly dynamic or non-stationary environments. Computational overhead, while reduced, can still be significant in very high-dimensional action spaces. The current analysis assumes independence of delays and rewards, limiting applicability in correlated or non-stationary settings. Extending to continuous actions and non-i.i.d. delays remains an open challenge, requiring further theoretical and empirical work.