Internet Congestion Control via Deep Reinforcement Learning
Aurora uses PPO-based deep RL for congestion control, achieving near-full utilization from 1–128 Mbps links.
Key Findings
Methodology
The paper casts congestion control as a partially observable sequential decision problem. At each monitor interval, a sender changes its rate using a bounded history of latency gradient, latency ratio, and sending ratio derived from ACKs. Aurora uses a 32→16 fully connected tanh network, trained with PPO. Its reward is 10×throughput−1000×latency−2000×loss, balancing delivery rate, queuing delay, and packet loss.
Key Results
- On a 30 Mbps link with 1% random loss, Aurora distinguishes congestion-induced loss from independent random loss. TCP CUBIC halves its rate after either event, whereas Aurora often increases its rate after random loss and avoids overshooting when loss correlates with congestion.
- Although trained only on 1.2–6 Mbps links, Aurora was tested from 1 to 128 Mbps—over 20 times the upper training bandwidth. It achieved near-perfect utilization across capacities, with substantially lower self-inflicted latency than TCP CUBIC and performance comparable to PCC-Vivace.
- On a dynamic link whose capacity changed uniformly between 16 and 32 Mbps every five seconds, Aurora nearly matched BBR's throughput with lower latency. It outperformed PCC-Vivace on average; versus Copa, throughput was 4.2% higher but latency 16% worse.
Significance
This work shows that deep RL can learn useful temporal structure in Internet feedback, not merely solve games or robotics tasks. It addresses persistent weaknesses of rule-based TCP: confusing random and congestion losses, adapting poorly to changing capacity, and balancing throughput against delay. The result is also important methodologically: a small policy trained in a lightweight simulator can transfer to substantially different emulated environments, suggesting a practical path for data-driven protocol design.
Technical Contribution
The framework extends Performance-oriented Congestion Control by making rate control an RL policy over normalized ACK statistics and their history. Aurora uses a smooth multiplicative rate-update rule, a compact neural architecture, and PPO to handle delayed consequences and partial observability. The authors additionally release an OpenAI Gym environment and testing module. Unlike linear policies, bandit formulations, or hand-tuned protocols, this design learns nonlinear temporal mappings while remaining small enough for end-host deployment.
Novelty
The paper is an early systematic demonstration of deep RL for Internet congestion control and introduces Aurora as a reproducible implementation. Its fundamental difference from CUBIC, BBR, RemyCC, Copa, and PCC-Vivace is not merely a new objective; it learns the observation-to-rate policy from experience, using short histories to infer hidden network conditions. The contribution combines algorithmic formulation, a practical simulator, and cross-environment evaluation.
Limitations
- The main training and demonstrations focus on a single sender and simplified links. Multi-flow fairness, coexistence with heterogeneous protocols, and strategic or adversarial traffic are not fully addressed.
- RL provides no automatic safety guarantee. Out-of-distribution conditions, noisy observations, or an aggressive learned action could create queues or unfairness; deployment therefore requires rate caps, fallback controllers, and extensive real-network validation.
Future Work
Important directions include multi-agent fairness objectives, constrained or risk-sensitive RL, sim-to-real evaluation, and robustness to unseen routing, wireless, and queue dynamics. The released Gym environment enables controlled comparisons of PPO and alternative RL algorithms. Future systems should also combine learned policies with safety shields, interpretable diagnostics, and long-term deployment telemetry.
AI Executive Summary
Internet congestion control determines how quickly data sources transmit over shared links. Yet senders usually observe only ACK-derived feedback, not link capacity, queue policy, or competing flows. TCP CUBIC and related handcrafted protocols can mistake random wireless loss for congestion, underuse capacity, or create excessive queues when conditions change rapidly.
Jay and colleagues propose Aurora, a deep reinforcement-learning controller. Every monitor interval, it receives a short history of latency gradient, latency ratio, and sending ratio, then a 32→16 tanh network produces a rate adjustment. PPO trains the policy using a reward that favors throughput while penalizing latency and loss. A lightweight OpenAI Gym simulator randomizes bandwidth, delay, queue size, and loss rate.
Aurora generalizes far beyond its training distribution: trained on 1.2–6 Mbps links, it performs well from 1 to 128 Mbps. On a dynamic 16–32 Mbps link, it nearly matches BBR's throughput with lower latency, improves over PCC-Vivace, and achieves 4.2% higher throughput than Copa at a 16% latency cost. The work is promising but not deployment-ready: fairness, safety, multiple flows, and out-of-distribution behavior remain open systems problems.
Deep Analysis
Background
Congestion control has evolved from TCP heuristics toward performance-oriented designs. TCP CUBIC remains Linux's default; BBR models bottleneck bandwidth and round-trip time; RemyCC uses offline policy synthesis; PCC-Vivace and Copa optimize explicit throughput-latency objectives. These protocols address important failure modes, but mobile links, real-time applications, and heterogeneous competition make fixed rules difficult to tune universally.
Core Problem
A sender must choose rate using local feedback while hidden variables—capacity, buffer size, queue discipline, and other flows—change over time. Loss is ambiguous: it may reflect congestion or random channel errors. Actions also have delayed effects because excessive sending fills buffers before losses appear. The resulting problem is partially observable, long-horizon, continuous control with fairness and safety requirements.
Innovation
The paper contributes four linked ideas. First, it formulates congestion control as deep RL rather than a static heuristic or bandit. Second, it uses normalized statistics and bounded histories to improve transfer across link properties. Third, Aurora applies smooth multiplicative rate changes rather than unconstrained rate jumps. Fourth, a compact PPO policy trained in a simple simulator transfers to Mininet and Pantheon, while the authors release a Gym test suite for reproducibility.
Methodology
- �� Observation: construct vt from latency gradient, latency ratio, and sending ratio; use st=(vt−(k+d),…,vt−d).
- �� Action: for at≥0, xt=xt−1(1+αat); otherwise xt=xt−1/(1−αat), with α=0.025.
- �� Environment: simulate FIFO links with randomized bandwidth, latency, queue size, and random loss.
- �� Objective: optimize discounted return under 10 throughput−1000 latency−2000 loss.
- �� Policy: train a 32→16 tanh MLP with PPO; compare history k=1–10 and γ∈{0,0.5,0.99}.
- �� Validation: transfer the policy to emulated real packets and compare against established protocols.
Experiments
Training samples use 100–500 packets/s, 50–500 ms latency, 2–2981 packets of queue capacity, and 0–5% random loss; queue size is log-uniform while other variables are uniform. Evaluation uses two-minute single-flow tests in Mininet and Pantheon, with a standard 30 Mbps, 30 ms, 1000-packet, zero-loss configuration unless varied. Baselines include TCP CUBIC and PCC-Vivace, with BBR, Copa, and RemyCC in dynamic-link comparisons. Metrics are utilization, throughput, and self-inflicted latency.
Results
Aurora remains robust across 1–128 Mbps bandwidth, 1–512 ms latency, 1–10,000-packet queues, and up to 8% random loss. At 1 ms latency it performs poorly because emulation adds roughly 1 ms of unmodeled processing noise. Otherwise it generally approaches capacity and beats CUBIC on delay. γ=0 fails to learn, γ=0.5 eventually learns, and γ=0.99 learns fastest; k=1 is weaker, while k=2 is nearly as effective as longer histories.
Applications
Potential users include cellular video delivery, cloud gaming, interactive web services, IoT, and edge applications. Deployment requires ACK-statistics instrumentation, action bounds, a conservative fallback protocol, and staged testing. The small policy is attractive for end hosts, but operators must evaluate its behavior alongside existing TCP flows and under many-user competition.
Limitations & Outlook
The evidence is mainly simulation and emulation, not long-term Internet deployment. The model may miss routing changes, ACK compression, wireless scheduling, bursty arrivals, and multi-path interactions. The reward does not inherently guarantee fairness, safety, or stability across agents, and its coefficients are task-specific. Future work should use constrained multi-agent RL, distribution-shift tests, uncertainty monitoring, formal safety shields, and larger real-world trials.
Plain Language Accessible to non-experts
Imagine a highway with one narrow bridge and thousands of drivers. Each driver wants to cross quickly, but no driver can see the bridge's true capacity or how many others are approaching. Traditional TCP behaves like a driver who treats every fallen package as proof of a traffic jam and immediately cuts speed in half. A package might have fallen for an unrelated reason, so the bridge can remain partly empty.
Aurora acts like a traffic manager who learns from repeated observations. Every short period it checks how quickly cars arrive, whether waiting time is rising, and how many cars sent forward actually reach the other side. It remembers recent observations instead of reacting to one event. If faster driving causes waiting to rise, it slows down; if an isolated failure occurs without growing queues, it can continue using the road.
Researchers trained this manager on many computer-generated highways and then tested it on roads with very different widths, delays, queue sizes, and failure rates. It used the road almost fully and often caused less waiting than older drivers. However, several managers sharing one bridge still need rules for fairness, emergency braking, and cooperation. Learning alone is not a complete traffic law.
ELI14 Explained like you're 14
Think of an online game where everyone is trying to send messages through one tiny doorway. Send too slowly and the game freezes; send too quickly and messages pile up, arrive late, or disappear. TCP CUBIC uses a simple rule: if even one message disappears, it assumes the doorway is crowded and cuts its speed in half. But maybe that message was lost because of a random wireless glitch—not because the doorway was full!
Aurora is like a player who learns by practicing. Every few moments it checks three clues: how long messages take, whether that time is increasing, and how many sent messages are confirmed as received. A small neural network studies the recent clues and chooses whether to speed up or slow down. PPO is the coach: it gives better scores for fast delivery and lower delay and loss.
Here is the surprising part: Aurora trained on links from 1.2 to 6 Mbps, but tests ranged from 1 to 128 Mbps. It still used the available connection very well. On a connection changing between 16 and 32 Mbps, it had nearly BBR-level throughput with lower delay. Compared with Copa, it sent 4.2% more data but had 16% more delay.
Is it perfect? Not yet! It may not know that many other players are sharing the doorway. Before real Internet use, it needs speed limits, safety brakes, fairness rules, and backup behavior.
Glossary
Deep Reinforcement Learning
A learning paradigm in which an agent interacts with an environment and improves decisions from rewards. Deep networks approximate policies or value functions in large or continuous spaces.
Aurora learns rate-control decisions from network feedback.
Congestion Control
The process of regulating sending rates so shared network capacity is used efficiently without excessive queues, loss, or delay. Feedback commonly arrives through acknowledgements.
It is the paper's target systems problem.
PPO
Proximal Policy Optimization is a policy-gradient method that limits update magnitude to stabilize learning. It is used here for continuous rate-adjustment decisions.
Aurora is trained with PPO in stable-baselines.
Monitor Interval
A time window during which a sender maintains a rate and collects feedback. At the next interval it selects a rate change.
The interval defines the action and observation cycle.
PCC-Vivace
A performance-oriented congestion-control protocol that explicitly optimizes a utility involving throughput, latency, and loss. It is a major state-of-the-art baseline.
Aurora is compared with it in robustness tests.
Self-inflicted Latency
Extra delay created by the sender filling a bottleneck queue through excessive transmission. It measures how much delay the controller itself causes.
Figures 6 and 7 report this latency metric.
Open Questions Unanswered questions from this research
- 1 Fairness under competition remains unresolved. Multiple Aurora agents, or Aurora competing with TCP, may optimize local rewards by taking disproportionate capacity. Multi-agent objectives and enforceable fairness constraints are needed.
- 2 Safety under distribution shift is unproven. Unseen wireless events, routing changes, or observation noise could trigger aggressive actions. Reliable deployment needs uncertainty estimation, safety shields, fallback controllers, and long-duration Internet trials.
Applications
Immediate Applications
Cellular video and cloud gaming
Content providers could prototype Aurora at the end host to avoid unnecessary rate reductions after random wireless loss and to respond faster to changing capacity. Required safeguards include rate caps, ACK instrumentation, a CUBIC-like fallback, and staged A/B evaluation.
Congestion-control research testbed
Researchers can use the released OpenAI Gym simulator to randomize bandwidth, latency, queues, and loss, then validate policies with Mininet or Pantheon. This supports reproducible comparisons among PPO, alternative RL methods, and handcrafted protocols.
Long-term Vision
Cooperative learning-based networks
Future network endpoints could coordinate throughput, latency, energy, and fairness rather than optimize isolated flows. With operator policies and safety constraints, learned controllers might adapt cellular, edge, and data-center networks more continuously than fixed protocols.
Abstract
We present and investigate a novel and timely application domain for deep reinforcement learning (RL): Internet congestion control. Congestion control is the core networking task of modulating traffic sources' data-transmission rates to efficiently utilize network capacity, and is the subject of extensive attention in light of the advent of Internet services such as live video, virtual reality, Internet-of-Things, and more. We show that casting congestion control as RL enables training deep network policies that capture intricate patterns in data traffic and network conditions, and leverage this to outperform the state-of-the-art. We also highlight significant challenges facing real-world adoption of RL-based congestion control, including fairness, safety, and generalization, which are not trivial to address within conventional RL formalism. To facilitate further research and reproducibility of our results, we present a test suite for RL-guided congestion control based on the OpenAI Gym interface.