AI for Distributed Systems Design: Scalable Cloud Optimization Through Repeated LLMs Sampling And Simulators
Repeated LLM sampling with Eudoxia verification improved simulated FaaS throughput by up to 371.1% over FIFO.
Key Findings
Methodology
The paper implements an iterative generate-and-verify loop. A black-box LLM generates a Python scheduler from prompts, examples, and prior feedback; Eudoxia executes it deterministically on fixed workloads; a context manager summarizes throughput, errors, traces, and optional safety violations for the next iteration. Algorithm 1 repeats this process for 50 iterations and returns the policy with the highest median throughput across traces.
Key Results
- Across six standardized traces generated with Eudoxia, the best GPT5 policy improved throughput by 371.1% relative to FIFO; Sonnet 4.5 improved it by 313.2%, and Opus 4.1 by 263.2%. GPT5-mini achieved 0% improvement.
- Using the best of three independent runs, Sonnet cost $4.587 and took 2,785 seconds; Opus cost $37.27 and took 2,158 seconds; GPT5 cost $9.92 and took 8,292 seconds; GPT5-mini cost $1.65 and took 7,669 seconds.
- These are feasibility results, not a generalization benchmark or model ranking. Large cross-model variation shows that readable code plus deterministic simulation can expose both semantic failures and useful scheduling ideas.
Significance
The work reframes distributed-systems policy design as program search rather than manual rule engineering. This addresses a persistent industry problem: B2B cloud customers differ in workloads, concurrency, and resources, making one hand-crafted policy difficult to transfer. Academically, the paper offers a concrete neuro-symbolic pattern linking LLM reasoning, executable code, and simulator-based verification. Industrially, it enables low-risk pre-deployment exploration before changing production schedulers. However, the reported gains are only as trustworthy as Eudoxia’s fidelity to Bauplan and real workloads; simulated throughput must not be interpreted as production improvement.
Technical Contribution
The main contribution is a modular interface between stochastic code generation and deterministic systems evaluation, rather than a newly trained model. The policy generator performs LLM sampling, parsing, and validation; Eudoxia runs Generator–Scheduler–Executor simulations; the context manager synthesizes structured feedback and compresses long histories. LLMs, simulators, and managers can be independently replaced. Every candidate is logged as executable Python, enabling inspection, asynchronous debugging, and human modification. This creates an engineering substrate for searchable, interpretable policy evolution.
Novelty
The novelty lies in operationalizing repeated inference-time sampling as a search over distributed-systems programs and using a domain simulator as the verifier for machine-generated policies. Unlike supervised learning, reinforcement learning, or fixed heuristics, the method needs no policy-training dataset and does not expose the production system to untested code. Its fundamental combination is stochastic proposal, deterministic replay, structured feedback, and human-readable artifacts.
Limitations
- Eudoxia is an abstraction of Bauplan. If it misrepresents contention, tail latency, failures, or execution times, the reported 371.1% gain may disappear in a real cluster.
- The study uses only six traces, 50 iterations, and throughput as the main objective; it does not establish statistical confidence, cross-customer transfer, or latency–cost–fairness trade-offs.
- Quality and budget are unstable: GPT5-mini failed to improve FIFO, while model costs and runtimes differed substantially.
Future Work
The authors propose prompt evolution, richer context management, and stronger use of Eudoxia’s fine-grained logs. Evolutionary search could structure exploration, while reinforcement-trained smaller models might reduce cost. The verifier should be statistically calibrated against real deployments and extended to multi-tenant, Lambda-like serverless systems. The most ambitious direction is AI-assisted bootstrapping of new simulators from specifications, logs, and system code.
AI Executive Summary
Cloud schedulers are usually built from expert-written rules. Such rules are understandable and fast to execute, but they are expensive to design and often fail to transfer across customers whose workloads, resource pools, and latency expectations differ. Bauplan, a data-oriented Function-as-a-Service runtime, makes this challenge acute: short interactive queries and multi-hour pipelines compete for stateless virtual machines while respecting function-DAG dependencies.
The paper proposes repeated LLM sampling combined with deterministic verification. A policy generator writes a Python scheduler; Eudoxia runs it on fixed traces and measures throughput, p99 latency, failures, and optional safety conditions; a context manager converts the outcome into structured feedback for the next prompt. In 50 iterations over six traces, GPT5 improved throughput over FIFO by 371.1%, Sonnet 4.5 by 313.2%, and Opus 4.1 by 263.2%; GPT5-mini improved by 0%.
The result is a feasibility demonstration, not evidence of production gains or a model leaderboard. GPT5 required 8,292 seconds, whereas Opus cost $37.27 but took 2,158 seconds, illustrating quality–cost–latency trade-offs. The central scientific caution is simulator validity: a policy can optimize an inaccurate model. Future work must calibrate Eudoxia, test broader workloads and objectives, and investigate whether AI can help construct new verifiers.
Deep Analysis
Background
Distributed platforms traditionally rely on expert heuristics, including Kubernetes autoscaling policies. Learning-based systems such as FaaSRank and LAVA optimize selected decisions, but often remain within a predefined policy space. Bauplan represents SQL and data pipelines as function DAGs executed through a unified FaaS model. Eudoxia, introduced as a FaaS scheduling simulator, provides repeatable evaluation before costly production changes.
Core Problem
The task is to generate a scheduler that outperforms FIFO while handling arrivals, dependencies, resource pools, preemption, execution time, and failures. Manual design does not scale across B2B customers; direct LLM code generation produces syntax errors, incorrect API assumptions, and unverifiable performance claims. A safe search procedure therefore needs both flexible proposal generation and deterministic evaluation.
Innovation
First, the work treats LLM code generation as repeated black-box policy sampling rather than one-shot prompting. Second, it repurposes Eudoxia as a verifier, connecting generated programs to measurable system outcomes. Third, structured context stores policies, metrics, traces, and failures so later samples make targeted changes. Fourth, explicit interfaces decouple the LLM, policy generator, simulator, and context manager, allowing each component to evolve independently.
Methodology
- �� Initialization: system and user prompts describe Bauplan, Eudoxia, APIs, metrics, and a FIFO starting point.
- �� Generation: LiteLLM requests a Python scheduler; parsing, syntax, and interface checks reject invalid code.
- �� Simulation: valid policies run through Eudoxia’s Workload Generator, Scheduler, and Executor loop under fixed parameters.
- �� Measurement: the system records throughput, p99 latency, failed DAGs, traces, and optional monitors such as no oversubscription and bounded waiting.
- �� Feedback: the context manager summarizes outcomes and appends policy–metric–feedback records; overlong contexts are compressed.
- �� Selection: after 50 iterations, the policy with the best median throughput across six traces is returned.
Experiments
The authors generated six realistic Eudoxia traces: three parameter sets, each with two different random seeds. FIFO was the baseline because it is simple, inspectable, and operationally valid. Sonnet 4.5 and Opus 4.1 used temperature 0.7; GPT5 and GPT5-mini used high reasoning effort. Each model was run for 50 iterations, with three independent runs and the best result reported. Evaluation used median throughput across traces; LiteLLM supplied total cost and wall-clock time.
Results
GPT5 produced the largest improvement, 371.1% over FIFO, but required 8,292 seconds. Sonnet 4.5 reached 313.2% at the lowest cost, $4.587. Opus 4.1 reached 263.2%, cost $37.27, and finished in 2,158 seconds. GPT5-mini cost $1.65, took 7,669 seconds, and achieved no improvement. Thus repeated sampling can discover strong candidates, but quality is not monotonic in price, runtime, or nominal reasoning effort.
Applications
Cloud operators can use the loop to screen schedulers against replayed workloads before deployment. Relevant settings include FaaS platforms, serverless ETL, mixed batch–interactive pipelines, and multi-pool resource managers. Requirements are a credible simulator, a constrained policy API, representative traces, and safety monitors. Human review and canary deployment remain necessary because simulator scores do not guarantee production behavior.
Limitations & Outlook
The main threat is simulation-to-reality error: Eudoxia may omit contention, correlated failures, tail effects, or customer interactions. The small six-trace study optimizes mainly throughput and does not establish broad generalization or confidence intervals. LLMs may misunderstand APIs, repeat local strategies, or consume substantial inference budgets. Future work should add multi-objective evaluation, more workloads and failures, prompt evolution, evolutionary search, smaller RL-trained models, and multi-tenant serverless simulators.
Plain Language Accessible to non-experts
Imagine a busy restaurant with a few cooks and many orders. Some customers need a quick drink; others request a complicated meal that occupies a stove for hours. A simple first-come, first-served rule is easy to understand, but one huge order can block many small ones. The paper asks whether an AI assistant can invent better kitchen rules.
The assistant writes each proposed rule as a small Python program. Instead of risking the real restaurant, a simulator replays the same set of orders every time and reports how many meals were completed, how long customers waited, and whether anything failed. The assistant reads that report, changes its rule, and tries again for 50 rounds. Because the orders stay controlled, the comparison is fair, and because the programs are readable, a human can inspect them.
On six simulated order streams, GPT5 found a rule with 371.1% higher throughput than the basic FIFO rule; Sonnet 4.5 reached 313.2%. But the simulator is only a model of a restaurant. If it forgets equipment breakdowns or rush-hour behavior, a winning recipe may fail in real life. The work is therefore an automated test kitchen, not an autonomous restaurant manager.
ELI14 Explained like you're 14
Think of a school cafeteria with only a few counters and a huge line. Some students want a snack and leave quickly; others need a complicated meal. If everyone is served strictly in arrival order, one giant order can make the whole line crawl. The research question is: can an AI invent a smarter serving plan?
The researchers give an AI a safe practice game called Eudoxia. The AI writes a Python rule for choosing which cloud task goes next. Eudoxia feeds it the same six pretend workloads, counts how many tasks finish, and reports mistakes or waiting problems. The AI then uses the feedback to write a new rule. This happens 50 times, like replaying a level until a better strategy appears.
GPT5’s best rule beat the basic FIFO strategy by 371.1% in simulated throughput. Sonnet 4.5 improved it by 313.2%, while GPT5-mini improved nothing. Sounds incredible, right? But there is a catch: simulated results are not automatically real-world results. A game map may miss surprise traffic, broken machines, or unfair waiting.
The coolest idea is the safety net: let AI experiment in a virtual world, save its readable code, and have people check it before trying anything on real servers. It is like testing a new gaming strategy in practice mode before using it in a tournament!
Glossary
Large Language Model (LLM)
A model that generates text or code from context. Here it acts as a black-box proposal engine rather than being retrained for scheduling.
LLMs repeatedly generate Eudoxia-compatible Python policies through LiteLLM.
Function-as-a-Service (FaaS)
A cloud model in which users submit functions and the platform allocates resources to execute them. The scheduler manages queues, resources, dependencies, and preemption.
Bauplan uses FaaS for SQL and data pipelines.
Eudoxia
A deterministic simulator for FaaS scheduling. It models arrivals, resource allocation, execution time, and completion tracking.
It serves as the verifier for generated schedulers.
FIFO
First-In, First-Out: tasks are served in arrival order. It is transparent and valid, but can perform poorly for mixed short and long jobs.
It is the experimental baseline.
Throughput
The amount of work completed per unit time. It measures processing capacity rather than individual waiting time.
The primary optimization target; GPT5 improved it by 371.1% over FIFO.
Generate-and-verify
A workflow that proposes a candidate program, tests it with a verifier, and feeds the result into the next proposal. It combines stochastic generation with deterministic checking.
This is the paper’s central discovery loop.
Open Questions Unanswered questions from this research
- 1 How closely does Eudoxia predict real clusters? Answering this requires production replay, statistical calibration, and tests of contention, tail latency, failures, and customer interference.
- 2 Can policies transfer to unseen customers and workloads? Six traces are insufficient to establish generalization, and the study does not separate familiar from novel workload distributions.
- 3 Does throughput improvement harm p99 latency, fairness, cost, or safety? Multi-objective evaluation and explicit constraints are needed.
Applications
Immediate Applications
Pre-deployment scheduler screening
Cloud teams can replay historical traces in Eudoxia, ask several LLMs to generate Python schedulers, and compare them before production use. They need a stable simulator, a constrained API, safety monitors, and human review, followed by a canary rollout.
FaaS workload tuning
Serverless ETL, data-lake pipelines, and mixed batch–interactive platforms can search for customer-specific queue, priority, preemption, and resource-pool policies. Expected benefits include less manual rule maintenance, but latency and failure rates must be checked alongside throughput.
Long-term Vision
AI-assisted simulator construction
LLMs may eventually help derive simulators from specifications, logs, and system code, extending the approach to multi-tenant Lambda-like platforms. The central obstacle is verifying the verifier itself; otherwise AI may optimize behavior in an inaccurate virtual world.
Abstract
We explore AI-driven distributed-systems policy design by combining stochastic code generation from large language models (LLMs) with deterministic verification in a domain-specific simulator. Using a Function-as-a-Service runtime (Bauplan) and its open-source simulator (Eudoxia) as a case study, we frame scheduler design as an iterative generate-and-verify loop: an LLM proposes a Python policy, the simulator evaluates it on standardized traces, and structured feedback steers subsequent generations. This setup preserves interpretability while enabling targeted search over a large design space. We detail the system architecture and report preliminary results on throughput improvements across multiple models. Beyond early gains, we discuss the limits of the current setup and outline next steps; in particular, we conjecture that AI will be crucial for scaling this methodology by helping to bootstrap new simulators.