Building2Building: A Large Scale Benchmark for Generalizable Real-World Reinforcement Learning
B2B uses EnergyPlus to generate 6,000 buildings and evaluates RL transfer across goals, dynamics, action spaces, and domains.
Key Findings
Methodology
Building2Building (B2B) combines EnergyPlus with parametric building generators to create Gymnasium-compatible environments. It contains 1,000 instances of each of six building types, totaling 6,000 environments. Tasks independently vary reward goals, building dynamics, controllable actuators, and building domains. PPO provides core baselines, while an Amorpheus-inspired architecture uses a structured morphology representation, type-specific encoders/decoders, and shared Transformer layers for heterogeneous spaces.
Key Results
- In dynamics adaptation, 100 per-building PPO specialists were compared with a non-specialist PPO and a parameterized PPO. Specialists performed best on their own buildings, while conditioning on floor area, construction year, actuator count, and unit count improved transfer over an agent without building context.
- For cross-domain transfer, an Amorpheus variant was trained jointly on retail stores, restaurants, small offices, and medium offices for 1M environment steps, then evaluated on 20 unseen test buildings. It handled heterogeneous interfaces and exceeded the ASHRAE Guideline 36-inspired reactive baseline on some buildings.
- B2B explicitly defines action shifts: unitary systems change from 5 to 10 dimensions and central systems from 33 to 36, with reverse shifts also included. Reward normalization makes a shared energy weight wE more comparable across building scales and equipment.
Significance
B2B turns HVAC control from a collection of single-building case studies into a large, reproducible test of real-world generalization. It addresses gaps left by RLBench, Meta-World, Sinergym, and CityLearn, which either focus on robotics or provide relatively few building models and limited interface diversity. Academically, it enables controlled studies of goal adaptation, dynamics adaptation, action-space transfer, and cross-domain learning. Industrially, training over building distributions could reduce the need to construct a highly accurate digital twin and retrain a controller from scratch for every facility. Because buildings consume roughly one-third of global energy and HVAC represents nearly half of building usage, improved control could have substantial economic and climate benefits.
Technical Contribution
The main contribution is an integrated generation and evaluation stack: parametric EnergyPlus models, automatic conversion into Gymnasium environments, standardized tasks, and structured morphology metadata. A morphology contains common attributes, typed nodes with local attributes, and split/join mappings between global spaces and node-level observations/actions. Thermal zones and HVAC components become graph nodes, supporting graph neural networks, heterogeneous Transformers, and type-specific encoder-decoder models. The benchmark also implements ASHRAE Guideline 36-inspired PI and Trim-and-Respond controllers and provides compute_normalized_score, which divides cumulative return by the reactive controller’s return on the same simulation window.
Novelty
The novelty is primarily infrastructural and scientific rather than a new optimization algorithm. B2B unifies physically grounded HVAC simulation, large-scale procedural building diversity, and heterogeneous-interface transfer in one benchmark. Unlike Meta-World, RLBench, and Procgen, where variations often concern goals, friction, or visual layouts, B2B simultaneously varies climate, envelope, building topology, HVAC equipment, observation spaces, action spaces, and comfort-energy objectives. It therefore makes cross-morphology generalization a first-class evaluation problem.
Limitations
- The evidence is simulation-based. EnergyPlus cannot fully reproduce sensor noise, actuator latency, communication failures, equipment aging, occupant overrides, or safety-critical operating constraints, so successful benchmark transfer does not yet establish reliable real-building deployment.
- Only PPO and one Amorpheus-inspired policy are experimentally demonstrated. The paper does not provide a broad comparison with meta-RL, offline RL, world models, successor features, or continual-learning methods, and several reported comparisons are primarily graphical.
- The environment distribution is centered on North American ASHRAE archetypes and locations. Random setpoints increase difficulty but do not represent typical occupant behavior.
Future Work
Future work should add calibrated real-building models, sensor and actuator imperfections, safety constraints, faults, and broader climates and archetypes. Important comparisons include successor features, meta-RL, hypernetworks, offline RL, graph Transformers, and continual learning. Evaluations should report seasonal and multi-year energy savings, comfort violations, sample efficiency, compute cost, statistical uncertainty, and integration cost before field deployment.
AI Executive Summary
Reinforcement-learning controllers often excel only in the building or task on which they were trained. A change in climate, envelope, equipment, sensors, actuators, or comfort policy can invalidate the learned behavior. Existing generalization benchmarks such as Meta-World and RLBench are dominated by robotics, while HVAC platforms including Sinergym and CityLearn offer relatively few buildings. This makes it difficult to distinguish genuine transfer ability from specialization.
Building2Building (B2B) addresses this gap with EnergyPlus-based environments and parametric generators. The suite covers houses, restaurants, warehouses, retail stores, small offices, and medium offices, with 1,000 environments per type and 6,000 total. A morphology representation describes common attributes and typed graph nodes for thermal zones and HVAC equipment, allowing observations and actions to vary while remaining Gymnasium-compatible. Four tasks isolate goal, dynamics, action-space, and cross-domain adaptation.
The experiments show that parameterized PPO benefits from building context in dynamics adaptation. A morphology-aware Amorpheus variant trained for 1M environment steps across four building types successfully evaluated on 20 unseen buildings and outperformed the ASHRAE Guideline 36-inspired reactive baseline on some cases. B2B’s central value is methodological: it makes transfer measurable at scale. Its main caveat is that evidence remains simulated, leaving real-world safety, robustness, deployment cost, and broader algorithmic comparisons unresolved.
Deep Analysis
Background
RL generalization has been studied with Procgen, RLBench, Meta-World, DM Control, and related suites, but these benchmarks largely emphasize games and robotics with fixed interfaces. HVAC research commonly selects one building and demonstrates comfort-energy control in EnergyPlus or another simulator. Policies trained this way often fail to transfer across buildings. B2B uses HVAC because buildings share thermodynamic principles while differing substantially in climate, structure, equipment, and objectives.
Core Problem
The benchmark asks whether a controller can maintain zone temperatures and manage energy when rewards, dynamics, observation dimensions, actuator sets, or building types change. Thermal inertia, multi-zone coupling, equipment interactions, and long horizons make the problem difficult. A fixed-size MLP cannot naturally consume heterogeneous interfaces, while reward weights are sensitive to building scale and equipment energy consumption.
Innovation
- �� A 6,000-environment EnergyPlus suite spanning six building classes.
- �� Four controlled tasks separating goal, dynamics, action-space, and domain shifts.
- �� A morphology abstraction with common data, typed graph nodes, local attributes, and split/join mappings.
- �� Baseline-dependent reward normalization so one wE has more consistent meaning.
- �� Reproducible ASHRAE Guideline 36-inspired reactive controllers and normalized scoring.
Methodology
- �� Generation: a residential generator models Quebec housing statistics; a commercial generator samples ASHRAE 90.1 archetypes across 16 North American locations, varying size, window-to-wall ratio, insulation, and infiltration.
- �� Interface: observations include outdoor temperature/humidity, calendar signals, zone temperatures, and floor-area-normalized HVAC energy. Actions control supply-air temperature, airflow, dampers, reheat, or thermostatic setpoints.
- �� Reward: r_t=−(1/(Zτ_T))Σ_z(T_z−T_z^target)^2−w_E E_HVAC/τ_E.
- �� Learning: PPO is used for core adaptation studies; cross-domain learning uses type-specific linear blocks around shared acausal Transformer layers in an Amorpheus variant.
- �� Evaluation: returns are normalized by the reactive controller, with temperature RMS and transfer failures examined across buildings.
Experiments
The dataset contains 1,000 generated environments per building type and is split into training and test sets. Dynamics adaptation uses single-zone houses with a constant 21°C setpoint and wE=0, comparing 100 specialists, a resampled non-specialist PPO, and a parameterized PPO. Cross-domain training uses retail, restaurant, small-office, and medium-office buildings, resampling every 10 PPO updates for 1M steps and testing 20 unseen buildings. Action transfer includes 5→10 and 33→36 dimension changes, plus reversals.
Results
Per-building specialists achieve the strongest normalized returns on their training buildings, as expected. The parameterized policy improves over the non-specialist by using building attributes, although both show long-tail failures among 100 test houses while specialists remain more consistent. The cross-domain Amorpheus policy handles variable observation and action structures and exceeds the normalized reactive baseline of 1.0 on some of 20 unseen buildings, demonstrating useful but incomplete transfer.
Applications
Building operators could pretrain a policy over a B2B-like distribution and adapt it using limited data from a target facility. Relevant systems include heat pumps, packaged rooftop units, VAV boxes, central air loops, dampers, and reheat coils. Practical deployment requires calibrated simulation, sensor integration, actuator bounds, supervisory safety logic, human override, and continuous monitoring of comfort and equipment constraints.
Limitations & Outlook
EnergyPlus provides physically grounded simulation but omits many deployment complications, including noisy sensors, delays, faults, aging equipment, occupant behavior, and communication failures. The empirical study is narrow, focusing on PPO and one Amorpheus-inspired model rather than a comprehensive algorithmic leaderboard. North American archetypes also limit geographic and architectural coverage. Future work should combine real data, risk-sensitive RL, fault testing, safe online adaptation, multi-season evaluation, and explicit reporting of compute and integration costs.
Plain Language Accessible to non-experts
Imagine a company that manages climate in thousands of restaurants. Every restaurant wants customers to feel comfortable while spending as little electricity as possible, but each location has a different floor plan, number of rooms, machine setup, and local weather. A manager trained in one restaurant may not know what to do in another.
B2B is a giant practice laboratory for this problem. EnergyPlus acts like a detailed digital rehearsal room, creating 6,000 different buildings. The learning controller observes weather, time, room temperatures, and energy use, then adjusts airflow, supply temperature, and dampers. Researchers deliberately change the comfort target, the building materials, the climate, and even which controls are available.
Its modular description is like labeling every kitchen station and dining room. One building may have two controls and another thirty-six, but the controller can still recognize the kinds of rooms and machines involved instead of expecting an identical set of buttons. Experiments show that giving the controller basic building information helps it move between buildings, and a policy trained across four building types beats a traditional rule-based manager in some unseen cases. The important warning is that a rehearsal is not the real world: noisy sensors, broken equipment, and human behavior still require field testing.
ELI14 Explained like you're 14
Picture a video game where you control the air in every classroom of a school. You want students to feel comfortable, but you also want to save electricity. The catch? Every school has different room numbers, wall materials, air machines, and weather. A strategy that works perfectly at your school might be terrible at another one!
The researchers built B2B, a huge training world for this challenge. Using EnergyPlus, they created 6,000 simulated buildings: houses, restaurants, warehouses, shops, and offices. The computer watches outdoor weather, the date, room temperatures, and energy use. Then it chooses how strongly to blow air, how warm the supplied air should be, or which valves should open. The researchers also change the goal: maximum comfort, lower energy use, or changing temperature targets.
Here’s the clever part: buildings do not all have the same number of “buttons.” A normal neural network is like a game controller designed for exactly five buttons. B2B describes rooms and machines as labeled pieces, so a smarter model can work with different layouts and equipment. A PPO controller that knew building details transferred better than one that did not. Another model trained on four building types tested on 20 new buildings and beat the usual rule-based controller in some cases.
But remember: simulated buildings are practice levels, not real schools! Real sensors can be noisy, machines can break, and people can open windows. The next challenge is proving the system is safe, reliable, and genuinely saves energy outside the computer!
Glossary
Reinforcement Learning
A learning paradigm in which an agent interacts with an environment and improves actions using rewards. Technically, it seeks a policy maximizing expected discounted cumulative return.
B2B trains HVAC policies, mainly with PPO.
EnergyPlus
A high-fidelity building-energy simulator supported by NREL. It models thermal behavior, weather exposure, equipment, and energy consumption.
It supplies the physical dynamics for every B2B environment.
PPO
Proximal Policy Optimization is a policy-gradient algorithm that limits excessively large policy updates, commonly through a clipped surrogate objective. This improves practical training stability.
It is used for goal and dynamics adaptation baselines.
Morphology
A structured description of one environment’s entities, types, attributes, and mappings between local and global spaces. It is analogous to a typed graph for the building and its equipment.
B2B represents thermal zones and HVAC components as morphology nodes.
Goal Adaptation
Transfer under a changed reward or target while the environment interface and dynamics remain fixed. It tests whether a policy can change behavioral priorities.
B2B varies energy weight wE and constant, occupancy-based, or random setpoints.
ASHRAE Guideline 36
An industry guideline describing high-performance HVAC sequences of operation. The paper derives PI-loop and Trim-and-Respond reactive controllers from it.
These controllers provide the principal engineering baseline.
Open Questions Unanswered questions from this research
- 1 It remains unknown how much benchmark transfer survives in operating buildings with sensor noise, actuator delay, equipment faults, occupant overrides, and hard safety constraints. This requires calibrated models and controlled field studies.
- 2 B2B does not yet establish a broad ranking across meta-RL, offline RL, world models, successor features, hypernetworks, and continual learning, nor does it fully quantify sample efficiency and compute cost.
- 3 The distribution is based on North American ASHRAE archetypes and locations. Generalization to other countries, extreme climates, informal construction, and old buildings remains an open empirical question.
Applications
Immediate Applications
Controller initialization for new buildings
Facility operators could pretrain on a diverse simulated fleet, then fine-tune with limited data from a target building. Required safeguards include calibrated models, temperature and weather sensors, actuator limits, supervisory fallback logic, human override, and comfort alarms.
Standardized HVAC algorithm evaluation
Researchers can use the 6,000-environment suite to compare PPO, graph networks, meta-learning, and hypernetworks under the same goal, dynamics, action-space, and cross-domain protocols, avoiding bespoke single-building setups.
Long-term Vision
Fleet-scale adaptive energy management
A mature morphology-aware policy could operate across heterogeneous building portfolios, rapidly adapting to new rooms and equipment. This could reduce digital-twin construction and controller-training costs, although validation, cybersecurity, safety certification, and occupant acceptance remain major barriers.
Abstract
Reinforcement learning (RL) has achieved strong results in control, yet learned policies remain brittle to changes in dynamics, action spaces, observation spaces, or goals, a critical limitation for real-world deployment. Existing benchmarks offer limited diversity and complexity, making it difficult to rigorously study transfer, multi-task learning, and meta-learning in RL. We introduce Building2Building (B2B), a large-scale suite of realistic Heating, Ventilation, and Air Conditioning (HVAC) control environments built on EnergyPlus, a state-of-the-art building simulator. B2B is fully compatible with the Gymnasium interface and features a parametric building generator, enabling the systematic generation of diverse building configurations with heterogeneous observation and action spaces. Based on this suite, we define benchmark tasks targeting key open challenges in RL, including goal adaptation, dynamics adaptation, action-space shifts, and cross-domain transfer. By providing a large-scale, diverse, and physically grounded testbed with standardized evaluation protocols, B2B enables systematic investigation of generalization and transfer in continuous control. Beyond advancing research on generalization in RL, this new benchmark also carries significant societal implications by enabling improved HVAC control at scale, one of the most energy-intensive systems in buildings.