Coachable agents for interactive gameplay
Style-conditioned UVFAs with Cat-RAC enable runtime behavior control, validated across 57,000 Horizon Forbidden West battles.
Key Findings
Methodology
The framework decomposes reward as r(s,a;θ)=r_task(s,a)+r_style(s,a,θ), where a multidimensional style vector θ controls reward weights, thresholds, gates, and setpoints. A style-conditioned UVFA represents a family of policies rather than one optimum. Training combines scenario training, data augmentation, style-specific replay buffers, and a new SAC-family algorithm, Categorical Regularized Actor-Critic (Cat-RAC), developed for Horizon Forbidden West’s complex action space.
Key Results
- In HFW, five random seeds were trained for 4.5 million gradient steps and evaluated over 20 styles, 19 machines, and three locations. Ten samples per machine–style–location–seed combination produced 57,000 battles. The damage matrix shows a strong diagonal: requested weapon, melee, trap, and elemental styles reliably changed the inflicted damage type.
- For GT Sophy in Gran Turismo 7, experiments used the Red Bull X2019 Competition on Lago Maggiore—Full. Fuel- or tire-wear penalties extended the time until a pit stop by up to 60%, while maximal conservation cost approximately 2–3 seconds per lap, yielding a runtime-adjustable pace–resource trade-off.
- Most HFW styles retained high task performance; the default style achieved a reported win rate of 0.998. However, melee, Sunscourge, and IceStorm styles won less than half of fights against Thunderjaw or Slaughterspine. Part-removal styles removed about twice as many components but sometimes timed out because the auxiliary reward dominated the task reward.
Significance
Standard reinforcement learning usually discovers one near-optimal behavior, leaving users little control over how success is achieved. This paper addresses that long-standing gap by treating preferences as modifiers of a shared task rather than separate tasks. Demonstrations across AAA combat, racing strategy, and humanoid locomotion suggest a common design pattern for interactive AI. The approach could make game agents more expressive and accessible, while giving robots a mechanism for adapting to context—for example, prioritizing quiet operation when a child is sleeping or speed when time is limited.
Technical Contribution
The central technical contribution is extending UVFAs from goal-conditioned learning to high-dimensional, fixed-task behavioral control. The style vector can encode linear reward mixtures, thresholds, gating/ramping rules, and target setpoints. Scenario construction and style-aware replay address sparse coverage and instability, while augmentation improves data efficiency. Cat-RAC extends Soft Actor-Critic for HFW’s mixed, highly structured controller actions. Crucially, the policy is not retrained at deployment: modifying θ or style weights changes the executed behavior and exposes an empirical Pareto frontier between style adherence and task success.
Novelty
The novelty lies less in any isolated ingredient than in their integration and scale. Prior work used reward shaping or reward conditioning in small turn-based games, OpenAI Gym, or DMC Vision, often relying on data generated only under the task reward. This study operationalizes style-conditioned UVFAs in AAA, open-world control, with carefully engineered scenarios, sampling, replay, and Cat-RAC. It also demonstrates compositional styles and scripted runtime switching, rather than merely selecting among separately trained policies.
Limitations
- The method is sensitive to reward scaling. In HFW, a high part-removal reward encouraged agents to chase components instead of ending fights, causing timeouts against weaker enemies and exposing a conflict between stylistic fidelity and the core objective.
- Some requested styles are intrinsically costly. Against Thunderjaw and Slaughterspine, melee, Sunscourge, and IceStorm win rates fell below 50%, showing that user control is constrained by game mechanics, equipment strength, and opponent dynamics.
Future Work
Future research should automate style-reward design, calibrate θ from natural-language preferences or demonstrations, and impose safety or success constraints during runtime tuning. Larger compositional style spaces require conflict detection and better extrapolation. Real-robot studies should test sensor noise, contact safety, energy use, long-horizon adaptation, and human feedback, while reporting compute costs and systematic ablations more fully.
AI Executive Summary
Reinforcement learning has produced agents that can race, fight and walk, but most learn one dominant solution. That is inadequate when users care about manner as well as outcome: a robot may need to clean quietly, a race car may need to preserve tires, and a game character may be asked to avoid a powerful weapon. Treating each preference as a separate task loses their shared structure and offers no smooth control over intensity.
The authors propose style-conditioned Universal Value Function Approximators. The fixed reward captures the core objective—winning, finishing or walking—while a style reward, parameterized by θ, modifies behavior. Scenario training, data augmentation and specialized replay improve coverage; Cat-RAC, a new Soft Actor-Critic variant, addresses HFW’s complex controller. At deployment, users change θ or reward weights rather than retraining the agent.
The evidence spans three domains. HFW used five seeds, 4.5 million gradient steps, 19 enemies, three locations and 20 styles, yielding 57,000 battles; the strong diagonal of the damage matrix shows high style adherence, while default win rate reached 0.998. GT7 extended pit-stop intervals by up to 60%, at a cost of roughly 2–3 seconds per lap. DMC Humanoid combined three arm poses with gait lengths from 0.3 to 0.65 metres. Failures reveal the central challenge: excessive style rewards can undermine success, so controllability must be calibrated rather than maximized blindly.
Deep Analysis
Background
Deep RL systems such as Soft Actor-Critic and GT Sophy have achieved strong results in games and robotics, while DMC Humanoid provides a standard locomotion benchmark. Yet conventional policies optimize one return and usually express one behavior. Multi-task, multi-objective and goal-conditioned methods change tasks or goals, but do not naturally represent a spectrum of ways to perform one task. This paper frames such modifiers as styles: adverbs attached to a stable verb.
Core Problem
The objective is to preserve core competence while allowing real-time, continuous and compositional behavioral control. The problem is difficult because style rewards may be sparse or conflicting, high-dimensional actions require broad exploration, and training must cover many opponents and contexts. In HFW, dozens of attacks, weapons, ammunition types, elemental states and removable components create long-horizon dependencies that small-domain reward-conditioning methods do not capture.
Innovation
The work contributes five linked innovations. First, style-conditioned UVFAs encode a policy family with θ instead of independent policies. Second, rewards separate task success from stylistic preferences and support weights, thresholds, gates and setpoints. Third, scenario training, augmentation and style-specific replay target rare behaviors. Fourth, Cat-RAC adapts the SAC family to HFW. Fifth, experiments test compositional styles, runtime switching and out-of-distribution enemies, demonstrating contextual adaptation rather than simple action repetition.
Methodology
- �� Input: environment observation s plus style request vector θ; output: controller action a.
- �� Objective: optimize r(s,a;θ)=r_task(s,a)+r_style(s,a,θ), with the relative scale determining the task–style trade-off.
- �� Data: construct scenarios spanning enemies, locations and requested styles; collect trajectories and store them in replay buffers, including style-specific tables.
- �� Learning: train a UVFA-conditioned policy and critic; use Cat-RAC for HFW’s complex action representation.
- �� Stabilization: apply data augmentation, targeted sampling and carefully designed practice situations.
- �� Deployment: keep network parameters fixed and modify θ or style weights, enabling continuous tuning, composition and scripted switching.
Experiments
The domains were Gran Turismo 7, Horizon Forbidden West and DeepMind Control Suite Humanoid. GT7 used the Red Bull X2019 Competition on Lago Maggiore—Full, with fuel consumption and tire wear independently manipulated. HFW used a late-stage Aloy configuration with 700 HP, a fully unlocked skill tree and six upgraded weapons. Five seeds trained for 4.5 million gradient steps; evaluation covered 20 styles, 19 enemies and three locations, with win rate, damage type and style score recorded across 57,000 battles. Humanoid tested three arm poses and continuous gait lengths.
Results
HFW’s damage matrix showed a strong requested-style diagonal across melee, traps, six weapons and seven elemental styles. Elemental agents selected state-inducing ammunition and sometimes switched to stronger weapons after the state was reached, indicating contextual reasoning. Performance remained strong on five OOD situations. Part-removal produced roughly twice as many removed components but could cause timeouts. In GT7, resource conservation delayed pit stops by up to 60% while costing 2–3 seconds per lap; Humanoid composed arm poses with 0.3–0.65 m gait lengths.
Applications
Game studios can use these agents for controllable NPCs, automated QA, accessibility assistance, difficulty adaptation and design iteration. Racing agents can expose strategic controls over pace, fuel and tire life. Robotics systems could switch among quiet, fast and cautious modes according to household context. Deployment requires measurable task and style rewards, a stable controller interface, scenario coverage and safeguards preventing auxiliary objectives from overwhelming safety or completion.
Limitations & Outlook
Evidence is limited to three environments and does not establish robustness in physical robots, long-horizon household tasks or natural-language coaching. Style rewards are manually engineered and scale-sensitive; some styles conflict inherently with winning. AAA training is computationally expensive, and the supplied text reports limited systematic ablations and user studies. Next steps include constrained optimization, automatic reward design, language-to-θ grounding, risk-sensitive learning, broader compositional generalization and transparent compute reporting.
Plain Language Accessible to non-experts
Imagine a master chef preparing one signature dish. The dish itself is the task: it must be cooked correctly. Customers may then ask for less oil, faster service, less spice or a more elegant presentation. Those requests are styles. A conventional cooking robot knows one standard recipe; this paper trains a chef who understands the dish and can adjust several knobs without learning the entire kitchen again.
The knobs are represented by θ. The kitchen records two things separately: whether the dish succeeded and whether it followed the customer’s preference. For a racing car, that means finishing well while saving fuel or tires. For a combat character, it means defeating the machine while using traps, frost or a chosen weapon. Training includes many customers, kitchens and orders, with rare orders deliberately practiced and remembered.
At runtime, the customer can turn a knob. More fuel saving makes the car slower but postpones the pit stop; a frost request changes the weapon sequence. The chef can even combine requests. But the analogy also exposes the danger: if presentation matters too much, the chef may spend so long arranging the plate that the meal arrives late. Good coaching therefore balances preference with success.
ELI14 Explained like you're 14
Suppose you are playing a huge game. The goal is to defeat a robot, but you can tell your character, “Use close combat,” “set traps,” or “freeze it first.” You might even change your mind halfway through the battle. A normal AI is like a brilliant player who only knows one strategy: powerful, yes, but not very coachable.
This paper builds a coachable AI. It sees the game state and your style request, then chooses controller actions. It is not merely memorizing button sequences. For example, it may use frost ammunition to make an enemy brittle, then switch to a stronger weapon while that condition lasts. That is like a basketball player hearing “play safer” and changing decisions without forgetting to score.
The researchers tested it in Horizon Forbidden West, Gran Turismo 7 and a virtual humanoid. HFW included 57,000 battles. In racing, stronger conservation could delay a pit stop by as much as 60%, though each lap became about 2–3 seconds slower. The humanoid combined three arm poses with steps between 0.3 and 0.65 metres.
There is a catch! If the AI is rewarded too heavily for removing enemy parts, it may keep lining up shots instead of finishing the fight. So the big lesson is not “give the AI unlimited instructions.” It is “teach it how to balance instructions with the main mission.”
Glossary
Universal Value Function Approximator (UVFA)
A value model conditioned on both the current state and an additional goal or preference variable. Technically, it approximates expected return across a family of conditions with shared parameters.
The paper replaces conventional goals with style vectors to obtain style-conditioned UVFAs.
Style-conditioned UVFA
A UVFA whose conditioning variable specifies how a fixed task should be performed. It controls behavioral characteristics rather than changing the underlying task.
It drives fuel saving, weapon choice, elemental combat and gait variation.
Categorical Regularized Actor-Critic (Cat-RAC)
A new member of the Soft Actor-Critic family designed for the paper’s difficult control setting. An actor proposes actions while a critic estimates their long-term value under regularization.
It was developed for Horizon Forbidden West’s complex action space.
Scenario training
Deliberately constructing practice situations that expose an agent to important states, opponents and objectives. It focuses experience collection instead of relying only on generic exploration.
HFW scenarios span 19 machines, three locations and multiple style requests.
Pareto curve
The boundary of best achievable trade-offs between competing objectives. Improving one objective beyond the boundary generally sacrifices another.
The paper plots style performance against HFW win rate as style weight changes.
Out-of-distribution (OOD)
Data or situations whose distribution differs from training conditions. OOD evaluation tests whether learned behavior generalizes beyond familiar examples.
The HFW agent faced unseen machines and behaviors such as flying attacks.
Open Questions Unanswered questions from this research
- 1 How can natural-language preferences or demonstrations be converted automatically into reliable style rewards and θ? The current framework depends heavily on expert reward engineering.
- 2 Compositional style spaces may grow combinatorially. The paper demonstrates selected combinations, but conflict detection, extrapolation and safety across arbitrary combinations remain open.
- 3 Physical robots introduce sensor noise, contact hazards, energy limits and wear. Success in DMC Humanoid is encouraging but cannot substitute for real-world validation.
Applications
Immediate Applications
Controllable NPCs and accessibility
Game developers can request weapon preferences, aggression levels or elemental tactics for NPCs, while players can ask an agent to assist during difficult encounters. A game must expose measurable rewards and a safe action API. Expected benefits include richer characters, automated testing and broader accessibility.
Runtime racing strategy
A GT-style system can select sprinting, fuel-saving or tire-preserving behavior during a race. Engineers tune preference weights instead of retraining separate policies, supporting pit-stop planning, competitive balancing and regression testing under controlled resource conditions.
Long-term Vision
Preference-aware household robots
Future robots could separate the core job of cleaning or carrying from preferences such as quietness, speed and caution, switching modes as people and environments change. Major obstacles include language grounding, safety constraints, physical generalization, accountability and dependable operation over long horizons.
Abstract
Reinforcement learning has proven to be a valuable tool in the creation of advanced AI and robotic systems, contributing to everything from game playing to robotics to foundation models. Through trial-and-error, these AI systems typically learn one, near-optimal behavior to solve their tasks. However, there are many use cases in which one would like to assert some level of control, preferably in real time, over how the task is solved. We refer to these modifications of a core task as styles. We combine universal value function approximators (UVFAs) with carefully selected training scenarios, learning algorithms, and data augmentation to create a framework for coaching agents that exhibit styles in complex domains. We demonstrate the framework's application in the AAA video games Horizon Forbidden West and Gran Turismo, and in an open-source humanoid test domain. Despite the different nature of the domains -- car racing, stylized game combat, and humanoid walking -- each agent shows strong coherence to the style requests while still satisfying the main task in its domain. Importantly, the techniques outlined in this paper allow an end user to choose the final behavior at run time, giving them flexible control over the final executed performance.