SocialRL: Refining LLMs' Social Intelligence through Multi-turn Reinforcement Learning and Reward Design
SocialRL combines multi-turn PPO with dynamic process rewards, raising goal achievement by an average of 9.2 percentage points.
Key Findings
Methodology
SocialRL treats a complete social dialogue as a finite-horizon MDP and optimizes full trajectories rather than isolated utterances. PPO, a value network, and GAE propagate delayed outcomes across turns. A reward model evaluates six dimensions: goal advancement, strategic positioning, relational attunement, persona consistency, contextual coherence, and turn quality. It generates 2–4 binary criteria per dimension, while stage-aware weights emphasize rapport early, goal progress mid-dialogue, and balanced closure late.
Key Results
- On SOTOPIA-π, SocialRL with Qwen2.5-7B raises average Goal Achievement from 42.3% for Base to 52.3%, while Relationship Change rises from 0.311 to 0.358. The strongest trained backbone, LLaMA3.1-8B, reaches 59.6% Goal Achievement.
- On AgentSense, Qwen3-8B improves from 73.3%/0.392 for Base to 80.5%/0.444 for SocialRL in Goal Achievement/Relationship Change. Gemma-3-4B shows an especially large increase, from 46.3% to 68.1% Goal Achievement.
- Across four benchmarks and four trained backbones, the paper reports an average Goal Achievement gain of 9.2 percentage points; benchmark-specific gains are 10.725 points on SOTOPIA-π and 9.625 on AgentSense.
Significance
The work reframes social intelligence as sustained decision-making rather than single-turn politeness. It addresses three persistent problems: local optimization, sparse terminal feedback, and the unresolved tension between pursuing private goals and preserving relationships. Academically, it offers interpretable process supervision tied to social dimensions and interaction stages. Industrially, it provides a route toward more dependable negotiation, customer-service, educational, and collaborative agents that know when to listen, persuade, compromise, or stop.
Technical Contribution
The central contribution is the joint use of full-trajectory PPO, dense multidimensional process rewards, and context-dependent stage weights. The composite reward is ˜R_t=αR_process,t before termination and αR_process,t+βR_outcome at the final turn, with α=0.3 and β=1. GAE uses λ=0.95 and PPO clipping uses ϵ=0.2. Unlike single-turn Sotopia-RL with GRPO, SocialRL uses a state value function to estimate future returns, reducing baseline mismatch when identical turn indices correspond to negotiation, repair, compromise, or failure.
Novelty
The novelty lies in the combination, not in inventing PPO or LLM judging independently. In this social-dialogue setting, SocialRL jointly optimizes complete trajectories, six inspectable process dimensions, and stage-dependent priorities. This differs from Sotopia-RL’s single-turn GRPO, ArCHer’s macro-level return modeling, and BC or SDPO’s static supervision. The key conceptual advance is to encode a changing policy: build rapport, advance the objective, then close without sacrificing the relationship.
Limitations
- The reward model generates rubrics and binary judgments with an LLM, so cultural bias, evaluator instability, and reward hacking remain possible. Subtle norms involving status, age, or indirectness may be scored inconsistently.
- Evaluation relies mainly on simulated or script-derived benchmarks—SOTOPIA-π, SOTOPIA-All, SOTOPIA-Hard, and AgentSense. Real users, long-term relationships, deployment safety, and the computational cost of per-turn rubric generation are not fully tested.
- gpt-4o remains ahead in several settings, showing that SocialRL does not remove gaps caused by scale, priors, and deeper social reasoning ability.
Future Work
Future work should evaluate real users, cross-cultural interactions, and longitudinal relationships while adding human feedback, uncertainty calibration, and explicit safety constraints. Promising directions include causal credit assignment, adaptive stage inference, multi-agent negotiation, and cheaper reward models that preserve the interpretability of binary criteria without requiring extensive per-turn generation.
AI Executive Summary
Social intelligence requires more than producing a polite sentence. An agent must infer context, read intentions, pursue a private objective, and preserve cooperation over many turns. Existing methods such as single-turn Sotopia-RL optimize local responses, whereas outcome-focused ArCHer provides little guidance about which intermediate action helped or harmed the final result.
SocialRL treats the entire conversation as a reinforcement-learning trajectory. PPO, a value network, and Generalized Advantage Estimation propagate delayed outcomes to earlier turns. In parallel, a reward model scores each utterance using six dimensions: goal advancement, strategic positioning, relational attunement, persona consistency, contextual coherence, and turn quality. The weighting changes with the dialogue stage—rapport early, goal progress in the middle, and balanced closure near the end.
The empirical gains are substantial. On SOTOPIA-π, Qwen2.5-7B improves from 42.3% to 52.3% Goal Achievement and from 0.311 to 0.358 Relationship Change. On AgentSense, Qwen3-8B rises from 73.3% to 80.5%. Across four benchmarks and four backbones, the average gain is 9.2 percentage points. The result suggests that social competence depends not only on better wording, but on long-horizon credit assignment, inspectable process feedback, and stage-sensitive strategy.
Deep Analysis
Background
As LLMs become autonomous collaborators, social intelligence becomes central to trustworthy interaction. SOTOPIA, SOTOPIA-π, and AgentSense test negotiation, empathy, information verification, and latent social inference. Behavior Cloning can imitate fluent style; Sotopia-RL applies GRPO to single turns; ArCHer models macro-level returns. Yet none directly supplies dense, stage-sensitive supervision for balancing goal pursuit against relationship maintenance across an evolving dialogue.
Core Problem
Social dialogue is a state-dependent sequential decision problem with hidden intentions and delayed outcomes. Persistence may advance a goal but damage trust; immediate concession may preserve rapport but guarantee failure. Single-turn rewards cannot capture how early actions shape later states, while terminal rewards are sparse and high variance. The agent therefore needs both long-horizon planning and turn-level explanations of what socially effective behavior looks like.
Innovation
- ��Full-trajectory optimization: PPO updates the policy over complete dialogues, with a value network estimating future returns.
- ��Six process dimensions: goal advancement and strategic positioning represent the goal side; relational attunement and persona consistency represent the relationship side; contextual coherence and turn quality provide general validity constraints.
- ��Dynamic rubrics: the reward model generates 2–4 inspectable binary criteria for every dimension after each turn.
- ��Stage-aware weighting: early dialogue prioritizes rapport, middle dialogue prioritizes objective progress, and late dialogue balances achievement with respectful closure.
Methodology
- ��Input: scenario S, private goal G, persona information C, and dialogue history H_t; the action is a complete utterance rather than an isolated token.
- ��Scoring: r_t,i=(1/M_i)Σ_j pass_i,j, the pass rate of binary criteria for dimension i.
- ��Reward construction: R_process,t=Σ_i w_t,i r_t,i, with non-negative normalized weights; the final turn adds outcome reward using α=0.3 and β=1.
- ��Optimization: δ_t=˜R_t+γV(s_{t+1})−V(s_t); GAE with λ=0.95 estimates advantages, and PPO updates the policy with clipping ϵ=0.2.
- ��Stage inference: the reward model predicts Early, Mid, or Late from context, then applies whitelist filtering, clipping, caps, fallback priors, overflow redistribution, and normalization.
Experiments
The study uses SOTOPIA-π, with 1,773 scenarios and 259 held-out tests; SOTOPIA-All with 90 scenarios; SOTOPIA-Hard with 14 difficult scenarios; and AgentSense with 1,225 scenarios organized into 245 templates. Policies use Qwen2.5-7B-Instruct, Qwen3-8B, LLaMA3.1-8B, or Gemma-3-4B. Opponents include Qwen2.5-7B, Qwen3-8B, Qwen3.5-35B, and gpt-4o. Baselines include Base, BC, SDPO, Sotopia-RL, and ArCHer. Metrics are Goal Achievement and Relationship Change; each pair is repeated five times.
Results
On SOTOPIA-π, SocialRL-Qwen2.5-7B averages 52.3%/0.358 versus 42.3%/0.311 for Base; SocialRL-LLaMA3.1-8B reaches 59.6%/0.341. On AgentSense, SocialRL-Qwen3-8B reaches 80.5%/0.444 versus 73.3%/0.392 for Base, while Gemma-3-4B rises from 46.3%/0.065 to 68.1%/0.234. gpt-4o remains strong at 69.0% on SOTOPIA-π and 91.7% on AgentSense, preserving a frontier-model gap.
Applications
Potential uses include negotiation assistants, customer service, team coordination, educational tutoring, and emotional-support interfaces. Deployment requires explicit goal and safety constraints, cultural validation of the reward model, and human monitoring. The practical advantage is strategic timing: an agent can first acknowledge the partner, then propose a solution, and finally close without trading away either task success or trust.
Limitations & Outlook
The approach depends on an LLM evaluator that generates rubrics, so evaluator bias, stage misclassification, and reward hacking can distort learning. Simulated and script-derived tasks may not represent real long-term relationships. Full rollouts, value learning, and per-turn rubric generation also increase computational cost. Future systems should add real-user feedback, cultural diversity, uncertainty-aware rewards, longitudinal evaluation, and more efficient batched judging.
Plain Language Accessible to non-experts
Imagine two colleagues trying to finish an important project together. A short-sighted assistant sees only the next sentence: if the colleague says they are busy, it either pushes harder or gives up. The first choice may damage trust; the second may leave the project unfinished. SocialRL acts more like an experienced project leader who remembers the entire collaboration.
It evaluates every step in the journey. Did the project move forward? Was the suggestion tactful? Did the speaker understand the colleague’s feelings? Did the conversation remain consistent and clear? Did the sentence fit what came before? At the beginning, the leader builds trust; in the middle, they work toward the objective; at the end, they confirm the result and leave both people comfortable.
The system also looks backward from the final outcome. If a respectful compromise eventually succeeded, earlier listening and clarification receive credit. If repeated pressure caused failure, those choices receive less credit. In the paper, Qwen2.5-7B’s success rate on SOTOPIA-π rises from 42.3% to 52.3%, showing that good social behavior comes from managing the whole collaboration, not merely polishing isolated sentences.
ELI14 Explained like you're 14
Picture a group project at school. You need a classmate’s help, but they are already busy. If you immediately say, “You have to help me!”, they may get annoyed. If you instantly say, “Never mind,” everyone stays friendly, but the project may fail. A clever teammate first checks what is possible, suggests a fair trade, and later confirms the plan.
SocialRL trains an AI to behave more like that clever teammate. It does not judge only one reply. It watches the whole chat and asks: Did the task move forward? Did the AI understand the other person? Did it keep its personality consistent? Was the reply connected and useful? Did the relationship get better or worse?
The AI also changes its strategy during the conversation. At the start, it acts like someone making friends. In the middle, it focuses on solving the problem. At the end, it tries to finish successfully without making the other person feel used. PPO helps it learn from the entire conversation, while a scoring model gives feedback after each turn.
The results are impressive! On SOTOPIA-π, Qwen2.5-7B’s goal success climbs from 42.3% to 52.3%. On AgentSense, Qwen3-8B goes from 73.3% to 80.5%. So social intelligence is not just “sounding nice”—it is knowing what to say, when to say it, and how today’s choice changes the next move!
Glossary
Multi-turn PPO
Proximal Policy Optimization is a reinforcement-learning algorithm that limits overly large policy updates. Here, the optimized unit is a complete dialogue trajectory rather than one isolated response.
Used to propagate delayed social outcomes across turns and update the dialogue policy.
GAE
Generalized Advantage Estimation combines temporal-difference errors across multiple steps to estimate how much better an action is than its baseline. It trades off estimator bias and variance.
SocialRL uses λ=0.95 to compute utterance-level advantages.
Process reward
A process reward evaluates intermediate behavior instead of waiting for the final result. It provides denser and more interpretable learning feedback.
The paper constructs it from six social dimensions.
Stage-aware weighting
Stage-aware weighting changes the importance of criteria as an interaction evolves. Early turns favor rapport, middle turns favor progress, and late turns favor balanced closure.
The reward model infers Early, Mid, or Late from dialogue context.
Goal Achievement
A benchmark metric measuring whether the agent accomplishes its private social objective. It is reported as a percentage or a 0–10 score depending on the benchmark.
It is the principal performance metric in the experiments.
Relationship Change
A metric for the change in relationship quality produced by the interaction. Higher values generally indicate better relationship maintenance.
It is reported alongside Goal Achievement to quantify the goal–relationship trade-off.
Open Questions Unanswered questions from this research
- 1 Can LLM-generated binary rubrics remain reliable across cultures, ages, and power relations? The current benchmarks do not adequately test how different communities define tact, respect, or appropriate persistence.
- 2 How much of the gain comes from full-trajectory PPO, dense process rewards, or stage weighting? More systematic component-level ablations and cost analyses are still needed.
- 3 Can an agent preserve honest, safe, and consistent strategies over weeks or months of real interaction? Short benchmark episodes cannot fully test long-term trust, memory, or repair after serious mistakes.
Applications
Immediate Applications
Collaborative and customer-service agents
Organizations can use SocialRL for multi-turn support, scheduling, and task negotiation after defining goals, privacy boundaries, and escalation rules. Relationship Change can complement task success to detect excessive pressure, premature concession, or deteriorating cooperation.
Education and negotiation coaching
Learning platforms can simulate teachers, classmates, or negotiation partners and train users to acknowledge constraints, propose alternatives, and confirm agreements. Human review remains necessary because cultural politeness norms may be mis-scored by the reward model.
Long-term Vision
Persistent human–AI collaborators
Future agents could maintain trust and adapt goals across projects, becoming dependable partners in research, management, and daily life. Realizing this vision requires long-term memory, culturally diverse evaluation, verifiable safety constraints, and continuous human feedback.
Abstract
Social intelligence enables agents to read social context, infer intent, and adapt over sustained dialogue. As language models become autonomous collaborators, it is central to building effective and trustworthy human-AI interaction. Existing reinforcement learning methods optimize single-turn utterances and sparse outcome rewards, producing short-sighted policies that struggle to manage goal-relationship tensions across multi-turn interactions. We propose SocialRL, a multi-turn reinforcement learning framework addressing both challenges. First, we apply multi-turn reinforcement learning using PPO that propagates delayed outcome rewards back to each turn, enabling long-horizon planning. Second, we design six process reward dimensions capturing the goal-relationship trade-off, including goal advancement, relational attunement, contextual coherence, etc. A reward model dynamically generates fine-grained scoring criteria for each dimension, while a stage-aware weight schedule prioritizes relationship-building in early turns, goal advancement mid-way, and balanced closure late. Across multiple social-dialogue benchmarks, SocialRL improves Goal Achievement by an average of 9.2 percentage points over the corresponding Base models. These results demonstrate the effectiveness of SocialRL across synthetic and real social scenes, as well as standard and challenging social scenarios.