Disclosure-Gated User Simulation for Companion-Agent Evaluation

The rapid evolution of dialogue systems, driven by large language models like GPT-4 and PaLM, has revolutionized human-computer interaction. Early evaluation relied on small-scale human studies, which were costly and lacked scalability. Recent advances introduced user simulators trained on massive corpora, aiming to automate evaluation. However, these simulators tend to over-cooperate, asking too many questions and inflating success metrics, which compromises fairness. Existing solutions, such as restricting prompt information or hard-coding behaviors, are limited by lack of verifiability and robustness. Benchmarks like CompanionBench have standardized evaluation but still face issues of simulation fidelity and stability. The current work introduces a hierarchical gating mechanism to address these challenges, providing a controllable, verifiable environment that better reflects human behaviors.

Current large-scale user simulators often exhibit excessive cooperation, asking numerous questions to score higher, which diverges from real human interaction patterns. This leads to unreliable rankings and unstable scores across different models and environments. The core challenge is to design a mechanism that can regulate information disclosure dynamically, mimicking human-like gradual sharing and withdrawal, ensuring that evaluation results are both fair and reproducible. Without such a mechanism, the evaluation risks favoring models that exploit over-cooperative behaviors rather than genuinely understanding user engagement. Therefore, establishing a controllable, ablatable, and environment-integrated gating system becomes crucial.

The primary innovation is the development of a five-level ordered disclosure gate integrated into the dialogue environment, inspired by social penetration theory. This gate controls information flow based on behavior states, allowing the simulator to emulate human-like disclosure patterns. Unlike prompt-based constraints, the gate is trained internally on synthetic data, internalizing behavior dynamics. It can be ablated or modified without affecting the environment’s core, enabling rigorous evaluation of its impact. The design combines social cognition principles with a state transition model, ensuring that the ranking remains order-preserving and scale-stable. This approach offers a new paradigm for controllable, verifiable user simulation, addressing the over-cooperation problem effectively.

• �� Define five ordered gates (opening, asked_or_natural, felt_heard, felt_safe, earned_deep_trust) merged into three depth layers.
• �� Collect annotated corpora: real Chinese conversations with per-item disclosure labels and synthetic trajectories with behavior scripts.
• �� Train the simulator on synthetic data to internalize gating behaviors, while real data provides authentic reactions.
• �� Incorporate the gate as a load-bearing environment component, enabling ablation by removing gate info from training or inference prompts.
• �� Use a state machine to enforce gate transition rules based on behavior classes, ensuring order and asymmetry in retreat.
• �� Conduct ablation studies: remove synthetic branch, strip gate info at training or runtime, vary model scale.
• �� Evaluate on CompanionBench, measuring ranking stability and score scale consistency across manipulations.

The experiments utilize the CompanionBench benchmark, assessing 12 dialogue systems. The key setup involves training models with and without gate information, then measuring ranking shifts and score variations. Ablation studies remove synthetic data, strip gate info during training or inference, and alter model scales. Human conversations validate the realism of behaviors. Metrics include rank order preservation, score scale stability, and environment robustness. Results show that the proposed gate mechanism maintains ranking integrity even when gate info is removed, unlike prompt-only models which shift scores upward. The correlation with the original simulator reaches 0.993, confirming high fidelity. These experiments demonstrate the mechanism’s effectiveness in producing stable, realistic evaluations.

The proposed simulator, trained with internalized gating, maintains ranking order and score stability across multiple manipulations. Removing gate info from training causes rank shifts exceeding the noise threshold, validating its load-bearing role. The environment’s design ensures that the ranking remains order-preserving and scores are scale-stable, outperforming prompt-only models that shift scores without affecting rankings. The correlation coefficient with the original simulator is 0.993, indicating near-identical behavior. Ablation results confirm that synthetic trajectories are essential for gating behavior, and environment manipulations validate the mechanism’s robustness. Overall, the results establish a new standard for reliable, fair evaluation of dialogue systems.

This gating mechanism can be integrated into dialogue system evaluation pipelines, especially for applications requiring human-like interaction modeling, such as emotional companions, customer service bots, and mental health assistants. It enables fairer comparisons of models by controlling for over-cooperation and ensuring reproducibility. In industry, it can improve model tuning and deployment by providing more realistic user simulations. Future extensions include multi-modal data integration, long-term interaction modeling, and multi-task evaluation, broadening its applicability to real-world scenarios.

The approach relies heavily on detailed annotated corpora, which are costly and time-consuming to produce. Its performance in out-of-distribution or unseen scenarios remains untested, risking limited generalization. The gate design adds complexity, increasing training and inference costs, potentially hindering scalability. Further work is needed to simplify the architecture, reduce annotation requirements, and validate robustness across diverse domains and modalities.