Actions Speak Louder than Words: Measuring Cross-Lingual Policy Retention in Tool-Using Agents
This paper introduces a cross-lingual policy retention metric based on action traces, revealing that four models retain 71-73% of their strategies across languages after bias correction.
Key Findings
Methodology
The study employs an action trace-based evaluation framework, extracting tool call sequences from 8 models, 6 benchmarks, and 41 languages, totaling 2.38 million rollouts. To ensure robustness, multiple bias correction techniques are applied, including exclusion of empty traces, length-matching, double-seed repetition, and normalization by model reproducibility. The core metric, normalized policy retention ˜I, is computed as the ratio of cross-lingual to within-language trace similarity, with bootstrap sampling used at the task level for statistical confidence. The approach isolates structural behavioral differences from sampling noise, revealing that the observed divergence is a genuine property of models' internal strategies rather than artifacts of randomness.
Key Results
- Under greedy decoding, four state-of-the-art models—Gemma-3-27B, Qwen-3-235B, Llama-4-Maverick, Sarvam-M—demonstrate a policy retention rate of approximately 71-73% across languages after bias correction, with model identity explaining only 5.7% of variance. This indicates a strong structural regularity in their behavior. The results hold across different parameter scales, architectures, and training regimes, emphasizing that the core behavior is model-independent and driven by the underlying task structure.
- The study uncovers a causal link: models predominantly route non-English tasks through English, a behavior confirmed by pre-registered predictions and robust across four models. Even explicit instructions to abandon this pivot are largely ignored, demonstrating its deep integration into the models' behavior. Additionally, a single regex-based trace extraction error artificially inflated the perceived multilingual failure of one model by 26 times, highlighting the importance of measurement robustness.
- Parameter scale influences the observed regularity: models below approximately 10B parameters show larger behavioral variance, with a structural boundary where policy retention converges at the frontier. Correcting for sampling bias reveals that the apparent differences among smaller models are largely artifacts, and true policy retention is closer to the 71-73% range for large models. This insight challenges simple scaling laws and underscores the importance of bias correction in behavioral evaluation.
Significance
This research shifts the paradigm from output-only evaluation to an analysis of intermediate behaviors, providing a more nuanced understanding of model behavior consistency across languages. It addresses critical issues in deploying multilingual agents, such as cost, failure modes, and safety, by quantifying how much their internal strategies are preserved. The methodology enables fairer comparisons among models of different sizes and architectures, and offers tools to diagnose and mitigate biases. Ultimately, it advances the development of reliable, interpretable, and equitable multilingual AI systems, with implications for safety, transparency, and fairness in real-world applications.
Technical Contribution
The paper introduces a novel measurement protocol based on action trace similarity, incorporating multiple bias correction steps—length matching, empty trace exclusion, reproducibility normalization, and bootstrap-based estimation—to isolate genuine behavioral divergence. It combines these with a rigorous statistical framework, enabling quantification of policy retention that is model-agnostic and bias-resistant. The approach extends existing evaluation paradigms by focusing on intermediate actions rather than final outputs, providing a new lens for analyzing model behavior in multilingual settings. The methodology is validated across a large-scale dataset, demonstrating its robustness and generality.
Novelty
This work is the first to systematically quantify cross-lingual policy retention via action traces, moving beyond traditional output-based metrics. The introduction of a bias-corrected, normalized measure—˜I—addresses longstanding challenges in behavioral evaluation, such as confounding factors like trace length, empty outputs, and sampling noise. Unlike prior work that primarily assesses final answers or factual retrieval, this study emphasizes the structural similarity of intermediate actions, revealing deep behavioral regularities. Its causal validation of English routing behavior further distinguishes it from existing approaches, offering a comprehensive framework for analyzing model strategies across languages.
Limitations
- The evaluation primarily relies on greedy decoding; behavior under stochastic decoding or other sampling strategies remains less explored, which may affect the generality of the findings.
- The trace extraction regex, while effective, may introduce biases or miss subtler action differences, especially in more complex tasks or models with different output formats.
- The focus on tool call sequences assumes a shared symbolic alphabet, which may not fully capture all aspects of model behavior, especially in models with more implicit or distributed strategies.
- Models below 10B parameters show larger variance, indicating that the observed regularity may not extend to smaller or less optimized models, limiting the universality of the conclusions.
Future Work
Future research will extend the framework to include stochastic decoding strategies, explore internal model representations for deeper causal analysis, and evaluate more diverse model architectures and training regimes. Additionally, integrating behavioral metrics with internal interpretability tools could uncover the mechanisms behind policy retention. The authors also plan to investigate how training data composition influences cross-lingual strategy consistency and develop interventions to enhance behavioral alignment. Ultimately, the goal is to establish a comprehensive, bias-resistant evaluation suite that guides the development of safer, more reliable multilingual agents.
AI Executive Summary
As artificial intelligence models evolve into complex multilingual agents, understanding their internal strategies across languages becomes crucial. Traditional evaluation methods focus predominantly on the correctness of final answers, overlooking the intermediate steps and behaviors that lead to those answers. This oversight can mask significant differences in how models operate internally, which in turn impacts costs, failure modes, and safety considerations in real-world deployments.
This paper introduces a pioneering approach that shifts the focus from output accuracy to the analysis of action policies—the sequences of tool calls and decision steps that models perform during task execution. By extracting and comparing these action traces across multiple models, benchmarks, and languages, the authors aim to quantify the degree to which models retain their strategies when switching languages. The core challenge was to develop a measurement that is both robust and fair, accounting for confounding factors such as trace length, empty outputs, and sampling noise.
To address this, the authors propose a bias-corrected, normalized metric called policy retention ˜I. This measure compares the similarity of action traces in cross-lingual settings to within-language baselines, using bootstrap sampling and multiple correction steps to eliminate artifacts. The results reveal a surprising regularity: despite differences in architecture, size, and training data, four leading models retain approximately 71-73% of their action policies across languages when decoded greedily. This indicates a structural, rather than random, basis for behavioral divergence.
Further analysis uncovers that models predominantly route non-English tasks through English, a behavior confirmed as causally significant. Even explicit instructions to abandon this pivot are largely ignored, highlighting its deep integration into model strategies. The study also identifies a scale-dependent boundary: models below roughly 10B parameters exhibit greater behavioral variability, while larger models converge towards the observed regularity.
These findings have profound implications. They suggest that models possess an inherent structural tendency to maintain certain strategies across languages, which can be leveraged for safer and more predictable multilingual AI deployment. The methodology offers a new toolset for diagnosing and improving model behavior, emphasizing the importance of intermediate actions over final outputs. Looking ahead, future work will explore internal representations, training influences, and broader model architectures to deepen our understanding of cross-lingual strategy retention, ultimately guiding the development of more aligned and trustworthy AI systems.
Deep Dive
Abstract
When a tool-using agent is given the same task in a different language, does it still take the same steps? Multilingual evaluation rarely asks: it compares final answers and discards the actions. Yet those actions are the product: they fix cost and latency, decide how the system fails, and are the only auditable part of its behaviour. We make the action policy the measured object across 8 models, 6 parallel benchmarks and 41 languages (2.38M rollouts). The naive measurement fails: five confounds sit between raw trace similarity and any defensible claim, each able to flip a conclusion. Short traces score higher, empty traces score perfectly, unrelated traces agree by chance over half the time, the gap is capped by each model's reproducibility, and a model asked the same question twice in one language answers differently, leaving no baseline. We remove all five, and every correction makes the effect larger. Divergence proves structural, not sampling noise: it survives greedy decoding in every cell and stays flat as temperature rises, even as models grow less self-consistent. Normalised by their own reproducibility, four very different frontier models converge under greedy decoding, each keeping 71-73% of its action policy across languages, with model identity explaining only 5.7% of the variance. Below roughly 10B parameters it breaks down, and the ordering among smaller models is largely an artifact of a chance floor we measure by permutation rather than assume. Agents route non-English tasks through English; this pivot is causally load-bearing, confirmed by a pre-registered prediction across four models, and models will not abandon it when told to. Finally, a single trace-extraction regex, not the model, manufactured a multilingual failure: two worked examples raise one model's measured accuracy twenty-sixfold while its accuracy on readable outputs barely moves.
References (20)
Efficient Memory Management for Large Language Model Serving with PagedAttention
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang et al.
Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontier
John Dang, Shivalika Singh, Daniel D'souza et al.
Gemma 3 Technical Report
Gemma Team Aishwarya Kamath, Johan Ferret, Shreya Pathak et al.
gpt-oss-120b&gpt-oss-20b Model Card
OpenAI Sandhini Agarwal, L. Ahmad, Jason Ai et al.
Markov Decision Processes: Discrete Stochastic Dynamic Programming
M. Puterman
Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting
Melanie Sclar, Yejin Choi, Yulia Tsvetkov et al.
The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants
Lucas Bandarkar, Davis Liang, Benjamin Muller et al.
On the Cross-lingual Transferability of Monolingual Representations
Mikel Artetxe, Sebastian Ruder, Dani Yogatama
MLQA: Evaluating Cross-lingual Extractive Question Answering
Patrick Lewis, Barlas Oğuz, Ruty Rinott et al.
XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages
Tahmid Hasan, Abhik Bhattacharjee, Md Saiful Islam et al.
Do Llamas Work in English? On the Latent Language of Multilingual Transformers
Chris Wendler, Veniamin Veselovsky, Giovanni Monea et al.
No Language Left Behind: Scaling Human-Centered Machine Translation
Nllb team, M. Costa-jussà, James Cross et al.
How do Large Language Models Handle Multilingualism?
Yiran Zhao, Wenxuan Zhang, Guizhen Chen et al.
XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning
E. Ponti, Goran Glavavs, Olga Majewska et al.
Holistic Evaluation of Language Models
Percy Liang, Rishi Bommasani, Tony Lee et al.
XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
Junjie Hu, Sebastian Ruder, Aditya Siddhant et al.
Chain of Thought Prompting Elicits Reasoning in Large Language Models
Jason Wei, Xuezhi Wang, Dale Schuurmans et al.
GAIA: a benchmark for General AI Assistants
G. Mialon, Clémentine Fourrier, Craig Swift et al.
XNLI: Evaluating Cross-lingual Sentence Representations
Alexis Conneau, Guillaume Lample, Ruty Rinott et al.