We usually measure agents in English
When we say an AI agent can manage an inbox, compare apartments, or update a calendar, we almost always mean that it can do those things in English.
We wanted to see what happens when the language changes and the intended task stays the same. An agent still has to understand the request, keep track of its goal, use the right tools, and know when to ask a question. Any misunderstanding can affect every action that follows.
Our new paper, OmnilingualGAIA2: Evaluating the Multilingual Gap in Frontier AI Agents, measures how well these abilities transfer from English to ten other languages.
For all seven agents, mean pass@3 across the ten target languages was below the English score. The pooled gap ranged from 8.8 to 18.4 percentage points.
Four GAIA2 capabilities, in ten more languages
OmnilingualGAIA2 builds on GAIA2. In GAIA2, an agent works inside a simulated digital life. It can use email, messages, contacts, calendars, shops, cabs, cloud files, housing services, and city information. These apps share data, and the environment can change while the agent is working.
We machine-translated four of GAIA2's seven capability splits into Spanish, German, French, Italian, Portuguese, Indonesian, Turkish, Mandarin Chinese, Japanese, and Hindi. Time, Noise, and Agent2Agent remain out of scope because they require translation or verifier paths not covered by this release. We focused on four abilities that remain difficult even in English:
Retrieve and combine information across apps.
Turn a request into the correct sequence of actions.
Revise a plan when the environment changes.
Recognise missing or conflicting information.
The difficult part is consistency
This is not a collection of questions with short answers. A person, event, or product can appear in the request, in several apps, and in the expected result. If its name changes between those places, the task may become impossible even though each sentence looks fine on its own.
Our pipeline has three steps. It first translates the text stored in each app. It then builds a shared table for recurring names and terms. Finally, it uses that table while translating the request and the expected answer. In a head-to-head evaluation of the two shortlisted open-weight translators, Gemma 4 31B Instruct outperformed Qwen 3.6 27B in all ten languages. A second-model review pass produced no statistically measurable quality improvement, so we left it out.
We also adapted the verifier instead of assuming that an English judge would work unchanged. We rewrote its prompts and checked the GPT-OSS-120B judge against human annotations. On the 153 calibration traces whose verdict invokes an LLM checker, agreement with the fixed human-majority label ranges from κ = 0.886 to 0.971 across the eleven language versions. This measures the language-sensitive part of the verifier; most remaining checks are deterministic.
How we evaluated the agents
Each capability–language cell contains 160 matched scenarios. We run each scenario three times and report pass@3: a scenario is counted as solved if at least one valid run passes the verifier. The headline score averages the four capabilities, and the multilingual score pools the ten target languages. The paper also reports pass@1, all-three-run consistency, Wilson 95% confidence intervals, and paired McNemar tests against English.
How this relates to other multilingual agent benchmarks
OmnilingualGAIA2 complements multilingual web navigation and tool-use evaluations such as X-WebAgentBench, SEATauBench, MLCL, MAPS, PolyWorkBench, and GAIA-v2-LILT. Its specific contribution is to preserve GAIA2's executable task structure across ten languages, evaluate a heterogeneous seven-agent cohort with a shared calibrated verifier, and separate model failures from translation and verifier effects. The paper's related-work section gives the full comparison.
Every agent shows a pooled multilingual gap
We tested Claude 4.7 Opus, GPT-5.4, Gemini 3.1 Pro, Kimi 2.6, Gemma 4 31B Instruct, and two Qwen 3.6 variants. For every one of them, the average over the ten translated languages is lower than the English score. This pooled statement should not be read as saying that every individual language–capability cell is significantly worse.
Scores are percentages. The multilingual value is the mean over the ten languages other than English.
Mandarin Chinese, Japanese, and Hindi are particularly difficult. At the aggregate level, the ordering of the four abilities remains stable: search > execution > adaptability > ambiguity. Language changes the scores without changing the broad difficulty ordering, although individual model–language cells vary.
The gap is about orchestration, not counting
We decomposed verifier failures for Claude 4.7 Opus, GPT-5.4, and Kimi 2.6. Failure rates on quantitative and categorical checks barely change across languages. The additional failures are instead concentrated in two places: the agent performs the wrong number of write actions, or its final reply does not contain the expected information.
Their behaviour changes too. Outside English, they spend more of their tool calls looking around and fewer of them taking action. The share of write actions falls by 2.1 points for Claude, 2.5 for GPT-5.4, and 1.3 for Kimi. In other words, the agents search more and commit less.
How much of this is bad translation?
This was the first question we asked ourselves. If a translated task is broken, a lower score tells us nothing about the agent.
For Claude 4.7 Opus, an automatic diagnostic protocol compared target-language failures with matched English runs and assigned them to the agent, the translation, or the verifier. The estimate excludes infrastructure failures—8.5% of runs—and stratifies deterministic and stochastic regressions before reweighting them to the observed failure volume. Scenario-clustered bootstrap intervals quantify uncertainty.
Five native or fluent reviewers collectively re-adjudicated 149 sampled items across seven languages, independently of the automatic protocol. Nine were undecidable; on the remaining 140, the human judgment matched the automatic fault-side label in 128 cases (91.4%, 95% CI 85.6–95.0). Agreement was lower for Japanese—68.2%—where several agent-failure labels were reassigned to translation defects.
Agent failureWrong, missing, or extra actions on a fair task
Translation defectThe translated task changes or loses a constraint
Verifier errorA correct action is rejected by the evaluator
Scenario-clustered 95% CIs: agent 50.3–60.7%; translation 29.5–39.1%; verifier 7.1–13.5%. The 6.4% result uses a different denominator: all scenario–language pairs.
Translation accounts for an estimated 34.5% of the measured performance gap. Across the whole benchmark, the estimated bound is 6.4% of scenario–language pairs made unsolvable by machine translation.
In the infrastructure-clean Claude 4.7 Opus analysis, the target-language agent succeeds at least once on 74.5% of scenario–language pairs; under the paper's unsolvability definition, those pairs cannot be translation-contaminated. We audited samples from the remaining outcome cells—including tasks that also fail in English—to estimate the 6.4% benchmark-wide bound. Translation defects are material, but they do not account for most of the measured gap.
A larger model is not enough
A separate four-checkpoint Qwen 3.5 ablation reports pass@1, rather than the headline pass@3. Average performance rises across the tested ladder, but the English-minus-target gaps are 8.1, 7.5, 10.4, and 12.7 points: scale does not consistently close the gap. This ablation uses a different Qwen generation from the Qwen 3.6 models above. A smaller Claude 4.7 Opus effort ablation likewise finds that increasing reasoning effort does not remove most of the gap.
Some cues do not survive translation
The most interesting failures are often not obviously bad translations. They happen when another language expresses a constraint differently, or does not state it explicitly at all.
This part of the study is qualitative: three linguists inspected traces for 16 scenarios spanning the four capabilities, and the paper presents six representative cases. The example below illustrates one mechanism; it is not a prevalence estimate.
“Delete my contact from the US.”The environment contains two matching US contacts.
The mismatch is visible. The agent usually asks which contact the user means.
No destructive action“My contact” can read as a generic set. The agent sees no conflict and deletes both.
Unsafe actionWe saw the same pattern elsewhere. Translation can make an existing ambiguity worse. A preposition can remove a directional constraint. Number and article markers can disappear, changing whether the agent knows it should ask for clarification. Names and categories can also drift when the prompt uses one translation and an app uses another.
A separate model-assisted audit used Claude 5 Sonnet to examine Claude 4.7 Opus failures across 939 Mandarin, Hindi, and Japanese scenarios. It classified 88 failures (9%) as script-related, often because the evaluated agent noticed that a name existed in both native and Latin forms but searched for only one. The paper estimates an impact of roughly two pass@3 points for this agent; this audit does not establish the same rate for all seven systems.
What this benchmark does not establish
- It covers four of seven GAIA2 capabilities. Time, Noise, and Agent2Agent require additional translation or verifier work.
- It is machine-translated. The full corpus was not professionally human-post-edited, so a bounded translation-defect floor remains.
- The judge has a human-reference ceiling. High judge agreement means parity with an imperfect annotation set, not absolute correctness.
- The leaderboard is harness-dependent. Kimi uses a different harness, and proprietary systems use provider-default reasoning effort; the effort ablation covers only one model and three target languages.
- Human audits are limited. They cover subsets of languages and scenarios rather than the complete matrix.
- Translation is not cultural localisation. These scores do not measure culturally appropriate agent behaviour.
What this means
An English-only score is insufficient to characterise multilingual agent performance. For all seven evaluated agents, the ten-language mean is below English. The size of the loss varies, and the additional failures concentrate on using tools and carrying out actions rather than simple calculation.
OmnilingualGAIA2 gives us a way to measure that problem. The release includes the dataset, evaluation code, multilingual verifier, and human annotations. We hope multilingual agent evaluation becomes a normal part of model reporting.
English should be one evaluation language, not the definition of the task.
Links
Citation
@misc{caciolai2026omnilingualgaia2,
title={OmnilingualGAIA2: Evaluating the Multilingual Gap
in Frontier AI Agents},
author={Andrea Caciolai and Pere-Lluís Huguet Cabot and
Chierh Cheng and Albert Ventayol-Boada and Gabriel Mejia Gonzalez
and Christophe Ropers and Lucas Bandarkar and Sebastian Ruder
and Darlene Sakakihara and Elliot Yun and Pierre Andrews
and Grégoire Mialon and Romain Froger and Marta R. Costa-jussà},
year={2026},
eprint={2608.08775},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2608.08775}
}