Google’s Android Bench 2.0 evaluates frontier AI models on multi-day coding tasks to determine how well agents handle complex engineering.
The new benchmark aligns its testing structure with the Harbor framework. Instead of grading simple bug patches, the system introduces long-horizon tasks that require multiple days or an entire week of human engineering labour.
Assignments in the benchmark require AI models to update project dependencies, build applications from scratch, implement multi-step features, and port cross-platform code bases directly to native Android. Early benchmark runs focused strictly on incremental code modifications, where top models routinely achieved pass rates near 91 percent. The introduction of long-horizon tasks sharply reduces those figures.
The highest pass rate on these multi-day tasks currently reaches approximately 28 percent, with OpenAI’s GPT-6 Astra leading the public leaderboard:
Matthew McCullough, VP of Product Management for Android Developer, said: “On multi-day engineering tasks, binary pass or fail grading doesn’t capture the full picture.”
Google replaces binary evaluation metrics with Android Bench 2.0
Under the revised methodology, evaluators apply continuous scoring to gauge partial architectural success. A strict pass-fail approach previously marked a run as zero percent if a single edge-case assertion failed. This occurred even when an agent converted 40 screens to Jetpack Compose, structured database tables, and satisfied 90 percent of functional requirements.
The updated evaluation formula calculates completion rates by assessing functional correctness, visual fidelity, and regression prevention. Evaluators apply objective point deductions whenever a model disobeys instructions or breaks structural project constraints. Model cards display these completion rates alongside traditional pass percentages and average computed expenses per task.
The benchmark data indicates clear differences between code synthesis and code maintenance. Models perform better when authoring fresh files than when refactoring existing code bases, where success depends on architectural hierarchy rather than sheer code volume.
Tested systems demonstrate high consistency on deterministic conversions. They translate Java classes to Kotlin, swap Retrofit dependencies for Ktor, and configure ViewModel architecture patterns across more than 125 files and 8,000 lines of code.
System reliability drops during dynamic runtime validation checks, such as unmapped dependency injection graphs. Models struggle with breaking framework revisions and unreleased library dependencies.
Converting cross-platform software to native Android remains an open challenge. No model achieved a 100 percent pass rate on cross-platform tasks, while leading systems peaked at an 80 percent completion score.
Provider agent pairings reduce token consumption
Android Bench 2.0 incorporates agentic evaluations to inspect how models operate within actual developer tooling environments. Initial evaluations pair frontier models with native agents developed by their respective model providers.
Engineers tested OpenAI’s GPT 5.6 Sol through Codex and paired Gemini 3.8 Flash with Google Antigravity. Harness design dictates operational efficiency. Prompt caching and compact tool windowing within the harness produce measurable token reductions during complex multi-step sessions.
The updated leaderboard includes Gemini 3.8 Flash, Gemini 3.7 Flash, OpenAI’s GPT-6, Anthropic’s Fable 5.1, Kimi K3, and Qwen 3.8-Max. Android Bench plans to evaluate cross-provider agent combinations in future testing cycles.
See also: SmartBear embeds BearQ testing agent in Atlassian Jira
Want to learn more about AI and big data from industry leaders? Check out AI & Big Data Expo taking place in Amsterdam, California, and London. The comprehensive event is part of TechEx and is co-located with other leading technology events including the Cyber Security & Cloud Expo. Click here for more information.
Developer is powered by TechForge Media. Explore other upcoming enterprise technology events and webinars here.