Google released Android Bench 2.0 on Wednesday, introducing multi-day engineering tests and continuous scoring metrics to measure how artificial intelligence models handle complex, real-world application development.
Key points
- Google released Android Bench 2.0 to evaluate AI coding models on multi-day software engineering challenges.
- The benchmark abandons binary pass-fail grading for a continuous completion rate based on functionality and visual fidelity.
- OpenAI's GPT-6 Astra leads the new leaderboard with a 28.0% pass rate, followed by Claude Fable 5.1 at 22.7%.
- No evaluated model achieved a 100% pass rate when porting cross-platform applications directly to Android.
- The framework tests complete agent harnesses including OpenAI Codex and Google's Antigravity SDK alongside the underlying models.
The revised benchmark shifts away from small, isolated bug fixes and instead challenges AI models with assignments that typically take human engineers several days to a full week to finish. As software teams increasingly rely on autonomous tools to modify large codebases, the update provides a standardized measurement of how effectively these systems manage sustained technical labor on the Android operating system.
Continuous Scoring Replaces Pass-Fail Grades
The original benchmark assessed model output using binary pass or fail criteria. Under that framework, an agent that completed most of an extensive migration but missed a single edge condition received a zero, obscuring partial success on complex assignments.
On multi-day engineering tasks, binary pass or fail grading doesn’t capture the full picture. For example, an agent might refactor 40 screens to Jetpack Compose, set up database tables, and pass 90% of requirements, but fail a single edge
case assertion. Binary scoring rates this run as 0%.
According to the Android Bench 2.0 announcement authored by Matthew McCullough, VP of Product Management for Android Developer, the system now calculates a continuous completion rate. The metric measures functional execution, visual fidelity, and the absence of regressions, while applying penalties if a model deviates from explicit instructions or structural constraints.
Long-Horizon Tasks Lower Top AI Scores
The transition to multi-day task profiles reduced overall success rates across all leading models. While frontier AI tools frequently scored above 90% on previous incremental bug-fix benchmarks, the highest-performing models struggled to complete the expanded challenges.
| Model | Framework Harness | Pass Rate |
|---|---|---|
| OpenAI GPT-6 Astra | OpenAI Codex | 28.0% |
| Anthropic Claude Fable 5.1 | Claude-Code | 22.7% |
| Gemini 3.8 Flash | Antigravity SDK | Sub-20% |
OpenAI’s GPT-6 Astra took the top position on the updated leaderboard with a 28.0% pass rate. Anthropic’s Claude Fable 5.1 followed in second place with 22.7%. AI systems performed reliably on deterministic assignments, such as converting Java files to Kotlin or swapping network layers from Retrofit to Ktor. Models also managed feature creation from clean slates. Difficulties emerged during architectural refactoring, runtime verification, handling unreleased libraries, and porting cross-platform codebases to native Android, where completion rates peaked at 80% and zero models reached 100%.
Evaluating Developer Harnesses and Agent Workflows
Android Bench 2.0 expands testing beyond standalone foundational models to measure complete developer agents. Google paired models with their respective operational harnesses, including OpenAI Codex, Claude-Code, the Antigravity SDK, Kimi-Code, and Qwen-Coder, to measure real-world performance.
- Evaluates end-to-end agentic workflows rather than isolated raw API prompts.
- Measures token consumption and context window management across multi-turn sessions.
- Assesses developer environment integration, build-system repairs, and automated terminal commands.
The updated framework builds on the original Android Bench introduced in March 2026 and incorporates the sandboxed testing environments standardized by Google under the Harbor framework in July 2026. The testing suite continues to run models against isolated virtual devices to verify visual and operational fidelity during grading runs.
