Android Instrumentation Testing in Continuous Integration: Practices, Patterns, and Performance

A new study accepted at the IEEE International Conference on Software Testing, Verification and Validation (ICST 2026) provides the first large-scale analysis of how Android developers handle instrumentation testing in continuous integration. Researchers Hamid Parsazadeh, Taher A. Ghaleb, and Safwat Hassan analyzed 4,518 open-source Android repositories using CI, examining workflow files, scripts, and build configurations. They found that only 10.6% (481 projects) actually run instrumentation tests—the end-to-end tests that run on real devices or emulators and catch integration issues unit tests miss. The primary barrier is the notorious fragility of emulator setup and configuration drift over time.

The research reveals three key patterns in how teams approach this challenge. First, projects typically use either reusable community components (like GitHub Actions for Android) or custom repository-specific scripts. Second, when setups do change, teams tend to migrate from custom scripts toward these more standardized community components. Third, by analyzing commit histories and GitHub Actions metadata, the study found each approach has distinct performance characteristics: community-based setups are most reliable for daily checks, third-party device labs work for scheduled regression testing but fail more often, and custom scripting offers flexibility but correlates with significantly more test reruns. This data-driven analysis gives engineering teams concrete evidence for choosing and evolving their mobile testing infrastructure.