Flaky Test Statistics: What the Evidence Shows in 2026
Flaky Test Statistics: What the Evidence Shows in 2026
A CI failure can be technically harmless and still derail your release: someone reruns it, waits, investigates, and decides whether the red build is real. That is why flaky test statistics matter—but only if you keep track of what each number counts.
The short answer: there is no representative, universal 2026 flaky-test rate. The strongest evidence measures different things—runs, test inventories, commits, repository failures, teams, survey responses, and probability models. Read the denominator with the figure, and the data still gives you a useful picture of prevalence, CI impact, and likely failure mechanisms.
Key flaky test statistics at a glance
Google reported that about 1.5% of all test runs produced a flaky result in May 2016, while almost 16% of its tests had some level of flakiness. Google defined a flaky result as the same code both passing and failing; the first figure is run-level, while the second is a test-inventory measure. It also said about 84% of observed pass-to-fail transitions in its post-submit CI involved a flaky test. Google Testing Blog
In Google’s April 2017 follow-up, around 63,000 of approximately 4.2 million CI tests had a flaky run during one week—less than 2% of tests. In that same weekly population, Google reported flakiness in 0.5% of its small tests, 1.6% of medium tests, and 14% of large tests; its size labels were engineer-selected and subjective. Google Testing Blog
GitHub reported that 1 in 11 commits in its monolith had at least one red build caused by a flaky test in its 2020 baseline. After six weeks using its flaky-build management system, GitHub reported fewer than 0.5% of commits, or 1 in 200, with that outcome—an 18× improvement by its calculation. This is a commit-level observation from one company, not an industry rate. GitHub Engineering
Atlassian reported flaky tests caused as much as 21% of master-build failures in Jira Frontend, and approximately 15% of repository failures in Jira Backend. In its December 2025 account of those repositories, Atlassian also said reruns and investigation wasted more than 150,000 developer hours per year. These are company-specific repository and labor measures. Atlassian
Bitrise reported that the share of teams experiencing any test flakiness rose from 10% in 2022 to 26% in 2025. Its aggregated, anonymized dataset covered more than 10 million builds collected from January 2022 through June 2025. This is a team-level yes/no measure on Bitrise’s platform, not a percentage of tests or test executions that flaked. Bitrise Mobile Insights
A 2025 study of 24 Java projects on GitHub identified 810 flaky tests across 10,000 suite runs; 75% belonged to a failure cluster, with a mean cluster size of 13.5 tests. That is evidence about the study’s open-source Java population, not all test suites. Systemic Flakiness study
A 2025 FSE study found that 46.5% of flaky tests in its dataset were resource-affected flaky tests. The study covered 52 Java, JavaScript, and Python projects across 27 resource configurations, so the result identifies a dimension worth measuring rather than a universal cause share. FSE 2025
In a survey collected from July 21 through October 31, 2023, 1,615 qualified respondents from 70 countries reported spending an additional 7.8% of QA time fixing flaky tests. The same survey reported that 58% of respondents’ teams had more than 1% flaky test runs. These are self-reported survey results, not CI telemetry. TestMu AI Future of QA Survey
Nx modelled a 39% broken-CI outcome with 50 VMs, 10 average tests per VM, and a 0.1% flaky-test probability. The 500-test configuration is a probability scenario published in March 2024, not an observed flaky-test prevalence measurement. Nx
Get the Mobile Testing Playbook Used by 800+ QA Teams
Discover 50+ battle-tested strategies to catch critical bugs before production and ship 5-star apps faster.
Why there is no single flaky-test rate
A flaky test is a test that can pass and fail without a relevant code change. That definition is straightforward. The measurement is not.
Google’s 1.5% figure counts test runs. Its almost-16% figure counts tests in an inventory that showed some flakiness. GitHub counts commits with at least one affected build. Atlassian reports shares of repository or master-build failures. Bitrise counts teams with any flakiness. TestMu AI records respondents’ reported time. Nx models the chance that a CI configuration breaks.
Those measurements can all be true at the same time. They do not combine into an average “industry flake rate.” A low run-level probability can coexist with frequent disruption when your suite runs at high volume, while a large share of affected tests may produce little day-to-day friction if those tests run rarely.
Use three checks before you reuse any flaky test statistic in a planning document:
Check the unit. Is the numerator flaky runs, tests, transitions, builds, commits, or teams?
Check the population. Does it describe one repository, one vendor platform, selected open-source projects, or a survey sample?
Check the observation period and evidence type. A 2016 internal measurement, a 2023 survey, a 2025 repository report, and a modelled scenario answer different questions.
This denominator-first approach is the practical difference between a citable statistic and a persuasive-looking but misleading one.
What observed flaky-test data says about CI impact
The oldest large-scale figures here are still useful because they separate several ways flakiness appears. Google’s 2016 account shows that an execution-level rate, an inventory-level share, and a share of pass-to-fail transitions tell different operational stories. Its note that flaky-test insertion and fix rates were approximately equal is also a warning: active cleanup does not automatically reduce the backlog if new flaky tests arrive at a similar pace.
Google’s 2017 size breakdown adds a distributional signal. In its one-week sample, the large-test category had a much higher reported flaky share than small and medium categories. You should not infer that test size alone caused the difference, particularly because the labels were subjective. You can infer that your suite may not have one evenly distributed problem.
GitHub’s 2020 baseline answers a different question: how often did flakiness block a commit? Its reported move from 1 in 11 commits to fewer than 1 in 200 after six weeks shows why a commit-level metric can be valuable to your delivery process. It measures the experience your developers have when deciding whether a red build deserves attention.
Atlassian’s figures make the cost side visible. A share of master-build or repository failures is not equivalent to the share of flaky tests, but it can be much closer to the queue of reruns and investigations your team experiences. The more than 150,000 annual developer hours Atlassian reported are not a general cost estimate for your organization; they show the scale that repeated uncertainty can reach in a large operating context.
Bitrise offers the most recent mobile-adjacent signal in this set. Its 10% to 26% movement tells you how broadly teams on that platform reported any test flakiness from 2022 to 2025. It cannot tell you the mobile UI-test flake rate, because it does not use tests or executions as its denominator. It does show that the issue is not confined to a small number of teams in that dataset.
What recent research says about causes
Recent research suggests that many flaky failures are systemic rather than isolated. The 2025 systemic-flakiness study found three quarters of its identified flaky tests in clusters. If multiple tests fail together, repeatedly repairing each test in isolation can miss the shared condition behind them.
The researchers identified networking issues and unstable external dependencies as predominant systemic causes after manually inspecting stack traces and error messages in their dataset. That finding is specific to the 24 Java GitHub projects studied. Still, it gives you a useful investigation order: look for the service, dependency, or shared environment before assuming every failing test has its own defect.
The FSE result on resource-affected flaky tests adds another variable: computational resources. When 46.5% of flaky tests in that selected dataset were affected by resources, CPU, memory, contention, and configuration became part of the test evidence—not merely background conditions.
For mobile testing, the same discipline applies without assuming the studies prove a mobile-specific rate. Separate the app flow from the execution conditions: physical device or emulator, network state, operating-system state, available resources, and external services. If your test mix spans layers, a test-pyramid strategy can also help you record which layer produces the most disruptive reruns rather than treating every failure as equivalent.
How small flaky-test probabilities become broken pipelines
Nx’s 39% result is useful precisely because it is not telemetry. It makes the arithmetic visible: its scenario has 500 tests across 50 VMs, and each test has a 0.1% flaky-test probability. When many opportunities for a failure exist in the same CI run, the chance that at least one failure interrupts the pipeline becomes far larger than the individual probability suggests.
That is why “only a small percentage of runs” is not enough to prioritize or dismiss a problem. Your practical measure is often the rate at which a flaky result blocks a commit, forces a rerun, or creates a human investigation. GitHub’s commit-level result and Atlassian’s failure-share figures are useful complements to run-level measurements for that reason.
The TestMu AI survey captures the human side from another angle. Respondents reported spending additional QA time on flaky-test fixes. That does not establish a machine-measured time cost for every organization. It does make a planning point clear: your flakiness metric should include the recovery work after the failed run, not just the failed run itself.
If you need to reduce that recovery work, start with your own evidence: record the test layer, environment, suspected failure class, rerun result, and time spent investigating. For a diagnostic workflow after you have those records, see this guide to isolating the cause of a flaky mobile test.
What the statistics prove—and what they do not
The evidence supports three conclusions.
First, flakiness is documented in large CI systems, vendor-platform populations, and open-source research datasets. Second, its impact can be larger than a run-level percentage implies because it can block commits, consume investigation time, and recur in clusters. Third, dependencies and resource conditions can be material parts of the failure mechanism.
The evidence does not support a representative 2026 flaky-test rate across mobile, web, backend, unit, integration, and end-to-end testing. No source above measures all of those populations using one common denominator and observation period.
It also does not establish that a particular framework, device strategy, or testing product prevents flakiness in every environment. A study of clustered Java failures, a vendor’s team-level mobile CI data, and a company’s monolith intervention are evidence for their stated populations. Treating them as universal rankings would exceed the evidence.
The mobile benchmark that is still missing
No first-party Quash data covers flaky-test statistics, and Quash has not published a flaky-test telemetry dataset for this question. There is no honest Quash-specific percentage to add to this report.
The larger public-data gap is equally specific: this evidence set contains no verified benchmark across real mobile applications that reports flaky-test prevalence alongside test layer, execution environment, observation period, and failure cause. Bitrise is informative for platform-level team breadth, but it does not fill that gap.
A reproducible mobile flaky-test study would need to report at least:
Test layer: unit, UI, integration, or end-to-end.
Execution environment: emulator, simulator, physical device, or hosted infrastructure.
Observation period: a stated week, month, release cycle, or longer period.
Failure classification: app logic, timing, resources, network, dependency, or environment state.
Severity: the rerun, developer-time, or release-blocking consequence of each failure class.
Until that benchmark exists, the responsible choice is to use the closest statistic for the question you are asking—and retain its denominator.
Methodology
This report prioritizes first-party company accounts for their own CI data, an original research paper and conference program for study findings, a named survey for self-reported time, and a clearly labelled vendor probability model. It does not treat competitor roundups as evidence for their underlying figures.
Figures are grouped by their measurement unit rather than ranked by size. Historical measurements retain their original dates. Survey responses remain survey responses. The Nx result remains a model. Those labels make the figures easier to compare without making them falsely interchangeable.
Conclusion
The most useful flaky test statistic is not necessarily the largest percentage. It is the number whose unit matches the decision in front of you.
Use run-level data to understand execution reliability, commit- or build-level data to understand delivery disruption, and investigation time to understand operational cost. Then split failures by layer and environment before choosing what to repair. That gives you a measurement system that can turn a red build from recurring noise into a specific engineering decision.
Continue reading
More articles from the Quash editorial desk.
