flakescopeOpen source

A flaky test fails sometimes and passes sometimes, and the useful question is what it depends on. flakescope answers that for Go packages. It varies configurations, not goroutine interleavings: Go has no seedable goroutine scheduler outside testing/synctest, so no tool can replay a particular ordering. What it can vary is test order, available processors, and the race detector, and it reports which of those a failure tracks.

A configuration matrix laid out over cells, one per shuffle arm and GOMAXPROCS value, with runs dealt across cells in rotation so every arm collects enough observations to state a failure rate.

A runner that parses the go test -json event stream, runs each configuration in its own process group, and treats build failures and all-timeout runs as tool errors (exit 2) rather than findings.

A classifier that separates flaky tests from always-failing ones and labels each flaky test order-dependent, load-dependent, both, or undetermined, printing the evidence rates next to the label.

Failure clustering by normalized signature with a minimal reproducing configuration per cluster, plus a machine-readable --json report with a versioned schema.

flakescope generates the configuration matrix, runs go test -json for each configuration across parallel workers, and folds the event streams into per-test pass and fail counts broken out by axis. Classification, clustering, and minimal configuration selection all run over that aggregate. Its own test suite replays recorded go test -json streams from testdata instead of invoking go test, so the tool that detects flaky tests does not have flaky tests of its own.

Dependence is decided by failure rate, not presence. Before v0.3.0 the order rule only asked whether every failure happened under shuffle; with most runs shuffled, that was true by chance for almost any failing test, so nearly everything came back order-dependent. The fix compares rates between arms and requires at least four observations per arm, a 2x ratio, and a pooled two-proportion z of 2.

Clustering prefers splitting over merging. A split shows up as two near-identical clusters you can read; a merge hides a second bug behind the first. So only run-varying noise such as goroutine IDs, addresses, and temp paths is normalized, and integers in messages are left alone.

The race detector runs on one configuration in seven. Race builds dominate wall-clock time and answer a yes or no question that a sample handles fine, which cuts matrix cost from 5.5x to 2.3x of a race-free run. Seven is odd so raced runs do not all land in the same shuffle arm.

Minimality is per cluster. A test with two failure modes gets two repro commands, because one command would reproduce only one bug and hide the other.

On a 4-core machine, 1,000 configurations take about 171 seconds warm and 700 finish in 116 seconds. Both are floors, since the fixture's tests do almost no work.

That 1,000-configuration run produced 2,174 individual failures across three tests and reported them as four clusters, including a load-dependent test split into its two distinct failure modes at GOMAXPROCS 2 and 4.

The report states its own resolution limit: at the default 20 runs it can only separate rates above roughly 38 to 50 percent, and it prints the rate a run could have resolved so the next --runs is a number rather than a guess.