Repository navigation
Test significance across processes with --runs - #574
Open
thomasklemm wants to merge 1 commit into
Open
thomasklemm wants to merge 1 commit into
thomasklemm wants to merge 1 commit into
Conversation
--pvalue ran Welch's t-test over the measured iterations of one process per executable. Iterations of one process share its heap layout, GC timing, hash seed and JIT decisions, so they are not independent samples: the test sees only the noise inside a process, while two processes of the same Ruby differ by more than that. On an A/A run of ten identical processes per benchmark (YJIT and the interpreter, railsbench, lobsters, fib, respond_to, str_concat), 214 of 450 pairs of identical builds were marked significant at p < 0.05, where 5% is expected. --runs N now runs each benchmark in N processes per executable, interleaved, alternating the executable order every round, and the t-test compares the N per-process means. On the same processes, 5 against 5 gave 103 significant results out of 2,000 random groupings (5%). End to end, four identical Rubies with --runs 5 gave 2 of 30 significant comparisons, while the single-process test called 566 of the 950 pairs among those same 100 processes significant. The default stays at one process, so runs that don't ask for p-values take no longer. With one process the p-value is N/A and --pvalue warns, rather than showing stars it can't support. The timing columns report all measured iterations of the N processes, so their stddev includes the variation between processes, and the 1st itr ratio averages the first iteration of each process. The JSON keeps the concatenated series and records the iterations of each process in warmup_itrs_per_run and bench_itrs_per_run; other values, such as JIT stats, come from the first process. --runs refuses the Ractor harness for now, because its per-Ractor-count results are not merged across processes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
--pvalueruns Welch's t-test over the measured iterations of one process per Ruby. Those iterations share one heap layout, hash seed, GC schedule and set of JIT decisions, so the test only sees the noise inside a process. Two processes of the same Ruby differ by more than that, and the test reports that difference as significant. Comparing identical Rubies, about half the comparisons got a star:--runs 5This adds
--runs N: each benchmark runs in N processes per Ruby, interleaved, with the order of the Rubies flipped every round, and the t-test compares the N per-process means. With one process, still the default, the p-value is N/A and--pvalueprints a warning, rather than showing stars it can't support. The default stays at one process because--runs Ntakes N times as long. The timing columns report all measured iterations of the N processes, the 1st itr ratio averages each process's first iteration, and the JSON records each process's iteration counts inwarmup_itrs_per_runandbench_itrs_per_run.The test still finds differences that are larger than the variation between processes. With
--runs 5against YJIT, the interpreter got***on railsbench, fib and str_concat. An added spin loop made fib 6.6% slower (***, p = 4e-5) and str_concat 11% slower (**), but railsbench only 1.9%, where it was not detected (p = 0.31): railsbench processes vary by about 3% on the 4-vCPU VM used here, so five runs did not detect this 2% difference.How this was measured
Ruby 4.0.9 with YJIT and the interpreter,
--no-sudo --interleave --pvalue, default iterations, on mainad63bf8and on this branch; railsbench, lobsters, fib, respond_to and str_concat. Main: one run with-e yjit01::ruby --yjit … -e yjit10::ruby --yjit, one with ten interpreter executables. "Pairs" are all pairs of processes of a benchmark, tested with main's single-process test. "Groupings" are 1,000 random 5-against-5 splits of the same ten processes per mode, tested on their per-process means; they reuse processes, so they are not independent (103 of 2,000). Branch:--runs 5 --pvalue -e A::ruby --yjit … -e D::ruby --yjit, all 6 pairs per benchmark. Slowdown: a preloaded module that spins after each iteration for a set share of its time; the ratios above are measured, not nominal. Wall time grows with N: the branch A/A ran 100 benchmark processes in 2,156 s; one process per Ruby would have been 20 processes, about 430 s.