Thanks for these, not sure if you noticed the notes from each run e.g. need >= 4 samples to detect a difference at alpha level 0.05.
Basically run the benchmark with a -count=6 or more and then run the tool, and you get a the comparison values which are typically the interesting bit On Thu, 25 Apr 2024 at 00:29, Robert Engels <rob...@me.com> wrote: > │ /Users/robertengels/go1.21.5.txt │ > /Users/robertengels/go1.22.2.txt │ > │ sec/op │ sec/op vs base > │ > AddFixed-8 0.9603n ± ∞ ¹ 0.7931n ± ∞ ¹ ~ > (p=1.000 n=1) ² > AddDecimal-8 66.41n ± ∞ ¹ 66.27n ± ∞ ¹ ~ > (p=1.000 n=1) ² > AddBigInt-8 9.452n ± ∞ ¹ 10.650n ± ∞ ¹ ~ > (p=1.000 n=1) ² > AddBigFloat-8 63.26n ± ∞ ¹ 66.33n ± ∞ ¹ ~ > (p=1.000 n=1) ² > MulFixed-8 3.519n ± ∞ ¹ 3.939n ± ∞ ¹ ~ > (p=1.000 n=1) ² > MulDecimal-8 65.98n ± ∞ ¹ 67.07n ± ∞ ¹ ~ > (p=1.000 n=1) ² > MulBigInt-8 10.69n ± ∞ ¹ 10.49n ± ∞ ¹ ~ > (p=1.000 n=1) ² > MulBigFloat-8 23.72n ± ∞ ¹ 24.12n ± ∞ ¹ ~ > (p=1.000 n=1) ² > DivFixed-8 3.675n ± ∞ ¹ 3.661n ± ∞ ¹ ~ > (p=1.000 n=1) ² > DivDecimal-8 460.8n ± ∞ ¹ 469.6n ± ∞ ¹ ~ > (p=1.000 n=1) ² > DivBigInt-8 34.82n ± ∞ ¹ 34.90n ± ∞ ¹ ~ > (p=1.000 n=1) ² > DivBigFloat-8 110.4n ± ∞ ¹ 113.6n ± ∞ ¹ ~ > (p=1.000 n=1) ² > CmpFixed-8 0.2529n ± ∞ ¹ 0.2784n ± ∞ ¹ ~ > (p=1.000 n=1) ² > CmpDecimal-8 6.883n ± ∞ ¹ 6.475n ± ∞ ¹ ~ > (p=1.000 n=1) ² > CmpBigInt-8 4.779n ± ∞ ¹ 4.805n ± ∞ ¹ ~ > (p=1.000 n=1) ² > CmpBigFloat-8 4.411n ± ∞ ¹ 5.081n ± ∞ ¹ ~ > (p=1.000 n=1) ² > StringFixed-8 50.36n ± ∞ ¹ 50.64n ± ∞ ¹ ~ > (p=1.000 n=1) ² > StringNFixed-8 53.41n ± ∞ ¹ 49.66n ± ∞ ¹ ~ > (p=1.000 n=1) ² > StringDecimal-8 197.6n ± ∞ ¹ 197.0n ± ∞ ¹ ~ > (p=1.000 n=1) ² > StringBigInt-8 98.17n ± ∞ ¹ 98.00n ± ∞ ¹ ~ > (p=1.000 n=1) ² > StringBigFloat-8 386.2n ± ∞ ¹ 395.2n ± ∞ ¹ ~ > (p=1.000 n=1) ² > WriteTo-8 31.82n ± ∞ ¹ 31.71n ± ∞ ¹ ~ > (p=1.000 n=1) ² > geomean 22.01n 22.28n +1.26% > ¹ need >= 6 samples for confidence interval at level 0.95 > ² need >= 4 samples to detect a difference at alpha level 0.05 > > > > │ > /Users/robertengels/go1.21.5.txt │ /Users/robertengels/go1.22.2.txt │ > │ B/op │ B/op vs base > │ > AddFixed-8 0.000 ± ∞ ¹ 0.000 ± ∞ ¹ ~ > (p=1.000 n=1) ² > AddDecimal-8 80.00 ± ∞ ¹ 80.00 ± ∞ ¹ ~ > (p=1.000 n=1) ² > AddBigInt-8 0.000 ± ∞ ¹ 0.000 ± ∞ ¹ ~ > (p=1.000 n=1) ² > AddBigFloat-8 48.00 ± ∞ ¹ 48.00 ± ∞ ¹ ~ > (p=1.000 n=1) ² > MulFixed-8 0.000 ± ∞ ¹ 0.000 ± ∞ ¹ ~ > (p=1.000 n=1) ² > MulDecimal-8 80.00 ± ∞ ¹ 80.00 ± ∞ ¹ ~ > (p=1.000 n=1) ² > MulBigInt-8 0.000 ± ∞ ¹ 0.000 ± ∞ ¹ ~ > (p=1.000 n=1) ² > MulBigFloat-8 0.000 ± ∞ ¹ 0.000 ± ∞ ¹ ~ > (p=1.000 n=1) ² > DivFixed-8 0.000 ± ∞ ¹ 0.000 ± ∞ ¹ ~ > (p=1.000 n=1) ² > DivDecimal-8 384.0 ± ∞ ¹ 384.0 ± ∞ ¹ ~ > (p=1.000 n=1) ² > DivBigInt-8 8.000 ± ∞ ¹ 8.000 ± ∞ ¹ ~ > (p=1.000 n=1) ² > DivBigFloat-8 24.00 ± ∞ ¹ 24.00 ± ∞ ¹ ~ > (p=1.000 n=1) ² > CmpFixed-8 0.000 ± ∞ ¹ 0.000 ± ∞ ¹ ~ > (p=1.000 n=1) ² > CmpDecimal-8 0.000 ± ∞ ¹ 0.000 ± ∞ ¹ ~ > (p=1.000 n=1) ² > CmpBigInt-8 0.000 ± ∞ ¹ 0.000 ± ∞ ¹ ~ > (p=1.000 n=1) ² > CmpBigFloat-8 0.000 ± ∞ ¹ 0.000 ± ∞ ¹ ~ > (p=1.000 n=1) ² > StringFixed-8 24.00 ± ∞ ¹ 24.00 ± ∞ ¹ ~ > (p=1.000 n=1) ² > StringNFixed-8 24.00 ± ∞ ¹ 24.00 ± ∞ ¹ ~ > (p=1.000 n=1) ² > StringDecimal-8 56.00 ± ∞ ¹ 56.00 ± ∞ ¹ ~ > (p=1.000 n=1) ² > StringBigInt-8 16.00 ± ∞ ¹ 16.00 ± ∞ ¹ ~ > (p=1.000 n=1) ² > StringBigFloat-8 176.0 ± ∞ ¹ 176.0 ± ∞ ¹ ~ > (p=1.000 n=1) ² > WriteTo-8 29.00 ± ∞ ¹ 28.00 ± ∞ ¹ ~ > (p=1.000 n=1) ³ > geomean ⁴ -0.16% > ⁴ > ¹ need >= 6 samples for confidence interval at level 0.95 > ² all samples are equal > ³ need >= 4 samples to detect a difference at alpha level 0.05 > ⁴ summaries must be >0 to compute geomean > > > > │ > /Users/robertengels/go1.21.5.txt │ /Users/robertengels/go1.22.2.txt │ > │ allocs/op │ allocs/op vs base > │ > AddFixed-8 0.000 ± ∞ ¹ 0.000 ± ∞ ¹ ~ > (p=1.000 n=1) ² > AddDecimal-8 2.000 ± ∞ ¹ 2.000 ± ∞ ¹ ~ > (p=1.000 n=1) ² > AddBigInt-8 0.000 ± ∞ ¹ 0.000 ± ∞ ¹ ~ > (p=1.000 n=1) ² > AddBigFloat-8 1.000 ± ∞ ¹ 1.000 ± ∞ ¹ ~ > (p=1.000 n=1) ² > MulFixed-8 0.000 ± ∞ ¹ 0.000 ± ∞ ¹ ~ > (p=1.000 n=1) ² > MulDecimal-8 2.000 ± ∞ ¹ 2.000 ± ∞ ¹ ~ > (p=1.000 n=1) ² > MulBigInt-8 0.000 ± ∞ ¹ 0.000 ± ∞ ¹ ~ > (p=1.000 n=1) ² > MulBigFloat-8 0.000 ± ∞ ¹ 0.000 ± ∞ ¹ ~ > (p=1.000 n=1) ² > DivFixed-8 0.000 ± ∞ ¹ 0.000 ± ∞ ¹ ~ > (p=1.000 n=1) ² > DivDecimal-8 12.00 ± ∞ ¹ 12.00 ± ∞ ¹ ~ > (p=1.000 n=1) ² > DivBigInt-8 1.000 ± ∞ ¹ 1.000 ± ∞ ¹ ~ > (p=1.000 n=1) ² > DivBigFloat-8 2.000 ± ∞ ¹ 2.000 ± ∞ ¹ ~ > (p=1.000 n=1) ² > CmpFixed-8 0.000 ± ∞ ¹ 0.000 ± ∞ ¹ ~ > (p=1.000 n=1) ² > CmpDecimal-8 0.000 ± ∞ ¹ 0.000 ± ∞ ¹ ~ > (p=1.000 n=1) ² > CmpBigInt-8 0.000 ± ∞ ¹ 0.000 ± ∞ ¹ ~ > (p=1.000 n=1) ² > CmpBigFloat-8 0.000 ± ∞ ¹ 0.000 ± ∞ ¹ ~ > (p=1.000 n=1) ² > StringFixed-8 1.000 ± ∞ ¹ 1.000 ± ∞ ¹ ~ > (p=1.000 n=1) ² > StringNFixed-8 1.000 ± ∞ ¹ 1.000 ± ∞ ¹ ~ > (p=1.000 n=1) ² > StringDecimal-8 4.000 ± ∞ ¹ 4.000 ± ∞ ¹ ~ > (p=1.000 n=1) ² > StringBigInt-8 1.000 ± ∞ ¹ 1.000 ± ∞ ¹ ~ > (p=1.000 n=1) ² > StringBigFloat-8 7.000 ± ∞ ¹ 7.000 ± ∞ ¹ ~ > (p=1.000 n=1) ² > WriteTo-8 0.000 ± ∞ ¹ 0.000 ± ∞ ¹ ~ > (p=1.000 n=1) ² > geomean ³ +0.00% > ³ > > > On Apr 24, 2024, at 6:20 PM, Steven Hartland <stevenmhartl...@gmail.com> > wrote: > > What’s it look like when your run it through > https://pkg.go.dev/golang.org/x/perf/cmd/benchstat which will provide a > nice side by side comparison? > > On Wed, 24 Apr 2024 at 19:26, 'Robert Engels' via golang-nuts < > golang-nuts@googlegroups.com> wrote: > >> I have a fairly stable project github.com/robaho/fixed which is almost >> 100% cpu bound. It doesn’t change so it makes a great way to compare the >> performance of different Go versions using the same hardware. I took the >> time to re-run the tests today. >> >> Using 1.21.17: >> >> BenchmarkAddFixed-8 2000000000 0.59 ns/op >> 0 B/op 0 allocs/op >> BenchmarkAddDecimal-8 5000000 243 ns/op >> 176 B/op 8 allocs/op >> BenchmarkAddBigInt-8 100000000 14.3 ns/op >> 0 B/op 0 allocs/op >> BenchmarkAddBigFloat-8 20000000 78.8 ns/op >> 48 B/op 1 allocs/op >> BenchmarkMulFixed-8 300000000 4.88 ns/op >> 0 B/op 0 allocs/op >> BenchmarkMulDecimal-8 20000000 72.0 ns/op >> 80 B/op 2 allocs/op >> BenchmarkMulBigInt-8 100000000 17.1 ns/op >> 0 B/op 0 allocs/op >> BenchmarkMulBigFloat-8 30000000 35.5 ns/op >> 0 B/op 0 allocs/op >> BenchmarkDivFixed-8 300000000 4.71 ns/op >> 0 B/op 0 allocs/op >> BenchmarkDivDecimal-8 2000000 779 ns/op >> 568 B/op 21 allocs/op >> BenchmarkDivBigInt-8 30000000 46.1 ns/op >> 8 B/op 1 allocs/op >> BenchmarkDivBigFloat-8 20000000 108 ns/op >> 24 B/op 2 allocs/op >> BenchmarkCmpFixed-8 2000000000 0.38 ns/op >> 0 B/op 0 allocs/op >> BenchmarkCmpDecimal-8 200000000 8.05 ns/op >> 0 B/op 0 allocs/op >> BenchmarkCmpBigInt-8 300000000 5.87 ns/op >> 0 B/op 0 allocs/op >> BenchmarkCmpBigFloat-8 300000000 5.46 ns/op >> 0 B/op 0 allocs/op >> BenchmarkStringFixed-8 20000000 57.4 ns/op >> 32 B/op 1 allocs/op >> BenchmarkStringNFixed-8 20000000 55.6 ns/op >> 32 B/op 1 allocs/op >> BenchmarkStringDecimal-8 10000000 218 ns/op >> 64 B/op 5 allocs/op >> BenchmarkStringBigInt-8 10000000 122 ns/op >> 24 B/op 2 allocs/op >> BenchmarkStringBigFloat-8 3000000 416 ns/op >> 192 B/op 8 allocs/op >> BenchmarkWriteTo-8 30000000 45.8 ns/op >> 18 B/op 0 allocs/op >> >> >> and version 1.21.5: >> >> BenchmarkAddFixed-8 1000000000 0.9735 ns/op >> 0 B/op 0 allocs/op >> BenchmarkAddDecimal-8 14311995 69.99 ns/op >> 80 B/op 2 allocs/op >> BenchmarkAddBigInt-8 100000000 13.42 ns/op >> 0 B/op 0 allocs/op >> BenchmarkAddBigFloat-8 17506702 63.84 ns/op >> 48 B/op 1 allocs/op >> BenchmarkMulFixed-8 313983104 3.732 ns/op >> 0 B/op 0 allocs/op >> BenchmarkMulDecimal-8 18046520 66.59 ns/op >> 80 B/op 2 allocs/op >> BenchmarkMulBigInt-8 100000000 10.79 ns/op >> 0 B/op 0 allocs/op >> BenchmarkMulBigFloat-8 49186024 24.30 ns/op >> 0 B/op 0 allocs/op >> BenchmarkDivFixed-8 306888069 3.721 ns/op >> 0 B/op 0 allocs/op >> BenchmarkDivDecimal-8 2510688 462.4 ns/op >> 384 B/op 12 allocs/op >> BenchmarkDivBigInt-8 33993822 37.02 ns/op >> 8 B/op 1 allocs/op >> BenchmarkDivBigFloat-8 9415330 111.5 ns/op >> 24 B/op 2 allocs/op >> BenchmarkCmpFixed-8 1000000000 0.2548 ns/op >> 0 B/op 0 allocs/op >> BenchmarkCmpDecimal-8 168714549 7.086 ns/op >> 0 B/op 0 allocs/op >> BenchmarkCmpBigInt-8 234895634 4.952 ns/op >> 0 B/op 0 allocs/op >> BenchmarkCmpBigFloat-8 260814464 4.503 ns/op >> 0 B/op 0 allocs/op >> BenchmarkStringFixed-8 23725470 50.57 ns/op >> 24 B/op 1 allocs/op >> BenchmarkStringNFixed-8 23666628 50.67 ns/op >> 24 B/op 1 allocs/op >> BenchmarkStringDecimal-8 5665790 200.1 ns/op >> 56 B/op 4 allocs/op >> BenchmarkStringBigInt-8 10596398 100.2 ns/op >> 16 B/op 1 allocs/op >> BenchmarkStringBigFloat-8 2922332 391.2 ns/op >> 176 B/op 7 allocs/op >> BenchmarkWriteTo-8 45734523 31.53 ns/op >> 23 B/op 0 allocs/op >> >> which is pretty impressive across the board. >> >> Only 2 tests show any degradation and most show significant improvement. >> >> On the two that degrade, AddFixed is a fairly trivial add of 2 longs, so >> that is surprising. Strangely, WriteTo shows a different number of B/op on >> different runs (all of the other tests are stable). >> >> -- >> You received this message because you are subscribed to the Google Groups >> "golang-nuts" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to golang-nuts+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/golang-nuts/A6412E4C-03BD-477B-A725-06510DDD618A%40me.com >> <https://groups.google.com/d/msgid/golang-nuts/A6412E4C-03BD-477B-A725-06510DDD618A%40me.com?utm_medium=email&utm_source=footer> >> . >> > > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAA38peaa510kwqPuZUubJztLbNAR-jsBKMgF3svBb4zEJoZCrg%40mail.gmail.com.
