BTW, depending on what the problem is we are actually talking about, even the numbers from the initial post might be fine. With those numbers, sorting the 2GB should take roughly half an hour. A significant amount of time, yes, but if you only need to do it once (or once a day or week or whatever) it might not be worth spending any significant amount of time making it go faster. Seems counter-intuitive, I know, but CPU time is very cheap and human time is very expensive :)
On Wed, Nov 29, 2017 at 8:08 PM, Michael Jones <michael.jo...@gmail.com> wrote: > here's a little more detail on system / direct-type / and parallel > direct-type sorting: > > celeste:double mtj$ go test -run=NONE -bench Sort > goos: darwin > goarch: amd64 > pkg: double > BenchmarkSSort250-8 100000 11285 ns/op > BenchmarkSSort500-8 50000 31001 ns/op > BenchmarkSSort1000-8 20000 93048 ns/op > BenchmarkSSort2000-8 10000 218677 ns/op > BenchmarkSSort4000-8 3000 473944 ns/op > BenchmarkSSort8000-8 2000 1033446 ns/op > BenchmarkSSort16000-8 1000 2208783 ns/op > BenchmarkSSort32000-8 300 4599316 ns/op > BenchmarkSSort64000-8 200 9571278 ns/op > BenchmarkSSort128000-8 100 20524194 ns/op > BenchmarkSSort256000-8 30 42263166 ns/op > BenchmarkSSort512000-8 20 89330628 ns/op > BenchmarkSSort1024000-8 10 190409724 ns/op > BenchmarkQSort250-8 500000 2978 ns/op > BenchmarkQSort500-8 200000 10271 ns/op > BenchmarkQSort1000-8 30000 36474 ns/op > BenchmarkQSort2000-8 20000 93520 ns/op > BenchmarkQSort4000-8 10000 217781 ns/op > BenchmarkQSort8000-8 3000 470050 ns/op > BenchmarkQSort16000-8 2000 1020604 ns/op > BenchmarkQSort32000-8 1000 2184270 ns/op > BenchmarkQSort64000-8 300 4750204 ns/op > BenchmarkQSort128000-8 200 9841153 ns/op > BenchmarkQSort256000-8 100 21644805 ns/op > BenchmarkQSort512000-8 30 46220023 ns/op > BenchmarkQSort1024000-8 20 89012565 ns/op > BenchmarkPSort250-8 500000 3009 ns/op > BenchmarkPSort500-8 200000 10284 ns/op > BenchmarkPSort1000-8 50000 36435 ns/op > BenchmarkPSort2000-8 30000 48991 ns/op > BenchmarkPSort4000-8 20000 87163 ns/op > BenchmarkPSort8000-8 10000 156933 ns/op > BenchmarkPSort16000-8 5000 289609 ns/op > BenchmarkPSort32000-8 2000 616701 ns/op > BenchmarkPSort64000-8 1000 1183778 ns/op > BenchmarkPSort128000-8 500 2602444 ns/op > BenchmarkPSort256000-8 300 5492333 ns/op > BenchmarkPSort512000-8 100 10974195 ns/op > BenchmarkPSort1024000-8 50 22786689 ns/op > PASS > ok double 104.041s > > ...which has parallel sorting an additional 2x faster on average. More > CPUs helps, but not anywhere close to scalably. The real issue here is > cache effects, memory bandwidth, and bounds checking. > > On Wed, Nov 29, 2017 at 10:47 AM, Michael Jones <michael.jo...@gmail.com> > wrote: > >> Using a personal sort to avoid the standard sort's abstraction... >> >> // 50331648 bytes, 6291456 8-byte elements, 0.687317 <06873%2017> >> seconds (48MB of uint64) >> // 50331648 bytes, 50331648 1-byte elements, 1.605258 seconds (48MB >> of uint8) >> // 2147483648 <(214)%20748-3648> bytes, 268435456 8-byte elements, >> 35.262593 seconds (2GB of uint64) >> // 2147483648 <(214)%20748-3648> bytes, 2147483648 <(214)%20748-3648> >> 1-byte elements, 68.793937 seconds (2GB of uint8) >> >> ...is good for another 2-3x. Running in parallel mode would get you >> another 3x-5x, but i expect 5x this rate is as good as you will get. (at >> least with battery, laptop, pyjamas, and reclining on a sofa :-) >> -- >> Michael T. Jones >> michael.jo...@gmail.com >> > > > > -- > Michael T. Jones > michael.jo...@gmail.com > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
