agree!
On Wed, Nov 29, 2017 at 11:37 AM, 'Axel Wagner' via golang-nuts <
golang-nuts@googlegroups.com> wrote:
> BTW, depending on what the problem is we are actually talking about, even
> the numbers from the initial post might be fine. With those numbers,
> sorting the 2GB should take roughly ha
BTW, depending on what the problem is we are actually talking about, even
the numbers from the initial post might be fine. With those numbers,
sorting the 2GB should take roughly half an hour. A significant amount of
time, yes, but if you only need to do it once (or once a day or week or
whatever)
here's a little more detail on system / direct-type / and parallel
direct-type sorting:
celeste:double mtj$ go test -run=NONE -bench Sort
goos: darwin
goarch: amd64
pkg: double
BenchmarkSSort250-8 10 11285 ns/op
BenchmarkSSort500-8 5 31001 ns/op
BenchmarkSSort
Using a personal sort to avoid the standard sort's abstraction...
// 50331648 bytes,6291456 8-byte elements, 0.687317 seconds (48MB of
uint64)
// 50331648 bytes, 50331648 1-byte elements, 1.605258 seconds (48MB of
uint8)
// 2147483648 bytes, 268435456 8-byte elements, 35.262593 seconds (2
I'm not sure that the sort package is your your problem, here.
First, the actual size (in bytes) of the data set matters, but is not
*that* important. The number of elements is much more important. And I
wrote a naive program to measure how long it takes to sort different
in-memory data sets and ge
Hi,
Yes measuring it now shall get back, on a side note do we have any big
datafile parsing package part of go or some library?
The basic sort.Sort() what I have is taking 44sec for 48MB of data, I have
to parse 2G of data.
Regards,
Subu K
On Wed, Nov 29, 2017 at 8:03 PM, Jan Mercl <0xj...@gmai
On Wed, Nov 29, 2017 at 3:19 PM Subramanian K wrote:
> To run 2GB of data it takes really long time, I am trying to split these
to buckets and make it run concurrently, finally need to collate results of
all these small sorted buckets.
Have you measured and detected where the bottleneck is? If i
Hi
I am using native sort provided part of the package and to process 48MB of
slice data, it takes ~45sec.
To run 2GB of data it takes really long time, I am trying to split these to
buckets and make it run concurrently, finally need to collate results of
all these small sorted buckets.
Do we
