I'm not sure that the sort package is your your problem, here. First, the actual size (in bytes) of the data set matters, but is not *that* important. The number of elements is much more important. And I wrote a naive program to measure how long it takes to sort different in-memory data sets and get
48MB of uint64 (~6M elements): ~1s 48MB of uint8 (~50M elements): ~4s 2GB of uint64 (~270M elements): ~1m13s 2GB of uint8 (~2B elements): ~2m54s Neither of these seem particularly bad to me. For example, in this post <http://iyedb.github.io/algorithms/2014/05/09/sort-10-million.html> a C++ implementation of quick sort sorts 10M integers in 2.5s; my time is ~4s for 50M elements. Yes, the hardware is probably very different, but it doesn't suggest a huge problem with the speed of the sort package to me. I would assume the issue here is more that your comparison function is slow, that the way you are reading the values is inefficient (e.g. do you leave them on-disk?) or that your expectations about the time it takes to sort hundreds of millions of elements are unrealistic. On Wed, Nov 29, 2017 at 3:42 PM, Subramanian Karunanithi <subub...@gmail.com > wrote: > Hi, > > Yes measuring it now shall get back, on a side note do we have any big > datafile parsing package part of go or some library? > The basic sort.Sort() what I have is taking 44sec for 48MB of data, I have > to parse 2G of data. > > > Regards, > Subu K > > > On Wed, Nov 29, 2017 at 8:03 PM, Jan Mercl <0xj...@gmail.com> wrote: > >> On Wed, Nov 29, 2017 at 3:19 PM Subramanian K <subub...@gmail.com> wrote: >> >> > To run 2GB of data it takes really long time, I am trying to split >> these to buckets and make it run concurrently, finally need to collate >> results of all these small sorted buckets. >> >> Have you measured and detected where the bottleneck is? If it's in any of >> the sort.Interface methods, concurrency might not be the best approach to >> consider. >> >> >> >> -- >> >> -j >> > > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
