I have done something quite similar, only that the structure consumed hundred of GB's of RAM and was the main bottleneck of the application. Attempting it on the jvm (either java or clojure) resulted in a lot of wasted bits per bits and GC hangs, so I tried to use ByteBuffers (Direct, Mapped, etc). I ended up writing the structure in C and exposing it to clojure via jni, thus reducing the heap size to the bare minimum and getting ultra fast performance with the convenience of clojure.
Shlomi On Friday, June 27, 2014 7:04:34 AM UTC+3, Vjeran Marcinko wrote: > > Hi, > > I am planning to play with implementing some giant in-memory index that is > basically tree-like structure containing counters on certain tree nodes, > and can aggregate billion data points and will probably consume tens of GBs > of RAM. > > Since space (memory)-efficiency is crucial here, I was wondering how good > Clojure is for this problem, and should I better just stick to plain java, > because it is well known that clojure's persistent data structures > sacrifice space (and some speed, but that is not such a big issue here) for > sake of immutability and good development practice? > > Regards, > Vjeran > > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
