Re: Loading a huge graph

2012-04-13 Thread László Török
Thanks, I know about transients, but I'm already using mutable arrays for speed :-) On Apr 13, 2012 8:01 PM, "Robert Marianski" wrote: > > If the jvm does have enough memory, you may want to try building up the > map using a transient. > > And not sure if this is faster, (maybe it's slower), but y

Re: Loading a huge graph

2012-04-13 Thread Robert Marianski
If the jvm does have enough memory, you may want to try building up the map using a transient. And not sure if this is faster, (maybe it's slower), but you can spell the function you pass to reduce more succinctly: (fn [G [v1 v2]] (update-in G [v1] (fnil conj []) v2)) Robert On Thu, Apr 12, 201

Re: Loading a huge graph

2012-04-13 Thread László Török
Excellent! Thank you, that did the trick! 2012/4/13 bOR_ > I have this option in my project.clj file, which does the trick if you are > developing from emacs+swank+clojure-jack-in, and using large networks > > :jvm-opts ["-Xmx4000m"] > > And yes, one of the things to do when working with the j

Re: Loading a huge graph

2012-04-13 Thread bOR_
I have this option in my project.clj file, which does the trick if you are developing from emacs+swank+clojure-jack-in, and using large networks :jvm-opts ["-Xmx4000m"] And yes, one of the things to do when working with the jvm is learning how to use jconsole or visualvm to see why your progr

Re: Loading a huge graph

2012-04-13 Thread László Török
Thanks, having a C++ background, there is definitely a lot to learn about the JVM. I'm wrote a script in Python that ran in about 10-12s and used up to 320MB of memory. I'm running a clojure repl from Emacs via jack-in. What's the best way to adjust the heap size? swank-clojure and clojure-mode

Re: Loading a huge graph

2012-04-12 Thread Alex Robbins
Yeah, sounds like it could definitely be a memory issue. This is one part where the JVM works a lot differently than I expected coming from a python background. Everybody may already know this, but the JVM only takes 64mb for the heap by default. You'll get an out of memory error if your program u

Re: Loading a huge graph

2012-04-12 Thread David Nolen
How much memory do Python & Go consume when you do this? Are you giving the JVM enough memory? On Thu, Apr 12, 2012 at 6:17 PM, László Török wrote: > Hi, > > I'm trying figure out how to load a huge file that contains some 800k pair > of integers (two integers per line) which represent edges of