I have this option in my project.clj file, which does the trick if you are
developing from emacs+swank+clojure-jack-in, and using large networks
:jvm-opts ["-Xmx4000m"]
And yes, one of the things to do when working with the jvm is learning how
to use jconsole or visualvm to see why your program is slow :-), and know
how to adapt the jvm options. If I remember correctly, java 1.7 has an
option for a more flexible heap size, which might be a nice default.
On Friday, 13 April 2012 10:32:24 UTC+2, Las wrote:
>
> Thanks,
>
> having a C++ background, there is definitely a lot to learn about the JVM.
>
> I'm wrote a script in Python that ran in about 10-12s and used up to 320MB
> of memory.
>
> I'm running a clojure repl from Emacs via jack-in.
>
> What's the best way to adjust the heap size? swank-clojure and
> clojure-mode don't say.
>
> If I spin up the repl via lein swank, I can use the LEIN_JVM_OPTS, however
> it affects only one of the JVMs started by lein swank, the one with jline.
>
> Any advice? :)
>
>
> 2012/4/13 Alex Robbins <alexander.j.robb...@gmail.com>
>
>> Yeah, sounds like it could definitely be a memory issue. This is one
>> part where the JVM works a lot differently than I expected coming from
>> a python background.
>>
>> Everybody may already know this, but the JVM only takes 64mb for the
>> heap by default. You'll get an out of memory error if your program
>> uses more than that. In contrast, python just takes all the memory it
>> needs. As your program gets closer to the JVM memory limit it'll spend
>> more and more time doing garbage collection, with less and less real
>> work getting done. You can pass an -Xmx flag to give java access to
>> more memory, which many (most?) programs do.
>>
>>
>> On Thu, Apr 12, 2012 at 5:22 PM, David Nolen <dnolen.li...@gmail.com>
>> wrote:
>> > How much memory do Python & Go consume when you do this? Are you giving
>> the
>> > JVM enough memory?
>> >
>> >
>> > On Thu, Apr 12, 2012 at 6:17 PM, László Török <ltoro...@gmail.com>
>> wrote:
>> >>
>> >> Hi,
>> >>
>> >> I'm trying figure out how to load a huge file that contains some 800k
>> pair
>> >> of integers (two integers per line) which represent edges of a directed
>> >> graph.
>> >>
>> >> So if the ith line has x and y, it means that there is an edge between
>> x
>> >> and y vertex in the graph.
>> >>
>> >> The goal is to load it in an array of arrays representation, where the
>> kth
>> >> array contains all the nodes, where there is a directed edge from the
>> kth
>> >> node to those nodes.
>> >>
>> >> I've attempted multiple variants of with-open reader and line-seq etc.
>> but
>> >> almost always ended up with OutMemoryException or sg VERY slow.
>> >>
>> >> My latest attempt that also does not work on the large input:
>> >>
>> >> (defn load-graph [input-f]
>> >> (with-open [rdr (io/reader input-f)]
>> >> (->> (line-seq rdr)
>> >> (map (fn [row]
>> >> (let [[v1str v2str] (str/split row #"\s")]
>> >> [ (Integer/parseInt v1str) (Integer/parseInt v2str)
>> ]))
>> >> )
>> >> (reduce (fn [G [v1 v2]]
>> >> (if-let [vs (get G v1)]
>> >> (update-in G [v1] #(conj % v2))
>> >> (assoc G v1 [v2]))) { } ))))
>> >>
>> >> I'm getting a bit frustrated as there are Python, Go implementations
>> that
>> >> load the graph in less the 5 seconds.
>> >>
>> >> What am I doing wrong?
>> >>
>> >> Thanks
>> >>
>> >> --
>> >> László Török
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> >> Groups "Clojure" group.
>> >> To post to this group, send email to clojure@googlegroups.com
>> >> Note that posts from new members are moderated - please be patient with
>> >> your first post.
>> >> To unsubscribe from this group, send email to
>> >> clojure+unsubscr...@googlegroups.com
>> >> For more options, visit this group at
>> >> http://groups.google.com/group/clojure?hl=en
>> >
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups "Clojure" group.
>> > To post to this group, send email to clojure@googlegroups.com
>> > Note that posts from new members are moderated - please be patient with
>> your
>> > first post.
>> > To unsubscribe from this group, send email to
>> > clojure+unsubscr...@googlegroups.com
>> > For more options, visit this group at
>> > http://groups.google.com/group/clojure?hl=en
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>>
>
>
>
> --
> László Török
>
>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
