Thanks,
having a C++ background, there is definitely a lot to learn about the JVM.
I'm wrote a script in Python that ran in about 10-12s and used up to 320MB
of memory.
I'm running a clojure repl from Emacs via jack-in.
What's the best way to adjust the heap size? swank-clojure and clojure-mode
don't say.
If I spin up the repl via lein swank, I can use the LEIN_JVM_OPTS, however
it affects only one of the JVMs started by lein swank, the one with jline.
Any advice? :)
2012/4/13 Alex Robbins <alexander.j.robb...@gmail.com>
> Yeah, sounds like it could definitely be a memory issue. This is one
> part where the JVM works a lot differently than I expected coming from
> a python background.
>
> Everybody may already know this, but the JVM only takes 64mb for the
> heap by default. You'll get an out of memory error if your program
> uses more than that. In contrast, python just takes all the memory it
> needs. As your program gets closer to the JVM memory limit it'll spend
> more and more time doing garbage collection, with less and less real
> work getting done. You can pass an -Xmx flag to give java access to
> more memory, which many (most?) programs do.
>
>
> On Thu, Apr 12, 2012 at 5:22 PM, David Nolen <dnolen.li...@gmail.com>
> wrote:
> > How much memory do Python & Go consume when you do this? Are you giving
> the
> > JVM enough memory?
> >
> >
> > On Thu, Apr 12, 2012 at 6:17 PM, László Török <ltoro...@gmail.com>
> wrote:
> >>
> >> Hi,
> >>
> >> I'm trying figure out how to load a huge file that contains some 800k
> pair
> >> of integers (two integers per line) which represent edges of a directed
> >> graph.
> >>
> >> So if the ith line has x and y, it means that there is an edge between x
> >> and y vertex in the graph.
> >>
> >> The goal is to load it in an array of arrays representation, where the
> kth
> >> array contains all the nodes, where there is a directed edge from the
> kth
> >> node to those nodes.
> >>
> >> I've attempted multiple variants of with-open reader and line-seq etc.
> but
> >> almost always ended up with OutMemoryException or sg VERY slow.
> >>
> >> My latest attempt that also does not work on the large input:
> >>
> >> (defn load-graph [input-f]
> >> (with-open [rdr (io/reader input-f)]
> >> (->> (line-seq rdr)
> >> (map (fn [row]
> >> (let [[v1str v2str] (str/split row #"\s")]
> >> [ (Integer/parseInt v1str) (Integer/parseInt v2str)
> ]))
> >> )
> >> (reduce (fn [G [v1 v2]]
> >> (if-let [vs (get G v1)]
> >> (update-in G [v1] #(conj % v2))
> >> (assoc G v1 [v2]))) { } ))))
> >>
> >> I'm getting a bit frustrated as there are Python, Go implementations
> that
> >> load the graph in less the 5 seconds.
> >>
> >> What am I doing wrong?
> >>
> >> Thanks
> >>
> >> --
> >> László Török
> >>
> >> --
> >> You received this message because you are subscribed to the Google
> >> Groups "Clojure" group.
> >> To post to this group, send email to clojure@googlegroups.com
> >> Note that posts from new members are moderated - please be patient with
> >> your first post.
> >> To unsubscribe from this group, send email to
> >> clojure+unsubscr...@googlegroups.com
> >> For more options, visit this group at
> >> http://groups.google.com/group/clojure?hl=en
> >
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Clojure" group.
> > To post to this group, send email to clojure@googlegroups.com
> > Note that posts from new members are moderated - please be patient with
> your
> > first post.
> > To unsubscribe from this group, send email to
> > clojure+unsubscr...@googlegroups.com
> > For more options, visit this group at
> > http://groups.google.com/group/clojure?hl=en
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
>
--
László Török
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
