Hmmm. I did some additional experiments using the JRockit JVM on
Windows XP, and JRockit not only has equivalent memory usage for
Clojure 1.2 and 1.3 alpha1, but it can also run these programs with
significantly less memory than HotSpot. HotSpot is similar, but not
identical, in its memory requirements on all of Mac, Linux, and
Windows XP.
This particular program has lots of small things allocated (e.g. one
java.lang.String per line, and also a LazySeq and Cons created by lazy-
seq in line-seq), and while it is traversing that str is internally
using StringBuilder to create one big large string of the whole input
file's contents. I am guessing that HotSpot's default GC parameters
are not good for this memory usage pattern, and somehow JRockit gets
it right by default.
Anyone know any command line options for HotSpot that make it work
harder to compact things down into the smallest space possible, no
matter what? I am aware of several pages of documentation on
HotSpot's GC options, and I've been reading some of them, but it is a
lot to wade through.
Thanks,
Andy
On Sep 26, 2010, at 4:30 PM, Andy Fingerhut wrote:
While updating the benchmark programs I wrote for the shootout web
site for 1.3 alpha1, I came across a program that required much more
memory to complete in 1.3 than it did in 1.2. I boiled it down to a
simpler program that has similar properties.
----------------------------------------
(ns filestr2
(:gen-class))
(defn -main [& args]
(with-open [br (java.io.BufferedReader. *in*)]
(let [dna-str (apply str (line-seq br))]
(println "len=" (count dna-str))))
(. System (exit 0)))
----------------------------------------
I've checked that the function apply, str, and line-seq are
identical between those two Clojure versions, except for a few minor
things like type declarations and static declarations added in 1.3.0-
alpha1.
I'm curious if anyone can explain the cause of this significant
extra memory. As an example of what I mean, when run on an input
file with about 125 million characters, it requires 932 Mbytes with
Clojure 1.2, but 1,736 Mbytes with Clojure 1.3 alpha1 (using the -
Xmx command line option to the JVM). That was on my Mac, and I've
seen similar but not identical results on an Ubuntu system. You can
read more details if you are curious in the README file of this
tarball:
http://homepage.mac.com/jafingerhut/files/private/toomuch1.tar.bz2
It also includes a program that uses binary search to find the
smallest memory required in order for the program to succeed, to the
nearest 8 MB.
Thanks,
Andy
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
