Re: Space usage of lazy seqs

2009-12-03 Thread David Brown
On Wed, Dec 02, 2009 at 08:18:33PM -0800, Dave M wrote: >On Dec 2, 9:09 pm, David Brown wrote: >... >> If you're running JDK 6, you can run the virtualvm, or jconsole to get >> a better handle on the memory usage, and even dig into what it might >> used for. > >Google does not return useful refer

Re: Space usage of lazy seqs

2009-12-03 Thread Aaron Cohen
On Thu, Dec 3, 2009 at 9:13 AM, Johann Hibschman wrote: > On Dec 2, 9:59 pm, Johann Hibschman wrote: >> On Dec 2, 9:09 pm, David Brown wrote: >> >> > You can tune the max with -Xmx1G for example, to limit it to one GB. >> >> That's a good idea; then I'll know for sure if it's keeping a handle >>

Re: Space usage of lazy seqs

2009-12-03 Thread Johann Hibschman
On Dec 2, 9:59 pm, Johann Hibschman wrote: > On Dec 2, 9:09 pm, David Brown wrote: > > > You can tune the max with -Xmx1G for example, to limit it to one GB. > > That's a good idea; then I'll know for sure if it's keeping a handle > to the entire file. Ok, that's a relief. First of all, -Xmx1G

Re: Space usage of lazy seqs

2009-12-02 Thread Dave M
On Dec 2, 9:09 pm, David Brown wrote: ... > If you're running JDK 6, you can run the virtualvm, or jconsole to get > a better handle on the memory usage, and even dig into what it might > used for. Google does not return useful references to a tool called virtualvm; perhaps you mean VisualVM (j

Re: Space usage of lazy seqs

2009-12-02 Thread Johann Hibschman
On Dec 2, 9:09 pm, David Brown wrote: > How much memory do you have on your machine.  A recent Sun JVM on a > machine with a bunch of memory will consider it to be a "server" > machine.  It will set the heap max to 1/4 of total physical memory > (which suggests you might have 16GB of RAM). I have

Re: Space usage of lazy seqs

2009-12-02 Thread David Brown
On Wed, Dec 02, 2009 at 02:01:36PM -0800, Johann Hibschman wrote: >There is a qualitative difference between the runs, though. I can run >test-split-3 five times in a row, all with similar times, without >having the java process size get bigger than 0.6 GB. When I run any of >the others, the size

Re: Space usage of lazy seqs

2009-12-02 Thread Johann Hibschman
On Dec 2, 2:50 pm, ataggart wrote: > After reading the code, I'm inclined to not trust those numbers.  Note > that the time metrics for test-split* are all in the same ballpark, > creating the same number of superfluous, intermediate String > instances, but the memory numbers you list are wildly

Re: Space usage of lazy seqs

2009-12-02 Thread ataggart
On Dec 2, 10:50 am, Johann Hibschman wrote: > I don't understand Clojure's space requirements when processing lazy > sequences. Are there some rules-of-thumb that I could use to better > predict what will use a lot of space? > > I have a 5.5 GB pipe-delimited data file, containing mostly floats

Re: Space usage of lazy seqs

2009-12-02 Thread ataggart
On Dec 2, 10:50 am, Johann Hibschman wrote: > I don't understand Clojure's space requirements when processing lazy > sequences. Are there some rules-of-thumb that I could use to better > predict what will use a lot of space? > > I have a 5.5 GB pipe-delimited data file, containing mostly floats

Space usage of lazy seqs

2009-12-02 Thread Johann Hibschman
I don't understand Clojure's space requirements when processing lazy sequences. Are there some rules-of-thumb that I could use to better predict what will use a lot of space? I have a 5.5 GB pipe-delimited data file, containing mostly floats (14 M rows of 40 cols). I'd like to stream over that fil