Re: another game of exploding heap, via clojure.string/split

2013-09-18 Thread Rob Lally
I've used YourKit extensively for profiling and analysis and would highly recommend it . Rob. On 17 Sep 2013, at 08:56, Andy Fingerhut wrote: > Some of the hits point at commercial tools, which you didn't mention. > > I've heard positive comments about YourKit in the past. It is commerci

Re: another game of exploding heap, via clojure.string/split

2013-09-17 Thread Andy Fingerhut
Some of the hits point at commercial tools, which you didn't mention. I've heard positive comments about YourKit in the past. It is commercial, but it looks pretty easy to get a 15-day evaluation license. I haven't used it, but it claims to have some features to aid in detecting and analyzing me

Re: another game of exploding heap, via clojure.string/split

2013-09-17 Thread Brian Craft
I did, of course, spend a lot of time with google before posting. All of the hits point to jconsole, jmap, and visualvm. None of these tools work reliably. They hang, they crash, they spit up errors, they generate useless results. You'll note in another thread this morning another developer hav

Re: another game of exploding heap, via clojure.string/split

2013-09-17 Thread Brian Craft
Awesome, thanks! I'd forgotten about the hits for MAT, because I don't use eclipse, and I had not found YourKit. On Tuesday, September 17, 2013 9:13:17 AM UTC-7, Andy Fingerhut wrote: > > I didn't notice before posting my previous message that YourKit also has a > free "for open source project u

Re: another game of exploding heap, via clojure.string/split

2013-09-17 Thread Andy Fingerhut
I didn't notice before posting my previous message that YourKit also has a free "for open source project use only" license for their tool. Click on the "Open Source" or "License Comparison" tabs on this page: http://www.yourkit.com/purchase/index.jsp On Tue, Sep 17, 2013 at 8:56 AM, Andy Fi

Re: another game of exploding heap, via clojure.string/split

2013-09-17 Thread Julien
You might want to give Eclipse MAT (http://www.eclipse.org/mat/) a try. It can be used as a standalone tool. Julien Le mardi 17 septembre 2013 12:45:41 UTC-3, Brian Craft a écrit : > > I did, of course, spend a lot of time with google before posting. All of > the hits point to jconsole, jmap, a

Re: another game of exploding heap, via clojure.string/split

2013-09-17 Thread Brian Craft
On Thursday, September 12, 2013 7:47:02 PM UTC-7, Cedric Greevey wrote: > > On Thu, Sep 12, 2013 at 3:33 PM, Andy Fingerhut > > > wrote: > >> I have just added some discussion of this on ClojureDocs.org for the >> function clojure.core/subs, and references to that discussion for several >> ot

Re: another game of exploding heap, via clojure.string/split

2013-09-17 Thread Andy Fingerhut
Another possibility: The people who know aren't reading this thread. I'd tell you if I knew, but I haven't needed to track down a problem like this for several years, and forgotten whatever tool I used at the time (it was probably jmap). Suggestion: Google search "java memory leak" and see what t

Re: another game of exploding heap, via clojure.string/split

2013-09-14 Thread Brian Craft
Very interesting. Thanks. On Saturday, September 14, 2013 5:48:57 AM UTC-7, TheBusby wrote: > > If you're working with large text files, tsv especially, you may encounter > Java's memory overhead for Strings as well. > I remember parsing a 1GB TSV file into EDN, and watching it consume 10GB > of

Re: another game of exploding heap, via clojure.string/split

2013-09-14 Thread Alan Busby
If you're working with large text files, tsv especially, you may encounter Java's memory overhead for Strings as well. I remember parsing a 1GB TSV file into EDN, and watching it consume 10GB of RAM. I'd recommend taking a look at the Iota library to see if it would be of any help to you, https://

Re: another game of exploding heap, via clojure.string/split

2013-09-12 Thread Brian Craft
Ouch. Thanks for the explanation. On Thursday, September 12, 2013 9:46:47 AM UTC-7, Andy Fingerhut wrote: > > Clojure's substr, and many other functions that return substrings of a > larger one (e.g. re-find, re-seq, etc) are based on Java's > java.lang.String/substring() method behavior. > > Be

Re: another game of exploding heap, via clojure.string/split

2013-09-12 Thread Cedric Greevey
On Thu, Sep 12, 2013 at 3:33 PM, Andy Fingerhut wrote: > I have just added some discussion of this on ClojureDocs.org for the > function clojure.core/subs, and references to that discussion for several > other Clojure functions that I am pretty sure are affected, e.g. re-find, > re-seq, re-matches

Re: another game of exploding heap, via clojure.string/split

2013-09-12 Thread Andy Fingerhut
I have just added some discussion of this on ClojureDocs.org for the function clojure.core/subs, and references to that discussion for several other Clojure functions that I am pretty sure are affected, e.g. re-find, re-seq, re-matches, clojure.string/split, replace, replace-first http://cloju

Re: another game of exploding heap, via clojure.string/split

2013-09-12 Thread Andy Fingerhut
Clojure's substr, and many other functions that return substrings of a larger one (e.g. re-find, re-seq, etc) are based on Java's java.lang.String/substring() method behavior. Before Java version 7u6 or thereabouts, this was implemented in O(1) time by creating a String object that referred to an

another game of exploding heap, via clojure.string/split

2013-09-12 Thread Brian Craft
After working around the seq + closure = death problem, I still had a severe memory leak in my code, which took many hours to find. Holding a reference to a string returned by clojure.string/split is somehow retaining a reference to the original string. In my case I needed to hold the first col