It may be time to return to a more radical idea, one that is more technically feasible now (in release 5.2) than it was in the past.
Get rid of page pooling. I'm not saying to re-create each page for each request; I don't think that would scale. However, it may be possible to change Tapestry so that you need only ONE instance of a given page (per locale). All transient (per-request) state for the page could be accessed indirectly, stored in a per-thread object (or inside the Request, as attributes). I can envision some small amount of extra overhead per request (due to extra levels of indirection). However, you could then process any number of threads against the same, single page object with no extra memory consumption: for example, no more duplicated Binding objects, and no more extra Maps to hold them all. Some parts of the Tapestry's internal implementation (InternalComponentResourcesImpl and ComponentPageElementImpl) would need some changing, i.e., a bit more synchronization around some critical sections. It's an exciting idea ... when will I have time to investigate it? On Tue, Jun 29, 2010 at 10:02 AM, Howard Lewis Ship <hls...@gmail.com> wrote: > Just to check ... which version of Tapestry? > > On Tue, Jun 29, 2010 at 8:48 AM, Blower, Andy > <andy.blo...@proquest.co.uk> wrote: >> I doubt anyone remembers this thread except me, but we're still having >> problems that I could do with help on. >> >> So, two months later, we have managed to improve things significantly, but >> still have very large pages and we're having a lot of trouble with the page >> pool size. What we see when we push the app too hard is that the heap fills >> until the gc thread is permanently running, and the app is essentially dead >> and unresponsive. I've been testing with 2g & 4g heaps up to now, with 100ms >> soft-wait & 200 hard limit because we need a lot of instances of some pages, >> and were running out with a 100 limit. I know I need to understand and do >> more with these values, but it's worrying because our tests are only >> requesting English pages at present and we'll have 17 languages for launch! >> So I'm rather concerned to say the least... >> >> I have a theory about what is happening that I'd like to run by the list and >> see what others think. I'm having some trouble figuring out what's happening >> at load. >> >> What I think is that when the heap gets full-ish at high load, the gc kicks >> in more and more often, slowing request processing down a little. (there's a >> fair amount of soft referenced stuff it can boot) This means that pages are >> returned to the page pool a bit later causing more page instances to be >> created which fills the heap and enters an vicious cycle with the gc trying >> to free memory and the T5 page pool trying to create more page instances. >> The reason I suspect this is because heap dumps show about 40 PageImpl's >> that have not completed loading, and many more that have but are much >> smaller in size than others for the same page. (e.g. a Results PageImpl >> retaining 28k when a normal one retains 7.5Mb - very big I know, but I've >> cut it down from 14Mb) >> >> Our site has 223 pages which consume 213Mb if 1 instance of each is >> instantiated. Multiply this by 200 (hard limit) and then by 17 languages and >> worst case we'll need a heap size of 725Gb which is a little ridiculous! I >> will do more work on reducing the size of the pages but I've already done >> the easy stuff. >> >> Here's a list of the heap dominators using Eclipse MAT, >> http://lh6.ggpht.com/_YwJn8TJTqJU/TCoNT11ouZI/AAAAAAAAACI/G8eTGsth4zM/HeapDominators.png >> >> Anyone think I'm on the right track, or barking up the wrong tree completely? >> >> Thanks, >> >> Andy >> >>> -----Original Message----- >>> From: Blower, Andy [mailto:andy.blo...@proquest.co.uk] >>> Sent: 20 April 2010 17:42 >>> To: 'Tapestry users' >>> Subject: RE: Tapestry using 1.3Gb of heap space after capacity testing >>> >>> Thanks for the link. As I said before, I don't think that we've fallen >>> prey to that exactly. Doing some local testing and looking at one of >>> our largest pages which displays search results, it has 76 component >>> objects, each of those will have components nested within (pretty >>> heavily in some cases) but nothing which has a bunch of components >>> where only one is used and the others are redundant. (what I think is >>> this uber-component thing) >>> >>> It looks like explicit repeated use of a component, rather than using a >>> single component in a loop, creates a lot more component objects in the >>> heap. I've never seen anything warning against this, and we do usually >>> use loops but it's not always the best solution. >>> >>> Again, looking at our results page, a single instance seems to get >>> 11,283 ComponentPageElementImpl instances created for it. I'm finding >>> it hard to find a good view of the tree to figure out what component(s) >>> are the main culprits for this huge number of instances. >>> >>> Thanks for any help or guidance you can give me. >>> >>> Andy >>> >>> > -----Original Message----- >>> > From: Thiago H. de Paula Figueiredo [mailto:thiag...@gmail.com] >>> > Sent: 20 April 2010 15:56 >>> > To: Tapestry users >>> > Subject: Re: Tapestry using 1.3Gb of heap space after capacity >>> testing >>> > >>> > On Tue, 20 Apr 2010 11:41:25 -0300, Blower, Andy >>> > <andy.blo...@proquest.co.uk> wrote: >>> > >>> > > Uber-component anti-pattern, not to my knowledge, but we have a lot >>> > of >>> > > pages and components now. And a lot of logic in them. The project's >>> > > large enough I can't be sure. Do you have a handy link to Howard's >>> > > description so I can get my team to check? >>> > >>> > It's here: >>> > http://old.nabble.com/-T5.0.18--Out-of-Memory-Error---Potential-Leak- >>> > (doesn't-reduce-after-forced-GC)-to25403474s302.html#a25497441 >>> > That thread is interesting, by the way. The problem is more about how >>> > components and pages are written than the number of them. >>> > What Tapestry version are you using? >>> > >>> > > Page pool config is default, I don't know enough to fiddle yet. >>> Also, >>> > > the test is on a single stand alone server with no clustering, we >>> are >>> > > using Tomcat clustering for production. >>> > >>> > You should take a look a them, as they can affect the memory >>> > consumption >>> > directly. >>> > >>> > > I did fire the GC manually before getting concerned BTW. A lot of >>> > heap >>> > > seems to be taken up with 100-300k descriptions on the >>> > > InternalClassTransformationImpl class. An example of one is >>> attached >>> > > below - seems wrong to me and more like the debug that can be >>> > requested >>> > > through log4j rather than a description. >>> > >>> > They look really bigger than they should. 300k is a lot. I have one >>> > application with a reasonable number of pages and components and it >>> > runs >>> > inside a 512 MB slide at SliceHost without using the whole memory. >>> > >>> > -- >>> > Thiago H. de Paula Figueiredo >>> > Independent Java, Apache Tapestry 5 and Hibernate consultant, >>> > developer, >>> > and instructor >>> > Owner, software architect and developer, Ars Machina Tecnologia da >>> > Informação Ltda. >>> > http://www.arsmachina.com.br >>> > >>> > --------------------------------------------------------------------- >>> > To unsubscribe, e-mail: users-unsubscr...@tapestry.apache.org >>> > For additional commands, e-mail: users-h...@tapestry.apache.org >>> > >> >> > > > > -- > Howard M. Lewis Ship > > Creator of Apache Tapestry > > The source for Tapestry training, mentoring and support. Contact me to > learn how I can get you up and productive in Tapestry fast! > > (971) 678-5210 > http://howardlewisship.com > -- Howard M. Lewis Ship Creator of Apache Tapestry The source for Tapestry training, mentoring and support. Contact me to learn how I can get you up and productive in Tapestry fast! (971) 678-5210 http://howardlewisship.com --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tapestry.apache.org For additional commands, e-mail: users-h...@tapestry.apache.org
