Thanks for your response, here are my answers to your questions... In our test environment, we have a 2 server cluster, but our front-end Apache server is only hitting one of them. Regardless of that fact, all sessions created on either server get replicated to the other. So as they are created on the single test path, they get replicated to the second tomcat server. With only 2 cluster members, this isn't really a big deal (except for the load generators being able to create sessions faster than Tomcat can get rid of them). However, in our production environment we have 4 cluster members, so it only takes 1/4 of the load on each server to max out all 4 servers in terms of session creation speed. Keep in mind that during this situation, the _only_ problem we're seeing is that the sessions eventually fill up the heap. Performance remains good until the heap fills up and we start thrashing on Full GCs. (Clearly, more memory will help - by extending the duration we can sustain of peak loads, but a better solution will be to stop creating sessions for stuff that doesn't need it.)
Your point is well taken about not creating the short-duration sessions, but alas, Tapestry is the chosen framework - and it uses the session as a mechanism to pass (more-or-less) "global" values between components of the page creation process. So ripping out that capability in Tapestry would require a massive change to our infrastructure. Your notes on the clustering (changes to DeltaManager) are right on target. So we adjusted when the DeltaManager "creates" the session until the end of processing, so that we know the timeout before we attempt to replicate it to other cluster members. That way, we already know the actual timeout duration before we decide whether to replicate. All of this would be unnecessary if there was some sort of "session cleanup" process that could dump expired sessions "en masse" every so often. Near as we can tell, each session has to be individually expired, which is why it lags so far behind the creation process. -- Robin D. Wilson Director of Web Development KingsIsle Entertainment, Inc. CELL: 512-426-3929 DESK: 512-623-5913 www.KingsIsle.com -----Original Message----- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Tuesday, January 12, 2010 3:06 PM To: Tomcat Users List Subject: Re: More on Tomcat Sessions - limiting cluster session replication to sessions that will last longer than 'n' duration -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Robin, On 1/12/2010 1:42 PM, Robin Wilson wrote: > Earlier this week I posted a question about how to prevent sessions > from being created in our Tapestry pages, and/or how to get Tomcat to > get rid of a bunch of '1-second' sessions we're creating during a > load test because the sessions eventually fill up the heap. (They are > being created faster than Tomcat can clean them out - even though > they expire faster than we create them.) Well, they are eligible for expiration faster than they are being created. That session expiration thread can't be running /all/ the time :) > So, my lead developer thinks he has found a way to alleviate our > problems (at least for our production Tomcat 6.0.20 cluster of 4 > servers). We will not replicate sessions to other cluster members > unless they have a duration longer than a 'threshold' we set. Aah... I didn't realize that you were experiencing a session replication "storm" in your test setup. I thought you were only using a single app server for load testing. Did I misunderstand? > We are > altering the DeltaManager to make this change, so that simply > creating a session doesn't automatically guarantee that it is > replicated to other nodes in the cluster. The session duration will > also have to be greater than a "sessionDurationMinThreshold" value we > will set in the 'server.xml' file for the new DeltaManager. The idea > is that sessions created that have very short durations are really > 'transient' anyway, so there is no need to pass them off to the other > members of the cluster. One could argue that sessions limited to that length of time are not necessary at all. Why bother creating them in the first place? > The question I have, is there anything we should watch out for in > making this adjustment to the DeltaManager? We will test this pretty > heavily before we deploy it to our production environment, but I'm > worried about things we should be looking for in that testing (other > than just validating that our "useful" session data can be available > across multiple cluster members). Give some of the cluster gurus on the list a chance to read your question. Feel free to start your implementation: I'm guessing it's a pretty simple change to simply ignore sessions with low timeouts. Depending on how to set up your sessions, you may not be able to prevent the replication (or replication of "useful" sessions may not occur). I've never looked at the code for the DeltaManager, but I can imagine a scenario like this: 0. Your web.xml configures the default session timeout to be 1 second 1. Upon session creation, DeltaManager is notified that a session has been created and will need to be replicated 2. DeltaManager sees that the "sessionDurationMinThreshold" is only 5s, and so it decides not to replicate that session throughout the cluster 3. Your code (still during the request that created the session) determines that this will be a useful session and changes the timeout, say, to 30 minutes Result: a useful session is never replicated Conversely, the opposite situation could occur if your default timeout is 30 minutes and you intentionally reduce the timeout for sessions expected to be useless: in this case, replication occurs all the time. Beware! - -chris -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAktM5B4ACgkQ9CaO5/Lv0PDj5wCfZ4CGx2XZbq+qsFx9/GVK6eCy 6HQAoJvD81ghtJl7L9KGCvKyTXN9LF0j =2nq7 -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
