On Feb 3, 2014, at 9:00 AM, Maor Yosef <maoryo...@gmail.com> wrote: > On Mon, Feb 3, 2014 at 2:10 PM, Daniel Mikusa <dmik...@gopivotal.com> wrote: > >> On Feb 3, 2014, at 3:30 AM, Maor Yosef <maoryo...@gmail.com> wrote: >> >>> Hi. >>> >>> 1. We are aware that 6.0.26 is old, but since there is a large >> operational >>> impact, we wont upgrade the tomcat until we will know its definetly an >>> issue in this specific version >> >>>> While I understand what you're saying, I disagree. If you need to sell >> the change to management, you should take a look at the security problems >> that have been fixed since 6.0.26 and >>weigh the cost of upgrading versus >> a security breach or manually applying mitigations, if that's even possible. >> >>>> http://tomcat.apache.org/security-6.html > > > I agree that an upgrade is needed, but still before we will determine > to which version to upgrade, we need to solve this issue > >> >> >>> 2. You are right, it was my mistake, it causes OOM and not stack >> overflow, >>> when we see high sessions count we get exceptions saying "unable to >> create >>> new native thread" >> >>>> This is telling you that there's not enough memory to allocate any more >> threads. >> >>>> 1.) Have you tried setting "-Xss"? This will set the thread stack size, >> i.e. how much memory each thread has available for its stack. Often times >> you don't need nearly as much as the >>default allocated by the JVM, so you >> can lower it and get more threads out of the same available memory. Maybe >> try 256k or even lower. You'll know you went too low if you see >>>> StackOverflowErrors. >> > > >>>> 2.) How many threads are being created / used? Perhaps you're creating >> too many threads? You'll probably want to do some monitoring and see how >> many the Tomcat is creating / using >>and how many your application is >> creating / using. Perhaps you've got a problem where too many threads are >> being created or where threads are being created and not properly cleaned >> up. >>Tools that could help here: jconsole / jvisualvm or thread dumps >> >> The memory configuration doesn't seems to be the issue,
I would beg to differ. It’s certainly one issue. You seem to have multiple. As I said previously, I don’t think your OOME is related to your session problem. At least it doesn’t seem to be based on the information you’ve given to date. >> as mentioned > before we see a direct correlation between the amount of opened sessions > (which can go as high as 100k+) > even for a simple application that is just running a simple jsp file, > which only prints back to the screen "OK", we still see a large number of > sessions for it (also after adding a web.xml with session expiration time = > 1) Here’s what I’d suggest. 1.) In a test environment, install Tomcat 6.0.39. It’s the latest. 2.) Deploy your “simple application”, if it’s really as simple as you say it shouldn’t be a problem to install. 3.) See if you can replicate the problem. If so then report your findings here, just include a test app your steps to reproduce. If you’re unable to reproduce the issue, then consider upgrading or compare your configuration’s to see if your current config is doing something wrong. Also, please realize that a JSP page, even one that simply prints out “OK” will create a session. This is by design and if you don’t want it to create a session you need to explicitly indicate that in your JSP. Ex: <%@ page session="false" %> This is important in scenarios where you’re doing load testing and using custom HTTP clients, because these client’s may not be handling sessions correctly and thus end up creating a new session every time they access the page. Another way to handle misbehaving clients is, what Konstantin mentioned in his earlier message about web bots, the CrawlerSessionManagerValve. > >> 3. Looking at the sessions manager, we see a lot of sessions with a >>> negative TTL - meaning they werent expired, if I manually expire them >> then >>> the sessions count decreases. >> >>>> This doesn't sound related, although it's hard to say. Might be helpful >> if you can include your configuration, minus comments. Are you going to provide your configuration? There’s only so much we can tell you without knowing how you have your environment configured. Dan >> >>> 4. Can you point me to an article on how to configure different >> background >>> thread for each container? is it configured in tomcat or should be >>> implemented in the application? >> >>>> What exactly do you mean here? You can control Tomcat's thread usage >> with an Executor [1] or directly on the connector [2]. >> >>>> Dan >> >>>> [1] - http://tomcat.apache.org/tomcat-6.0-doc/config/executor.html >>>> [2] - http://tomcat.apache.org/tomcat-6.0-doc/config/http.html >> >> >>> >>> Thanks, >>> >>> >>> On Sun, Feb 2, 2014 at 7:38 PM, Konstantin Kolinko >>> <knst.koli...@gmail.com>wrote: >>> >>>> 2014-02-02 Maor Yosef <maoryo...@gmail.com>: >>>>> Hi, >>>> >>>> 1. 6.0.26 is old. >>>> >>>>> We are facing issues where the sessions are not being expired >>>>> and eventually causing a stack overflow. >>>> >>>> 2. Non-expiring sessions may cause OOM, but they cannot cause a stack >>>> overflow. >>>> >>>> What are your evidences? >>>> >>>>> We see that at some point the sessions get pilled up and we need to >>>>> manually expire those sessions through the manager application, or >>>>> until we will restart tomcat but after a few hours / days, sessions >>>>> will start to get stuck again >>>> >>>> 3. Increasing count of session does not mean that sessions do not >> expire. >>>> >>>> Your evidence = ? >>>> >>>>> We see it in all the applications that are deployed on tomcat, >>>>> including the manager application - this rules out (in my opinion) the >>>>> possibility that its an issue with our application. >>>> >>>> 4. Sessions are expired by a background thread. If the thread is stuck >>>> somewhere (e.g. doing auto-deployment work) it will affect expiration >>>> of sessions. >>>> >>>> Your thread dump = ? >>>> >>>> By default there is one background thread that is shared by all >>>> container levels in Tomcat, but you can configure a separate one in >>>> each container (container = Context, Host or Engine). >>>> >>>> 5. Web bots that do not support cookies may generate multiple sessions. >>>> >>>> See CrawlerSessionManagerValve >>>> >>>> Best regards, >>>> Konstantin Kolinko >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>> >>>> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >> --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
