-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Leon,
On 7/9/14, 3:16 AM, Leon Rosenberg wrote:
> On Wed, Jul 9, 2014 at 4:47 AM, Hernán Marsili
> <her...@cmsmedios.com> wrote:
>> For the past 4 years we has been working with a 'stable'
>> configuration in which we put APACHE in front of TOMCAT7
>> (previously Tomcat6) with mod_jk connector. We usually serve high
>> traffic sites with about 7000 to 10.000 concurrent users per box
>> (8gb RAM / 4 vcpu) (50.000 active users total).
>>
>> We are OK with the performance, but sometimes we notice Tomcat
>> stops responding normally while there are at least 2 full CPU
>> left to be consumed (JAVA memory is fine).
>>
>
> Hard to tell from here, but dropping performance while still
> having resources is often an indicator for synchronization issues.
> You should analyze your thread usage. You could do it with jstack
> (save multiple stacks in short slots, like every 10 seconds for 2-3
> minutes). Check what threads are in what states: - do you have any
> threads in BLOCKED state? If yes, what they are waiting for? - what
> are you RUNNABLE threads doing? Are they waiting for something,
> even not blocked - for example reading the database or reading the
> incoming request. - is your amount of TIMED_WAITING threads
> sufficient? If you have non, your thread pool is probably out of
> threads.
>
>
>
>>
>> This is the configuration we use for the connector:
>>
>> <Connector port="8009" protocol="AJP/1.3" address="127.0.0.1"
>> emptySessionPath="true" redirectPort="8443" maxThreads="1024"
>> minSpareThreads="32" enableLookups="false"
>> request.registerRequests="false" />
>>
>
> Generally removing apache httpd can increase performance. I assume
> you have a hardware loadbalancer in front of things, so httpd
> doesn't do you any good.
I would agree in general. Hernán, is there another reason to use httpd
in front?
>> I have a couple of questions: 1) should we set a particular
>> connector or let Tomcat7 decide? I understand using
>> protocol="AJP/1.3" the auto-switch kicks in. But, for non-SSL
>> high concurrency sites maybe is best to fixed on APR?
>>
>
> Warning, flame war is about to begin, but personally I always found
> that the plain old java connector is the best option for speed. And
> the simplest to use too. If you insist of having apache httpd in
> front, you may want to try mod_proxy (or was it mod_proxy_ajp?)
> instead.
All other things being equal, the BIO ("plain old java connector")
connector offers the best performance. But in this setup, there are
many simultaneous requests.
It looks like 5 httpd instances (7-10k connections x 5 ~= 50k
connections) and 5 backend Tomcat instances (please correct me if I'm
wrong). If every httpd had to talk to every Tomcat, then you will need
enough connections/threads on every Tomcat to handle all httpd
instances contacting it at once.
So, given the default prefork MPM of 250 simultaneous connections,
each Tomcat would need to be able to handle 250 x 5 = 1250
connections. With only 1024 threads, stalls can indeed happen.
If one uses the APR or NIO connector (still using AJP), reads on
incoming connections (e.g. waiting for the next request to come across
the wire) are done asynchronously, so you only need a number of
threads on your Tomcat instance to support the number of *active
requests*, not the number of connections over which active requests
could come.
The upshot is that you can get away with fewer threads on the Tomcat
side and (likely) still serve the same amount of traffic and either
eliminate or minimize stalling.
>> 2) how many THREADS can we have? can we go beyond the 1024?
>>
>
> Depends on your OS. On modern linuxes sky is the limit. However
> context switches can kill you too.
Your thread stack size + your memory limit will probably limit you
before context switches do.
> If you have very fast connections, got for a smaller amount. If you
> have keepalive and slow connections, remember that every connection
> can hold 1-2 threads without doing anything at all.
Hmm?
> But you can go up to 4096 without second thought, if you are
> really out of threads. However, if your problem are blocked threads
> or a slow DB, you will make things much much much worse.
+1
>> 3) is there any advantage on using processorCache?
I've never used that, but the documentation suggests that
processorCache = max(maxThreads, max expected simultaneous
connections) would be a good value to try. I would try other
techniques to improve performance, first. It sounds like this comes
down to a memory/GC issue, so it should incrementally improve
performance rather than fix something like stalling.
>> 4) We are not defining a CONNECTION TIMEOUT not a KEEP ALIVE. Any
>> advice on this one? The average user session is 7 minutes.
>
> If playing with CONNECTION_TIMEOUT, than better on OS level. But
> again that depends on what is really happening within your
> application and with the connections. You could monitor it with
> netstat and see if you have too many CLOSE_WAITs or something. That
> it's easier to decide what to do.
For AJP, you should set the connection timeout to infinite (the
default). If you don't do this, Tomcat will close AJP connections
between httpd and itself. If you decide to change the connection
timeout, make sure you make a similar configuration change on the
httpd side, otherwise you'll start getting errors in httpd because
Tomcat has hung up the phone and httpd is still trying to talk.
I have the same recommendation for keepAliveTimeout when using AJP
connections. HTTP keepalives are essentially irrelevant when using AJP
because, really, every request "feels" like a keepalive request over
AJP because the connection (from httpd!) is expected to remain open
forever.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJTvp6hAAoJEBzwKT+lPKRYITgP/jPc7Y8/fqB31fmtmKZh1dAZ
AQRzgjalAVbFLZn3L7l7YOvv3IjZP2VlzyHLC3T6A3p7VFyxadVppEUCHHiwIKkg
8SK/AVAh3m2pUEUBr4MWqHY1J7Ftz2Z1sbvrUOGZK0S/XHglfBf7QhYScwwHuR0K
pXFy0NWtS6OKvBbxP3zaEKA55k+M6YJ2nYqSgPbF/dJfhkdhpVnEi71S0jCnpvIr
0xWaQtRT2v9QeQvZoiNZ/kvJ6GwrPuJ1UZTe/CjGdIaX4cQShFEYAZcIkxeZAyJd
fxHZdEGvB26ywIY1DEQmT2CVDm+ROi/2U0/z+fCna9wv6F7p0EFhU2bwoUfsMAhV
l5QNhft8GxuLKut5mUKQjW9XNNJa6D+0/jO8TJvEAhwxGiiMqGDAmnz6mofdFuxv
vyi4R5UqQq0X5Hn6fQaXvvdmKax28dnUxrqcCgyNpKwKW5euJgdp1JMiWYRXy6qW
3MOT9xiqqZYXzcRdgIWHmL7ZlI1nlGk6CdOyoZd3cU7PV+5FptE1E2B+srmmlFVo
H48aS6dVt2o0ssTOn6SokHmNy10Lh529fe0NQT2ogdtw3r69hM4/WqUH/Qq9ElpM
jfc3rYUoYL69d7M2CoCPmSi0tqDv03/r31EbVuF9cOkyv11PqvrE6bjO0BGkq1Tp
7lM98GK4wxN14lLnCtad
=ECkf
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
