-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Bruce,
On 3/6/14, 3:19 PM, Bruce Weertman wrote:
> Thanks for the replies. Here’s some more info.
>> Bruce,
>>
>> On 3/5/14, 1:11 PM, Bruce Weertman wrote:
>>> We have a load balancer talking with tomcat’s running on
>>> multiple backend tomcat servers. The tomcats are running web
>>> service applications. In doing testing, and as reported by
>>> customers, we occasionally see refused connections. Not real
>>> often, but enough to be a head ache.
>>>
>>> We’re not sure where the problem is coming from. Suspects
>>> include (1) firewall, (2) load balancer (3) tomcat and perhaps
>>> the (4) the web app. The volume of traffic is so high, that it
>>> becomes a needle in a hay stack issue.
>>
>> Can you describe your setup in a bit more detail? What kind of
>> lb? How many of them? Any fronting web server(s)? How many? How
>> many Tomcat instances?
>
> There’s no front end servers. We used to do that via AJP, mod_jk
> and apache but now we just have a single generic http load
> balancer. If a backend it unavailable, it will send traffic to
> another server. We have ~5 instances give or take.
Ok.
> Everything’s Linux except osx for development and the testing (see
> below).
>
>> What is/are your <Connector> configuration(s)?
>
> Pretty simple, nothing much to see here accept the keep alive
>
> <Connector port=“8080" protocol="HTTP/1.1"
> connectionTimeout="20000" maxKeepAliveRequests="1" compression="on"
> compressableMimeType="text/html,text/xml,text/plain,application/xml"
>
>
redirectPort="8443" bindOnInit="false”/>
Just curious: why do you have maxKeepAliveRequests=1? That's... not
terribly efficient.
Are you using the native/APR connector or the Java blocking-I/O (BIO)
connector? You can tell from the startup messages or if you do a
thread dump, the names of the threads will tell you apr versus bio.
>> Tomcat usually complains if you run out of threads.
>
> I don’t see that this is true. Here’s a simple test to illustrate
>
> I set up a tomcat with:
>
> <Connector port=“8080" protocol="HTTP/1.1"
> connectionTimeout="20000" maxKeepAliveRequests= "1" maxThreads="5"
> acceptCount="1" redirectPort="8443" bindOnInit="false”/>
>
> I then have very simple JSP that looks like this.
>
> <%@page contentType="text/plain" pageEncoding="UTF-8"%><% long t0 =
> System.currentTimeMillis(); Thread.sleep(5000l); long t1 =
> System.currentTimeMillis(); %>Slept for <%= t1 - t0 %>
> milliseconds
>
> I then have a very simple BASH script that looks like:
>
> #!/bin/ti for i in {0..20} do wget —tries=1 -O file${i}.txt
> "http://localhost:8080/myapp/sleep.jsp"; & done
>
> After running this file, I end up with something like 8 files
> successfully downloaded. Naively, I would think that I’d get 6.
You have no timeout set on wget, so they might wait forever if you let
them. Any requests that aren't immediately assigned to request
processors (that would be the first 5 that connect) are queued in the
TCP accept queue by the OS and eventually will connect and be handled.
> Changing localhost to the host name, I do in fact get 6. So going
> to localhost must have a connection queuing effect (at the OS
> level) that is different from going through the interface card. The
> point is, the thread limit did it’s thing. Requests were rejected.
>
> LOOKING in the logs directory, specifically catalina.out, I do not
> see any indication that tomcat ran out of threads! It says nothing
> about rejected connections! EEEK. Bad stuff could be happening all
> the time and I would never know. What am I missing? There’s got to
> be some way to see this stuff.
Hmm... Tomcat used to complain when you hit maxConnections. Perhaps
that it gone, or perhaps it only was for certain connectors, or maybe
without using an executor (and an executor is always used since Tomcat
7... maybe Tomcat 6).
>>> In production maxConnections is the default of 200, but when
>>> looking at the number of connections being processed by tomcat
>>> it’s rarely more than a couple dozen per instance. There’s
>>> always the possibility that it occasionally spikes.
>>
>> What is the resolution of the data you have? Are you sampling
>> with some regularity?
>
> It’s really just spot checking using JMX and looking at the tomcat
> server-status page.
You might want to set up something like Nagios to watch all the time.
You can set alarms for high-usage of the connection pool.
>>> Anyway, the question is: Is there any good way to see if
>>> tomcat is going into a state where it’s not accepting
>>> connections? How would would I see this? Is it possible to see
>>> this ?
>>
>> Sure: connect to a specific Tomcat. If you get a "connection
>> refused", then you are in this state.
>
> Well of course, but this doesn’t happen very often.
Right: the TCP accept queue will hold the connection indefinitely
unless the client decides to terminate the connection (usually due to
a timeout). So unless you saturate both your maxConnections plus the
accept queue, you won't get (many) connection failures.
>> Your load balancer should have a setting where you can choose a
>> back-end server explicitly if you want to go through the lb, or
>> you can just contact one directly.
>
> Of course, I can talk to back ends directly or not.
I was wondering if you could connect through the lb to a specific
backend server, because it's possible that the lb is the one refusing
the connections for some reason and not Tomcat itself.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJTGfREAAoJEBzwKT+lPKRY3aEQAJDPN+BjyhHZgm9FQvrsnk2H
IYxsiVSmWgpF/ncMvBJEL9XNFInZDzN6R9U7frwLP8oRMOvpF9kyHaRM3KjV2jb7
wOebWdyo6tMOTDabptBS3zrDpIz0UYTE4qiVmv8kKZOgEZlvwxuMCtzt57M7UHNG
c0ZqDtsf9/St1crs27rm7pjCSsiD+tzjluqDqQ10NXZ/oOFHo4BQBsTLT75WfqrQ
eCvU20ARV5p8gVvzTLi1lgBmTqPdHy1N5mlulrKOJ2bgV2X2XiG3oEe0RmTPk0qo
6g9mjHsLZdpbWT68wVQbzl7FgeDpkzor2cItr8gI7RYAXuZmI7QvioCufBud/7mH
IUfTgQzg9J9k0HfV4U8nAN0NgRqrVmRkkWvvD5H/7yO7Jt5qU1uKtmQEGIWlDDOx
kGIUK3TwFjp/ttYBwyI2GHB19RLZkhQF9+kmK9OLOaGkisK+loKioFI9J2ZMke5f
utLXfOhdcTCjLPhftA6HO3I5G/p2+3VN5MCSJ/N5W+hgrCcynSo+wLY9ySTU2rGc
JlvuQ+cfOwS7fVsstkRImJATXrrOAy7x61x27vEm5I1QkilB8hOmTAL01ioe32FL
9CzpaP1hXVGfN+LxtcNrTSCo0fzY1vci7/T7PpoAcojdTn75qgkiQ11N44metx48
gGlNy1imoZYtRbYmSuCZ
=xiij
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
