Hello
I have a website which during peak time (peak lasts around ~4 hours with
14 pageviews/sec, 140 http requests/sec) starts to drop pageview requests.
My setup is:
- SUSE Linux Enterprise Server 10 (x86_64) with 8 cpus, 16GB ram
- 1 instance of apache 2.2.10 with mpm prefork
- mod_jk/1.2.28
- 1 instance of tomcat 6.0.20
the O/S, tomcat and mysql seem to be completely underutilized:
- cpu is at 2% to 4%
- tomcat page render time is very low, around 50-200ms
- iostat shows minimal disk utilization
- lots of free disk space
even when apache is not saturated, 1 out of ~6 requests (to be routed to
tomcat) will never return (or will return after 5-10 seconds).
mod_status at that point shows me the following which means that it can
handle more requests:
KK.K_____CKK___KC_KK.__C__K___K__._CCKKKC_K__KK_KW_KK____KW..KC_
KKK_KCK_KCC_.___W_CC__CC__KC.CK_K.CKK___KK_C_CKKK_KKKKCCKK.KKKK_
K___KW_CK_K.CK_.C____C.C..KCRW.._KC.._.K....K..K...K_.._..K..K.K
.KC._W.C...._.K_CC._.K__K..KC_._K...K.K._..._..._._......_CKK_..
_.CKK_K_.KK...............R.....................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
All requests to static content (not served via tomcat) have no problem.
my apache config is:
KeepAlive On
MaxKeepAliveRequests 1024
KeepAliveTimeout 5
JkLogFile logs/mod_jk.log
JkShmFile logs/jk-runtime-status
JkLogLevel info
JkWorkerProperty worker.list=tc1
JkWorkerProperty worker.tc1.type=ajp13
JkWorkerProperty worker.tc1.host=localhost
JkWorkerProperty worker.tc1.port=8010
<VirtualHost *:80>
ServerName www.foobar.com
DocumentRoot ...
JkMount /* tc1
JkUnMount /*.css tc1
JkUnMount /*.js tc1
...
JkUnMount /*.jpg tc1
</VirtualHost>
<IfModule mpm_prefork_module>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
ServerLimit 1024
MaxClients 1024
MaxRequestsPerChild 100000
</IfModule>
my tomcat config is very plain:
<Service name="Catalina">
<Connector port="8010" protocol="AJP/1.3" redirectPort="8443"
address="127.0.0.1" />
<Engine name="Catalina" defaultHost="www.foobar.com">
<Host name="www.foobar.com" appBase="webapps/foobar" />
</Engine>
</Service>
After I stop/start apache I start receiving these logs many times on
mod_jk.log:
[5449:902382544] [info] ajp_process_callback::jk_ajp_common.c (1748):
Writing to client aborted or client network problems
[5449:902382544] [info] ajp_service::jk_ajp_common.c (2407): (tc1)
sending request to tomcat failed (unrecoverable), because of client
write error (attempt=1)
[5449:902382544] [info] jk_handler::mod_jk.c (2469): Aborting connection
for worker=tc1
Exactly 200 seconds after the restart (confirmed it 10 times) these
start appearing as well:
[5761:902382544] [info] jk_open_socket::jk_connect.c (593): connect to
127.0.0.1:8010 failed (errno=110)
[5761:902382544] [info] ajp_connect_to_endpoint::jk_ajp_common.c (922):
Failed opening socket to (127.0.0.1:8010) (errno=110)
[5761:902382544] [error] ajp_send_request::jk_ajp_common.c (1467): (tc1)
connecting to backend failed. Tomcat is probably not started or is
listening on the wrong port (errno=110)
These logs may be unrelated to the slow or dropped responses from mod_jk.
On tomcat catalina.out I'm getting lots of:
org.apache.jk.common.ChannelSocket processConnection
WARNING: processCallbacks status 2
Any ideas on where should I look at?
Will the mod_jk status worker give me any insights on this?
Is there any possibility that mod_jk is getting "blocked" at the O/S
level (maybe from iptables)?
Also, I'm not sure if I can successfully reproduce the whole situation
locally with JMeter since I don't have these many hosts to hit a staging
server with.
All I'm left with is wait each day for the peak time and fiddle around
with the settings of the production server.
thanks a lot
Ioannis
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
