Hi This is my prod setup
Client -> Internet -> IIS7.5/NLB (2 node cluster) with the 1.2.32 tomcat connector -> 2 node tomcat cluster. We had done a fair bit of testing on this setup enough to move into production. But we ran into a problem ... when IIS recycle its application pool. Luckily we had set this to manually happen after hours So I setup a test rig to see if I could simulate the problem. The test rig is Client -> 1 node of IIS7.5 + 1.2.32 tomcat connector -> 2 node of development tomcat :) The client works by creating a connection and keeping it open and making lots of requests. We have jerry rigged a client to make multiple of these as the error occurs when the system is under load. So the test scenario is start client login >>> this creates 1 connection strat stress test >>> this creates 500 connection and does a little bit of polling My initial setup is 4 process worker threads for the application pool Allow overlapping recycle >>>> my understanding is this means the 4 threads are not all terminated at once, but one at a time as a new thread is restarted I use isapi_redirect.properties file ============================ extension_uri=/jakarta/isapi_redirect.dll log_file=C:\ThisNow\Local\dev.ThisNow.com\Logs\ajpconfisapi_redirect.log log_level=warn log_filesize=20M worker_file=C:\ThisNow\Shared\dev.ThisNow.com\ajpconfig\workers.properties worker_mount_file=C:\ThisNow\Shared\dev.ThisNow.com\ajpconfig\uriworkermap.properties rewrite_rule_file=C:\ThisNow\Shared\dev.ThisNow.com\ajpconfig\\rewrite.properties C:\ThisNow\Shared\dev.ThisNow.com\ajpconfig\workers.properties ==================================================== worker.maintain=60 worker.list=jbclb worker.jbclb.type=lb worker.jbclb.balance_workers=worker1, worker2 worker.jbclb.sticky_session=true worker.jbclb.method=Request worker.jbclb.lock=pessimistic # Template for all worker threads worker.template.type=ajp13 worker.template.port=8009 worker.template.ping_mode=A worker.template.ping_timeout=30000 worker.template.connection_pool_size=1000 worker.template.connection_pool_minsize=100 worker.template.socket_keepalive=true worker.template.lock=pessimistic worker.template.max_packet_size=65536 # worker 1 worker.worker1.host=10.172.204.30 worker.worker1.route=node1 worker.worker1.reference=worker.template # worker 1 worker.worker2.route=node2 worker.worker2.host=10.172.204.31 worker.worker2.reference=worker.template worker.list=jkstatus # Define a 'jkstatus' worker using status worker.jkstatus.type=status C:\ThisNow\Shared\dev.ThisNow.com\ajpconfig\uriworkermap.properties ========================================================= /dev.ThisNow.com/ThisNowLive|/*=jbclb /dev.ThisNow.com/wls|/*=jbclb /dev.ThisNow.com/jkmanager|/*=jkstatus /10.172.204.33/jkmanager|/*=jkstatus /ThisNowodevrp01/jkmanager|/*=jkstatus C:\ThisNow\Shared\dev.ThisNow.com\ajpconfig\rewrite.properties =================================================== # rewrite rules # Convert single node ~/jb./(.*)=/$1 /web-debts-1.0.0-nossl/=/web-debts-1.0.0/ I can see there are 500 connection to IIS and ~500 connection through to tomcat. Check with perfmon, netstat... When I recycle the application pool. IIS starts to send 500's back to the server.... it complain about the tomcat connector returning an error errorcode 2147942401, internal server error Incorrect Function 0c80070001 What I can see from the network side of things. Is the client is trying to recreate all 500 connections at once. What I can see in my tomcat connector logs... some times there is 4 for each process and sometimes 2 or 3, if its crashed into this 500 error loop Examples of what I see (2 log files) Ajp file 1 === [Tue Mar 06 14:46:46.572 2012] [5872:5028] [error] ajp_worker_factory::jk_ajp_common.c (2997): allocating ajp worker record from shared memory [Tue Mar 06 14:46:46.572 2012] [5872:5028] [error] wc_create_worker::jk_worker.c (151): factory for ajp13 failed for worker2 m shared memory [Tue Mar 06 14:46:46.572 2012] [5872:5028] [error] validate::jk_lb_worker.c (1615): Failed creating worker worker2 worker2 [Tue Mar 06 14:46:46.587 2012] [2888:5708] [error] ajp_worker_factory::jk_ajp_common.c (2997): allocating ajp worker record from shared memory [Tue Mar 06 14:46:46.603 2012] [5872:5028] [error] validate::jk_lb_worker.c (1663): NULL parameters [Tue Mar 06 14:46:46.603 2012] [5872:5028] [error] wc_create_worker::jk_worker.c (163): validate failed for jbclb worker2 [Tue Mar 06 14:46:46.619 2012] [5872:5028] [error] build_worker_map::jk_worker.c (262): failed to create worker jbclb [Tue Mar 06 14:46:46.619 2012] [5872:5028] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.619 2012] [5872:5028] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jbclb' in uri map post processing. [Tue Mar 06 14:46:46.634 2012] [5872:5028] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.634 2012] [5872:5028] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jbclb' in uri map post processing. [Tue Mar 06 14:46:46.634 2012] [1536:1332] [error] validate::jk_lb_worker.c (1663): NULL parameters d not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.634 2012] [5872:5028] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.634 2012] [5872:5028] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jbclb' in uri map post processing. [Tue Mar 06 14:46:46.634 2012] [1536:1332] [error] wc_create_worker::jk_worker.c (163): validate failed for jbclb [Tue Mar 06 14:46:46.634 2012] [5872:5028] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jbclb' in uri map post processing. [Tue Mar 06 14:46:46.634 2012] [5872:5028] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.650 2012] [5872:5028] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.650 2012] [1536:1332] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.650 2012] [1536:1332] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jbclb' in uri map post processing. [Tue Mar 06 14:46:46.665 2012] [1536:1332] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.665 2012] [1536:1332] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jbclb' in uri map post processing. [Tue Mar 06 14:46:46.665 2012] [1536:1332] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.665 2012] [2888:5708] [error] build_worker_map::jk_worker.c (262): failed to create worker jbclb with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.665 2012] [1536:1332] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jbclb' in uri map post processing. [Tue Mar 06 14:46:46.665 2012] [2888:5708] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.681 2012] [2888:5708] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jbclb' in uri map post processing. [Tue Mar 06 14:46:46.681 2012] [2888:5708] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.681 2012] [2888:5708] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jbclb' in uri map post processing. [Tue Mar 06 14:46:46.681 2012] [2888:5708] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.681 2012] [2888:5708] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.681 2012] [2888:5708] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jbclb' in uri map post processing. [Tue Mar 06 14:46:46.681 2012] [2888:5708] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jbclb' in uri map post processing. [Tue Mar 06 14:46:46.681 2012] [2888:5708] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.697 2012] [2888:5708] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. Ajp log file 2 ============== [Tue Mar 06 14:46:46.650 2012] [5872:3624] [warn] init_jk::jk_isapi_plugin.c (2738): The optimal shared memory size can now be determined automatically. [Tue Mar 06 14:46:46.650 2012] [5872:3624] [warn] init_jk::jk_isapi_plugin.c (2740): You can remove the shm_size attribute if you want to use the optimal size. [Tue Mar 06 14:46:46.665 2012] [5872:3624] [error] wc_create_worker::jk_worker.c (151): factory for lb failed for jbclb [Tue Mar 06 14:46:46.665 2012] [5872:3624] [error] build_worker_map::jk_worker.c (262): failed to create worker jbclb [Tue Mar 06 14:46:46.665 2012] [5872:3624] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.665 2012] [5872:3624] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jbclb' in uri map post processing. [Tue Mar 06 14:46:46.665 2012] [5872:3624] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.665 2012] [5872:3624] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jbclb' in uri map post processing. [Tue Mar 06 14:46:46.665 2012] [5872:3624] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.665 2012] [5872:3624] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.681 2012] [5872:3624] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jbclb' in uri map post processing. [Tue Mar 06 14:46:46.681 2012] [5872:3624] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jbclb' in uri map post processing. [Tue Mar 06 14:46:46.681 2012] [5872:3624] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.681 2012] [5872:3624] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.697 2012] [5872:2836] [warn] init_jk::jk_isapi_plugin.c (2738): The optimal shared memory size can now be determined automatically. [Tue Mar 06 14:46:46.697 2012] [2888:4792] [warn] init_jk::jk_isapi_plugin.c (2738): The optimal shared memory size can now be determined automatically. [Tue Mar 06 14:46:46.697 2012] [5872:2836] [warn] init_jk::jk_isapi_plugin.c (2740): You can remove the shm_size attribute if you want to use the optimal size. [Tue Mar 06 14:46:46.712 2012] [2888:4792] [warn] init_jk::jk_isapi_plugin.c (2740): You can remove the shm_size attribute if you want to use the optimal size. [Tue Mar 06 14:46:46.712 2012] [2888:4792] [error] wc_create_worker::jk_worker.c (151): factory for lb failed for jbclb w be determined automatically. [Tue Mar 06 14:46:46.728 2012] [2888:4792] [error] build_worker_map::jk_worker.c (262): failed to create worker jbclb te if you want to use the optimal size. [Tue Mar 06 14:46:46.728 2012] [2888:4792] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.728 2012] [5872:2836] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.728 2012] [2888:4792] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jbclb' in uri map post processing. [Tue Mar 06 14:46:46.728 2012] [1536:1492] [error] build_worker_map::jk_worker.c (262): failed to create worker jbclb [Tue Mar 06 14:46:46.728 2012] [2888:4792] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.728 2012] [1536:1492] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.744 2012] [5872:2836] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.744 2012] [2888:4792] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jbclb' in uri map post processing. [Tue Mar 06 14:46:46.744 2012] [1536:1492] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jbclb' in uri map post processing. [Tue Mar 06 14:46:46.744 2012] [5872:2836] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jbclb' in uri map post processing. [Tue Mar 06 14:46:46.744 2012] [2888:4792] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.744 2012] [1536:1492] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.744 2012] [5872:2836] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.744 2012] [2888:4792] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.744 2012] [5872:2836] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.744 2012] [2888:4792] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jbclb' in uri map post processing. [Tue Mar 06 14:46:46.759 2012] [1536:1492] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.759 2012] [5872:2836] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jbclb' in uri map post processing. [Tue Mar 06 14:46:46.759 2012] [2888:4792] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jbclb' in uri map post processing. [Tue Mar 06 14:46:46.759 2012] [1536:1492] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.759 2012] [5872:2836] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jbclb' in uri map post processing. [Tue Mar 06 14:46:46.759 2012] [2888:4792] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.759 2012] [1536:1492] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jbclb' in uri map post processing. [Tue Mar 06 14:46:46.759 2012] [5872:2836] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.759 2012] [1536:1492] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jbclb' in uri map post processing. [Tue Mar 06 14:46:46.759 2012] [5872:2836] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.775 2012] [1536:1492] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.775 2012] [1536:1492] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.775 2012] [5872:3380] [warn] init_jk::jk_isapi_plugin.c (2738): The optimal shared memory size can now be determined automatically. [Tue Mar 06 14:46:46.790 2012] [5872:3380] [warn] init_jk::jk_isapi_plugin.c (2740): You can remove the shm_size attribute if you want to use the optimal size. [Tue Mar 06 14:46:46.790 2012] [1536:1272] [warn] init_jk::jk_isapi_plugin.c (2740): You can remove the shm_size attribute if you want to use the optimal size. [Tue Mar 06 14:46:46.790 2012] [2888:1548] [warn] init_jk::jk_isapi_plugin.c (2740): You can remove the shm_size attribute if you want to use the optimal size. [Tue Mar 06 14:46:46.790 2012] [1536:1272] [error] wc_create_worker::jk_worker.c (151): factory for lb failed for jbclb [Tue Mar 06 14:46:46.790 2012] [2888:1548] [error] wc_create_worker::jk_worker.c (151): factory for lb failed for jbclb [Tue Mar 06 14:46:46.806 2012] [1536:1272] [error] build_worker_map::jk_worker.c (262): failed to create worker jbclb with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.806 2012] [2888:1548] [error] build_worker_map::jk_worker.c (262): failed to create worker jbclb [Tue Mar 06 14:46:46.806 2012] [5872:3380] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jbclb' in uri map post processing. [Tue Mar 06 14:46:46.806 2012] [1536:1272] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.806 2012] [2888:1548] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.806 2012] [5872:3380] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.806 2012] [1536:1272] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jbclb' in uri map post processing. [Tue Mar 06 14:46:46.806 2012] [2888:1548] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jbclb' in uri map post processing. [Tue Mar 06 14:46:46.822 2012] [5872:3380] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jbclb' in uri map post processing. [Tue Mar 06 14:46:46.822 2012] [2888:1548] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.822 2012] [1536:1272] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jbclb' in uri map post processing. [Tue Mar 06 14:46:46.822 2012] [2888:1548] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jbclb' in uri map post processing. [Tue Mar 06 14:46:46.822 2012] [5872:3380] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.822 2012] [2888:1548] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.837 2012] [1536:1272] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.837 2012] [2888:1548] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.837 2012] [1536:1272] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jbclb' in uri map post processing. [Tue Mar 06 14:46:46.837 2012] [2888:1548] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jbclb' in uri map post processing. [Tue Mar 06 14:46:46.837 2012] [1536:1272] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jbclb' in uri map post processing. [Tue Mar 06 14:46:46.837 2012] [2888:1548] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jbclb' in uri map post processing. [Tue Mar 06 14:46:46.853 2012] [1536:1272] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.853 2012] [5872:3380] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.853 2012] [1536:1272] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.853 2012] [2888:1548] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.869 2012] [2888:3536] [warn] init_jk::jk_isapi_plugin.c (2738): The optimal shared memory size can now be determined automatically. [Tue Mar 06 14:46:46.869 2012] [2888:3536] [warn] init_jk::jk_isapi_plugin.c (2740): You can remove the shm_size attribute if you want to use the optimal size. [Tue Mar 06 14:46:46.869 2012] [1536:5072] [warn] init_jk::jk_isapi_plugin.c (2738): The optimal shared memory size can now be determined automatically. [Tue Mar 06 14:46:46.884 2012] [1536:5072] [warn] init_jk::jk_isapi_plugin.c (2740): You can remove the shm_size attribute if you want to use the optimal size. [Tue Mar 06 14:46:46.884 2012] [2888:3536] [error] build_worker_map::jk_worker.c (262): failed to create worker jbclb [Tue Mar 06 14:46:46.884 2012] [1536:5072] [error] wc_create_worker::jk_worker.c (151): factory for lb failed for jbclb [Tue Mar 06 14:46:46.884 2012] [1536:5072] [error] build_worker_map::jk_worker.c (262): failed to create worker jbclb with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.884 2012] [1536:5072] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.884 2012] [2888:3536] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jbclb' in uri map post processing. [Tue Mar 06 14:46:46.884 2012] [1536:5072] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jbclb' in uri map post processing. [Tue Mar 06 14:46:46.900 2012] [2888:3536] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.900 2012] [1536:5072] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.900 2012] [2888:3536] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jbclb' in uri map post processing. [Tue Mar 06 14:46:46.900 2012] [2888:3536] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.900 2012] [2888:3536] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.915 2012] [2888:3536] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jbclb' in uri map post processing. [Tue Mar 06 14:46:46.915 2012] [2888:3536] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jbclb' in uri map post processing. [Tue Mar 06 14:46:46.915 2012] [2888:3536] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.915 2012] [1536:5072] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. [Tue Mar 06 14:46:46.915 2012] [2888:3536] [error] uri_worker_map_ext::jk_uri_worker_map.c (506): Could not find worker with name 'jkstatus' in uri map post processing. During normal startup I don't see anything logged at all. My initial testing was to move from overlapping recycle to non-overlapping recycle. This seems to have solved it for 99% of the time. But if I get enough connection trying to hit the server at 1 time, in my case > ~450 it will cause this corruption some of the times :) My best solution is to go to single process thread and non over lapping (although I would have thought non overlapping shouldn't have an effect, but maybe that is how IIS decides to kill first or afterwards!) So my conclusion / presumption is there is an issue with the shared memory locking and also what seems to be the initialisation code sequencing. Under load, IIS will first start 1 process and then if enough request another and another depending on load. So what I see is 4 process starting at about the same time and the shared memory is getting corrupted and from there it turns to poo. I can presume it get corrupted for 2 reasons. Firstly the connector logs faults unable to map a URI and then stop warning about that, I can presume that, the process in question hasn't reads its config and then eventual does ! If it hasn't read its config and it processing uri's and presumable touching the SHM, it is doing so under an optimistic locking method... That in windows land using in process locking methods, which don't work across processes ! (this is different to linux/unix world). So I am thinking 2 changes need to be made. The connector needs to default to pessimistic locking (maybe the default for windows).... It also depends on how to fix the init code. If you can guarantee that the config file will be processed before the Shared memory is touched, then maybe its okay to leave the default to optimistic. The second bug, the connector needs to be fully initialised before it starts to process any URI's... This is not good ... but I think it a problem with Web Gardens (this is the name given by MS for more than 1 process in the application pool) When it was architected I can guess this wouldn't have been the setup. I could presume (haven't looked at the init stuff yet), but if it find the SHM memory already created, it thinks it's ready to go ! I don't mind digging around the code and supplying patches, but I am more used to the linux world and well , I have tried to build the connector in VS 2010 and ..... Does somebody have a tar ball of a tree what I can just point to and compile :) (or maybe some instruction on how to get it to build from the svn tree in VS2010) I have logged a bug report https://issues.apache.org/bugzilla/show_bug.cgi?id=52659, I think it might be linked to https://issues.apache.org/bugzilla/show_bug.cgi?id=52286 Thanks Alex Sorry for the long email --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
