I wonder if this is related to the requestProcessor and mod_jk problem?
Travis
---- Original Message ----
From: Gerard BORREILL <[EMAIL PROTECTED]>
Sent: 2003-02-25
To: [EMAIL PROTECTED]
Subject: A robustness improvement
Hello,
I am using Tomcat 4.0.3. We have encountered several reliability problems, which
made the HttpProcessors be not reused.
* In several cases HttpProcessor may remain used because a user
request is performed, and may be not recycled. For example
o When a long request is performed (e.g. database access) the
HttpProcessor is used during all the request, even if the
user gives up.
o If the user disconnects (stops its navigator for example),
during a request.
In fact when using an applet which calls a servlet the problem becomes much more
irritating.
* If the HttpProcessor fails due to an IO error, or to a Java Error
it is never reused. This is much less probable since Tomcat 4.11.16.
Tomcat 4.11.16 makes the HttpProcessors more reliable but it does not prevent from
HttpProcessors starvation. I have fixed the problem and would like my fixes be
included in the current Tomcat version under developement.
Below my code and the server.xml configuration file changes. Attention I have only
made a patch for HTTP 1.1.
Thanks in advance for considering my proposal. I do not know if such an issue has
already been discussed, so anyway, please let me know if and when this could be
included in Tomcat.
Bests regards,
Gérard Borreill,
1) Configuration
----------------
In server.xml file (under jakarta-tomcat4.0.3/conf) you can add two
parameters to the HTTP1.1 HttpConnector configuration
<Connector className="org.apache.catalina.connector.http.HttpConnector"
port="8080" minProcessors="5" maxProcessors="75"
enableLookups="true" redirectPort="8443"
acceptCount="10" debug="1" connectionTimeout="20000"
maxActivityTime="900" activityCtrlSleepTime="120"/>
* maxActivityTime= the maximum allowed activity time for a
HttpProcess in seconds, here 900 = 1/4hour. (Default value is 5 minutes)
* activityCtrlSleepTime= the activity controller sleep time in seconds.
Here it sleeps 2 minutes (120s) and then checks the HttpProcessors,
and then sleeps again etc. (Default value is 1 minute.)
Default settings should be enough for most cases. The HTTP1.1 connection is
maintainned as far as a client sends data within a short time, for example if there
is a burst. So long activity times should not occur.
If an Application request takes a very long time, for example a database request,
the HttpProcessor is handled by this request as far as it does not return. So the
maximum activity time should be set according to this situation.
2) What the fix does:
* The maximum activity time is the maximum amount of time an
HttpProcessor, which processes a client HTTP1.1 requests, is used
by this client.
* If an HttpProcessor hangs or is used during a long period, then it
will be destroyed and replaced by another one.
* The robustness patch guarantees that if a HttpProcessor is lost
due to an Error or an unexpected exception, it is automatically
replaced.
* The ActivityController is the thread related to the HttpConnector,
which checks HttpProcessors activity time. If necessary it kills
and replaces HttpProcessors.
* Attention: in my code log levels may have been changed, and I make
a difference between unexpected Exceptions and Errors, in the
Tomcat logs.
2) Below my specific code:
In HttpProcessor:
*************
/** The HttpProcess activity start time. If set to 0, the HttpProcess
thread
* is available for further requests.
*/
private long activityStartTime = 0;
/** Specifies whether the stop now is required (true) or not.*/
private boolean stopNowRequired = false;
/** Returns the activity start time, or 0 if the process is waiting
for a new
* socket.
* @return the activity start time.
*/
long getActivityStartTime()
{
return activityStartTime;
}
// ---------------------------------------------- Background Thread
Methods
/**
* The background thread that listens for incoming TCP/IP
connections and
* hands them off to an appropriate processor.
*/
public void run()
{
stopNowRequired = false;
// Process requests until we receive a shutdown signal
while (!stopped) {
try
{
activityStartTime = 0;
// Wait for the next socket to be assigned
Socket socket = await();
if (socket == null)
continue;
// Activity start time is set.
activityStartTime = System.currentTimeMillis();
// Process the request from this socket
process(socket);
if (!stopNowRequired)
{
// Finish up this request
connector.recycle(this);
}
else
{
log("stop required now log=> removes every
registered listeners.");
lifecycle.removeAllListeners();
}
}
catch(Throwable e)
{
log("run() exits with ", e);
connector.recycle(this);
}
}
// Tell threadStop() we have shut ourselves down successfully
synchronized (threadSync) {
threadSync.notifyAll();
}
log("thread stopped");
}// run()
/** Returns the process id.
* @return the process id.
*/
int getId() {
return id;
}
/** Tries to stop the thread now. This method is called if the
HttpProcess
* thread is being performed and the ActivityController tries to
stop it
* because the activity lasts a too long time.
* An HttpProcess stopped this way should never be reused.
* @see HttpConnector
*/
void stopNow()
{
try
{
if (!started)
throw new LifecycleException
(sm.getString("httpProcessor.notStarted"));
started = false;
stopNowRequired = true;
threadStop();
}
catch(LifecycleException e)
{
log("activity lasted a too long time => stop now", e);
}
}
I provide a StandardLoader and WebappLoader which provides more
information when their thread fail.
My fix specific code is identified by a "Patch" comment.
e.g.:
// Patch specific code
Regards,
Gérard Borreill
In HttpConnector.java
**********************
package org.apache.catalina.connector.http;
import java.io.IOException;
import java.net.BindException;
import java.net.InetAddress;
import java.net.ServerSocket;
import java.net.Socket;
import java.security.AccessControlException;
import java.util.Stack;
import java.util.Vector;
import java.util.Enumeration;
...
/**
* Implementation of an HTTP/1.1 connector.
*
* @author Craig R. McClanahan
* @author Remy Maucherat
* @version $Revision: 1.7 $ $Date: 2002/10/01 10:08:09 $
*/
public final class HttpConnector
implements Connector, Lifecycle, Runnable {
// ----------------------------------------------------- Instance Variables
...
//-------------------------------- Patch specific code.
/** The maximum activity time, by default 5 minutes.*/
private int maxActivityTime = 300;
/** The activity sleep time, by default 2 minutes. */
private int activityCtrlSleep = 120;
/** The activity controller.*/
private ActivityController activityController;
/** The active HttpProcessors are referenced in the
* activeProcessors table, so that the ActivityCrtlThread can check the maximum
* activity time.
*/
private Vector activeProcessors = new Vector(maxProcessors);
//-------------------------------- End of patch specific code.
...
// ------------------------------------------------------------- Properties
...
/** Sets the maximum activity time.
* @param maxActivityTime the maximum activity time.
*/
public void setMaxActivityTime(int maxActivityTime)
{
this.maxActivityTime = maxActivityTime;
}
/** Returns the maximum activity time.
* @return the maximum activity time.
*/
public int getMaxActivityTime()
{
return maxActivityTime;
}
/** Sets the activity controller sleep time in seconds.
* @param the activity controller sleep time in seconds.
*/
public void setActivityCtrlSleepTime(int activityCtrlSleep)
{
this.activityCtrlSleep = activityCtrlSleep;
}
/** Returns the activity controller sleep time.
* @return the activity controller sleep time.
*/
public int getActivityCtrlSleepTime()
{
return activityCtrlSleep;
}
// --------------------------------------------------------- Public Methods
...
// -------------------------------------------------------- Package Methods
/**
* Recycle the specified Processor so that it can be used again.
*
* @param processor The processor to be recycled
*/
void recycle(HttpProcessor processor) {
// Patch specific code
activeProcessors.remove(processor);
if (debug >= 2)
log(processor+ " recycling processor, activeProcessors
count="+activeProcessors.size());
processors.push(processor);
}
// -------------------------------------------------------- Private Methods
/**
* Create (or allocate) and return an available processor for use in
* processing a specific HTTP request, if possible. If the maximum
* allowed processors have already been created and are in use, return
* <code>null</code> instead.
*/
private HttpProcessor createProcessor() {
synchronized (processors) {
if (processors.size() > 0) {
// if (debug >= 2)
// log("createProcessor: Reusing existing processor");
return ((HttpProcessor) processors.pop());
}
if ((maxProcessors > 0) && (curProcessors < maxProcessors)) {
// if (debug >= 2)
// log("createProcessor: Creating new processor");
return (newProcessor());
} else {
if (maxProcessors < 0) {
// if (debug >= 2)
// log("createProcessor: Creating new processor");
return (newProcessor());
} else {
// if (debug >= 2)
// log("createProcessor: Cannot create new processor");
return (null);
}
}
}
}
...
/** Patch Code delegated to newProcessor(int).
* Create and return a new processor suitable for processing HTTP
* requests and returning the corresponding responses.
*/
private HttpProcessor newProcessor() {
return newProcessor(curProcessors++);
}
/** Patch specific code.
* Create and return a new processor suitable for processing HTTP
* requests and returning the corresponding responses.
*/
private HttpProcessor newProcessor(int id) {
if (debug >= 2)
log("newProcessor: Creating new processor");
HttpProcessor processor = new HttpProcessor(this, id);
if (processor instanceof Lifecycle) {
try {
((Lifecycle) processor).start();
} catch (LifecycleException e) {
log("newProcessor", e);
return (null);
}
}
created.addElement(processor);
return (processor);
}
...
// ---------------------------------------------- Background Thread Methods
/**
* The background thread that listens for incoming TCP/IP connections and
* hands them off to an appropriate processor.
*/
public void run()
{
try
{
// Loop until we receive a shutdown command
while (!stopped) {
// Accept the next incoming connection from the server socket
Socket socket = null;
try {
// if (debug >= 3)
// log("run: Waiting on serverSocket.accept()");
socket = serverSocket.accept();
// if (debug >= 3)
// log("run: Returned from
serverSocket.accept()");
if (connectionTimeout > 0)
socket.setSoTimeout(connectionTimeout);
socket.setTcpNoDelay(tcpNoDelay);
} catch (AccessControlException ace) {
log("socket accept security exception", ace);
continue;
} catch (IOException e) {
// if (debug >= 3)
// log("run: Accept returned IOException", e);
try {
synchronized (threadSync) {
if (started && !stopped)
log("accept: ", e);
if (!stopped) {
// if (debug >= 3)
// log("run: Closing server
socket");
serverSocket.close();
// if (debug >= 3)
// log("run: Reopening
server socket");
serverSocket = open();
}
}
// if (debug >= 3)
// log("run: IOException processing
completed");
} catch (IOException ex) {
// If reopening fails, exit
log("socket reopen: ", ex);
break;
}
continue;
}
// Hand this socket off to an appropriate processor
HttpProcessor processor = createProcessor();
if (processor == null) {
try {
log(sm.getString("httpConnector.noProcessor")+'
'+activeProcessors.size()+" processors used");
socket.close();
} catch (IOException e) {
;
}
continue;
}
// References the processor so that the activity may be checked.
activeProcessors.add(processor);
// Patch , if we reach 95% of the connexions used, log this
if (debug >= 1)
{
int activeOnes = activeProcessors.size();
int createdOnes = created.size();
if (activeOnes > (int)(createdOnes*0.95))
{
log(activeOnes+"/"+createdOnes+" processors used");
}
}
processor.assign(socket);
// The processor will recycle itself when it finishes
}
}
// Patch specific code.
// We don't catch only Throwable as if an Exception is caught it must be
// treated specifically, and usage of Exceptions is detected at compilation
time.
catch(RuntimeException e)
{
log("thread exits with", e);
}
catch(Error e)
{
log("thread exits with", e);
}
finally
{
// Notify the threadStop() method that we have shut ourselves down
// if (debug >= 3)
// log("run: Notifying threadStop() that we have shut down");
synchronized (threadSync) {
threadSync.notifyAll();
}
}
}// run()
/**
* Start the background processing thread.
*/
private void threadStart() {
log(sm.getString("httpConnector.starting")+" -- catalina with Tomcat 4.0.3
patch 1.0");
thread = new Thread(this, threadName);
thread.setDaemon(true);
thread.start();
activityController = new ActivityController();
activityController.start();
}
/**
* Stop the background processing thread.
*/
private void threadStop() {
log(sm.getString("httpConnector.stopping"));
activityController.notifyStop();
stopped = true;
try {
threadSync.wait(5000);
} catch (InterruptedException e) {
;
}
thread = null;
}
// ------------------------------------------------------ Lifecycle Methods
...
/** Patch
* This thread checks that the HttpProcessors do not remain active more than
* the maximum allowed time.
*/
private class ActivityController extends Thread
{
/** Boolean that specifies whether the thread should stop or not.*/
private boolean shouldStop = false;
/** Constructs an activity controller object. It sets the Thread class
* as a daemon.
*/
ActivityController()
{
setDaemon(true);
}
/** The main thread method.
* This thread must not be killed.*/
public void run()
{
log("ActivityController started!");
while(!shouldStop)
{
try
{
long maxActivityTimeMs = (long)(maxActivityTime*1000);
synchronized(activeProcessors)
{
long currentTime = System.currentTimeMillis();
long activityStartTime;
for (int i = activeProcessors.size() -1; i >= 0; i--)
{
HttpProcessor processor =
(HttpProcessor)activeProcessors.get(i);
activityStartTime = processor.getActivityStartTime();
if (activityStartTime == 0)
{
continue;
}
// The HttpProcessor is destroyed
if (currentTime - activityStartTime > maxActivityTimeMs)
{
synchronized(processors)
{
// The id must be reused.
int id = processor.getId();
// stop the processor now.
processor.stopNow();
// The Processor is now inactive and should no
more
// be referenced.
activeProcessors.remove(processor);
created.remove(processor);
// Create a new processor that replaces this one.
processors.push(newProcessor(id));
log("activity timeout reached:
HttpProcessor["+id+"] replaced");
}
// Current time is computed again as this branch
// lasts a certain time.
currentTime = System.currentTimeMillis();
}
// Wait a little to time in order to take the less
// CPU as possible.
try{Thread.sleep(1);}
catch(InterruptedException e){};
}
}// synchronized()
// Sleeps before checking again the active HttpProcessors.
try { Thread.sleep((long)(activityCtrlSleep*1000)); }
catch(InterruptedException e) { }
}// try
catch(RuntimeException e)
{
log("ActivityController fails with :", e);
}
catch(Error e)
{
log("ActivityController fails with :", e);
}
}// while()
log("ActivityController stopped!");
}// run()
/** Notify the Thread it should stop.
*/
public void notifyStop()
{
shouldStop = true;
interrupt();
}
}// class ActivityController
}
In StandardLoader.java and in WebAppLoader.java I have added a try/ catch block in the
run() methods in order to catch RuntimeExceptions and Errors, and to log them.
***********************
// ------------------------------------------------------ Background Thread
/**
* The background thread that checks for session timeouts and shutdown.
*/
public void run() {
try
{
if (debug >= 1)
log("BACKGROUND THREAD Starting");
DirContextURLStreamHandler.bindThread(this.container.getResources());
// Loop until the termination semaphore is set
while (!threadDone) {
// Wait for our check interval
threadSleep();
// Perform our modification check
if (!classLoader.modified())
continue;
// Handle a need for reloading
notifyContext();
break;
}
DirContextURLStreamHandler.unbindThread();
}
// Patch specific code.
catch(RuntimeException e)
{
log("thread exits with", e);
}
catch(Error e)
{
log("thread exits with", e);
}
if (debug >= 1)
log("BACKGROUND THREAD Stopping");
}
In WebappLoader.java
********************
// ------------------------------------------------------ Background Thread
/**
* The background thread that checks for session timeouts and shutdown.
*/
public void run() {
try
{
if (debug >= 1)
log("BACKGROUND THREAD Starting");
// Loop until the termination semaphore is set
while (!threadDone) {
// Wait for our check interval
threadSleep();
try {
// Perform our modification check
if (!classLoader.modified())
continue;
} catch (Exception e) {
log(sm.getString("webappLoader.failModifiedCheck"), e);
continue;
}
// Handle a need for reloading
notifyContext();
break;
}
}
// Patch specific code.
catch(RuntimeException e)
{
log("thread exits with", e);
}
catch(Error e)
{
log("thread exits with", e);
}
if (debug >= 1)
log("BACKGROUND THREAD Stopping");
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
