- how is your iowait ? maybe apache is waiting for disk access. #vmstat 1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 1 3 2536 698992 195040 211940 0 0 4 187 2 3 14 14 54 17 1 0 2536 698276 195064 211984 0 0 20 0 8746 9432 27 13 43 17 0 0 2536 697444 195084 211964 0 0 20 0 8428 8802 26 11 62 1 0 0 2536 697196 195088 211960 0 0 4 0 7206 6812 19 8 73 0 1 3 2536 706232 195092 211956 0 0 4 4068 5886 5305 12 7 67 14 0 82 2536 706300 195092 211956 0 0 0 844 4534 3498 9 4 24 64 4 0 2536 705516 195092 211956 0 0 0 584 7017 6394 17 10 14 59 1 0 2536 705460 195092 211956 0 0 0 0 7994 8138 18 10 72 0 0 0 2536 712564 195100 212016 0 0 8 0 6763 7027 17 8 74 0 0 0 2536 712692 195112 212004 0 0 12 0 6201 8271 15 7 77 1 1 1 2536 712256 195116 212000 0 0 20 2372 6276 5845 14 6 78 2 1 32 2536 712436 195128 211988 0 0 12 1584 6633 7025 15 7 9 69 2 4 2536 722980 195128 211988 0 0 0 592 7114 7063 16 10 10 64 1 0 2536 722796 195140 212044 0 0 32 0 7215 7184 17 9 71 4 5 0 2536 722300 195164 212088 0 0 24 168 8066 9076 25 11 61 2 1 0 2536 722180 195168 212084 0 0 4 0 7529 7952 19 10 70 0 2 27 2536 720712 195168 211268 0 0 0 3792 6718 6542 20 11 28 41 4 7 2536 730984 195168 211268 0 0 0 1616 10398 10589 44 22 0 35 3 5 2536 731864 195172 211264 0 0 4 740 8842 10712 27 14 4 54 1 2 2536 732116 195184 211320 0 0 12 40 7049 7312 17 9 43 32 This is when apache is running normally. Iowait does not seem to be a problem, but I did not check during the crashes. - are your sure that you don't have a network bottleneck ? load balancer, firewall, network interface,.. We have a Twisted server running on the same machine, and it works OK during the crashes- so it does not seem to be a network bottleneck. Plus, system log shows only this during crashes, which indicates apache-specific bottleneck: Oct 1 13:21:56 web10 kernel: possible SYN flooding on port 80. Sending cookies. Oct 1 13:22:56 web10 kernel: possible SYN flooding on port 80. Sending cookies. Oct 1 13:23:57 web10 kernel: possible SYN flooding on port 80. Sending cookies. - do you serve static files only with you apache ? if you use php, cgi,mod_jk,mysql,... You should have a look to them maybe they are hanging up. We are using mod_python. I posted to mod_python list as well. - if you use thread safe modules (not php) and if you have to serve a lot of little requests, maybe you can consider to use mpm worker instead of prefork. This is a good idea. We will try it. Does it give significant performance bonus? Our average request size is only 1-2K. Alec From: Bj [mailto:[EMAIL PROTECTED] Sent: Monday, October 01, 2007 3:15 AM To: users@httpd.apache.org Subject: Re: [EMAIL PROTECTED] requests time-out under load, no warnings in logs Hi, When your server is slow: - how is your iowait ? maybe apache is waiting for disk access. - are your sure that you don't have a network bottleneck ? load balancer, firewall, network interface,.. - do you serve static files only with you apache ? if you use php, cgi,mod_jk,mysql,... You should have a look to them maybe they are hanging up. - if you use thread safe modules (not php) and if you have to serve a lot of little requests, maybe you can consider to use mpm worker instead of prefork. -- Bj On 10/1/07, Alec Matusis <[EMAIL PROTECTED]> wrote: Hi Christian, > How does your apache look memory-wise in top/htop? When Apache is "slow", it has about 200 processes with 11-15MB RSS each, soat least 1GB of physical RAM should be unused. The load average drops from normal 16.0 (this is a 4 CPU machine) to only 6.0 when apache becomes "slow". It would be interesting to see an example of a reasonable choice for MaxMemFree, after 2 days of searching in Google I cannot find any guidelines for setting it. > -----Original Message----- > From: Christian Folini [mailto:[EMAIL PROTECTED] > Sent: Sunday, September 30, 2007 11:55 PM > To: users@httpd.apache.org > Subject: Re: [EMAIL PROTECTED] requests time-out under load, no warnings in > logs > > Hey Alec, > > This looks like a tricky problem. Have you looked at the traffic > using tcpdump or ethereal when the machine hangs? You might > also try out mod_forensic or mod_security and add mod_security > internal timestamps to the access-log. Using these logs you will > get a clearer idea where it hangs. > > How does your apache look memory-wise in top/htop? > Are you sure it is not an application issue - even if it shows > serving static files. A reverse proxy setup for testing purposes > could seperate application and static file serving to > give you a clearer idea in this regard. > > These are just a lot of ideas. I do not really know how to > solve your problem, but this would give you a clearer idea > of the problem. > > I am not familiar with best practice values for MaxMemFree. > > regs, > > Christian > > > > On Sun, Sep 30, 2007 at 03:54:59PM -0700, Alec Matusis wrote: > > We are running a busy Apache/2.0.54 server on 2.6.9 kernel, that > suddenly > > becomes very slow- requests either time out, or it takes 10s to serve > a 1K > > thumbnail. > > It is usually correlated with load spikes, but not perfectly (by > looking at > > the bandwidth graph) > > Then it requires a stop/start, and it becomes fast again. > > This has been happening before, and it was always accompanied with > > > > [error] server reached MaxClients setting, consider raising the > MaxClients > > setting > > > > in the apache log. > > > > We raised MaxClients from 200 to 300, and now it happens with no > warning > > in any logs at all, except with the usual > > > > web10 kernel: possible SYN flooding on port 80. Sending cookies. in > > /var/log/messages (this has always happened when apache hung from > overload). > > > > What worries me is that there are no log messages in apache logs at > all when > > this happens. > > > > This is the relevant part of httpd.conf > > > > ServerLimit 300 > > # we are using prefork MPM > > <IfModule prefork.c> > > StartServers 10 > > MinSpareServers 5 > > MaxSpareServers 20 > > MaxClients 300 > > MaxRequestsPerChild 10000 > > MaxMemFree 2560 > > </IfModule> > > > > The server has 4GB of RAM and 4GB of swap. When apache "crashes" i.e. > > becomes very slow, the swap size is still 0 and vmstat shows no > swapping. > > I suspect the problem may be in > > > > MaxMemFree 2560 > > > > but then I would expect some kind of "out of memory" errors in the > logs? > > Another question: what is the good choice of MaxMemFree value? We > want to > > leave 512MB-1GB or RAM on this machine for other tasks. > > > > > > > > > > > > --------------------------------------------------------------------- > > The official User-To-User support forum of the Apache HTTP Server > Project. > > See <URL: http://httpd.apache.org/userslist.html> for more info. > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > " from the digest: [EMAIL PROTECTED] > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > --------------------------------------------------------------------- > The official User-To-User support forum of the Apache HTTP Server > Project. > See <URL:http://httpd.apache.org/userslist.html> for more info. > To unsubscribe, e-mail: [EMAIL PROTECTED] > " from the digest: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] --------------------------------------------------------------------- The official User-To-User support forum of the Apache HTTP Server Project. See <URL: <http://httpd.apache.org/userslist.html> http://httpd.apache.org/userslist.html> for more info. To unsubscribe, e-mail: [EMAIL PROTECTED] " from the digest: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
