On 06/07/2011 10:59 PM, Rob Morin wrote:
Hello all was not sure how to word my subject line...
I posted a few weeks back on how could I increase performance of
apache... I received quite a few replies and they all helped a bit...
We even added an additional server to our server cluster to total 7
web servers now, all load balanced by HAProxy
We currently receive about 14 million impressions a day to about 60
different domains/websites....
Our hosting provider uses HAProxy, rather than an appliance, I assume
to reduce costs. However I feel like these HAProxy machines are not
reliable at all....Well not under the supervision of our hosting
company , anyways...J
On to my questions....
During peek hours, like a few mins ago, all of our web servers became
high in load, like close to 100! We still had plenty of RAM left and
for sure there was no swapping....
So what was the cause of the load ?
Examine dstat output while the load spikes and see if there is a high
percentage spent in IOWait.
Apache does ultimately serve content from disk; is your disk subsystem
up to the task ?
Please see my config below as quick snapshot of the httpd.conf file
on our Centos Servers.
The weird thing is when I would go see the server-status I would see
all the slots filed with either W or C and nothing else.... Then after
a what appears a random amount of time it would all clear up , and
then come back again. Almost like HAProxy would send a bunch of
requests to a server, then it would get full and then(haproxy) say, oh
that server is filled I will not send traffic there, for now, then it
would come back and see hey this server is empty, lets send a bunch
of traffic there...
So it may just as well be the frontend that causes these spikes; you
want to establish this with certainty.
Do all apache backend boxes spike at the same time ?
You really need to do a more thorough investigation.
Consider graphing the performance of all boxes, and comparing values
during different times of the day.
An rrdtool grapher like cacti or mrtg will help with this, at marginal
extra load.
If you consider that too heavy, munin has useful data to correlate CPU
with disk I/O and apache stats.
We also use memcache to cache sessions and mysql queries, along with
eAcclerator... I assume that those are functioning properly
That's a pretty large assumption. Don't.
Does this make sense, am I on drugs? J
You tell us.
<IfModule prefork.c>
<IfModule worker.c>
This tells us nothing. WHICH ONE are you actually using ?
If you're using prefork (which is the stupid choice for high load
machines), starting up a hundred new threads at peak times will take
MINUTES.
It looks as if you haven't tuned any of the really vital server
variables for this supposed high load you're trying to handle.
<Directory />
Options FollowSymLinks
AllowOverride None
</Directory>
You really want to deny access here.
<Directory "/home/www">
Options -Indexes FollowSymLinks
This does nothing.
AllowOverride All
Order allow,deny
Allow from all
</Directory>
<IfModule mod_userdir.c>
UserDir disable
</IfModule>
DirectoryIndex index.php index.html
AccessFileName .htaccess
<Files ~ "^\.ht">
Order allow,deny
Deny from all
</Files>
TypesConfig /etc/mime.types
DefaultType text/plain
<IfModule mod_mime_magic.c>
MIMEMagicFile conf/magic
</IfModule>
HostnameLookups Off
ErrorLog logs/error_log
LogLevel notice
LogFormat "%h" combined
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent
CustomLog "/dev/null" combined
That's asenine.
GeoIPEnable On
GeoIPDBFile "/usr/local/share/GeoIP/GeoIPCity.dat"
GeoIPScanProxyHeaders On
Is this in any way a heavy/expensive database ?
What is being logged by this ?
What is it used for ?
Depending on usage, it may be as bad as performing DNS lookups.
--
J.
