On 7/20/2011 4:08 PM, Grant wrote:
I ran into an out of memory problem. The first mention of it in the
kernel log is "mysqld invoked oom-killer". I haven't run into this
before. I do have a swap partition but I don't activate it based on
something I read previously that I later found out was wrong so I
suppose I should activate it. Is fstab the way to do that? I have a
commented line in there for swap.
Yes, just uncomment it and should be automatic. (you can use "swapon"
to enable it without rebooting)
Got it.
Can anyone tell how much swap this is:
/dev/sda2 80325 1140614 530145 82 Linux swap / Solaris
If it's something like 512MB, that may not have prevented me from
running out of memory since I have 4GB RAM. Is there any way to find
out if there was a memory leak or other problem that should be
investigated?
That's 512MB. You can also create a swap file to supplement the swap
partition if you don't want to or aren't able to repartition.
So I'm sure I have the concept right, is adding a 1GB swap partition
functionally identical to adding 1GB RAM with regard to the potential
for out-of-memory conditions?
I'd check the MySQL logs to see if it shows anything. Maybe check the
settings with regard to memory upper limits (Google it, there's a lot
of info about MySQL RAM management).
Nothing in the log and from what I read online, an error should be
logged if I reach mysql's memory limit.
If you're running any other servers that utilize MySQL like Apache or
something, check its access logs to see if you had an abnormal number
of connections. Bruteforce hacking or some kind of flooding/DOS attack
might cause it to use more memory than it ordinarily would.
It runs apache and I found some info there.
A Basic "what's using up my memory?" technique is to log the output of
"top" by using the -b command. Something like "top -b> toplog.txt".
Then you can go back to the time when the OOM occurred and see what
was using a lot of RAM at that time.
The kernel actually logged some top-like output and it looks like I
had a large number of apache2 processes running, likely 256 processes
which is the default MaxClients. The specified total_vm for each
process was about 67000 which means 256 x 67MB = 17GB???
I looked over my apache2 log and I was hit severely by a single IP
right as the server went down. However, that IP looks to be a
residential customer in the US and they engaged in normal browsing
behavior both before and after the disruption. I think that IP may
have done the refresh-100-times thing out of frustration as the server
started to go down.
Does it sound like apache2 was using up all the memory? If so, should
I look further for a catalyst or did this likely happen slowly? What
can I do to prevent it from happening again? Should I switch apache2
from prefork to threads?
Switching from prefork to threads and vice versa can be very difficult
depending on which modules and libraries your site uses. It is not on
the list of things you should try first. Or second. Maybe 37th.
I wouldn't expect adding swap to do much in this case. Your site gets
hit hard, Mysql is a bit slow, Apache processes start stacking up, the
system starts swapping, disk is really slow compared to RAM, and
everything grinds to a complete halt possibly locking the machine up.
The easiest thing to try is to turn off keepalives so child processes
aren't hanging around keeping connections up. Also lower the number of
Apache children to 8 * number of processors or a minimum of 32. Test a
bit. Turning off keep alive can cause problems for Flash based uploaders
to your site and code that expect the connection to stay up. For most
sites this shouldn't matter.
Next I'd look at tuning your Mysql config. If you've never touched
my.cnf, by default it's set to use 64MB IIRC. You may need to raise this
to get better performance. key_buffer and innodb_buffer_pool_size are
the only two I'd modify without knowing more.
kashani
