On Wed, 12 Jun 2013, [email protected] wrote:
On Wed, Jun 12, 2013 at 07:44:50PM +0530, Mahesh V wrote:
Hello,
The software i work, has syslog facility enabled for debug logs.
It generates a huge amount of logs (12000 lines in 2 minutes) - about 100
log entries per second and this is put into a mysql backend.
However, I see that after about 2 minutes of run, my application stalls
and if I disable (delete the SystemEvents table in Syslog database), my
application continues fine.
My my.cnf and rsyslog.conf is attached along.
Any ideas?
Thanks
Mahesh
Hi Mahesh,
You sent a lot of software configuration information, but none on your
DB hardware and I/O configurations. It sounds like you do not have enough
capacity there, but this is complete speculation without the actual details.
expanding on this, you don't provide the details of your disks. 100
transactions/sec sounds like you are doing this on a single 7200 rpm disk. Since
MySQL wants to keep the data safe, it does a fsync as part of each transaction.
With a 7200 rpm disk it is doing just over 100 rotations/sec and so it can't do
fwyncs any faster than that. For high performance database work you need a disk
controller card with battery backed cache configured for writeback. That way the
controller can tell the OS the write is finished as soon as it's in the cache
(where it will get written, even if the system crashes, as long as the disks and
controller get power again before the battery dies). This can change the
transaction rate to tens of thousands per second.
In addition, rsyslog has the ability to write more than one message per
transation (batch mode), but you need to move to rsyslog 7.4 (well 7.3.5 or
later). Once you do that you should experiment with the batch size. It's very
likely that you will want a batch size somewhere >1000 messages/batch, but it
will depend on how fast your disks are able to keep up.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
