On Thu, 13 Jun 2013, Mahesh V wrote:
Hello David,
Thanks for the reply.
Here is another set of data points.
A simple program (shamelessly copied from internet) which does nothing but
inserts in mysql through rsyslog.
#include <syslog.h>
#include <unistd.h>
void main()
{
int i = 0;
setlogmask (LOG_UPTO (LOG_NOTICE));
openlog ("exampleprog", LOG_PID | LOG_NDELAY, LOG_DAEMON);
for(i = 0; i < 10000; i++) {
syslog (LOG_NOTICE, "A tree falls in a forest %d", i);
usleep(100);
}
closelog ();
}
rsyslog.conf
----------------
$SystemLogRateLimitInterval 1
$SystemLogRateLimitBurst 5000
## Optional
program run
-------------------
-bash-4.1# date; ./a.out; date
Mon Jun 10 13:02:10 IST 2013
Mon Jun 10 13:02:12 IST 2013
Close to 5 minutes later
------------------------------**--------
mysql -u root -p
mysql> select message from SystemEvents;
...
...
imuxsock begins to drop messages from pid 16346 due to rate-limiting |
imuxsock lost 1383 messages from pid 16346 due to rate-limiting |
| A tree falls in a forest 9998 |
| A tree falls in a forest 9999 |
+-----------------------------**------------------------------**
-----------+
8619 rows in set (0.01 sec)
I found that out of 10000, some 8616 records were inserted in 2 seconds.
Wondering why imuxsock dropped messages even though rate limiting value
was
high.
well, you have the rate limit set to 5000 per second, but you are sending
double that within 2 seconds. It doesn't seem surprising that you could
loose ~1400 of those messages.
5000 per second isn't especially high as far as rsyslog is concerned.
personally, I never turn on rate limiting
Obviously there would be system limitation, but that should be the only
criteria.
raise the limit to something higher than the rate that you are sending
logs.
-bash-4.1# /sbin/rsyslogd -v
rsyslogd 5.8.10, compiled with:
this in no way gives disk spinning capacity but on a server with 4 cores,
would this be the max I can reach?
no, you just hit the max that you had configured.
go to a very simple configuration, remove any rate limiting, just put in
the minimum needed to read the input and output to a file. on your 4 core
box you will get hundreds of thousands, if not millions, of logs per second
(and really only be using 2-3 cores to do so)
David Lang
thanks
Mahesh
On Wed, Jun 12, 2013 at 9:33 PM, David Lang <[email protected]> wrote:
On Wed, 12 Jun 2013, [email protected] wrote:
On Wed, Jun 12, 2013 at 07:44:50PM +0530, Mahesh V wrote:
Hello,
The software i work, has syslog facility enabled for debug logs.
It generates a huge amount of logs (12000 lines in 2 minutes) - about
100
log entries per second and this is put into a mysql backend.
However, I see that after about 2 minutes of run, my application stalls
and if I disable (delete the SystemEvents table in Syslog database), my
application continues fine.
My my.cnf and rsyslog.conf is attached along.
Any ideas?
Thanks
Mahesh
Hi Mahesh,
You sent a lot of software configuration information, but none on your
DB hardware and I/O configurations. It sounds like you do not have
enough
capacity there, but this is complete speculation without the actual
details.
expanding on this, you don't provide the details of your disks. 100
transactions/sec sounds like you are doing this on a single 7200 rpm
disk.
Since MySQL wants to keep the data safe, it does a fsync as part of each
transaction. With a 7200 rpm disk it is doing just over 100 rotations/sec
and so it can't do fwyncs any faster than that. For high performance
database work you need a disk controller card with battery backed cache
configured for writeback. That way the controller can tell the OS the
write
is finished as soon as it's in the cache (where it will get written, even
if the system crashes, as long as the disks and controller get power
again
before the battery dies). This can change the transaction rate to tens of
thousands per second.
In addition, rsyslog has the ability to write more than one message per
transation (batch mode), but you need to move to rsyslog 7.4 (well 7.3.5
or
later). Once you do that you should experiment with the batch size. It's
very likely that you will want a batch size somewhere >1000
messages/batch,
but it will depend on how fast your disks are able to keep up.
David Lang
______________________________****_________________
rsyslog mailing list
http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog>
<http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/>
<http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
______________________________**_________________
rsyslog mailing list
http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
______________________________**_________________
rsyslog mailing list
http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
