Hello David,
Thanks for the reply.
Here is another set of data points.
A simple program (shamelessly copied from internet) which does nothing but
inserts in mysql through rsyslog.
#include <syslog.h>
#include <unistd.h>
void main()
{
int i = 0;
setlogmask (LOG_UPTO (LOG_NOTICE));
openlog ("exampleprog", LOG_PID | LOG_NDELAY, LOG_DAEMON);
for(i = 0; i < 10000; i++) {
syslog (LOG_NOTICE, "A tree falls in a forest %d", i);
usleep(100);
}
closelog ();
}
rsyslog.conf
----------------
$SystemLogRateLimitInterval 1
$SystemLogRateLimitBurst 5000
## Optional
program run
-------------------
-bash-4.1# date; ./a.out; date
Mon Jun 10 13:02:10 IST 2013
Mon Jun 10 13:02:12 IST 2013
Close to 5 minutes later
--------------------------------------
mysql -u root -p
mysql> select message from SystemEvents;
...
...
imuxsock begins to drop messages from pid 16346 due to rate-limiting |
imuxsock lost 1383 messages from pid 16346 due to rate-limiting |
| A tree falls in a forest 9998 |
| A tree falls in a forest 9999 |
+----------------------------------------------------------------------+
8619 rows in set (0.01 sec)
I found that out of 10000, some 8616 records were inserted in 2 seconds.
Wondering why imuxsock dropped messages even though rate limiting value was
high.
Obviously there would be system limitation, but that should be the only
criteria.
-bash-4.1# /sbin/rsyslogd -v
rsyslogd 5.8.10, compiled with:
this in no way gives disk spinning capacity but on a server with 4 cores,
would this be the max I can reach?
thanks
Mahesh
On Wed, Jun 12, 2013 at 9:33 PM, David Lang <[email protected]> wrote:
> On Wed, 12 Jun 2013, [email protected] wrote:
>
> On Wed, Jun 12, 2013 at 07:44:50PM +0530, Mahesh V wrote:
>>
>>> Hello,
>>>
>>> The software i work, has syslog facility enabled for debug logs.
>>> It generates a huge amount of logs (12000 lines in 2 minutes) - about 100
>>> log entries per second and this is put into a mysql backend.
>>> However, I see that after about 2 minutes of run, my application stalls
>>> and if I disable (delete the SystemEvents table in Syslog database), my
>>> application continues fine.
>>> My my.cnf and rsyslog.conf is attached along.
>>> Any ideas?
>>>
>>> Thanks
>>> Mahesh
>>>
>>
>>
>> Hi Mahesh,
>>
>> You sent a lot of software configuration information, but none on your
>> DB hardware and I/O configurations. It sounds like you do not have enough
>> capacity there, but this is complete speculation without the actual
>> details.
>>
>
> expanding on this, you don't provide the details of your disks. 100
> transactions/sec sounds like you are doing this on a single 7200 rpm disk.
> Since MySQL wants to keep the data safe, it does a fsync as part of each
> transaction. With a 7200 rpm disk it is doing just over 100 rotations/sec
> and so it can't do fwyncs any faster than that. For high performance
> database work you need a disk controller card with battery backed cache
> configured for writeback. That way the controller can tell the OS the write
> is finished as soon as it's in the cache (where it will get written, even
> if the system crashes, as long as the disks and controller get power again
> before the battery dies). This can change the transaction rate to tens of
> thousands per second.
>
> In addition, rsyslog has the ability to write more than one message per
> transation (batch mode), but you need to move to rsyslog 7.4 (well 7.3.5 or
> later). Once you do that you should experiment with the batch size. It's
> very likely that you will want a batch size somewhere >1000 messages/batch,
> but it will depend on how fast your disks are able to keep up.
>
> David Lang
>
> ______________________________**_________________
> rsyslog mailing list
> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
