Thanks a lot David and all. regards Mahesh
On Fri, Jun 14, 2013 at 1:36 AM, David Lang <[email protected]> wrote: > On Thu, 13 Jun 2013, Mahesh V wrote: > > Hello David, >> >> Thanks for the reply. >> >> Here is another set of data points. >> A simple program (shamelessly copied from internet) which does nothing but >> inserts in mysql through rsyslog. >> >> #include <syslog.h> >> #include <unistd.h> >> void main() >> { >> int i = 0; >> setlogmask (LOG_UPTO (LOG_NOTICE)); >> openlog ("exampleprog", LOG_PID | LOG_NDELAY, LOG_DAEMON); >> for(i = 0; i < 10000; i++) { >> syslog (LOG_NOTICE, "A tree falls in a forest %d", i); >> usleep(100); >> } >> closelog (); >> } >> >> rsyslog.conf >> ---------------- >> $SystemLogRateLimitInterval 1 >> $SystemLogRateLimitBurst 5000 >> ## Optional >> >> program run >> ------------------- >> -bash-4.1# date; ./a.out; date >> Mon Jun 10 13:02:10 IST 2013 >> Mon Jun 10 13:02:12 IST 2013 >> >> Close to 5 minutes later >> ------------------------------**-------- >> mysql -u root -p >> mysql> select message from SystemEvents; >> ... >> ... >> imuxsock begins to drop messages from pid 16346 due to rate-limiting | >> imuxsock lost 1383 messages from pid 16346 due to rate-limiting | >> | A tree falls in a forest 9998 | >> | A tree falls in a forest 9999 | >> +-----------------------------**------------------------------** >> -----------+ >> 8619 rows in set (0.01 sec) >> >> I found that out of 10000, some 8616 records were inserted in 2 seconds. >> Wondering why imuxsock dropped messages even though rate limiting value >> was >> high. >> > > well, you have the rate limit set to 5000 per second, but you are sending > double that within 2 seconds. It doesn't seem surprising that you could > loose ~1400 of those messages. > > 5000 per second isn't especially high as far as rsyslog is concerned. > > personally, I never turn on rate limiting > > > Obviously there would be system limitation, but that should be the only >> criteria. >> > > raise the limit to something higher than the rate that you are sending > logs. > > > -bash-4.1# /sbin/rsyslogd -v >> rsyslogd 5.8.10, compiled with: >> >> this in no way gives disk spinning capacity but on a server with 4 cores, >> would this be the max I can reach? >> > > no, you just hit the max that you had configured. > > go to a very simple configuration, remove any rate limiting, just put in > the minimum needed to read the input and output to a file. on your 4 core > box you will get hundreds of thousands, if not millions, of logs per second > (and really only be using 2-3 cores to do so) > > David Lang > > thanks >> Mahesh >> >> >> >> >> On Wed, Jun 12, 2013 at 9:33 PM, David Lang <[email protected]> wrote: >> >> On Wed, 12 Jun 2013, [email protected] wrote: >>> >>> On Wed, Jun 12, 2013 at 07:44:50PM +0530, Mahesh V wrote: >>> >>>> >>>> Hello, >>>>> >>>>> The software i work, has syslog facility enabled for debug logs. >>>>> It generates a huge amount of logs (12000 lines in 2 minutes) - about >>>>> 100 >>>>> log entries per second and this is put into a mysql backend. >>>>> However, I see that after about 2 minutes of run, my application stalls >>>>> and if I disable (delete the SystemEvents table in Syslog database), my >>>>> application continues fine. >>>>> My my.cnf and rsyslog.conf is attached along. >>>>> Any ideas? >>>>> >>>>> Thanks >>>>> Mahesh >>>>> >>>>> >>>> >>>> Hi Mahesh, >>>> >>>> You sent a lot of software configuration information, but none on your >>>> DB hardware and I/O configurations. It sounds like you do not have >>>> enough >>>> capacity there, but this is complete speculation without the actual >>>> details. >>>> >>>> >>> expanding on this, you don't provide the details of your disks. 100 >>> transactions/sec sounds like you are doing this on a single 7200 rpm >>> disk. >>> Since MySQL wants to keep the data safe, it does a fsync as part of each >>> transaction. With a 7200 rpm disk it is doing just over 100 rotations/sec >>> and so it can't do fwyncs any faster than that. For high performance >>> database work you need a disk controller card with battery backed cache >>> configured for writeback. That way the controller can tell the OS the >>> write >>> is finished as soon as it's in the cache (where it will get written, even >>> if the system crashes, as long as the disks and controller get power >>> again >>> before the battery dies). This can change the transaction rate to tens of >>> thousands per second. >>> >>> In addition, rsyslog has the ability to write more than one message per >>> transation (batch mode), but you need to move to rsyslog 7.4 (well 7.3.5 >>> or >>> later). Once you do that you should experiment with the batch size. It's >>> very likely that you will want a batch size somewhere >1000 >>> messages/batch, >>> but it will depend on how fast your disks are able to keep up. >>> >>> David Lang >>> >>> ______________________________****_________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog> >>> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> >>> > >>> http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/> >>> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/> >>> > >>> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >>> ______________________________**_________________ >> rsyslog mailing list >> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> >> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> >> ______________________________**_________________ > rsyslog mailing list > http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> > http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/> > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
