6307 root 20 0 46.0g 39g 1332 R 99.6 84.3 127:43.74 rs:main Q:Reg 6299 root 20 0 46.0g 39g 1332 S 2.6 84.3 4:43.52 in:imtcp 6304 root 20 0 46.0g 39g 1332 S 1.3 84.3 2:36.95 in:imtcp 6306 root 20 0 46.0g 39g 1332 S 1.3 84.3 2:25.13 in:imtcp 6305 root 20 0 46.0g 39g 1332 S 1.0 84.3 2:30.65 in:imtcp 6308 root 20 0 46.0g 39g 1332 S 1.0 84.3 2:21.18 in:imtcp
Ive been reading http://www.rsyslog.com/doc/queues.html trying to understand how to make it spawn more que threads. On Fri, Jun 14, 2013 at 2:59 PM, David Lang <[email protected]> wrote: > Well, it is always possible that you have some other bottleneck in your > systems. I assumed that you were reporting that the number of connections > made a huge difference, even if the volume of logs remained pretty much the > same. > > before focusing on the queue configuration (which seldom causes the > problem), the first thing I would be trying to do is to see where rsyslog > is spending it's time. > > run top and enable thread views (the 'H' key) and see if one of the > threads is using 100% CPU or something close to that. If so, we will want > to figure out what that thread is doing. > > If it's the imtcp thread, then you could be running into an input limit. > But if it's one of the output threads, then the problem is that with your > configuration, rsyslog is not able to process the messages as fast as they > are arriving. That will then cause the queue to fill up and devices sending > via TCP to block. Increasing the queue size will probably not help (the > large queue may take a few more seconds to fill, but will still fill) > > Instead we would need to look at optimizing the processing of the messages. > > > David Lang > > On Fri, 14 Jun 2013, Timothy Ehlers wrote: > > I think the queue is not properly configured could this backup to not >> allowing connections? >> >> rsyslogd-pstats: main Q: size=39999989 enqueued=98847738 full=1805696 >> discarded.full=0 discarded.nf=0 maxqsize=40000000 >> >> >> >> On Fri, Jun 14, 2013 at 1:51 PM, David Lang <[email protected]> wrote: >> >> take a look at the imptcp module. My understanding is that it handles >>> large numbers of connections better than the plain imtcp module >>> >>> I have done very high traffic volume tests with rsyslog, but have not >>> been >>> in a position to test thousands of connections to one box. Rainer will >>> need >>> to comment on this. >>> >>> David Lang >>> >>> >>> On Fri, 14 Jun 2013, Timothy Ehlers wrote: >>> >>> Is there a recommended connection count per port. >>> >>>> >>>> We are sending to rsyslog using syslog-ng and this was working for a >>>> while. >>>> Suddenly syslog-ng is sending data which rsyslog is not acknowledging >>>> (tcpdump). >>>> >>>> We have a very high server count and were at ~3000 connections on port >>>> 20514, currently rsyslog seems to drag after ~500 connections. But it is >>>> slowly building them back up. >>>> >>>> Are we hitting some kind of wall? >>>> >>>> Version info: >>>> rsyslog-7.2.5 >>>> >>>> Ulimit info: >>>> ulimit -a >>>> core file size (blocks, -c) 0 >>>> data seg size (kbytes, -d) unlimited >>>> scheduling priority (-e) 0 >>>> file size (blocks, -f) unlimited >>>> pending signals (-i) 385913 >>>> max locked memory (kbytes, -l) 64 >>>> max memory size (kbytes, -m) unlimited >>>> open files (-n) 65536 >>>> pipe size (512 bytes, -p) 8 >>>> POSIX message queues (bytes, -q) 819200 >>>> real-time priority (-r) 0 >>>> stack size (kbytes, -s) 10240 >>>> cpu time (seconds, -t) unlimited >>>> max user processes (-u) 1024 >>>> virtual memory (kbytes, -v) unlimited >>>> file locks (-x) unlimited >>>> >>>> >>>> ______________________________****_________________ >>>> >>> rsyslog mailing list >>> http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog> >>> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> >>> > >>> http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/> >>> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/> >>> > >>> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >>> >> >> >> >> ______________________________**_________________ > rsyslog mailing list > http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> > http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/> > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > -- Tim Ehlers _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
