On Fri, 14 Jun 2013, Timothy Ehlers wrote:
6307 root 20 0 46.0g 39g 1332 R 99.6 84.3 127:43.74 rs:main Q:Reg
6299 root 20 0 46.0g 39g 1332 S 2.6 84.3 4:43.52 in:imtcp
6304 root 20 0 46.0g 39g 1332 S 1.3 84.3 2:36.95 in:imtcp
6306 root 20 0 46.0g 39g 1332 S 1.3 84.3 2:25.13 in:imtcp
6305 root 20 0 46.0g 39g 1332 S 1.0 84.3 2:30.65 in:imtcp
6308 root 20 0 46.0g 39g 1332 S 1.0 84.3 2:21.18 in:imtcp
Ive been reading http://www.rsyslog.com/doc/queues.html trying to
understand how to make it spawn more que threads.
Ok, this tells us that the problem is not in the tcp input to rsyslog, but
rather in the output.
Before you go too far in trying to spawn multiple threads, can we look at your
configuration to see if there's a way to optimize it?
any idea how many messages/sec you are getting?
David Lang
On Fri, Jun 14, 2013 at 2:59 PM, David Lang <[email protected]> wrote:
Well, it is always possible that you have some other bottleneck in your
systems. I assumed that you were reporting that the number of connections
made a huge difference, even if the volume of logs remained pretty much the
same.
before focusing on the queue configuration (which seldom causes the
problem), the first thing I would be trying to do is to see where rsyslog
is spending it's time.
run top and enable thread views (the 'H' key) and see if one of the
threads is using 100% CPU or something close to that. If so, we will want
to figure out what that thread is doing.
If it's the imtcp thread, then you could be running into an input limit.
But if it's one of the output threads, then the problem is that with your
configuration, rsyslog is not able to process the messages as fast as they
are arriving. That will then cause the queue to fill up and devices sending
via TCP to block. Increasing the queue size will probably not help (the
large queue may take a few more seconds to fill, but will still fill)
Instead we would need to look at optimizing the processing of the messages.
David Lang
On Fri, 14 Jun 2013, Timothy Ehlers wrote:
I think the queue is not properly configured could this backup to not
allowing connections?
rsyslogd-pstats: main Q: size=39999989 enqueued=98847738 full=1805696
discarded.full=0 discarded.nf=0 maxqsize=40000000
On Fri, Jun 14, 2013 at 1:51 PM, David Lang <[email protected]> wrote:
take a look at the imptcp module. My understanding is that it handles
large numbers of connections better than the plain imtcp module
I have done very high traffic volume tests with rsyslog, but have not
been
in a position to test thousands of connections to one box. Rainer will
need
to comment on this.
David Lang
On Fri, 14 Jun 2013, Timothy Ehlers wrote:
Is there a recommended connection count per port.
We are sending to rsyslog using syslog-ng and this was working for a
while.
Suddenly syslog-ng is sending data which rsyslog is not acknowledging
(tcpdump).
We have a very high server count and were at ~3000 connections on port
20514, currently rsyslog seems to drag after ~500 connections. But it is
slowly building them back up.
Are we hitting some kind of wall?
Version info:
rsyslog-7.2.5
Ulimit info:
ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 385913
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 65536
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 1024
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
______________________________****_________________
rsyslog mailing list
http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog>
<http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/>
<http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
______________________________**_________________
rsyslog mailing list
http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
