what effect did this have on throughput and/or CPU load?
David Lang
On Thu, 20 Jun 2013, Boylan, James wrote:
The configuration maxes out at 1000. At 1000 I was able to see the rate of
closes drop for several thousand a minute to the hundreds. It was a
significant change.
- James
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of David Lang
Sent: Thursday, June 20, 2013 8:55 AM
To: rsyslog-users
Subject: Re: [rsyslog] imPTCP module
On Thu, 20 Jun 2013, Boylan, James wrote:
I had recently noticed the error from the config of
$InputPTCPServerHelperThreads and had commented it out when I did notice it.
Now that I know the correct option I've adjusted the configs
accordingly and it is running with the expected number of threads which is good
to see.
That aside, the increased DynafileCacheSize has definitely had a
positive improvement overall. We definitely appreciate the input we've
gotten trying to implement the tuning options.
how big a difference did this make?
David Lang
We're going to be looking at testing out 7.4.1 soon (We're running on 7.2.5 at
the moment) to see what kind of performance gains can be seen in the
improvements between those versions.
-- James
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of Rainer
Gerhards
Sent: Thursday, June 20, 2013 4:01 AM
To: rsyslog-users
Subject: Re: [rsyslog] imPTCP module
On Tue, Jun 18, 2013 at 3:30 PM, David Lang <[email protected]> wrote:
The overhead of the opens and closes is so high that I expect that
you just need to scale it to the point where you are keeping them open.
If it's set a lot larger than what you need it to be, it wastes
memory that you could use for other things (I don't know how much)
It's depending on buffer parameters. By default I think two 64k buffers (but I
may be wrong).
, and I guess if it's too large it could be expensive to search and
find that something isn't in there.
In current v7, that's no longer a problem, we have switched to a hash table
lookup. Seen some cases with low-thousands of open files and good performance
(that actually made us switch ;)).
But I would expect that these would be fairly minor effects. I don't
understand why the default is so low.
Stems back to pre-journald times, when we weighted SOHO vs. enterprise use
case. I should probably now go a bit higher.
Rainer
David Lang
On Tue, 18 Jun 2013, Boylan, James wrote:
We definitely do have many files being created.
I'm starting to do the strace and I see what you mean about tons of
open and close actions. At what point does increasing
DynaFileCacheSize actually start negatively impacting overall
performance? Is there a number that we should keep the cache size
under? Or does it just need to be scaled based on the performance of the
hardware it is running on?
-- James
-----Original Message-----
From:
[email protected].**com<[email protected]>[mailto:
rsyslog-bounces@lists.**adiscon.com
<[email protected]>]
On Behalf Of David Lang
Sent: Monday, June 17, 2013 4:07 PM
To: rsyslog-users
Subject: Re: [rsyslog] imPTCP module
On Mon, 17 Jun 2013, Boylan, James wrote:
Per David and Rainer's suggestion, I've cut us over to this module.
Definitely an improvement for performance.
I do have one question. The configuration option
$InputPTCPHelperThreads doesn't seem to do anything. I have it set
to 12 (It's a 23 core machine) but it only ever creates 3 threads for the
imptcp module.
I think it will use one thread per inbound connection, up to the max.
If I remember your prior posts, you only had a handful of systems
sending you connections, but they were sending them at very high
rates (I could very easily be mixing you up with the other team that
had thousands of hosts sending
connections)
But in any case, this shows that your bottleneck is not on the input
side (at least not with imptcp), it's on the output side where you
are using 8 threads, each using about 1/4 of a core.
This makes me think that you have problems in your ruleset that we
should look at optimizing.
Am I correct in remembering you as the one who started off with 480
very complex if statements and we simplified it down to ~30 if statements?
If so, one thing that you need to do is to increase the number of
different files that it keeps track of.
DynaFileCacheSize defaults to keeping track of 10 files. Since you
have
~500 files that you are writing to, I think that you need to set
this to
500 or higher.
I'll bet that if you were to do a strace of those main Q threads you
would find that they are doing a lot of opening and closing of files
(pretty close to every message), and increasing the
DynaFileCacheSize to something large enough to avoid that would
result in a very sharp decrease in the CPU needed, and an even
larger increase in the rate of messages written.
David Lang
26694 root 20 0 15.9g 7.9g 1480 S 26.8 16.8 3:44.63 rs:main
Q:Reg
26695 root 20 0 15.9g 7.9g 1480 R 26.3 16.8 3:44.89 rs:main
Q:Reg
26689 root 20 0 15.9g 7.9g 1480 S 23.8 16.8 3:46.23 rs:main
Q:Reg
26693 root 20 0 15.9g 7.9g 1480 S 23.5 16.8 3:45.76 rs:main
Q:Reg
26698 root 20 0 15.9g 7.9g 1480 S 23.5 16.8 3:44.26 rs:main
Q:Reg
26697 root 20 0 15.9g 7.9g 1480 S 22.8 16.8 3:43.07 rs:main
Q:Reg
26699 root 20 0 15.9g 7.9g 1480 S 22.8 16.8 3:45.14 rs:main
Q:Reg
26696 root 20 0 15.9g 7.9g 1480 S 22.0 16.8 3:46.56 rs:main
Q:Reg
26685 root 20 0 15.9g 7.9g 1480 S 1.8 16.8 0:48.19 in:imptcp
26690 root 20 0 15.9g 7.9g 1480 S 1.8 16.8 0:28.76 in:imptcp
26692 root 20 0 15.9g 7.9g 1480 S 1.0 16.8 0:26.70 in:imptcp
26682 root 20 0 15.9g 7.9g 1480 S 0.0 16.8 0:00.00 rsyslogd
26683 root 20 0 15.9g 7.9g 1480 S 0.0 16.8 0:00.00 in:immark
26684 root 20 0 15.9g 7.9g 1480 S 0.0 16.8 0:00.00 in:imudp
26686 root 20 0 15.9g 7.9g 1480 S 0.0 16.8 0:00.00 in:imuxsock
26687 root 20 0 15.9g 7.9g 1480 S 0.0 16.8 0:00.00 in:imklog
26688 root 20 0 15.9g 7.9g 1480 S 0.0 16.8 0:00.00 in:impstats
--James
______________________________**_________________
rsyslog mailing list
http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.ad
i
scon.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/**professional-services/<http://www.rsyslog.
c om/professional-services/> What's up with rsyslog? Follow
https://twitter.com/rgerhards NOTE
WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you DON'T LIKE THAT.
______________________________**_________________
rsyslog mailing list
http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adi
s
con.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/**professional-services/<http://www.rsyslog.c
o m/professional-services/> What's up with rsyslog? Follow
https://twitter.com/rgerhards NOTE WELL:
This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
______________________________**_________________
rsyslog mailing list
http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adi
s
con.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/**professional-services/<http://www.rsyslog.c
o m/professional-services/> What's up with rsyslog? Follow
https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing
list, posts are ARCHIVED by a myriad of sites beyond our control.
PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
______________________________**_________________
rsyslog mailing list
http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adis
c
on.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/**professional-services/<http://www.rsyslog.co
m /professional-services/> What's up with rsyslog? Follow
https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing
list, posts are ARCHIVED by a myriad of sites beyond our control.
PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is
a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our
control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites
beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is
a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our
control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
