Thanks, that helps a lot. two (hopefully) quick fixes
1. can you fix the documentation for DynaFileCacheSize on http://www.rsyslog.com/doc/omfile.html to say what it's default size is, and if it's defined in bytes or in the number of filenames cached (reading the omfile.c code, it looks like it's defined in the number of filenames)
2. the thread name (as seen by top) for main Q worker threads is always just "main Q", how about changing it to "<rulestname> Q"?
David Lang On Sat, 15 Jun 2013, Rainer Gerhards wrote:
Date: Sat, 15 Jun 2013 12:09:34 +0200 From: Rainer Gerhards <[email protected]> Reply-To: rsyslog-users <[email protected]> To: rsyslog-users <[email protected]> Subject: Re: [rsyslog] rulesets, threads, queues I have updated the blog post (and fixed the typo): http://blog.gerhards.net/2013/06/rsyslog-performance-main-and-action.html Rainer On Sat, Jun 15, 2013 at 11:57 AM, Rainer Gerhards <[email protected]>wrote:On Sat, Jun 15, 2013 at 9:49 AM, David Lang <[email protected]> wrote:In this particular case, there was one action in the ruleset (a rather ugly dynafile with a regex in the filename and two in the format string). Based on this writeup it sounds as if multiple threads wouldn't help (either on the mail or on the action queue) assuming the same number of actions, I am not understanding why there would be any difference between trying to use multiple threads on a main queue or an action queue.because more work happens on the main queue --> filtering, obtaining batches and a couple of minor other things. Rainerby the way, I think you mix up sync and async at one point in the post "The reason is that main queue workers process all filters for all rules inside the rule set, as well as full action processing for all actions that are not run on an synchronous (action) queue" shouldn't this be asynchronous (action) queue instead? David Lang On Sat, 15 Jun 2013, Rainer Gerhards wrote: I managed to do some quick writeup. It's far from perfect, but may beuseful: http://blog.gerhards.net/2013/**06/rsyslog-performance-main-** and-action.html<http://blog.gerhards.net/2013/06/rsyslog-performance-main-and-action.html> If you have suggestions/questions, we can always extend or add more. Family is now asking me to go off the computer ;) Rainer On Sat, Jun 15, 2013 at 10:00 AM, Rainer Gerhards <[email protected]>**wrote: The effect on action queues is limited. Main q workers are the primarytool. Explaination in blog later. Sent from phone, thus brief. Am 15.06.2013 09:46 schrieb "David Lang" <[email protected]>: after he setup action queues it was only doing one thread per actionqueue, even after the maxthreads for the actionqueue was set to 64. David Lang On Sat, 15 Jun 2013, Rainer Gerhards wrote: Action or main q threads?Sent from phone, thus brief. Am 15.06.2013 09:23 schrieb "David Lang" <[email protected]>: He had set the max threads to 64 and was still only seeing one threadworking. In a few of the later messages in the thread, his actionqueue config parameters are up near the top of the message. David Lang On Sat, 15 Jun 2013, Rainer Gerhards wrote: Requires a longer reply, need to lookup some things myself first - that's why I didn't throw in comments.But one very important thing: by default a queue has max 1 worker. This needs to be increased if multiple ones are desired. I guess that was the main point missing. If you create an async action queue, you also get one more thread. Probably that's why it looks how it looks. I don't know the parameters out of my head. Sent from phone, thus brief. Am 15.06.2013 03:40 schrieb "David Lang" <[email protected]>: clarifying some questions that have come up in discussions around rulesets, thread, and queuesWhen you tie a ruleset to a specific input, does that still use the same main queue? or will the logs from that input go to a different main queue (so the workers working on that queue will have no locking contention with workers working on logs from a different input), or do they get put on a single main queue. how do output threads and rulesets interact? from what was reported, it looks as if everything goes to one queue, and only if you set action queues per-ruleset will you end up with per-ruleset threads (in this case, actually per-actionqueue threads) Is omfile able to make use of multiple threads if you have a single action (in this case, using dynafiles) note, the documentation for omfile there is a bad entry http://www.rsyslog.com/doc/********omfile.html<http://www.rsyslog.com/doc/******omfile.html> <http://www.**rsyslog.com/doc/****omfile.**html<http://www.rsyslog.com/doc/****omfile.html><http://www.**rsyslog.com/doc/****omfile.html<http://rsyslog.com/doc/**omfile.html> <http://www.**rsyslog.com/doc/**omfile.html<http://www.rsyslog.com/doc/**omfile.html><http://www.**rsyslog.com/doc/****omfile.html<http://rsyslog.com/doc/**omfile.html><http://rsyslog.**com/doc/omfile.html<http://rsyslog.com/doc/omfile.html><http://www.**rsyslog.com/doc/**omfile.html<http://rsyslog.com/doc/omfile.html> <http://www.**rsyslog.com/doc/omfile.html<http://www.rsyslog.com/doc/omfile.html>* DynaFileCacheSize (not mandatory, default will be used)Defines a template to be used for the output. I think this is the wrong description and it would be good to say what the default is. David Lang ______________________________********_________________ rsyslog mailing list http://lists.adiscon.net/********mailman/listinfo/rsyslog<http://lists.adiscon.net/******mailman/listinfo/rsyslog> <http**://lists.adiscon.net/******mailman/listinfo/rsyslog<http://lists.adiscon.net/****mailman/listinfo/rsyslog><http:**//lists.adiscon.net/****mailman/**listinfo/rsyslog<http://lists.adiscon.net/**mailman/**listinfo/rsyslog> <htt**p://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog><http:**//lists.adiscon.net/****mailman/**listinfo/rsyslog<http://lists.adiscon.net/**mailman/**listinfo/rsyslog><htt**p://lists.adiscon.net/mailman/****listinfo/rsyslog<http://lists.adiscon.net/mailman/**listinfo/rsyslog><htt**p://lists.adiscon.net/**mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/**listinfo/rsyslog> <htt**p://lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>http://www.rsyslog.com/********professional-services/<http://www.rsyslog.com/******professional-services/> <http://**www.rsyslog.com/******professional-services/<http://www.rsyslog.com/****professional-services/><http://**www.rsyslog.com/******professional-services/<http://www.rsyslog.com/****professional-services/> <http://**www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/><http://**www.rsyslog.com/****professional-**services/<http://www.rsyslog.com/**professional-**services/><http:**//www.rsyslog.com/**professional-**services/<http://www.rsyslog.com/professional-**services/><http:**//www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/>What's up with rsyslog? Follow https://twitter.com/rgerhardsNOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ______________________________******_________________ rsyslog mailing listhttp://lists.adiscon.net/******mailman/listinfo/rsyslog<http://lists.adiscon.net/****mailman/listinfo/rsyslog> <http:**//lists.adiscon.net/**mailman/**listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog><http:**//lists.adiscon.net/**mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/**listinfo/rsyslog> <htt**p://lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>http://www.rsyslog.com/******professional-services/<http://www.rsyslog.com/****professional-services/><http://**www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/><http://**www.rsyslog.com/**professional-**services/<http://www.rsyslog.com/professional-**services/> <http:**//www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>What's up with rsyslog? Follow https://twitter.com/rgerhardsNOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ______________________________******_________________ rsyslog mailing listhttp://lists.adiscon.net/******mailman/listinfo/rsyslog<http://lists.adiscon.net/****mailman/listinfo/rsyslog> <http:**//lists.adiscon.net/**mailman/**listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog><http:**//lists.adiscon.net/**mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/**listinfo/rsyslog> <htt**p://lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>http://www.rsyslog.com/******professional-services/<http://www.rsyslog.com/****professional-services/><http://**www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/><http://**www.rsyslog.com/**professional-**services/<http://www.rsyslog.com/professional-**services/> <http:**//www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>What's up with rsyslog? Follow https://twitter.com/rgerhardsNOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ______________________________****_________________rsyslog mailing list http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/>What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ______________________________****_________________rsyslog mailing list http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/>What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.______________________________**_________________rsyslog mailing list http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ______________________________**_________________rsyslog mailing list http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT._______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
