I have updated the blog post (and fixed the typo): http://blog.gerhards.net/2013/06/rsyslog-performance-main-and-action.html
Rainer On Sat, Jun 15, 2013 at 11:57 AM, Rainer Gerhards <[email protected]>wrote: > > > > On Sat, Jun 15, 2013 at 9:49 AM, David Lang <[email protected]> wrote: > >> In this particular case, there was one action in the ruleset (a rather >> ugly dynafile with a regex in the filename and two in the format string). >> Based on this writeup it sounds as if multiple threads wouldn't help >> (either on the mail or on the action queue) >> >> assuming the same number of actions, I am not understanding why there >> would be any difference between trying to use multiple threads on a main >> queue or an action queue. >> >> > because more work happens on the main queue --> filtering, obtaining > batches and a couple of minor other things. > > Rainer > >> by the way, I think you mix up sync and async at one point in the post >> >> "The reason is that main queue workers process all filters for all rules >> inside the rule set, as well as full action processing for all actions that >> are not run on an synchronous (action) queue" >> >> shouldn't this be asynchronous (action) queue instead? >> >> >> David Lang >> >> On Sat, 15 Jun 2013, Rainer Gerhards wrote: >> >> I managed to do some quick writeup. It's far from perfect, but may be >>> useful: >>> >>> http://blog.gerhards.net/2013/**06/rsyslog-performance-main-** >>> and-action.html<http://blog.gerhards.net/2013/06/rsyslog-performance-main-and-action.html> >>> >>> >>> If you have suggestions/questions, we can always extend or add more. >>> Family >>> is now asking me to go off the computer ;) >>> >>> Rainer >>> >>> >>> On Sat, Jun 15, 2013 at 10:00 AM, Rainer Gerhards >>> <[email protected]>**wrote: >>> >>> The effect on action queues is limited. Main q workers are the primary >>>> tool. Explaination in blog later. >>>> >>>> Sent from phone, thus brief. >>>> Am 15.06.2013 09:46 schrieb "David Lang" <[email protected]>: >>>> >>>> after he setup action queues it was only doing one thread per action >>>> >>>>> queue, even after the maxthreads for the actionqueue was set to 64. >>>>> >>>>> David Lang >>>>> >>>>> On Sat, 15 Jun 2013, Rainer Gerhards wrote: >>>>> >>>>> Action or main q threads? >>>>> >>>>>> >>>>>> Sent from phone, thus brief. >>>>>> Am 15.06.2013 09:23 schrieb "David Lang" <[email protected]>: >>>>>> >>>>>> He had set the max threads to 64 and was still only seeing one thread >>>>>> >>>>>>> working. >>>>>>> >>>>>>> In a few of the later messages in the thread, his actionqueue config >>>>>>> parameters are up near the top of the message. >>>>>>> >>>>>>> David Lang >>>>>>> >>>>>>> On Sat, 15 Jun 2013, Rainer Gerhards wrote: >>>>>>> >>>>>>> Requires a longer reply, need to lookup some things myself first - >>>>>>> that's >>>>>>> >>>>>>> why I didn't throw in comments. >>>>>>>> >>>>>>>> But one very important thing: by default a queue has max 1 worker. >>>>>>>> This >>>>>>>> needs to be increased if multiple ones are desired. I guess that >>>>>>>> was >>>>>>>> the >>>>>>>> main point missing. If you create an async action queue, you also >>>>>>>> get >>>>>>>> one >>>>>>>> more thread. Probably that's why it looks how it looks. I don't >>>>>>>> know >>>>>>>> the >>>>>>>> parameters out of my head. >>>>>>>> >>>>>>>> Sent from phone, thus brief. >>>>>>>> Am 15.06.2013 03:40 schrieb "David Lang" <[email protected]>: >>>>>>>> >>>>>>>> clarifying some questions that have come up in discussions around >>>>>>>> >>>>>>>> rulesets, thread, and queues >>>>>>>>> >>>>>>>>> When you tie a ruleset to a specific input, does that still use the >>>>>>>>> same >>>>>>>>> main queue? or will the logs from that input go to a different main >>>>>>>>> queue >>>>>>>>> (so the workers working on that queue will have no locking >>>>>>>>> contention >>>>>>>>> with >>>>>>>>> workers working on logs from a different input), or do they get put >>>>>>>>> on a >>>>>>>>> single main queue. >>>>>>>>> >>>>>>>>> how do output threads and rulesets interact? >>>>>>>>> >>>>>>>>> from what was reported, it looks as if everything goes to one >>>>>>>>> queue, >>>>>>>>> and >>>>>>>>> only if you set action queues per-ruleset will you end up with >>>>>>>>> per-ruleset >>>>>>>>> threads (in this case, actually per-actionqueue threads) >>>>>>>>> >>>>>>>>> Is omfile able to make use of multiple threads if you have a single >>>>>>>>> action >>>>>>>>> (in this case, using dynafiles) >>>>>>>>> >>>>>>>>> >>>>>>>>> note, the documentation for omfile there is a bad entry >>>>>>>>> http://www.rsyslog.com/doc/********omfile.html<http://www.rsyslog.com/doc/******omfile.html> >>>>>>>>> <http://www.**rsyslog.com/doc/****omfile.**html<http://www.rsyslog.com/doc/****omfile.html> >>>>>>>>> > >>>>>>>>> <http://www.**rsyslog.com/doc/****omfile.html<http://rsyslog.com/doc/**omfile.html> >>>>>>>>> <http://www.**rsyslog.com/doc/**omfile.html<http://www.rsyslog.com/doc/**omfile.html> >>>>>>>>> > >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> <http://www.**rsyslog.com/doc/****omfile.html<http://rsyslog.com/doc/**omfile.html> >>>>>>>>> <http://rsyslog.**com/doc/omfile.html<http://rsyslog.com/doc/omfile.html> >>>>>>>>> > >>>>>>>>> >>>>>>>>> <http://www.**rsyslog.com/doc/**omfile.html<http://rsyslog.com/doc/omfile.html> >>>>>>>>> <http://www.**rsyslog.com/doc/omfile.html<http://www.rsyslog.com/doc/omfile.html> >>>>>>>>> > >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> * DynaFileCacheSize (not mandatory, default will be used) >>>>>>>>> Defines a template to be used for the output. >>>>>>>>> >>>>>>>>> >>>>>>>>> I think this is the wrong description and it would be good to say >>>>>>>>> what >>>>>>>>> the >>>>>>>>> default is. >>>>>>>>> >>>>>>>>> David Lang >>>>>>>>> ______________________________********_________________ >>>>>>>>> rsyslog mailing list >>>>>>>>> http://lists.adiscon.net/********mailman/listinfo/rsyslog<http://lists.adiscon.net/******mailman/listinfo/rsyslog> >>>>>>>>> <http**://lists.adiscon.net/******mailman/listinfo/rsyslog<http://lists.adiscon.net/****mailman/listinfo/rsyslog> >>>>>>>>> > >>>>>>>>> <http:**//lists.adiscon.net/****mailman/**listinfo/rsyslog<http://lists.adiscon.net/**mailman/**listinfo/rsyslog> >>>>>>>>> <htt**p://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog> >>>>>>>>> > >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> <http:**//lists.adiscon.net/****mailman/**listinfo/rsyslog<http://lists.adiscon.net/**mailman/**listinfo/rsyslog> >>>>>>>>> <htt**p://lists.adiscon.net/mailman/****listinfo/rsyslog<http://lists.adiscon.net/mailman/**listinfo/rsyslog> >>>>>>>>> > >>>>>>>>> <htt**p://lists.adiscon.net/**mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/**listinfo/rsyslog> >>>>>>>>> <htt**p://lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> >>>>>>>>> > >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> http://www.rsyslog.com/********professional-services/<http://www.rsyslog.com/******professional-services/> >>>>>>>>>> <http://**www.rsyslog.com/******professional-services/<http://www.rsyslog.com/****professional-services/> >>>>>>>>>> > >>>>>>>>>> >>>>>>>>> <http://**www.rsyslog.com/******professional-services/<http://www.rsyslog.com/****professional-services/> >>>>>>>>> <http://**www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/> >>>>>>>>> > >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> <http://**www.rsyslog.com/****professional-**services/<http://www.rsyslog.com/**professional-**services/> >>>>>>>>> <http:**//www.rsyslog.com/**professional-**services/<http://www.rsyslog.com/professional-**services/> >>>>>>>>> > >>>>>>>>> <http:**//www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/> >>>>>>>>> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/> >>>>>>>>> > >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>>> >>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>>>> myriad >>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>>>>>>> you >>>>>>>>> DON'T LIKE THAT. >>>>>>>>> >>>>>>>>> ______________________________******_________________ >>>>>>>>> >>>>>>>>> rsyslog mailing list >>>>>>>> http://lists.adiscon.net/******mailman/listinfo/rsyslog<http://lists.adiscon.net/****mailman/listinfo/rsyslog> >>>>>>>> <http:**//lists.adiscon.net/**mailman/**listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog> >>>>>>>> > >>>>>>>> <http:**//lists.adiscon.net/**mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/**listinfo/rsyslog> >>>>>>>> <htt**p://lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> >>>>>>>> > >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> http://www.rsyslog.com/******professional-services/<http://www.rsyslog.com/****professional-services/> >>>>>>>> <http://**www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/> >>>>>>>> > >>>>>>>> <http://**www.rsyslog.com/**professional-**services/<http://www.rsyslog.com/professional-**services/> >>>>>>>> <http:**//www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/> >>>>>>>> > >>>>>>>> >>>>>>>>> >>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>>> myriad >>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>>>>>> you >>>>>>>> DON'T LIKE THAT. >>>>>>>> >>>>>>>> ______________________________******_________________ >>>>>>>> >>>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/******mailman/listinfo/rsyslog<http://lists.adiscon.net/****mailman/listinfo/rsyslog> >>>>>>> <http:**//lists.adiscon.net/**mailman/**listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog> >>>>>>> > >>>>>>> <http:**//lists.adiscon.net/**mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/**listinfo/rsyslog> >>>>>>> <htt**p://lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> >>>>>>> > >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> http://www.rsyslog.com/******professional-services/<http://www.rsyslog.com/****professional-services/> >>>>>>> <http://**www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/> >>>>>>> > >>>>>>> <http://**www.rsyslog.com/**professional-**services/<http://www.rsyslog.com/professional-**services/> >>>>>>> <http:**//www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/> >>>>>>> > >>>>>>> >>>>>>>> >>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>> myriad >>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>>>>> you >>>>>>> DON'T LIKE THAT. >>>>>>> >>>>>>> ______________________________****_________________ >>>>>>> >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog> >>>>>> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> >>>>>> > >>>>>> http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/> >>>>>> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/> >>>>>> > >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>> myriad >>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>>> DON'T LIKE THAT. >>>>>> >>>>>> ______________________________****_________________ >>>>>> >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog> >>>>> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> >>>>> > >>>>> http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/> >>>>> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/> >>>>> > >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>> myriad >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>> DON'T LIKE THAT. >>>>> >>>>> >>>> ______________________________**_________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> >>> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >>> ______________________________**_________________ >> rsyslog mailing list >> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> >> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
