On Sat, Jun 15, 2013 at 9:49 AM, David Lang <[email protected]> wrote: > In this particular case, there was one action in the ruleset (a rather > ugly dynafile with a regex in the filename and two in the format string). > Based on this writeup it sounds as if multiple threads wouldn't help > (either on the mail or on the action queue) > > assuming the same number of actions, I am not understanding why there > would be any difference between trying to use multiple threads on a main > queue or an action queue. > > because more work happens on the main queue --> filtering, obtaining batches and a couple of minor other things.
Rainer > by the way, I think you mix up sync and async at one point in the post > > "The reason is that main queue workers process all filters for all rules > inside the rule set, as well as full action processing for all actions that > are not run on an synchronous (action) queue" > > shouldn't this be asynchronous (action) queue instead? > > > David Lang > > On Sat, 15 Jun 2013, Rainer Gerhards wrote: > > I managed to do some quick writeup. It's far from perfect, but may be >> useful: >> >> http://blog.gerhards.net/2013/**06/rsyslog-performance-main-** >> and-action.html<http://blog.gerhards.net/2013/06/rsyslog-performance-main-and-action.html> >> >> >> If you have suggestions/questions, we can always extend or add more. >> Family >> is now asking me to go off the computer ;) >> >> Rainer >> >> >> On Sat, Jun 15, 2013 at 10:00 AM, Rainer Gerhards >> <[email protected]>**wrote: >> >> The effect on action queues is limited. Main q workers are the primary >>> tool. Explaination in blog later. >>> >>> Sent from phone, thus brief. >>> Am 15.06.2013 09:46 schrieb "David Lang" <[email protected]>: >>> >>> after he setup action queues it was only doing one thread per action >>> >>>> queue, even after the maxthreads for the actionqueue was set to 64. >>>> >>>> David Lang >>>> >>>> On Sat, 15 Jun 2013, Rainer Gerhards wrote: >>>> >>>> Action or main q threads? >>>> >>>>> >>>>> Sent from phone, thus brief. >>>>> Am 15.06.2013 09:23 schrieb "David Lang" <[email protected]>: >>>>> >>>>> He had set the max threads to 64 and was still only seeing one thread >>>>> >>>>>> working. >>>>>> >>>>>> In a few of the later messages in the thread, his actionqueue config >>>>>> parameters are up near the top of the message. >>>>>> >>>>>> David Lang >>>>>> >>>>>> On Sat, 15 Jun 2013, Rainer Gerhards wrote: >>>>>> >>>>>> Requires a longer reply, need to lookup some things myself first - >>>>>> that's >>>>>> >>>>>> why I didn't throw in comments. >>>>>>> >>>>>>> But one very important thing: by default a queue has max 1 worker. >>>>>>> This >>>>>>> needs to be increased if multiple ones are desired. I guess that was >>>>>>> the >>>>>>> main point missing. If you create an async action queue, you also get >>>>>>> one >>>>>>> more thread. Probably that's why it looks how it looks. I don't know >>>>>>> the >>>>>>> parameters out of my head. >>>>>>> >>>>>>> Sent from phone, thus brief. >>>>>>> Am 15.06.2013 03:40 schrieb "David Lang" <[email protected]>: >>>>>>> >>>>>>> clarifying some questions that have come up in discussions around >>>>>>> >>>>>>> rulesets, thread, and queues >>>>>>>> >>>>>>>> When you tie a ruleset to a specific input, does that still use the >>>>>>>> same >>>>>>>> main queue? or will the logs from that input go to a different main >>>>>>>> queue >>>>>>>> (so the workers working on that queue will have no locking >>>>>>>> contention >>>>>>>> with >>>>>>>> workers working on logs from a different input), or do they get put >>>>>>>> on a >>>>>>>> single main queue. >>>>>>>> >>>>>>>> how do output threads and rulesets interact? >>>>>>>> >>>>>>>> from what was reported, it looks as if everything goes to one queue, >>>>>>>> and >>>>>>>> only if you set action queues per-ruleset will you end up with >>>>>>>> per-ruleset >>>>>>>> threads (in this case, actually per-actionqueue threads) >>>>>>>> >>>>>>>> Is omfile able to make use of multiple threads if you have a single >>>>>>>> action >>>>>>>> (in this case, using dynafiles) >>>>>>>> >>>>>>>> >>>>>>>> note, the documentation for omfile there is a bad entry >>>>>>>> http://www.rsyslog.com/doc/********omfile.html<http://www.rsyslog.com/doc/******omfile.html> >>>>>>>> <http://www.**rsyslog.com/doc/****omfile.**html<http://www.rsyslog.com/doc/****omfile.html> >>>>>>>> > >>>>>>>> <http://www.**rsyslog.com/doc/****omfile.html<http://rsyslog.com/doc/**omfile.html> >>>>>>>> <http://www.**rsyslog.com/doc/**omfile.html<http://www.rsyslog.com/doc/**omfile.html> >>>>>>>> > >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> <http://www.**rsyslog.com/doc/****omfile.html<http://rsyslog.com/doc/**omfile.html> >>>>>>>> <http://rsyslog.**com/doc/omfile.html<http://rsyslog.com/doc/omfile.html> >>>>>>>> > >>>>>>>> >>>>>>>> <http://www.**rsyslog.com/doc/**omfile.html<http://rsyslog.com/doc/omfile.html> >>>>>>>> <http://www.**rsyslog.com/doc/omfile.html<http://www.rsyslog.com/doc/omfile.html> >>>>>>>> > >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> * DynaFileCacheSize (not mandatory, default will be used) >>>>>>>> Defines a template to be used for the output. >>>>>>>> >>>>>>>> >>>>>>>> I think this is the wrong description and it would be good to say >>>>>>>> what >>>>>>>> the >>>>>>>> default is. >>>>>>>> >>>>>>>> David Lang >>>>>>>> ______________________________********_________________ >>>>>>>> rsyslog mailing list >>>>>>>> http://lists.adiscon.net/********mailman/listinfo/rsyslog<http://lists.adiscon.net/******mailman/listinfo/rsyslog> >>>>>>>> <http**://lists.adiscon.net/******mailman/listinfo/rsyslog<http://lists.adiscon.net/****mailman/listinfo/rsyslog> >>>>>>>> > >>>>>>>> <http:**//lists.adiscon.net/****mailman/**listinfo/rsyslog<http://lists.adiscon.net/**mailman/**listinfo/rsyslog> >>>>>>>> <htt**p://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog> >>>>>>>> > >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> <http:**//lists.adiscon.net/****mailman/**listinfo/rsyslog<http://lists.adiscon.net/**mailman/**listinfo/rsyslog> >>>>>>>> <htt**p://lists.adiscon.net/mailman/****listinfo/rsyslog<http://lists.adiscon.net/mailman/**listinfo/rsyslog> >>>>>>>> > >>>>>>>> <htt**p://lists.adiscon.net/**mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/**listinfo/rsyslog> >>>>>>>> <htt**p://lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> >>>>>>>> > >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> http://www.rsyslog.com/********professional-services/<http://www.rsyslog.com/******professional-services/> >>>>>>>>> <http://**www.rsyslog.com/******professional-services/<http://www.rsyslog.com/****professional-services/> >>>>>>>>> > >>>>>>>>> >>>>>>>> <http://**www.rsyslog.com/******professional-services/<http://www.rsyslog.com/****professional-services/> >>>>>>>> <http://**www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/> >>>>>>>> > >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> <http://**www.rsyslog.com/****professional-**services/<http://www.rsyslog.com/**professional-**services/> >>>>>>>> <http:**//www.rsyslog.com/**professional-**services/<http://www.rsyslog.com/professional-**services/> >>>>>>>> > >>>>>>>> <http:**//www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/> >>>>>>>> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/> >>>>>>>> > >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>> >>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>>> myriad >>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>>>>>> you >>>>>>>> DON'T LIKE THAT. >>>>>>>> >>>>>>>> ______________________________******_________________ >>>>>>>> >>>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/******mailman/listinfo/rsyslog<http://lists.adiscon.net/****mailman/listinfo/rsyslog> >>>>>>> <http:**//lists.adiscon.net/**mailman/**listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog> >>>>>>> > >>>>>>> <http:**//lists.adiscon.net/**mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/**listinfo/rsyslog> >>>>>>> <htt**p://lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> >>>>>>> > >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> http://www.rsyslog.com/******professional-services/<http://www.rsyslog.com/****professional-services/> >>>>>>> <http://**www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/> >>>>>>> > >>>>>>> <http://**www.rsyslog.com/**professional-**services/<http://www.rsyslog.com/professional-**services/> >>>>>>> <http:**//www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/> >>>>>>> > >>>>>>> >>>>>>>> >>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>> myriad >>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>>>>> you >>>>>>> DON'T LIKE THAT. >>>>>>> >>>>>>> ______________________________******_________________ >>>>>>> >>>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/******mailman/listinfo/rsyslog<http://lists.adiscon.net/****mailman/listinfo/rsyslog> >>>>>> <http:**//lists.adiscon.net/**mailman/**listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog> >>>>>> > >>>>>> <http:**//lists.adiscon.net/**mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/**listinfo/rsyslog> >>>>>> <htt**p://lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> >>>>>> > >>>>>> >>>>>>> >>>>>>> >>>>>>> http://www.rsyslog.com/******professional-services/<http://www.rsyslog.com/****professional-services/> >>>>>> <http://**www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/> >>>>>> > >>>>>> <http://**www.rsyslog.com/**professional-**services/<http://www.rsyslog.com/professional-**services/> >>>>>> <http:**//www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/> >>>>>> > >>>>>> >>>>>>> >>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>> myriad >>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>>> DON'T LIKE THAT. >>>>>> >>>>>> ______________________________****_________________ >>>>>> >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog> >>>>> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> >>>>> > >>>>> http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/> >>>>> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/> >>>>> > >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>> myriad >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>> DON'T LIKE THAT. >>>>> >>>>> ______________________________****_________________ >>>>> >>>> rsyslog mailing list >>>> http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog> >>>> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> >>>> > >>>> http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/> >>>> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/> >>>> > >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>> DON'T LIKE THAT. >>>> >>>> >>> ______________________________**_________________ >> rsyslog mailing list >> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> >> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> >> ______________________________**_________________ > rsyslog mailing list > http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> > http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/> > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
