with omfwd you need queues only when the file system is remote and either unreliable or slow. Multiple threads never make sense with omfwd - we have sequential writes, and so omfwd serializes them in order to avoid incomplete lines (which could happen by multiple non-serialized writes).
in short: do not use queues with omfwd HTH Rainer El mié, 7 abr 2021 a las 10:44, siddhartha dewan (<[email protected]>) escribió: > > Thank you for getting back. Makes sense. One more question related to this. > If we dont need logs in the order they were received, then what is beneficial > - multiple threads in ruleset queue vs multiple threads in action queue. I > read your blog but still have some doubts. > Assume we don’t need any filtering. And there are 2 omfwd actions sending > logs to tcp destinations within the ruleset. > In this scenario - should i create a ruleset queue with 4 threads(example) or > create the actions each with 2 threads(example). > Or is there a benefit to having both. Of course am assuming large mps like > 500k. > > Regards, > Sid > > On Wed, Apr 7, 2021 at 12:51 AM Rainer Gerhards <[email protected]> > wrote: >> >> The actions are simply far too fast to see any difference. For that >> case, a queue would just add overhead. It's not for actions that fast. >> >> HTH >> Rainer >> >> El mié, 7 abr 2021 a las 5:02, siddhartha dewan via rsyslog >> (<[email protected]>) escribió: >> > >> > Hi, >> > this is my 1st time reaching out to the public community - so pls advise if >> > am doing something wrong. >> > My question is related to ruleset queues and parallelism. After reviewing >> > all the documentation on queues and reading them multiple times - the whole >> > idea behind the queue with a fixed or linked list was to introduce >> > asynchronous processing of actions within the ruleset which also helps with >> > handling costly filter logic. >> > >> > In order to test the benefits of queues - i created a simple ruleset >> > without a queue 1st: >> > ruleset(name="test") { >> > action(type="omfile" file="/var/log/mtu3.log") >> > action(type="omfile" file="/var/log/mtu4.log") >> > } >> > >> > As per the documentation - my expectation was - because i didn't define any >> > queues - the action would be processed synchronously - or - it will be >> > processed 1 after another. Meaning - all the syslog packets will be stored >> > in the main queue - which then will invoke the 1st action and once the 1st >> > action is complete - it will then invoke the 2nd action. >> > >> > To my surprise - the data was written in parallel to both these files. And >> > now i am totally confused - if the data is being written in parallel - how >> > exactly is the cpu thread working: >> > is it writing to both files at once? If yes - then what really is the >> > benefit of a dedicated queue? >> > In the documentation - the biggest benefit seems to be avoiding a slow >> > output - and also helping provide parallel threads to do work while >> > increasing performance. >> > >> > Yes - I understand that the slow output will eventually cause a backlog in >> > the main queue causing the entire rsyslog to hang - but is that the only >> > benefit of a queue? Performance doesnt really seem to be a benefit if the >> > main queue can already do things in parallel - till rsyslog eventually >> > hangs due to backlog. >> > >> > Your team has done an awesome job helping out with an open source solution. >> > Pls keep it up. >> > _______________________________________________ >> > rsyslog mailing list >> > https://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com/professional-services/ >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> > DON'T LIKE THAT. _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
