Agreed. We don't need a direct queue before ruleset. I added it only to understand what's going with DIRECT Queues (Since its a very novel concept - Haven't heard something like this before)
If you remove the DIRECT queue from ruleset(in Scenario I), then modifications done in ruleset are visible to the main lane. It's working as expected. But since I am interested in understanding DIRECT Queues, I added it. >>> Also, although the maintainers will need to say for sure, user variable defined in a call may not be visible to the parent. Same as in most programming languages. Rsyslog works in a different way, in few scenarios, user variables defined in a call will be visible to the parent (If ruleset is not backed by Queue). So, its not similar to other languages ;) >>> "Note that mmnormalize should only be called once on each message. Behaviour is undefined if multiple calls to mmnormalize happen for the same message." Its a documentation bug, Rainer has raised a PR and will be resolved soon - https://github.com/rsyslog/rsyslog-doc/pull/931/files. Its safe to use it multiple times. On Fri, Sep 17, 2021 at 12:56 AM David Lang via rsyslog < [email protected]> wrote: > On Thu, 16 Sep 2021, Mariusz Kruk via rsyslog wrote: > > > > https://www.rsyslog.com/doc/v8-stable/configuration/modules/mmnormalize.html > > > > "Note that mmnormalize should only be called once on each message. > > Behaviour is undefined if multiple calls to mmnormalize happen for the > > same message." > > I called that out and it should have been removed from current > documentation. > > > From my tests it turns out that even with direct queue the message gets > > copied when entering a separate queue and thus the results are not > > inherited on the ruleset (and queue) exit. > > > > So it seems to be the mmnormalize that's causing the OP's variable to be > > retained after the ruleset exit. > > but if you don't specify any queue at all, then things inside a ruleset > will > affect things outside the ruleset. > > David Lang > > > On 16.09.2021 20:54, David Lang wrote: > >> There is no reason you can't call mmnormalize multiple times, you can > >> specify where to put the results (by default, they go under $!) > >> > >> I'm not really sure what a direct queue on a ruleset means, you should > >> just call the ruleset without it having any queue on it. > >> > >> David Lang > >> > >> On Thu, 16 Sep 2021, Mariusz Kruk via rsyslog wrote: > >> > >>> I believe it's not about queue, but about mmnormalize. I haven't used > >>> mmnormalize much myself but from the docs I understand that > >>> mmnormalize should only be called once on a message and is supposed > >>> to parse the message into a set of "global" properties. > >>> > >>> > >>> On 16.09.2021 15:21, rajeshksv via rsyslog wrote: > >>>> Hi Rsyslog Users, > >>>> > >>>> I am trying to understand how queues work in Rsyslog. In case of > >>>> non-direct > >>>> queues, message is copied and placed in the queue and message > >>>> modifications > >>>> done by queue workers won't have any impact on the original message. > >>>> Makes > >>>> sense here. > >>>> > >>>> However, I am a little confused when it comes to direct Queue. What > >>>> happens in Direct Queue ? Will the message be copied or is it the same > >>>> message ? I tested out two scenarios (wrt rulesets and actions) and > >>>> both > >>>> of them gave different results. > >>>> > >>>> When ruleset is backed by a direct Queue, message modifications done > in > >>>> ruleset don't reflect back in original flow where as in case of > actions > >>>> (such mmnormalize, mmkubernetes) which are by default backed by direct > >>>> Queue, message modifications done with action reflects in original > flow > >>>> > >>>> Scenario 1: > >>>> > >>>> template(name="abc" type="string" string="%$!var1% %$.var2% %msg%") > >>>> > >>>> ruleset(name="relay.htp1" queue.type="Direct") { > >>>> call rs1 > >>>> * // $!var1, $.var2 aren't available here* > >>>> action(type="omfile" file="/tmp/output.log" template="abc") > >>>> call relay.htp > >>>> } > >>>> > >>>> ruleset(name="rs1" queue.type="Direct"){ > >>>> set $!var1 = "hello"; > >>>> set $.var2 = "bye"; > >>>> } > >>>> > >>>> input(type="imfile" > >>>> File="/tmp/input.log" > >>>> Ruleset="relay.htp1" > >>>> Tag="tag") > >>>> > >>>> > >>>> Scenario 2: > >>>> > >>>> module(load = "mmnormalize") > >>>> ruleset(name = "relay.htp1" queue.type="Direct") { > >>>> action(type = "mmnormalize" > >>>> ruleBase="/etc/rsyslog.d/service.rulebase" > >>>> path="$!msg") > >>>> * // $!msg will be available here even though action is backed by a > >>>> default Queue. * > >>>> } > >>>> > >>>> input(type="imfile" > >>>> File="/tmp/input.log" > >>>> Ruleset="relay.htp1" > >>>> Tag="tag") > >>>> > >>>> > >>>> How come $!var1, $.var2 aren't available in scenario1 whereas $!msg is > >>>> available when both are using Direct Queue. Am I missing something > >>>> here ? > >>>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> https://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com/professional-services/ > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > >>> POST if you DON'T LIKE THAT. > >>> > > _______________________________________________ > > rsyslog mailing list > > https://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of > > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T > > LIKE THAT. > _______________________________________________ > rsyslog mailing list > https://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. -- Regards, Rajesh KSV _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
