Hmm...
OK, so a simple setup of:
[cut]
ruleset(name="whatever" queue.type="direct") {
set $.msg="whatever";
}
set $.msg=$msg;
call whatever
template using $.msg
output with the template
[cut]
Which from my tests shows that $.msg retains the $msg value even though
it was supposed to be overwritten by the "whatever" ruleset is caused
not by the queue mechanics but by non-linearity of execution and race?
MK
On 17.09.2021 08:55, Rainer Gerhards via rsyslog wrote:
The default queue mode is "direct" - no matter if it is explicitly
specified or not.
There are indeed semantic differences between the "direct" queue
(a.k.a. "no real queue exists") and any other (real) queue type. As
you say, in direct mode, the message object is NOT copied. So anything
done to it, will potentially affect other uses of the very same
message object. I real queues, the message is copied and thus no
processing outside the scope of the queue is affected.
NOTE WELL: if you have parts with direct AND real queues AND modify
messages in "direct queue constructs" AND have actions/rulesets with
real queues running in parallel, you have a race. If the modification
made by the direct queue part depends on whether processing of the
real queues has already started. For "regular" rule sets this is no
problem, because main processing is sequential. However, if you do
very complex setups with multipele rule sets calling into each other
and using different queue modes, this can affect you.
It is best, also for maintainability, to have a simple sequential
structure in top level processing, spawning off topic rule sets (if at
all required). then do all "global" modifications at the beginning.
And do local (intentionally non outside-visible) modifications in the
topic rule sets.
I hope this helps,
Rainer
El vie, 17 sept 2021 a las 6:17, rajeshksv via rsyslog
(<[email protected]>) escribió:
Agreed. We don't need a direct queue before ruleset. I added it only to
understand what's going with DIRECT Queues (Since its a very novel concept
- Haven't heard something like this before)
If you remove the DIRECT queue from ruleset(in Scenario I), then
modifications done in ruleset are visible to the main lane. It's working as
expected. But since I am interested in understanding DIRECT Queues, I added
it.
Also, although the maintainers will need to say for sure, user variable
defined in a call may not be visible to the parent. Same as in most
programming languages.
Rsyslog works in a different way, in few scenarios, user variables defined
in a call will be visible to the parent (If ruleset is not backed by
Queue). So, its not similar to other languages ;)
"Note that mmnormalize should only be called once on each message.
Behaviour is undefined if multiple calls to mmnormalize happen for the
same message."
Its a documentation bug, Rainer has raised a PR and will be resolved soon -
https://github.com/rsyslog/rsyslog-doc/pull/931/files. Its safe to use it
multiple times.
On Fri, Sep 17, 2021 at 12:56 AM David Lang via rsyslog <
[email protected]> wrote:
On Thu, 16 Sep 2021, Mariusz Kruk via rsyslog wrote:
https://www.rsyslog.com/doc/v8-stable/configuration/modules/mmnormalize.html
"Note that mmnormalize should only be called once on each message.
Behaviour is undefined if multiple calls to mmnormalize happen for the
same message."
I called that out and it should have been removed from current
documentation.
From my tests it turns out that even with direct queue the message gets
copied when entering a separate queue and thus the results are not
inherited on the ruleset (and queue) exit.
So it seems to be the mmnormalize that's causing the OP's variable to be
retained after the ruleset exit.
but if you don't specify any queue at all, then things inside a ruleset
will
affect things outside the ruleset.
David Lang
On 16.09.2021 20:54, David Lang wrote:
There is no reason you can't call mmnormalize multiple times, you can
specify where to put the results (by default, they go under $!)
I'm not really sure what a direct queue on a ruleset means, you should
just call the ruleset without it having any queue on it.
David Lang
On Thu, 16 Sep 2021, Mariusz Kruk via rsyslog wrote:
I believe it's not about queue, but about mmnormalize. I haven't used
mmnormalize much myself but from the docs I understand that
mmnormalize should only be called once on a message and is supposed
to parse the message into a set of "global" properties.
On 16.09.2021 15:21, rajeshksv via rsyslog wrote:
Hi Rsyslog Users,
I am trying to understand how queues work in Rsyslog. In case of
non-direct
queues, message is copied and placed in the queue and message
modifications
done by queue workers won't have any impact on the original message.
Makes
sense here.
However, I am a little confused when it comes to direct Queue. What
happens in Direct Queue ? Will the message be copied or is it the same
message ? I tested out two scenarios (wrt rulesets and actions) and
both
of them gave different results.
When ruleset is backed by a direct Queue, message modifications done
in
ruleset don't reflect back in original flow where as in case of
actions
(such mmnormalize, mmkubernetes) which are by default backed by direct
Queue, message modifications done with action reflects in original
flow
Scenario 1:
template(name="abc" type="string" string="%$!var1% %$.var2% %msg%")
ruleset(name="relay.htp1" queue.type="Direct") {
call rs1
* // $!var1, $.var2 aren't available here*
action(type="omfile" file="/tmp/output.log" template="abc")
call relay.htp
}
ruleset(name="rs1" queue.type="Direct"){
set $!var1 = "hello";
set $.var2 = "bye";
}
input(type="imfile"
File="/tmp/input.log"
Ruleset="relay.htp1"
Tag="tag")
Scenario 2:
module(load = "mmnormalize")
ruleset(name = "relay.htp1" queue.type="Direct") {
action(type = "mmnormalize"
ruleBase="/etc/rsyslog.d/service.rulebase"
path="$!msg")
* // $!msg will be available here even though action is backed by a
default Queue. *
}
input(type="imfile"
File="/tmp/input.log"
Ruleset="relay.htp1"
Tag="tag")
How come $!var1, $.var2 aren't available in scenario1 whereas $!msg is
available when both are using Direct Queue. Am I missing something
here ?
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T
LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
--
Regards,
Rajesh KSV
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
