On 07/15/2013 11:07 PM, David Lang wrote:
On Mon, 15 Jul 2013, Erik Steffl wrote:
On 07/15/2013 09:37 PM, David Lang wrote:
On Mon, 15 Jul 2013, Erik Steffl wrote:
Here's what I'm trying to do:
receive cee message, put it into a json data structure and forward
the whole thing to another server.
input: <135> time host tag @cee{"orignal":"message"}
output: <135> time host tag @cee{"cluster":"someName", "message":
{"original":"message"}}
So what I did was:
set $!myClusterName = "someName";
then use these as part of the template:
constant(value="\"cluster\":{")
property("$!myClusterName")
...
constant(value="\"message\":")
property("$!all-json")
and it almost works but $!myClusterName becomes part of the
$!all-json. Looked at docs but it seems all variables that are set
using set keyword become part of $!all-json.
using property("msg") also sort of works but it includes @cee (which
I don't think makes sense in that part of message, it's already all
json).
Any way to get $!all-json without the custom variables or "msg"
property without @cee? Or should I be forwarding it in some entirely
different way?
what if you just do something like:
set $!cluster = "somename";
won't this include the 'cluster:"somename"' string inside the all-json
string?
yes it will but I want exact opposite :) I am trying to not change
the original message (it might have a field named "cluster" already)
or it might not even be a cee json message etc. (I can figure out
whether the message is json formatted or not but I have to set the
variables before that since I need them no matter what the format it)
trying to achieve something like (as described above but this might
be more obvious), pseudo code follows:
set $!message = $!all-json
set $!cluster = "clusterName"
after the two lines above the $!all-json has two fields - "message"
where the value is original $!all-json and cluster, where the value is
my variable (which of course does not work cause I (as far as I know)
don't have a way to set the variable without changing $all-json.
But only if the message is a cee message (otherwise it, of course
does not make sense). If it's not cee message then I still need
cluster name but $!all-json!message is just the original message string.
I.e. I am not trying to add info to the parsed message (that works,
discovered that during my failed attempts), I would like the intact
parsed message to become part of outgoing message.
Does that explain what I'm trying to do? Any ideas how to achieve that?
Ok, that's a little simpler than what I was thinking you were doing.
As far as I know, this is something that rsyslog does not currently
handle well, we need some way to blacklist items so they don't appear in
all-json without deleteing them entirely. There has occasionally be talk
about implementing more control along these lines, but there hasn't been
enough discussion to find good examples and people needing this control
to discuss possible syntax options with.
In the short term, I think the way to do what you are trying to do is to
output in two formats, depending on if the json parse was successful
(there is a variable set that says if the parse found json to parse),
one where you output all-json, the other where you format your own
string, just including the message body.
I'm not understanding why you want the output to always be in json
format, but only include the variable if the input was json, I would
have thought you always wanted to have the variable there.
If you are concerned about overwriting an existing variable, then what
you need to do is to reserve a subtree, and if you find that the subtree
exists, copy the contents of the subtree to subtree!old and then put
your variable in subtree!variable (taking advantage of the fact that
json is a tree structure, not just a name=variable flat structure.
the problem with that is that I have to set variables somewhere (so
that I can use them no matter what, I have two templates but they need
the same info).
Only after that the message is parsed, so it's actually parsing of
the message that would overwrite my info.
I currently have two files, one is a simple file that only sets the
variables (generated at deployment/installation time) and the other ones
is a somewhat complex file (stored in git) that has the templates, sets
up the actions, json parsing etc.
file with variables /etc/rsyslog.d/30-my-vars.conf:
set $!cluster = "something";
the other file /etc/rsyslog.d/31-my.conf:
template(name="json-parsed-ok" type="list") {
... standard header (pri, time, host)
constant(value="@cee{\"original-message-json\":\"")
property(name="$!all-json")
constant(value=",\"cluster\":\"")
property(name="$!cluster")
constant(value="\"}\n")
}
template(name="json-parsing-failed" type="list") {
... standard header (pri, time, host)
constant(value="@cee{\"original-message-txt\":\"")
property(name="msg" format="json")
constant(value=",\"cluster\":\"")
property(name="$!cluster")
constant(value="\"}\n")
}
if prifilt("local0.*") then {
action(type="mmjsonparse")
if $parsesuccess == "OK" then {
action(template="json-parsed-ok" type="omfwd" ...)
} else {
action(template="json-parsing-failed" type="omfwd" ...)
}
stop
}
The json-parsing-failed template works fine since $!cluster does not
change property("msg"). The json-parsed-ok does nto work so well since
$!cluster becomes part of json. I could almost use property("msg") in
json-parsed-ok as well but it has @cee at the beginning.
It seems that the only way out of this is to dynamically generate the
templates themselves but I was trying to keep the generated part as
simple as possible and the templates are not very readable even when
typed directly.
Or maybe use substring of property("msg")?
Is there some other place to store the value of cluster name (in
reality it's few more items but nothing complex, just strings) other
than $!all-json?
thanks!
erik
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
