On Mon, 26 Sep 2022, Ben Hart wrote:
Yeah sorry, here’s a block of matched return:
FROMHOST: 'rsyslog-server', fromhost-ip: '127.0.0.1', HOSTNAME:
'rsyslog-server', PRI: 6,
syslogtag 'kernel:', programname: 'kernel', APP-NAME: 'kernel', PROCID: '-',
MSGID: '-',
TIMESTAMP: 'Sep 26 16:10:01', STRUCTURED-DATA: '-',
msg: '[2074456.628051] VCSA Syslog-IN IN=ens192 OUT=
MAC=00:50:56:a0:03:90:00:0c:29:cb:fc:14:08:00 SRC=1.2.160.115 DST=1.2.160.30
LEN=205 TOS=0x00 PREC=0x00 TTL=64 ID=65500 DF PROTO=UDP SPT=36775 DPT=20525
LEN=185 '
escaped msg: '[2074456.628051] VCSA Syslog-IN IN=ens192 OUT=
MAC=00:50:56:a0:03:90:00:0c:29:cb:fc:14:08:00 SRC=1.2.160.115 DST=1.2.160.30
LEN=205 TOS=0x00 PREC=0x00 TTL=64 ID=65500 DF PROTO=UDP SPT=36775 DPT=20525
LEN=185 '
inputname: imklog rawmsg: '[2074456.628051] VCSA Syslog-IN IN=ens192 OUT=
MAC=00:50:56:a0:03:90:00:0c:29:cb:fc:14:08:00 SRC=1.2.160.115 DST=1.2.160.30
LEN=205 TOS=0x00 PREC=0x00 TTL=64 ID=65500 DF PROTO=UDP SPT=36775 DPT=20525
LEN=185 '
$!:
$.:
$/:
thanks, that gives good info. Yes, the iptables logging is by/from the kernel,
so you would need to match VCSA or Syslog-IN as you note.
I do have another conf file that I found some excellent examples of that
utilizes a catch_all ruleset/queue with if/thens below it. I’d rather use the
modern syntaxes when I can so Im thinking of trying to adapt something like
that to this. Except, in this case the format I’d mimic is directing matches
to 3 different log files. I don’t need to do that, they all need to stay in
the firewalld file except with the VCSA match being rate limited.
note that the different files all get merged together before rsyslog looks at
them, start rsyslog with -o /path/to/file and the file will have the combined
file as rsyslog sees it.
Because of how the files are combined, trying to put things into an included
file may not have the effect you wish it did because of how the files are
combined.
you can put a ruleset on the inkern module, or you can just put the filter in
the default ruleset
This is a nice configuration, it keeps the remote messages out of the main
queue, putting them in their own queue, and then classifies the messages,
sending them to separate rulesets (which could have their own queues if needed)
Here's the one I’m talking about:
ruleset(name="f_all" queue.type="LinkedList" queue.workerthreads="4"
queue.size="100000") {
note that lots of worker threads is likely to hurt performance rather than help
it. Only increase worker threads if you see that one of them is maxing out a
core (running top, hit 'H" to see the threads and look for a thread that is
using 100% cpu), too many threads reduces batch size and causing thrashing on
the locks, eating CPU without increasing throughput.
# WLC
if ($hostname contains 'wlc') then{
call r_wlc
stop
}
# ESXI
if ($hostname contains 'vmhost') then{
call r_vmhost
stop
}
# Catch All
call r_catch_all
stop
}
# add vmhosts #
template(name="d_catch_all" type="string"
string="/var/log/remote-syslog/undefined.log")
template(name="d_wlc" type="string" string="/var/log/remote-syslog/wlc.log")
template(name="d_vmhost" type="string" string="/var/log/remote-syslog/esxi.log")
note that none of these templates include a variable in them, so you don't need
them to be dynamic (i.e. you could use file instead of dynafile), dynafile is if
you wanted something like /var/log/remote-syslog/type/hostname.log, and if you
wanted that you could do
if ($hostname contains 'wlc') then{
set $.type='wlc';
} else if ($hostname contains 'vmhost') then{
set $.type='esxi';
} else set $.type='undefined';
$template type_output,"/var/log/remotelog/%$.type%/%hostname%.log"
?type_output
(I much prefer the new syntax for conditionals, but for simple output and simple
templates, the old syntax is more compact and still very readable)
David Lang
ruleset(name="r_wlc"){
action(type="omfile" Dynafile="d_wlc")
}
ruleset(name="r_vmhost"){
action(type="omfile" Dynafile="d_vmhost")
}
ruleset(name="r_catch_all"){
action(type="omfile" Dynafile="d_catch_all")
}
input(type="imudp" port="514" ruleset="f_all")
From: David Lang <da...@lang.hm>
Date: Monday, September 26, 2022 at 5:31 PM
To: Ben Hart <ben.h...@jamf.com>
Cc: David Lang <da...@lang.hm>, Ben Hart via rsyslog <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] Action params with exec.OnlyEveryNthTime
ok, I don't see the sample of the debug format message, please send one or two
of these to the body of the message rather than trying to use an attachment (I
assume that's what you tried)
your original approach to filter these out to a separate file and then throw
them away is the place to start, you just need to do that before any config
lines that do other things with the logs.
remember that you can filter on any variable, and you can use the simple
if...then format (where you can combine multiple criteria) rather than the old
format you used below.
David Lang
On Mon, 26 Sep 2022, Ben Hart wrote:
Date: Mon, 26 Sep 2022 21:19:43 +0000
From: Ben Hart <ben.h...@jamf.com>
To: David Lang <da...@lang.hm>
Cc: David Lang <da...@lang.hm>,
Ben Hart via rsyslog <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] Action params with exec.OnlyEveryNthTime
OK I imagine I did something wrong.. changed the firewalld.conf to exactly what
you suggested, bounced rsyslog and gave it a minute. The new fierwalld-debug
file remained empty. Do I need to paste that DebugFormat template in
somewhere? Also tried pasting the template definition into the bottom of this
same file. It did not correct the problem.
I do see numerous omfile suspension warnings in systemctl status rsyslog, I
presume it’s referring to the -debug file. Rsyslogd -N1 gives no errors. Also
the rules no longer matching on the “” value. Goes back to defaults which is
only logging denied packets to the file and all else into syslog.
After struggling a bit, then trying to simplify it I just appended the template
onto the tail of the original line. I let it run for a few minutes then
reverted back to original. But I now have a sample of log with the debugformat
matching my “” below. And it’s using the imklog module, but reading about that
module I get the impression that it’s nor normal nor probably a good idea to be
handling it. So with that in mind, maybe it’s a better idea to try to keep this
data out of that module entirely.
But, TBH I have no idea where to go from here. I still need to limit this
traffic but since it’s hitting the kernel, and it’s local, and I cannot
restrict it from the source… all suggestions welcome.
Thanks
From: David Lang <da...@lang.hm>
Date: Monday, September 26, 2022 at 4:13 PM
To: Ben Hart <ben.h...@jamf.com>
Cc: David Lang <da...@lang.hm>, Ben Hart via rsyslog <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] Action params with exec.OnlyEveryNthTime
currently you have
:msg,contains," Syslog-In " /var/log/firewalld
& stop
change this to
:msg,contains," Syslog-In " {
/var/log/firewalld
/var/log/firewalld-debug;RSYSLOG_DebugFormat
stop
}
(a little easier to read than using &)
then look at the firewalld-debug file
David Lang
On Mon, 26 Sep 2022, Ben Hart wrote:
Date: Mon, 26 Sep 2022 20:09:40 +0000
From: Ben Hart <ben.h...@jamf.com>
To: David Lang <da...@lang.hm>
Cc: David Lang <da...@lang.hm>,
Ben Hart via rsyslog <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] Action params with exec.OnlyEveryNthTime
Ok here’s a sanitized blurb from the debug file grepped to the events I’m
trying to limit.
2165.510789426:main Q:Reg/w0 : ruleset.c: Filter: check for property 'msg'
(value '[2070020.637509] VCSA Syslog-IN IN=ens192 OUT=
MAC=00:50:56:a0:03:90:00:0c:29:cb:fc:14:08:00 SRC=1.2.3.4160.115
DST=1.2.3.4160.30 LEN=361 TOS=0x00 PREC=0x00 TTL=64 ID=29068 DF PROTO=UDP
SPT=52225 DPT=9999 LEN=341 ') contains 'filter_': FALSE
2165.510961295:main Q:Reg/w0 : rainerscript.c: rainerscript: (string) var 1:
'[2070020.637509] VCSA Syslog-IN IN=ens192 OUT=
MAC=00:50:56:a0:03:90:00:0c:29:cb:fc:14:08:00 SRC=1.2.3.4160.115
DST=1.2.3.4160.30 LEN=361 TOS=0x00 PREC=0x00 TTL=64 ID=29068 DF PROTO=UDP
SPT=52225 DPT=9999 LEN=341 '
2165.512976011:main Q:Reg/w0 : omfile.c: omfile: write to stream, pData->pStrm
0x7f9e88001890, lenBuf 254, strt data Sep 26 14:56:05 rsyslog-server kernel:
[2070020.637509] VCSA Syslog-IN IN=ens192 OUT=
MAC=00:50:56:a0:03:90:00:0c:29:cb:fc:14:08
2165.513114940:main Q:Reg/w0 : omfile.c: omfile: write to stream, pData->pStrm
0x7f9e88004040, lenBuf 254, strt data Sep 26 14:56:05 rsyslog-server kernel:
[2070020.637509] VCSA Syslog-IN IN=ens192 OUT=
MAC=00:50:56:a0:03:90:00:0c:29:cb:fc:14:08
2165.545162782:vcsa9999:Reg/w0: rainerscript.c: 2165.545175184:main Q:Reg/w0
: ruleset.c: processBATCH: next msg 0: [2070020.676314] VCSA Syslog-IN
IN=ens192 OUT= MAC=00:50:56:a0:03:90:00:0c:29:cb:fc:14:08:00 SRC=1.2.3.4160.115
DST=1.2.3.4160.3
So it’s def hitting the main queue, and it’s matching correctly. Could not find
any mention of imuxsock, or the word socket along with firewalld. Oh wait a
minute, FWD natively logs to syslog so maybe I can still use an input for
imuxsock but then possibly filter my programname or msg to impose the limits?
From: Ben Hart <ben.h...@jamf.com>
Date: Monday, September 26, 2022 at 3:51 PM
To: David Lang <da...@lang.hm>
Cc: David Lang <da...@lang.hm>, Ben Hart via rsyslog <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] Action params with exec.OnlyEveryNthTime
Understood, So I’ve never actually changed the default template, is there a way
to engage that from within a .conf or does it need to be in rsyslog.conf? This
server is currently handling logs from about 15 sources and SplunkUF is handing
them off to Splunk Cloud so, unless I have to I’ don’t wanna mess with those.
From: David Lang <da...@lang.hm>
Date: Monday, September 26, 2022 at 3:47 PM
To: Ben Hart <ben.h...@jamf.com>
Cc: David Lang <da...@lang.hm>, Ben Hart via rsyslog <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] Action params with exec.OnlyEveryNthTime
log the message with the template RSYSLOG_DebugFormat and it will show you most
of the parsed properties from the message.
(I think you are correct, but it's best to be sure)
David Lang
On Mon, 26 Sep 2022, Ben Hart wrote:
(Apologies, somehow I sent this before it was complete.)
David, So yeah you are right.. I didnʼt dig into omfile and assumed from the
Actions page those were available to most. So I have a .conf with 13 rulesets
with matching inputs and Iʼve found that you can ratelimit on the input side
however this source is local (firewalld).
If I am loading modules imudp, imtcp and imuxsock and since I do have a .conf
to send all of the logs from that source to a specific file:
:msg,contains," Syslog-In " /var/log/firewalld
& stop
Am I correct in thinking that an input type for this would be imuxsock? And if
so, could I setup a ruleset with matching inputs to then attempt to rate limit
it?
Ben
From: David Lang <da...@lang.hm>
Date: Monday, September 26, 2022 at 2:52 PM
To: Ben Hart via rsyslog <rsyslog@lists.adiscon.com>
Cc: Ben Hart <ben.h...@jamf.com>
Subject: Re: [rsyslog] Action params with exec.OnlyEveryNthTime
I think it's a valid parameter to ommail, but I would be surprised to see it
under omfile (and if it is, check what version you are running vs the version
you are seeing it documented)
rsyslog is not an event correlation engine, it has some minimal things, but they
really aren't very good.
What I do in a situation like yours is to filter the messages and send them to
an external event correlation engine (I tend to use Simple Event Correlator) and
then have it generate alert or summary messages, feeding them back in to rsyslog
(watch out that you don't generate a loop in this process)
David Lang
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
