An another side note, I wonder if my liblognorm may be too old. I was not able to find what version rsyslog 7.4.2 requires.
Here is the version I am using, the one supplied in the ubuntu repo's. ii liblognorm-dev 0.3.2-1 Log normalizing library ii liblognorm0 0.3.2-1 Log normalizing library On Thu, Jul 18, 2013 at 7:48 AM, Todd Mortensen <[email protected]>wrote: > This is the sample line. > > [1084540.211910] Denied: IN=eth0 OUT= MAC=00:30:48:90:cc:a6:00:30: > 48:da:48:e8:08:00 SRC=10.10.10.10 DST=10.10.10.11 LEN=60 TOS=0x10 > PREC=0x00 TTL=64 ID=28843 DF PROTO=TCP SPT=44075 DPT=444 WINDOW=14600 > RES=0x00 SYN URGP=0 > > I will try to get a run going with valgrind and send the output. > > Todd > > > > On Thu, Jul 18, 2013 at 7:24 AM, Rainer Gerhards <[email protected] > > wrote: > >> I am not sure if I manage to look at it before vacation, but could you >> send >> me a log sample that generates the garbagge? Could you run it under >> valgrind control? I guess there is some mis-access taking place... >> >> Rainer >> >> >> On Wed, Jul 17, 2013 at 4:03 AM, Todd Mortensen <[email protected] >> >wrote: >> >> > I have made some progress in getting it to work, I removed the leading >> > space in the rule and it now matches. I totally missed that on log >> output. >> > >> > rule=:%kerntime:word% Denied: IN=%in:word% OUT=%out:word% MAC=%mac:word% >> > SRC=%src-ip:ipv4% DST=%dst-ip:ipv4% LEN=%len:number% TOS=%tos:word% >> > PREC=%prec:word% TTL=%ttl:number% ID=%id:number% %DF:word% >> > PROTO=%proto:word% SPT=%src-port:number% DPT=%dst-port:number% >> > WINDOW=%window:number% RES=%res:word% %pkt-type:word% URGP=%urgp:number% >> > >> > I did find how to use the iptables type but my results where not very >> good. >> > >> > using this rule my output was has a lot of garbage output in the log, >> looks >> > like it wrote out binary data: >> > >> > rule=:%kerntime:word% Denied: %iptables:iptables% >> > >> > { "IN": "eth0", "OUT": "", "MAC": >> > "00:30:48:90:cc:a6:00:30:48:da:48:e8:08:00", "SRC": "10.10.10.10", >> "DST": >> > "10.10.10.11", "LEN": "60", "TOS": "0x10", "PREC": "0x00", "TTL": "64", >> > "ID": "10181", "DF": "[*PRESENT*]", "PROTO": "TCP", "SPT": "51765", >> "DPT": >> > "444", "WINDOW": "14600", "RES": "0x00", "SYN": "[*PRESENT*]", "URGP": >> "0", >> > >> > >> "U)@00:30:48:90:cc:a6:00:30:48:da:48:e8:08:00ü«^?xü«^?ÿÿÿÿÿÿÿÿ%\u0003\u0010LENü«^?<80>¸ü«^?%\u0002\u001060ü«^?¼ü«^?%LEN«^?xü«^?@µü«^?%60ü«^?xü«^?¹ü«^?%\býý\u0001p¾ü«^?<80>ºü«^?%\u0003\bLENü«^?<80>»ü«^?%`´ü«^?ð¿ü«^?ðbü«^?%\u0003\u0010TOSü«^?ÿÿÿÿÿÿÿÿ%\u0004\u00100x10«^?ÿÿÿÿÿÿÿÿ%TOS«^?xü«^?¸ü«^?%0x10^?xü«^?ÿÿÿÿÿÿÿÿ%\býý\u0001P¿ü«^?ÿÿÿÿÿÿÿÿ%\u0003\bTOSü«^?»ü«^?%": >> > "[*PRESENT*]", "kerntime": "[1151929.993763]" } >> > >> > Using the full rule I get results like this, nice and clean. >> > >> > { "urgp": "0", "pkt-type": "SYN", "res": "0x00", "window": "14600", >> > "dst-port": "444", "src-port": "51954", "proto": "TCP", "DF": "DF", >> "id": >> > "31890", "ttl": "64", "prec": "0x00", "tos": "0x10", "len": "60", >> "dst-ip": >> > "10.10.10.10", "src-ip": "10.10.10.11", "mac": >> > "00:30:48:90:cc:a6:00:30:48:da:48:e8:08:00", "in": "eth0", "kerntime": >> > "[1152127.460873]" } >> > >> > I will continue my journey using the other method for now. >> > >> > >> > On Tue, Jul 16, 2013 at 3:19 PM, Todd Mortensen <[email protected] >> > >wrote: >> > >> > > I am not sure, I honestly stole the examples from the sagan project. >> > > >> > > I just tried using the following to test. >> > > >> > > rule=: %kerntime:word% Denied: %iptables% >> > > rule=: [%kerntime:word%] Denied: %iptables% >> > > >> > > I also tried adding [] to the above full line as well. Still doesn't >> > > return any parsed values. >> > > >> > > I forgot to mention this is version 7.4.2 on ubuntu 12.04 from my ppa >> > > >> > > ii rsyslog-mmnormalize 7.4.2-0ubuntu1ppa2 >> > > >> > > >> > > >> > > >> > > On Tue, Jul 16, 2013 at 3:12 PM, David Lang <[email protected]> wrote: >> > > >> > >> it looks to me like your ruleset is not matching the rule >> > >> >> > >> are you sure that %kerntime:word% includes the square brackets and it >> > >> shouldn't be [%kerntime:word%] instead? >> > >> >> > >> there is a special type 'iptables' that was created explicitly for >> > >> matching the name=value format of iptables rules. >> > >> >> > >> I think your rule can be simplified to something like: >> > >> >> > >> rule=: [%kerntime:word%] Denied: %iptables% >> > >> >> > >> David Lang >> > >> >> > >> On Tue, 16 Jul 2013, Todd Mortensen wrote: >> > >> >> > >> Date: Tue, 16 Jul 2013 14:53:36 -0700 >> > >>> From: Todd Mortensen <[email protected]> >> > >>> Reply-To: rsyslog-users <[email protected]> >> > >>> To: rsyslog-users <[email protected]> >> > >>> Subject: [rsyslog] mmnormalize and iptable logs >> > >>> >> > >>> >> > >>> I am trying my first attempts at using mmnormalize. >> > >>> >> > >>> My goal is to parse out my iptable firewall logs into a cee format >> so I >> > >>> can >> > >>> then send them to elasticsearch. >> > >>> >> > >>> But so far my logs just show unparsed items. Can anyone shed any >> light >> > >>> to >> > >>> where I am going wrong here. >> > >>> >> > >>> Relevant sections from rsyslog.conf >> > >>> >> > >>> module(load="mmnormalize") >> > >>> action(type="mmnormalize" UseRawMsg="off" >> > >>> ruleBase="/etc/rsyslog.d/**normalize.rb") >> > >>> >> > >>> module(load="mmjsonparse") >> > >>> >> > >>> >> > >>> #template(name="cee" type="string" string="%$!all-json%\n") >> > >>> template(name="cee" type="string" string="%$!%\n") >> > >>> >> > >>> *.* { action (name="cee" type="omfile" >> > >>> Template="cee" file="/var/log/cee.log") } >> > >>> >> > >>> I have also tried a couple variations of this but they still don't >> > match >> > >>> >> > >>> normalize.rb >> > >>> rule=: %kerntime:word% Denied: IN=%in:word% OUT= MAC=%mac:word% >> > >>> SRC=%src-ip:ipv4% DST=%dst-ip:ipv4% LEN=%len:number% TOS=%tos:word% >> > >>> PREC=%prec:word% TTL=%ttl:number% ID=%id:number% %DF:word% >> > >>> PROTO=%proto:word% SPT=%src-port:number% DPT=%dst-port:number% >> > >>> WINDOW=%window:number% RES=%res:word% %pkt-type:word% >> > URGP=%urgp:number% >> > >>> >> > >>> I also tried to use the iptales type, but I am not able to find many >> > docs >> > >>> on its usage. >> > >>> >> > >>> rule=: %kerntime:word% Denied: %iptables:iptables% >> > >>> >> > >>> >> > >>> Sample log line: >> > >>> >> > >>> [1084540.211910] Denied: IN=eth0 OUT= >> > >>> MAC=00:30:48:90:cc:a6:00:30:**48:da:48:e8:08:00 SRC=10.10.10.10 >> > >>> DST=10.10.10.11 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=28843 DF >> PROTO=TCP >> > >>> SPT=44075 DPT=444 WINDOW=14600 RES=0x00 SYN URGP=0 >> > >>> >> > >>> >> > >>> This is what I see in my cee logfile and it shows unpared-data. >> > >>> >> > >>> # { "originalmsg": "[1084540.211910] Denied: IN=eth0 OUT= >> > >>> MAC=00:30:48:90:cc:a6:00:30:**48:da:48:e8:08:00 SRC=10.10.10.10 >> > >>> DST=10.10.10.11 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=28843 DF >> PROTO=TCP >> > >>> SPT=44075 DPT=444 WINDOW=14600 RES=0x00 SYN URGP=0 ", >> "unparsed-data": >> > >>> "[1084540.211910] Denied: IN=eth0 OUT= >> > >>> MAC=00:30:48:90:cc:a6:00:30:**48:da:48:e8:08:00 SRC=10.10.10.10 >> > >>> DST=10.10.10.11 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=28843 DF >> PROTO=TCP >> > >>> SPT=44075 DPT=444 WINDOW=14600 RES=0x00 SYN URGP=0 " } >> > >>> ______________________________**_________________ >> > >>> rsyslog mailing list >> > >>> http://lists.adiscon.net/**mailman/listinfo/rsyslog< >> > http://lists.adiscon.net/mailman/listinfo/rsyslog> >> > >>> http://www.rsyslog.com/**professional-services/< >> > http://www.rsyslog.com/professional-services/> >> > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >> > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> > myriad >> > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> you >> > >>> DON'T LIKE THAT. >> > >>> >> > >>> ______________________________**_________________ >> > >> rsyslog mailing list >> > >> http://lists.adiscon.net/**mailman/listinfo/rsyslog< >> > http://lists.adiscon.net/mailman/listinfo/rsyslog> >> > >> http://www.rsyslog.com/**professional-services/< >> > http://www.rsyslog.com/professional-services/> >> > >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> myriad >> > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> you >> > >> DON'T LIKE THAT. >> > >> >> > > >> > > >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com/professional-services/ >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> > DON'T LIKE THAT. >> > >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
