This is the sample line. [1084540.211910] Denied: IN=eth0 OUT= MAC=00:30:48:90:cc:a6:00:30: 48:da:48:e8:08:00 SRC=10.10.10.10 DST=10.10.10.11 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=28843 DF PROTO=TCP SPT=44075 DPT=444 WINDOW=14600 RES=0x00 SYN URGP=0
I will try to get a run going with valgrind and send the output. Todd On Thu, Jul 18, 2013 at 7:24 AM, Rainer Gerhards <[email protected]>wrote: > I am not sure if I manage to look at it before vacation, but could you send > me a log sample that generates the garbagge? Could you run it under > valgrind control? I guess there is some mis-access taking place... > > Rainer > > > On Wed, Jul 17, 2013 at 4:03 AM, Todd Mortensen <[email protected] > >wrote: > > > I have made some progress in getting it to work, I removed the leading > > space in the rule and it now matches. I totally missed that on log > output. > > > > rule=:%kerntime:word% Denied: IN=%in:word% OUT=%out:word% MAC=%mac:word% > > SRC=%src-ip:ipv4% DST=%dst-ip:ipv4% LEN=%len:number% TOS=%tos:word% > > PREC=%prec:word% TTL=%ttl:number% ID=%id:number% %DF:word% > > PROTO=%proto:word% SPT=%src-port:number% DPT=%dst-port:number% > > WINDOW=%window:number% RES=%res:word% %pkt-type:word% URGP=%urgp:number% > > > > I did find how to use the iptables type but my results where not very > good. > > > > using this rule my output was has a lot of garbage output in the log, > looks > > like it wrote out binary data: > > > > rule=:%kerntime:word% Denied: %iptables:iptables% > > > > { "IN": "eth0", "OUT": "", "MAC": > > "00:30:48:90:cc:a6:00:30:48:da:48:e8:08:00", "SRC": "10.10.10.10", "DST": > > "10.10.10.11", "LEN": "60", "TOS": "0x10", "PREC": "0x00", "TTL": "64", > > "ID": "10181", "DF": "[*PRESENT*]", "PROTO": "TCP", "SPT": "51765", > "DPT": > > "444", "WINDOW": "14600", "RES": "0x00", "SYN": "[*PRESENT*]", "URGP": > "0", > > > > > "U)@00:30:48:90:cc:a6:00:30:48:da:48:e8:08:00ü«^?xü«^?ÿÿÿÿÿÿÿÿ%\u0003\u0010LENü«^?<80>¸ü«^?%\u0002\u001060ü«^?¼ü«^?%LEN«^?xü«^?@µü«^?%60ü«^?xü«^?¹ü«^?%\býý\u0001p¾ü«^?<80>ºü«^?%\u0003\bLENü«^?<80>»ü«^?%`´ü«^?ð¿ü«^?ðbü«^?%\u0003\u0010TOSü«^?ÿÿÿÿÿÿÿÿ%\u0004\u00100x10«^?ÿÿÿÿÿÿÿÿ%TOS«^?xü«^?¸ü«^?%0x10^?xü«^?ÿÿÿÿÿÿÿÿ%\býý\u0001P¿ü«^?ÿÿÿÿÿÿÿÿ%\u0003\bTOSü«^?»ü«^?%": > > "[*PRESENT*]", "kerntime": "[1151929.993763]" } > > > > Using the full rule I get results like this, nice and clean. > > > > { "urgp": "0", "pkt-type": "SYN", "res": "0x00", "window": "14600", > > "dst-port": "444", "src-port": "51954", "proto": "TCP", "DF": "DF", "id": > > "31890", "ttl": "64", "prec": "0x00", "tos": "0x10", "len": "60", > "dst-ip": > > "10.10.10.10", "src-ip": "10.10.10.11", "mac": > > "00:30:48:90:cc:a6:00:30:48:da:48:e8:08:00", "in": "eth0", "kerntime": > > "[1152127.460873]" } > > > > I will continue my journey using the other method for now. > > > > > > On Tue, Jul 16, 2013 at 3:19 PM, Todd Mortensen <[email protected] > > >wrote: > > > > > I am not sure, I honestly stole the examples from the sagan project. > > > > > > I just tried using the following to test. > > > > > > rule=: %kerntime:word% Denied: %iptables% > > > rule=: [%kerntime:word%] Denied: %iptables% > > > > > > I also tried adding [] to the above full line as well. Still doesn't > > > return any parsed values. > > > > > > I forgot to mention this is version 7.4.2 on ubuntu 12.04 from my ppa > > > > > > ii rsyslog-mmnormalize 7.4.2-0ubuntu1ppa2 > > > > > > > > > > > > > > > On Tue, Jul 16, 2013 at 3:12 PM, David Lang <[email protected]> wrote: > > > > > >> it looks to me like your ruleset is not matching the rule > > >> > > >> are you sure that %kerntime:word% includes the square brackets and it > > >> shouldn't be [%kerntime:word%] instead? > > >> > > >> there is a special type 'iptables' that was created explicitly for > > >> matching the name=value format of iptables rules. > > >> > > >> I think your rule can be simplified to something like: > > >> > > >> rule=: [%kerntime:word%] Denied: %iptables% > > >> > > >> David Lang > > >> > > >> On Tue, 16 Jul 2013, Todd Mortensen wrote: > > >> > > >> Date: Tue, 16 Jul 2013 14:53:36 -0700 > > >>> From: Todd Mortensen <[email protected]> > > >>> Reply-To: rsyslog-users <[email protected]> > > >>> To: rsyslog-users <[email protected]> > > >>> Subject: [rsyslog] mmnormalize and iptable logs > > >>> > > >>> > > >>> I am trying my first attempts at using mmnormalize. > > >>> > > >>> My goal is to parse out my iptable firewall logs into a cee format > so I > > >>> can > > >>> then send them to elasticsearch. > > >>> > > >>> But so far my logs just show unparsed items. Can anyone shed any > light > > >>> to > > >>> where I am going wrong here. > > >>> > > >>> Relevant sections from rsyslog.conf > > >>> > > >>> module(load="mmnormalize") > > >>> action(type="mmnormalize" UseRawMsg="off" > > >>> ruleBase="/etc/rsyslog.d/**normalize.rb") > > >>> > > >>> module(load="mmjsonparse") > > >>> > > >>> > > >>> #template(name="cee" type="string" string="%$!all-json%\n") > > >>> template(name="cee" type="string" string="%$!%\n") > > >>> > > >>> *.* { action (name="cee" type="omfile" > > >>> Template="cee" file="/var/log/cee.log") } > > >>> > > >>> I have also tried a couple variations of this but they still don't > > match > > >>> > > >>> normalize.rb > > >>> rule=: %kerntime:word% Denied: IN=%in:word% OUT= MAC=%mac:word% > > >>> SRC=%src-ip:ipv4% DST=%dst-ip:ipv4% LEN=%len:number% TOS=%tos:word% > > >>> PREC=%prec:word% TTL=%ttl:number% ID=%id:number% %DF:word% > > >>> PROTO=%proto:word% SPT=%src-port:number% DPT=%dst-port:number% > > >>> WINDOW=%window:number% RES=%res:word% %pkt-type:word% > > URGP=%urgp:number% > > >>> > > >>> I also tried to use the iptales type, but I am not able to find many > > docs > > >>> on its usage. > > >>> > > >>> rule=: %kerntime:word% Denied: %iptables:iptables% > > >>> > > >>> > > >>> Sample log line: > > >>> > > >>> [1084540.211910] Denied: IN=eth0 OUT= > > >>> MAC=00:30:48:90:cc:a6:00:30:**48:da:48:e8:08:00 SRC=10.10.10.10 > > >>> DST=10.10.10.11 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=28843 DF > PROTO=TCP > > >>> SPT=44075 DPT=444 WINDOW=14600 RES=0x00 SYN URGP=0 > > >>> > > >>> > > >>> This is what I see in my cee logfile and it shows unpared-data. > > >>> > > >>> # { "originalmsg": "[1084540.211910] Denied: IN=eth0 OUT= > > >>> MAC=00:30:48:90:cc:a6:00:30:**48:da:48:e8:08:00 SRC=10.10.10.10 > > >>> DST=10.10.10.11 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=28843 DF > PROTO=TCP > > >>> SPT=44075 DPT=444 WINDOW=14600 RES=0x00 SYN URGP=0 ", > "unparsed-data": > > >>> "[1084540.211910] Denied: IN=eth0 OUT= > > >>> MAC=00:30:48:90:cc:a6:00:30:**48:da:48:e8:08:00 SRC=10.10.10.10 > > >>> DST=10.10.10.11 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=28843 DF > PROTO=TCP > > >>> SPT=44075 DPT=444 WINDOW=14600 RES=0x00 SYN URGP=0 " } > > >>> ______________________________**_________________ > > >>> rsyslog mailing list > > >>> http://lists.adiscon.net/**mailman/listinfo/rsyslog< > > http://lists.adiscon.net/mailman/listinfo/rsyslog> > > >>> http://www.rsyslog.com/**professional-services/< > > http://www.rsyslog.com/professional-services/> > > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > > myriad > > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > > >>> DON'T LIKE THAT. > > >>> > > >>> ______________________________**_________________ > > >> rsyslog mailing list > > >> http://lists.adiscon.net/**mailman/listinfo/rsyslog< > > http://lists.adiscon.net/mailman/listinfo/rsyslog> > > >> http://www.rsyslog.com/**professional-services/< > > http://www.rsyslog.com/professional-services/> > > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > >> DON'T LIKE THAT. > > >> > > > > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
