Hi Paolo, I tested it with a small subset of prefixes for now.... (only 4)
I did the following. Note that I used id=1001 instead of tag=1001, cause the example pretag.map seemed to use id= and not tag= pre_tag.map file contents: id=1001 filter='src net 10.204.192.0/18 or src net 10.206.64.0/18 or src net 10.247.192.0/21 or src net 10.55.128.0/22' id=1002 filter='dst net 10.204.192.0/18 or dst net 10.206.64.0/18 or dst net 10.247.192.0/21 or dst net 10.55.128.0/22' And in my pmacctd-in.conf : pre_tag_map: /usr/local/pmacct/etc/pre_tag.map pre_tag_filter[in]: !1001 And in my pmacctd-out.conf : pre_tag_map: /usr/local/pmacct/etc/pre_tag.map pre_tag_filter[out]: !1002 And effectively, traffic did not seem to be measured any longer..... I commented out the pre_tag_filter, and it worked again. I must admit we use an older version, 0.14.3 Any ideas ? Best regards, Wouter -----Original Message----- From: pmacct-discussion [mailto:[email protected]] On Behalf Of Wouter de Jong Sent: Friday, September 25, 2015 14:08 To: [email protected] Subject: Re: [pmacct-discussion] Question regarding excluding traffic between internal networks Hi Paolo, Any idea if it would be better to use one big 'tag filter', or use several small 'tag filters' ? While using the syntax like the following, I tried to add around 30 prefixes : tag=666 filter='src net 192.168.15.0/24 or src net 192.168.25.0/24 or src net 192.168.35.0/24 or src net .....etc, etc....' And that gave me : pmacctd[9616]: WARN ( default/core ): line too long (max 288 chars). Line 1 in map '/usr/local/pmacct/etc/pre_tag.map' ignored. pmacctd[9616]: WARN ( default/core ): line too long (max 288 chars). Line 2 in map '/usr/local/pmacct/etc/pre_tag.map' ignored. So, one big one seems to already be not an optioon.... So I could split it in 2 or 3 filters, or have a filter per prefix ..... and then have pre_tag_filter: !666,!667,!668,!669,etc,etc Just a bit puzzling what would be best for performance vs usability (usability is for me the latter, with one prefix per filter) I'll see if I can watch the load before and after... Thanks & best regards, Wouter -----Original Message----- From: pmacct-discussion [mailto:[email protected]] On Behalf Of Paolo Lucente Sent: Sunday, September 20, 2015 15:58 To: [email protected] Subject: Re: [pmacct-discussion] Question regarding excluding traffic between internal networks Hi Wouter, The solution you propose is aapplicable & yes, pre_tag_filter does support multiple tags. Wrt pre_tag_filter & load: there is a load impact proportional to the complexity of the filter(s) and this will be appreciated more in pmacctd as filters are avaluated per packet (rather than per flow or per sample). I recommend to carry out some testing first (needless to say if you could share, even privately, some load figures that would be awesome). Cheers, Paolo On Thu, Sep 17, 2015 at 02:03:10PM +0000, Wouter de Jong wrote: > Hi Paolo, > > Thanks for your response :) > > OK, so if I understand correctly, I could do something like : > > tag=666 filter='src net 192.168.15.0/24 or src net 192.168.25.0/24 or src net > 192.168.35.0/24' > tag=667 filter='dst net 192.168.15.0/24 or dst net 192.168.25.0/24 or dst net > 192.168.35.0/24' > > Together with the appropriate pre_tag_filters : > > inbound instance: pre_tag_filter[in]: !666 > outbound instance: pre_tag_filter[out]: !667 > > > I think this could be a solution, great :) > > > Is it possible to specify multiple filter tags, instead of one ? > > Eg. : > > pre_tag_filter[in]: !666,!668,!670 > > tag=666 filter='src net 192.168.15.0/24' > tag=668 filter='src net 192.168.25.0/24' > tag=670 filter='src net 192.168.35.0/24' > > > For example, to be able to split the 'supernets' in multiple filters, for > readability and managability ? > Or is this performance wise a bad idea ? > > > Does the pre_tag_filter have any CPU load we should care for ? > > > > Thanks ! > > Best regards, > > Wouter > > > > > > > -----Original Message----- > From: pmacct-discussion [mailto:[email protected]] On > Behalf Of Paolo Lucente > Sent: Thursday, September 17, 2015 13:29 > To: [email protected] > Subject: Re: [pmacct-discussion] Question regarding excluding traffic between > internal networks > > Hi Wouter, > > Great to read from you! > > I should be correct that the amount of your supernets is manageable > to put in a pcap-style filter. Plus the set of supernets should not > change much. In such a case you could use a pre_tag_map like: > > tag=666 filter=<pcap-style filter with your supernets> > > Then in your config file: > > ... > ! > pre_tag_map: /path/to/pretag.map > ! > pre_tag_filter[in]: !666 > pre_tag_filter[out]: !666 > ! > ... > > It's not really needed that you have the 'tag' primitive as part of > your 'aggregate', unless you are making use of it for some other > reason (not evident from the config you posted). Intuitively: you > tag the traffic between your supernets with '666' in the pre_tag_map > and then you filter it out with a !666 as pre_tag_filter. > > The nice thing about pre_tag_map is that you can reload it at runtime > without having to restart the daemon, making the filter changeable > by sending a SIGUSR2 to the daemon, ie. 'killall -USR2 <daemon>'. > > Cheers, > Paolo > > > > On Fri, Sep 11, 2015 at 02:53:46PM +0000, Wouter de Jong wrote: > > Hi, > > > > Let's say I use the following in a pcap pmacctd-in instance : > > > > plugins: mysql[in] > > ! > > networks_file[in]: /usr/local/pmacct/etc/networks.def > > aggregate[in]: tag,dst_host > > ! > > sql_db[in]: pmacct > > sql_recovery_backup_host[in]: 127.0.0.1 > > ! > > sql_table[in]: acct_v2_in_%Y%m%d > > sql_optimize_clauses[in]: true > > sql_table_schema[in]: /usr/local/pmacct/etc/acct_custom_in.schema > > > > > > And likewise a pcap pmacctd-out instance : > > > > plugins: mysql[out] > > ! > > networks_file[out]: /usr/local/pmacct/etc/networks.def > > aggregate[out]: tag,src_host > > ! > > sql_db[out]: pmacct > > sql_recovery_backup_host[out]: 127.0.0.1 > > ! > > sql_table[out]: acct_v2_out_%Y%m%d > > sql_optimize_clauses[out]: true > > sql_table_schema[out]: /usr/local/pmacct/etc/acct_custom_out.schema > > > > > > > > So networks.def contains our RIPE assigned prefixes (eg. /17, /18, /19, > > etc), > > and the traffic is aggregated per host (/32) of our own prefix only. > > > > The problem is that I have traffic from our own prefixes coming in from > > both sides of the link that is monitored. > > And I need to exclude traffic between our own prefixes. > > > > I'm not 100% suire how to accomplish this. > > > > Writing a pcap_filter is not a very viable solution, since networks.def > > contains a lot of prefixes.... so it would become very large, > > and hence very error prone... > > > > Same seems to be the case for aggregate_filter, unless.... I totally > > haven't understood what this option does and how to correctly write it :) > > > > Basically I'm looking for something like this : > > > > ignore_src_to_dst: /usr/local/pmacct/etc/networks.def > > > > And then in very basic pseudo code : > > > > if(match_ignore_src_to_dst(src_addr) && match_ignore_src_to_dst(dst_addr)) { > > # simply ignore this packet since src and dst matches our local network > > prefixes > > next; > > } > > > > > > Last but not least : I haven't built the above config myself, so I'm not > > sure if the 'tag' in 'aggregate' is necessary at this moment ? > > > > > > Thanks for any hints & advice :) > > > > Best regards, > > > > Wouter > > > > _______________________________________________ > > pmacct-discussion mailing list > > http://www.pmacct.net/#mailinglists > > _______________________________________________ > pmacct-discussion mailing list > http://www.pmacct.net/#mailinglists > > _______________________________________________ > pmacct-discussion mailing list > http://www.pmacct.net/#mailinglists _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
