Hi Tiernan,
One solution might be to simplify the whole configuration and instantiate
a single plugin - instead of in, out, infree, etc. This plugin can link a
networks_file map ('networks_file[<plugin name>]: /path/to/networks.lst')
with a content similar to the following:
=== networks.lst ===
65534,10.<whatever>/24
65534,2001:<whatever>::/56
...
===
The plugin is then running one of the following two aggregation methods:
1) 'aggregate[<plugin name>]: src_as, dst_as'. In case you don't need to
distinguish which subnet/IP address is making/receiving the traffic.
You end up with a compact tabular of three/four entries:
0 -> 65534 is inbound traffic; 65534 -> 0 is outbound traffic;
65534 -> 65534 is internal/free traffic; 0 -> 0 you should never see
this or it means the networks_file content is not up-to-date (or you
are transiting 3rd party traffic).
2) 'aggregate[<plugin name>]: src_as, dst_as, src_net, dst_net'. Same as
before but more verbose: this allows you to distinguish which subnet,
as defined in the networks_file, is generating/receiving traffic. If
a subnet is not defined in the file (ie. everything external to your
private/ public address space) it will be conveniently rewritten as
zeroes.
3) 'aggregate[<plugin name>]: src_as, dst_as, src_host, dst_host'. Same
as 2) but more verbose, down to the specific IP address. If going for
this option be careful at the size of the matrix you are creating ..
Cheers,
Paolo
On Thu, Jun 21, 2012 at 01:11:53AM +1000, Tiernan Messmer wrote:
> On 2012-06-20 1126, Paolo Lucente wrote:
> > Hi Tiernan,
> >
> > On Tue, Jun 12, 2012 at 05:18:36PM +1000, Tiernan Messmer wrote:
> >
> >> If you think the exact filter may be contributing, I can send those
> >> privately if you'd like. On an unrelated note, is it possible to filter
> >> based on traffic direction in/out of an interface rather than relying on
> >> matching based on network? May simplify the filters slightly if that's
> >> what the bottleneck is.
> >
> > Yes, long filters can be the culprit - can you experiment ie. removing
> > filters (and optionally defining 'none' aggregation method to plugins) ?
>
> Removing the filters reduced cpu usage to below 1%, I'd say we've found
> a cause.
>
> Is there a more efficient way of defining filters? At present the
> filters are used to categorise if it's traffic going in or out by
> matching dst/src to be a local address (checks if it matches one public
> IPv4 address, one private range IP net, or one public IPv6 net) for
> in/out respectively and matching if it's 'free' traffic or not by
> matching if it's talking to a machine mentioned here:
> https://customer-webtools-api.internode.on.net/unmetered_ip_address_list.txt
>
> aggregate_filter[in]: dst net 10.190.38.0/24 or dst net 2001:range::/56
> or dst host publicip or dst host 10.190.205.38 and src net not
> 10.190.38.0/24 and src net not 2001:range::/56 and not (net <ip> and net
> <ip> ...)
>
> For the infree filter it would be the same except without the last 'not'
> just before the open bracket and out is the same, except with dst
> replaced with src.
>
> Is there a more efficient way of doing this?
>
> > Given your capturing framework, can you rely on MAC addresses to simplify
> > the filtering? libpcap does not provide an indication of direction of the
> > traffic.
>
> Unfortunately not, it is on a PPPoE interface, so no MACs.
>
> > Cheers,
> > Paolo
>
> Cheers,
> Tiernan
>
>
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
