Hi Paolo,
I tried to implement it using the pretag filtering. My configuration is the
following:
/etc/sfacct/sfacct.conf:
!
! Daemon
!
daemonize: true
pidfile: /var/run/sfacctd.pid
!syslog: daemon
logfile: /var/log/sfacctd.log
sfacctd_port: 6343
!
!
! Selected plugins
!
plugins:mysql[in],mysql[out]
!
!
! Aggregation by plugin
!
aggregate[in]:vlan,dst_host
aggregate[out]:vlan,src_host
!
pre_tag_map:/etc/sfacct/pretag.map
pre_tag_filter[in]:1
pre_tag_filter[out]:2
!
debug: true
!
! Memory pool parameters
!
! SQL parameters
!
sql_host: localhost
sql_user: sfacct
sql_passwd:Wfk.aCc7
sql_db: test_pyxis2
sql_table: SFACCT
sql_refresh_time: 300
sql_history: 5m
sql_history_roundoff: m
sql_dont_try_update: true
sql_table_version: 7
sql_recovery_logfile: /var/lib/pmacct/recovery_sfacctd_log
!
/etc/sfacct/pretag.map:
! === rtr-of-01 routed vlans filtered by dst ip address
!
id=1 ip=94.125.26.124 filter='vlan 365' jeq=eval_ip_01
id=1 ip=94.125.26.124 filter='vlan 555' jeq=eval_ip_01
id=1 ip=94.125.26.124 filter='vlan 360' jeq=eval_ip_01
id=1 ip=94.125.26.124 filter='vlan 359' jeq=eval_ip_01
id=1 ip=94.125.26.124 filter='vlan 800' jeq=eval_ip_01
id=1 ip=94.125.26.124 filter='vlan 354' jeq=eval_ip_01
id=1 ip=94.125.26.124 filter='dst net 192.76.141.0/24 or dst net
194.55.246.0/23 or dst net 195.246.160.0/19 or dst net 88.215.224.0/19'
label=eval_ip_01
!
!
! === rtr-of-02 routed vlans filtered by dst ip address
!
id=1 ip=94.125.26.125 filter='vlan 365' jeq=eval_ip_02
id=1 ip=94.125.26.125 filter='vlan 555' jeq=eval_ip_02
id=1 ip=94.125.26.125 filter='vlan 360' jeq=eval_ip_02
id=1 ip=94.125.26.125 filter='vlan 359' jeq=eval_ip_02
id=1 ip=94.125.26.125 filter='vlan 800' jeq=eval_ip_02
id=1 ip=94.125.26.125 filter='vlan 354' jeq=eval_ip_02
id=1 ip=94.125.26.125 filter='dst net 192.76.141.0/24 or dst net
194.55.246.0/23 or dst net 195.246.160.0/19 or dst net 88.215.224.0/19'
label=eval_ip_02
!
!
! === rtr-of-01 routed vlans filtered by src ip address
!
id=2 ip=94.125.26.124 filter='vlan 365' jeq=eval_ip_03
id=2 ip=94.125.26.124 filter='vlan 555' jeq=eval_ip_03
id=2 ip=94.125.26.124 filter='vlan 360' jeq=eval_ip_03
id=2 ip=94.125.26.124 filter='vlan 359' jeq=eval_ip_03
id=2 ip=94.125.26.124 filter='vlan 800' jeq=eval_ip_03
id=2 ip=94.125.26.124 filter='vlan 354' jeq=eval_ip_03
id=2 ip=94.125.26.124 filter='src net 192.76.141.0/24 or src net
194.55.246.0/23 or src net 195.246.160.0/19 or src net 88.215.224.0/19'
label=eval_ip_03
!
!
! === rtr-of-02 routed vlans filtered by src ip address
!
id=2 ip=94.125.26.125 filter='vlan 365' jeq=eval_ip_04
id=2 ip=94.125.26.125 filter='vlan 555' jeq=eval_ip_04
id=2 ip=94.125.26.125 filter='vlan 360' jeq=eval_ip_04
id=2 ip=94.125.26.125 filter='vlan 359' jeq=eval_ip_04
id=2 ip=94.125.26.125 filter='vlan 800' jeq=eval_ip_04
id=2 ip=94.125.26.125 filter='vlan 354' jeq=eval_ip_04
id=2 ip=94.125.26.125 filter='src net 192.76.141.0/24 or src net
194.55.246.0/23 or src net 195.246.160.0/19 or src net 88.215.224.0/19'
label=eval_ip_04
I did some test transfers but unfortunately I don't get anything aggregated by
dst_host but flows aggregated by src_host with ip addresses that do not match
the filters.
*************************** 1. row ***************************
agent_id: 0
class_id: unknown
mac_src: 0:0:0:0:0:0
mac_dst: 0:0:0:0:0:0
vlan: 555
as_src: 0
as_dst: 0
ip_src: 88.215.253.10
ip_dst: 0.0.0.0
src_port: 0
dst_port: 0
tcp_flags: 0
ip_proto: ip
tos: 0
packets: 34
bytes: 51748
flows: 0
stamp_inserted: 2011-07-01 20:45:00
stamp_updated: 2011-07-01 20:50:01
*************************** 2. row ***************************
agent_id: 0
class_id: unknown
mac_src: 0:0:0:0:0:0
mac_dst: 0:0:0:0:0:0
vlan: 365
as_src: 0
as_dst: 0
ip_src: 78.47.201.50
ip_dst: 0.0.0.0
src_port: 0
dst_port: 0
tcp_flags: 0
ip_proto: ip
tos: 0
packets: 24
bytes: 1680
flows: 0
stamp_inserted: 2011-07-01 20:45:00
stamp_updated: 2011-07-01 20:50:01
*************************** 3. row ***************************
agent_id: 0
class_id: unknown
mac_src: 0:0:0:0:0:0
mac_dst: 0:0:0:0:0:0
vlan: 555
as_src: 0
as_dst: 0
ip_src: 88.215.253.10
ip_dst: 0.0.0.0
src_port: 0
dst_port: 0
tcp_flags: 0
ip_proto: ip
tos: 0
packets: 3
bytes: 4566
flows: 0
stamp_inserted: 2011-07-01 20:50:00
stamp_updated: 2011-07-01 20:50:15
*************************** 4. row ***************************
agent_id: 0
class_id: unknown
mac_src: 0:0:0:0:0:0
mac_dst: 0:0:0:0:0:0
vlan: 365
as_src: 0
as_dst: 0
ip_src: 78.47.201.50
ip_dst: 0.0.0.0
src_port: 0
dst_port: 0
tcp_flags: 0
ip_proto: ip
tos: 0
packets: 1
bytes: 70
flows: 0
stamp_inserted: 2011-07-01 20:50:00
stamp_updated: 2011-07-01 20:50:15
The Sflow files in pcap format can be downloaded from dropbox via
http://dl.dropbox.com/u/20778197/sflow.pcap
Cheers,
Bernd
Paolo Lucente <[email protected]> wrote:
> Hi Bernd,
>
> Well, you need to add an 'ip' clause to the statements, where 'ip'
> according to the documentation ('examples/pretag.map.example' in the pmacct
> standard distribution tarball):
>
> ! 'ip' In nfacctd it's compared against the source IP
> address
> ! of the device which is originating NetFlow packets;
> in
> ! sfacctd this is compared against the AgentId field
> of
> ! received sFlow samples.
>
> My fault not having mentioned it in my previous email - i've worked ouy
> those configuration bits on the fly.
>
> Cheers,
> Paolo
>
>
> On Sun, Jun 12, 2011 at 03:34:12PM +0000, Bernd Bornkessel wrote:
> > Hi Paolo,
> >
> > I tried to implement the pre tag filtering, but oviously I'm doing
> something wrong.
> >
> > My pretag.map:
> > id=1 filter='vlan 365' jeq=eval_ip
> > id=1 filter='vlan 1337' jeq=eval_ip
> > id=1 filter='(dst net 192.76.141.0/24 or dst net 194.55.246.0/23 or
> > dst net 195.246.160/19 or dst net 88.215.224.0/19)' label=eval_ip
> > id=2 filter='vlan 365' jeq=eval_ip
> > id=2 filter='vlan 1337' jeq=eval_ip
> > id=2 filter='(src net 192.76.141.0/24 or src net 194.55.246.0/23 or
> > src net 195.246.160/19 or src net 88.215.224.0/19)' label=eval_ip
> >
> > When I start sfacctd I get the following error in the logfile:
> > ERROR ( /etc/sfacct/pretag.map ): required key missing at line: 3.
> Required keys are: 'id', 'ip'.
> > ERROR ( /etc/sfacct/pretag.map ): required key missing at line: 4.
> Required keys are: 'id', 'ip'.
> > ERROR ( /etc/sfacct/pretag.map ): required key missing at line: 5.
> Required keys are: 'id', 'ip'.
> > ERROR ( /etc/sfacct/pretag.map ): required key missing at line: 6.
> Required keys are: 'id', 'ip'.
> > INFO ( default/core ): map '/etc/sfacct/pretag.map' successfully
> (re)loaded.
> >
> > Cheers,
> > Bernd
> >
> > > -----Urspr?ngliche Nachricht-----
> > > Von: [email protected] [mailto:pmacct-discussion-
> > > [email protected]] Im Auftrag von Paolo Lucente
> > > Gesendet: Freitag, 10. Juni 2011 15:52
> > > An: [email protected]
> > > Betreff: Re: [pmacct-discussion] Broken aggregate Filter
> > >
> > > Hi Bernd,
> > >
> > > An alternative to what Brent correctly suggested, should you really
> > > want to go for a filter, is to use a pre_tag_filter instead of an
> aggregate_filter.
> > > In the following fashion:
> > >
> > > == sfacctd.conf ==
> > > ...
> > > pre_tag_map: /path/to/pretag.map
> > > pre_tag_filter[...]: 1
> > > ...
> > > ==
> > >
> > > == pretag.map ==
> > > id=1 filter='vlan 365' jeq=eval_ip
> > > id=1 filter='vlan 1337' jeq=eval_ip
> > > ...
> > > id=1 filter='(dst net 192.76.141.0/24 or dst net [ ... ]' label=eval_ip
> ...
> > > ==
> > >
> > > So the logics would be to place a tag of 1 (default is zero indeed)
> > > to what you want to pass through; all the rest is filtered out. For
> > > the IP layer everything can be evaluated in a single filter; whereas
> > > for the VLAN part you have a filter per VLAN you want to match. JEQ,
> > > as you can expect, means jump on equal - so upon passing the VLAN
> > > check the sample is sent for IP layer check.
> > >
> > > Cheers,
> > > Paolo
> > >
> > > On Thu, Jun 09, 2011 at 08:07:04PM +0000, Bernd Bornkessel wrote:
> > > > Hi Chris,
> > > >
> > > > thanks. So vlan based filtering will not work with more than one
> > > > vlan. I'm
> > > thinking about a workaround.
> > > >
> > > > Something about the background:
> > > > We're using Arista 7120 switches in one of our DC locations. These
> > > switches mainly build the 10G layer-2 infrastructure for our vSphere
> > > environment connecting the VMware servers as well as an iSCSI
> > > storage. The second purpose for the switches is acting as our core
> routers.
> > > >
> > > > The problem is, that we only need to account traffic that is been
> > > > routed
> > > to the ISP uplinks. Unfortunately the sflow implementation for this
> > > switches is ingress only per each physical interface. Thus we need
> > > to activate sflow for each interface and filter out the required
> > > flows. So I need to filter for the vlans and our public networks.
> > > >
> > > > Two workarounds came to my mind.
> > > >
> > > > 1. We do not filter for the vlan, but use the vlan for aggregation
> > > instead. On a daily basis we remove the records for unneccessary
> > > vlans from the database.
> > > >
> > > > 2. I create an incoming and outgoing plugin/aggregate/filter for
> > > > each vlan
> > > that aggregate and write to the database independently. Is there a
> > > recommendation for a maximum plugin instances? We need at least 6
> > > vlans resulting in 12 instances.
> > > >
> > > >
> > > > Cheers,
> > > > Bernd
> > > >
> > > > > -----Urspr?ngliche Nachricht-----
> > > > > Von: [email protected]
> > > > > [mailto:pmacct-discussion- [email protected]] Im Auftrag von
> > > > > Chris Wilson
> > > > > Gesendet: Donnerstag, 9. Juni 2011 19:04
> > > > > An: [email protected]
> > > > > Betreff: Re: [pmacct-discussion] Broken aggregate Filter
> > > > >
> > > > > Hi Bernd,
> > > > >
> > > > > On Thu, 9 Jun 2011, Bernd Bornkessel wrote:
> > > > >
> > > > > > The working filter is:
> > > > > >
> > > > > > vlan and (dst net 192.76.141.0/24 or dst net 194.55.246.0/23
> > > > > > or dst net
> > > > > > 195.246.160/19 or dst net 88.215.224.0/19 or dst net
> > > > > > 62.93.212.0/23 or dst net 62.93.246.0/23 or dst net
> > > > > > 88.215.192.0/19)
> > > > > >
> > > > > > The non-working are:
> > > > > >
> > > > > > vlan and ((vlan 365 or vlan 1337) and (dst net 192.76.141.0/24
> > > > > > or dst net 194.55.246.0/23 or dst net 195.246.160/19 or dst
> > > > > > net
> > > > > > 88.215.224.0/19 or dst net 62.93.212.0/23 or dst net
> > > > > > 62.93.246.0/23 or dst net
> > > > > > 88.215.192.0/19))
> > > > > >
> > > > > > ((vlan 365 or vlan 1337) and (dst net 192.76.141.0/24 or dst
> > > > > > net
> > > > > > 194.55.246.0/23 or dst net 195.246.160/19 or dst net
> > > > > > 88.215.224.0/19 or dst net 62.93.212.0/23 or dst net
> > > > > > 62.93.246.0/23 or dst net
> > > > > > 88.215.192.0/19))
> > > > >
> > > > > I think you may be falling victim to this (from man pcap-filter(7)):
> > > > >
> > > > > vlan [vlan_id]
> > > > >
> > > > > True if the packet is an IEEE 802.1Q VLAN packet.
> > > > > If [vlan_id] is specified, only true if the packet has the
> > > > > specified
> > > vlan_id.
> > > > > Note that the first vlan keyword encountered in expression
> > > > > changes the decoding offsets for the remainder of expression on
> > > > > the assumption that the packet is a VLAN packet. The vlan
> > > > > [vlan_id] expression may be used more than once, to filter on
> > > > > VLAN hierarchies. Each use of that expression increments the
> > > > > filter offsets
> > > by 4.
> > > > >
> > > > > Therefore I don't think you can use the "vlan" keyword more than
> > > > > once in the same expression (unless you have vlan hierarchies).
> > > > > This appears to be a limitation (and a rather "unusual" one) of
> > > > > libpcap, not
> > > pmacct.
> > > > >
> > > > > If they really want to support nested vlans (and I would
> > > > > seriously question the sanity of anyone who used them) I would
> > > > > respectfully suggest that they modify the "vlan" keyword to not
> > > > > change the filter offset, and create a new keyword like "nested-
> vlan" which does.
> > > > >
> > > > > Cheers, Chris.
> > > > >
> > > > > _______________________________________________
> > > > > pmacct-discussion mailing list
> > > > > http://www.pmacct.net/#mailinglists
> > > >
> > > > _______________________________________________
> > > > pmacct-discussion mailing list
> > > > http://www.pmacct.net/#mailinglists
> > >
> > > _______________________________________________
> > > pmacct-discussion mailing list
> > > http://www.pmacct.net/#mailinglists
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
