I tried to configure lagg with the latest source from SVN head and it did not help.
Regards Adarsh -----Original Message----- From: owner-freebsd-...@freebsd.org [mailto:owner-freebsd-...@freebsd.org] On Behalf Of Adarsh Joshi Sent: Tuesday, July 10, 2012 10:53 AM To: Andrew Boyer Cc: freebsd-net@freebsd.org Subject: RE: lacp lagg port flags do not show correctly resulting in poor traffic distribution/performance Andrew, Here are the logs with LACP_DEBUG defined in ieee802.3ad_lacp.c, after typing Ifconfig lagg0 create ifconfig lagg0 laggproto lacp laggport ql0 laggport ql1 192.168.100.1 netmask 255.255.255.0 I compiled it as a standalone driver by the way. System 1: # ifconfig -v lagg0 lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=13b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,TSO4> ether 00:0e:1e:08:05:20 inet 192.168.100.1 netmask 0xffffff00 broadcast 192.168.100.255 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect status: active groups: lagg laggproto lacp lag id: [(8000,00-0E-1E-08-05-20,01D3,0000,0000), (8000,00-0E-1E-04-2C-F0,0213,0000,0000)] laggport: ql1 flags=18<COLLECTING,DISTRIBUTING> state=7D [(8000,00-0E-1E-08-05-20,01D3,8000,000D), (FFFF,00-00-00-00-00-00,0000,FFFF,0000)] laggport: ql0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3D [(8000,00-0E-1E-08-05-20,01D3,8000,000C), (8000,00-0E-1E-04-2C-F0,0213,8000,000E)] System 2: # ifconfig -v lagg0 lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=13b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,TSO4> ether 00:0e:1e:04:2c:f0 inet 192.168.100.2 netmask 0xffffff00 broadcast 192.168.100.255 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect status: active groups: lagg laggproto lacp lag id: [(8000,00-0E-1E-04-2C-F0,0213,0000,0000), (FFFF,00-00-00-00-00-00,0000,0000,0000)] laggport: ql1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=7D [(8000,00-0E-1E-04-2C-F0,0213,8000,000F), (FFFF,00-00-00-00-00-00,0000,FFFF,0000)] laggport: ql0 flags=18<COLLECTING,DISTRIBUTING> state=3D [(8000,00-0E-1E-04-2C-F0,0213,8000,000E), (8000,00-0E-1E-08-05-20,01D3,8000,000C)] System 1 logs : Jul 10 10:38:49 bsd-14 kernel: lacp_attach[738] : lacp attached Jul 10 10:38:49 bsd-14 kernel: lacp_attach[740] : lacp_defined Jul 10 10:38:49 bsd-14 kernel: lagg0: link state changed to UP Jul 10 10:38:49 bsd-14 kernel: ql0: media changed 0x0 -> 0x100033, ether = 1, fdx = 1, link = 1 Jul 10 10:38:49 bsd-14 kernel: ql0: -> UNSELECTED Jul 10 10:38:49 bsd-14 kernel: ql1: media changed 0x0 -> 0x100033, ether = 1, fdx = 1, link = 1 Jul 10 10:38:49 bsd-14 kernel: ql1: -> UNSELECTED Jul 10 10:38:49 bsd-14 kernel: lacp_select_tx_port: no active aggregator Jul 10 10:38:50 bsd-14 kernel: ql1: port lagid=[(8000,00-0E-1E-08-05-20,01D3,8000,000D),(0000,00-00-00-00-00-00,0000,0000,0000)] Jul 10 10:38:50 bsd-14 kernel: ql1: aggregator created Jul 10 10:38:50 bsd-14 kernel: ql1: aggregator lagid=[(8000,00-0E-1E-08-05-20,01D3,0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)] Jul 10 10:38:50 bsd-14 kernel: ql1: mux_state 0 -> 1 Jul 10 10:38:50 bsd-14 kernel: ql0: port lagid=[(8000,00-0E-1E-08-05-20,01D3,8000,000C),(0000,00-00-00-00-00-00,0000,0000,0000)] Jul 10 10:38:50 bsd-14 kernel: ql0: aggregator created Jul 10 10:38:50 bsd-14 kernel: ql0: aggregator lagid=[(8000,00-0E-1E-08-05-20,01D3,0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)] Jul 10 10:38:50 bsd-14 kernel: ql0: mux_state 0 -> 1 Jul 10 10:38:51 bsd-14 kernel: ql1: lacpdu transmit Jul 10 10:38:51 bsd-14 kernel: actor=(8000,00-0E-1E-08-05-20,01D3,8000,000D) Jul 10 10:38:51 bsd-14 kernel: actor.state=85<ACTIVITY,AGGREGATION,EXPIRED> Jul 10 10:38:51 bsd-14 kernel: partner=(0000,00-00-00-00-00-00,0000,0000,0000) Jul 10 10:38:51 bsd-14 kernel: partner.state=2<TIMEOUT> Jul 10 10:38:51 bsd-14 kernel: maxdelay=0 Jul 10 10:38:51 bsd-14 kernel: ql0: lacpdu transmit Jul 10 10:38:51 bsd-14 kernel: actor=(8000,00-0E-1E-08-05-20,01D3,8000,000C) Jul 10 10:38:51 bsd-14 kernel: actor.state=85<ACTIVITY,AGGREGATION,EXPIRED> Jul 10 10:38:51 bsd-14 kernel: partner=(0000,00-00-00-00-00-00,0000,0000,0000) Jul 10 10:38:51 bsd-14 kernel: partner.state=2<TIMEOUT> Jul 10 10:38:51 bsd-14 kernel: maxdelay=0 Jul 10 10:38:51 bsd-14 kernel: ql0: lacpdu receive Jul 10 10:38:51 bsd-14 kernel: actor=(8000,00-0E-1E-04-2C-F0,0213,8000,000E) Jul 10 10:38:51 bsd-14 kernel: actor.state=5<ACTIVITY,AGGREGATION> Jul 10 10:38:51 bsd-14 kernel: partner=(8000,00-0E-1E-08-05-20,01D3,8000,000C) Jul 10 10:38:51 bsd-14 kernel: partner.state=85<ACTIVITY,AGGREGATION,EXPIRED> Jul 10 10:38:51 bsd-14 kernel: maxdelay=0 Jul 10 10:38:51 bsd-14 kernel: ql0: old pstate 2<TIMEOUT> Jul 10 10:38:51 bsd-14 kernel: ql0: new pstate 5<ACTIVITY,AGGREGATION> Jul 10 10:38:51 bsd-14 kernel: ql0: partner timeout changed Jul 10 10:38:51 bsd-14 kernel: ql0: lacpdu receive Jul 10 10:38:51 bsd-14 kernel: actor=(8000,00-0E-1E-04-2C-F0,0213,8000,000E) Jul 10 10:38:51 bsd-14 kernel: actor.state=5<ACTIVITY,AGGREGATION> Jul 10 10:38:51 bsd-14 kernel: partner=(8000,00-0E-1E-08-05-20,01D3,8000,000C) Jul 10 10:38:51 bsd-14 kernel: partner.state=85<ACTIVITY,AGGREGATION,EXPIRED> Jul 10 10:38:51 bsd-14 kernel: maxdelay=0 Jul 10 10:38:52 bsd-14 kernel: ql1: lacp_sm_rx_timer: EXPIRED -> DEFAULTED Jul 10 10:38:52 bsd-14 kernel: ql1: partner timeout changed Jul 10 10:38:52 bsd-14 kernel: ql1: lacp_sm_mux_timer: aggregator [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)], pending 1 -> 0 Jul 10 10:38:52 bsd-14 kernel: ql1: collecting disabled Jul 10 10:38:52 bsd-14 kernel: lacp_aggregator_delref: lagid=[(8000,00-0E-1E-08-05-20,01D3,0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)], refcnt 1 -> 0 Jul 10 10:38:52 bsd-14 kernel: ql1: mux_state 1 -> 0 Jul 10 10:38:52 bsd-14 kernel: ql1: lacpdu transmit Jul 10 10:38:52 bsd-14 kernel: actor=(8000,00-0E-1E-08-05-20,01D3,8000,000D) Jul 10 10:38:52 bsd-14 kernel: actor.state=45<ACTIVITY,AGGREGATION,DEFAULTED> Jul 10 10:38:52 bsd-14 kernel: partner=(FFFF,00-00-00-00-00-00,0000,FFFF,0000) Jul 10 10:38:52 bsd-14 kernel: partner.state=3c<AGGREGATION,SYNC,COLLECTING,DISTRIBUTING> Jul 10 10:38:52 bsd-14 kernel: maxdelay=0 Jul 10 10:38:52 bsd-14 kernel: ql0: lacp_sm_mux_timer: aggregator [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)], pending 1 -> 0 Jul 10 10:38:52 bsd-14 kernel: ql0: collecting disabled Jul 10 10:38:52 bsd-14 kernel: lacp_aggregator_delref: lagid=[(8000,00-0E-1E-08-05-20,01D3,0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)], refcnt 1 -> 0 Jul 10 10:38:52 bsd-14 kernel: ql0: mux_state 1 -> 0 Jul 10 10:38:52 bsd-14 kernel: ql0: lacpdu transmit Jul 10 10:38:52 bsd-14 kernel: actor=(8000,00-0E-1E-08-05-20,01D3,8000,000C) Jul 10 10:38:52 bsd-14 kernel: actor.state=5<ACTIVITY,AGGREGATION> Jul 10 10:38:52 bsd-14 kernel: partner=(8000,00-0E-1E-04-2C-F0,0213,8000,000E) Jul 10 10:38:52 bsd-14 kernel: partner.state=5<ACTIVITY,AGGREGATION> Jul 10 10:38:52 bsd-14 kernel: maxdelay=0 Jul 10 10:38:53 bsd-14 kernel: ql1: port lagid=[(8000,00-0E-1E-08-05-20,01D3,8000,000D),(FFFF,00-00-00-00-00-00,0000,FFFF,0000)] Jul 10 10:38:53 bsd-14 kernel: ql1: aggregator created Jul 10 10:38:53 bsd-14 kernel: ql1: aggregator lagid=[(8000,00-0E-1E-08-05-20,01D3,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)] Jul 10 10:38:53 bsd-14 kernel: ql1: mux_state 0 -> 1 Jul 10 10:38:53 bsd-14 kernel: ql0: port lagid=[(8000,00-0E-1E-08-05-20,01D3,8000,000C),(8000,00-0E-1E-04-2C-F0,0213,8000,000E)] Jul 10 10:38:53 bsd-14 kernel: ql0: aggregator created Jul 10 10:38:53 bsd-14 kernel: ql0: aggregator lagid=[(8000,00-0E-1E-08-05-20,01D3,0000,0000),(8000,00-0E-1E-04-2C-F0,0213,0000,0000)] Jul 10 10:38:53 bsd-14 kernel: ql0: mux_state 0 -> 1 Jul 10 10:38:54 bsd-14 kernel: ql0: lacpdu receive Jul 10 10:38:54 bsd-14 kernel: actor=(8000,00-0E-1E-04-2C-F0,0213,8000,000E) Jul 10 10:38:54 bsd-14 kernel: actor.state=d<ACTIVITY,AGGREGATION,SYNC> Jul 10 10:38:54 bsd-14 kernel: partner=(8000,00-0E-1E-08-05-20,01D3,8000,000C) Jul 10 10:38:54 bsd-14 kernel: partner.state=5<ACTIVITY,AGGREGATION> Jul 10 10:38:54 bsd-14 kernel: maxdelay=0 Jul 10 10:38:54 bsd-14 kernel: ql0: old pstate 5<ACTIVITY,AGGREGATION> Jul 10 10:38:54 bsd-14 kernel: ql0: new pstate d<ACTIVITY,AGGREGATION,SYNC> Jul 10 10:38:55 bsd-14 kernel: ql1: lacp_sm_mux_timer: aggregator [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)], pending 1 -> 0 Jul 10 10:38:55 bsd-14 kernel: ql1: collecting disabled Jul 10 10:38:55 bsd-14 kernel: ql1: mux_state 1 -> 2 Jul 10 10:38:55 bsd-14 kernel: ql1: collecting enabled Jul 10 10:38:55 bsd-14 kernel: ql1: mux_state 2 -> 3 Jul 10 10:38:55 bsd-14 kernel: ql1: enable distributing on aggregator [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)], nports 0 -> 1 Jul 10 10:38:55 bsd-14 kernel: lacp_select_active_aggregator: Jul 10 10:38:55 bsd-14 kernel: [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)], speed=10000000000, nports=1 Jul 10 10:38:55 bsd-14 kernel: active aggregator changed Jul 10 10:38:55 bsd-14 kernel: old (none) Jul 10 10:38:55 bsd-14 kernel: new [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)] Jul 10 10:38:55 bsd-14 kernel: Set table 1 with 1 ports Jul 10 10:38:55 bsd-14 kernel: lacp_suppress_distributing Jul 10 10:38:55 bsd-14 kernel: ql1: marker transmit, port=13, sys=00:0e:1e:08:05:20, id=1 Jul 10 10:38:55 bsd-14 kernel: ql0: marker transmit, port=12, sys=00:0e:1e:08:05:20, id=1 Jul 10 10:38:55 bsd-14 kernel: ql1: mux_state 3 -> 4 Jul 10 10:38:55 bsd-14 kernel: ql1: lacpdu transmit Jul 10 10:38:55 bsd-14 kernel: ql0: marker response, port=12, sys=00:0e:1e:08:05:20, id=1 Jul 10 10:38:55 bsd-14 kernel: actor=(8000,00-0E-1E-08-05-20,01D3,8000,000D) Jul 10 10:38:55 bsd-14 kernel: actor.state=7d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING,DEFAULTED> Jul 10 10:38:55 bsd-14 kernel: partner=(FFFF,00-00-00-00-00-00,0000,FFFF,0000) Jul 10 10:38:55 bsd-14 kernel: partner.state=3c<AGGREGATION,SYNC,COLLECTING,DISTRIBUTING> Jul 10 10:38:55 bsd-14 kernel: maxdelay=0 Jul 10 10:38:55 bsd-14 kernel: ql0: lacp_sm_mux_timer: aggregator [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(8000,00-0E-1E-04-2C-F0,0213,0000,0000)], pending 1 -> 0 Jul 10 10:38:55 bsd-14 kernel: ql0: collecting disabled Jul 10 10:38:55 bsd-14 kernel: ql0: mux_state 1 -> 2 Jul 10 10:38:55 bsd-14 kernel: ql0: collecting enabled Jul 10 10:38:55 bsd-14 kernel: ql0: mux_state 2 -> 3 Jul 10 10:38:55 bsd-14 kernel: ql0: lacpdu transmit Jul 10 10:38:55 bsd-14 kernel: actor=(8000,00-0E-1E-08-05-20,01D3,8000,000C) Jul 10 10:38:55 bsd-14 kernel: actor.state=1d<ACTIVITY,AGGREGATION,SYNC,COLLECTING> Jul 10 10:38:55 bsd-14 kernel: partner=(8000,00-0E-1E-04-2C-F0,0213,8000,000E) Jul 10 10:38:55 bsd-14 kernel: partner.state=d<ACTIVITY,AGGREGATION,SYNC> Jul 10 10:38:55 bsd-14 kernel: maxdelay=0 Jul 10 10:38:55 bsd-14 kernel: ql0: lacpdu receive Jul 10 10:38:55 bsd-14 kernel: actor=(8000,00-0E-1E-04-2C-F0,0213,8000,000E) Jul 10 10:38:55 bsd-14 kernel: actor.state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING> Jul 10 10:38:55 bsd-14 kernel: partner=(8000,00-0E-1E-08-05-20,01D3,8000,000C) Jul 10 10:38:55 bsd-14 kernel: partner.state=1d<ACTIVITY,AGGREGATION,SYNC,COLLECTING> Jul 10 10:38:55 bsd-14 kernel: maxdelay=0 Jul 10 10:38:55 bsd-14 kernel: ql0: old pstate d<ACTIVITY,AGGREGATION,SYNC> Jul 10 10:38:55 bsd-14 kernel: ql0: new pstate 3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING> Jul 10 10:38:56 bsd-14 kernel: ql0: enable distributing on aggregator [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(8000,00-0E-1E-04-2C-F0,0213,0000,0000)], nports 0 -> 1 Jul 10 10:38:56 bsd-14 kernel: lacp_select_active_aggregator: Jul 10 10:38:56 bsd-14 kernel: [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)], speed=10000000000, nports=1 Jul 10 10:38:56 bsd-14 kernel: [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(8000,00-0E-1E-04-2C-F0,0213,0000,0000)], speed=10000000000, nports=1 Jul 10 10:38:56 bsd-14 kernel: active aggregator changed Jul 10 10:38:56 bsd-14 kernel: old [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)] Jul 10 10:38:56 bsd-14 kernel: new [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(8000,00-0E-1E-04-2C-F0,0213,0000,0000)] Jul 10 10:38:56 bsd-14 kernel: Set table 0 with 1 ports Jul 10 10:38:56 bsd-14 kernel: lacp_suppress_distributing Jul 10 10:38:56 bsd-14 kernel: ql1: marker transmit, port=13, sys=00:0e:1e:08:05:20, id=2 Jul 10 10:38:56 bsd-14 kernel: ql0: marker transmit, port=12, sys=00:0e:1e:08:05:20, id=2 Jul 10 10:38:56 bsd-14 kernel: ql0: mux_state 3 -> 4 Jul 10 10:38:56 bsd-14 kernel: ql0: marker response, port=12, sys=00:0e:1e:08:05:20, id=2 Jul 10 10:38:59 bsd-14 kernel: lacp_transit_expire ^C System 2 logs : Jul 10 02:38:24 bsd-15 kernel: lacp_attach[738] : lacp attached Jul 10 02:38:24 bsd-15 kernel: lacp_attach[740] : lacp_defined Jul 10 02:38:24 bsd-15 kernel: lagg0: link state changed to UP Jul 10 02:38:24 bsd-15 kernel: ql0: media changed 0x0 -> 0x100033, ether = 1, fdx = 1, link = 1 Jul 10 02:38:24 bsd-15 kernel: ql0: -> UNSELECTED Jul 10 02:38:24 bsd-15 kernel: ql1: media changed 0x0 -> 0x100033, ether = 1, fdx = 1, link = 1 Jul 10 02:38:24 bsd-15 kernel: ql1: -> UNSELECTED Jul 10 02:38:24 bsd-15 kernel: lacp_select_tx_port: no active aggregator Jul 10 02:38:25 bsd-15 kernel: ql1: port lagid=[(8000,00-0E-1E-04-2C-F0,0213,8000,000F),(0000,00-00-00-00-00-00,0000,0000,0000)] Jul 10 02:38:25 bsd-15 kernel: ql1: aggregator created Jul 10 02:38:25 bsd-15 kernel: ql1: aggregator lagid=[(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)] Jul 10 02:38:25 bsd-15 kernel: ql1: mux_state 0 -> 1 Jul 10 02:38:25 bsd-15 kernel: ql0: port lagid=[(8000,00-0E-1E-04-2C-F0,0213,8000,000E),(0000,00-00-00-00-00-00,0000,0000,0000)] Jul 10 02:38:25 bsd-15 kernel: ql0: aggregator created Jul 10 02:38:25 bsd-15 kernel: ql0: aggregator lagid=[(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)] Jul 10 02:38:25 bsd-15 kernel: ql0: mux_state 0 -> 1 Jul 10 02:38:26 bsd-15 kernel: ql0: lacpdu receive Jul 10 02:38:26 bsd-15 kernel: actor=(8000,00-0E-1E-08-05-20,01D3,8000,000C) Jul 10 02:38:26 bsd-15 kernel: actor.state=85<ACTIVITY,AGGREGATION,EXPIRED> Jul 10 02:38:26 bsd-15 kernel: partner=(0000,00-00-00-00-00-00,0000,0000,0000) Jul 10 02:38:26 bsd-15 kernel: partner.state=2<TIMEOUT> Jul 10 02:38:26 bsd-15 kernel: maxdelay=0 Jul 10 02:38:26 bsd-15 kernel: ql0: lacp_sm_rx_update_ntt: assert ntt Jul 10 02:38:26 bsd-15 kernel: ql0: old pstate 2<TIMEOUT> Jul 10 02:38:26 bsd-15 kernel: ql0: new pstate 85<ACTIVITY,AGGREGATION,EXPIRED> Jul 10 02:38:26 bsd-15 kernel: ql0: partner timeout changed Jul 10 02:38:26 bsd-15 kernel: ql0: lacpdu transmit Jul 10 02:38:26 bsd-15 kernel: actor=(8000,00-0E-1E-04-2C-F0,0213,8000,000E) Jul 10 02:38:26 bsd-15 kernel: actor.state=5<ACTIVITY,AGGREGATION> Jul 10 02:38:26 bsd-15 kernel: partner=(8000,00-0E-1E-08-05-20,01D3,8000,000C) Jul 10 02:38:26 bsd-15 kernel: partner.state=85<ACTIVITY,AGGREGATION,EXPIRED> Jul 10 02:38:26 bsd-15 kernel: maxdelay=0 Jul 10 02:38:26 bsd-15 kernel: ql1: lacpdu transmit Jul 10 02:38:26 bsd-15 kernel: actor=(8000,00-0E-1E-04-2C-F0,0213,8000,000F) Jul 10 02:38:26 bsd-15 kernel: actor.state=85<ACTIVITY,AGGREGATION,EXPIRED> Jul 10 02:38:26 bsd-15 kernel: partner=(0000,00-00-00-00-00-00,0000,0000,0000) Jul 10 02:38:26 bsd-15 kernel: partner.state=2<TIMEOUT> Jul 10 02:38:26 bsd-15 kernel: maxdelay=0 Jul 10 02:38:26 bsd-15 kernel: ql0: collecting disabled Jul 10 02:38:26 bsd-15 kernel: lacp_aggregator_delref: lagid=[(8000,00-0E-1E-04-2C-F0,0213, 0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)], refcnt 1 -> 0 Jul 10 02:38:26 bsd-15 kernel: ql0: mux_state 1 -> 0 Jul 10 02:38:26 bsd-15 kernel: ql0: lacpdu transmit Jul 10 02:38:26 bsd-15 kernel: actor=(8000,00-0E-1E-04-2C-F0,0213,8000,000E) Jul 10 02:38:26 bsd-15 kernel: actor.state=5<ACTIVITY,AGGREGATION> Jul 10 02:38:26 bsd-15 kernel: partner=(8000,00-0E-1E-08-05-20,01D3,8000,000C) Jul 10 02:38:26 bsd-15 kernel: partner.state=85<ACTIVITY,AGGREGATION,EXPIRED> Jul 10 02:38:26 bsd-15 kernel: maxdelay=0 Jul 10 02:38:27 bsd-15 kernel: ql0: lacpdu receive Jul 10 02:38:27 bsd-15 kernel: actor=(8000,00-0E-1E-08-05-20,01D3,8000,000C) Jul 10 02:38:27 bsd-15 kernel: actor.state=5<ACTIVITY,AGGREGATION> Jul 10 02:38:27 bsd-15 kernel: partner=(8000,00-0E-1E-04-2C-F0,0213,8000,000E) Jul 10 02:38:27 bsd-15 kernel: partner.state=5<ACTIVITY,AGGREGATION> Jul 10 02:38:27 bsd-15 kernel: maxdelay=0 Jul 10 02:38:27 bsd-15 kernel: ql0: old pstate 85<ACTIVITY,AGGREGATION,EXPIRED> Jul 10 02:38:27 bsd-15 kernel: ql0: new pstate 5<ACTIVITY,AGGREGATION> Jul 10 02:38:27 bsd-15 kernel: ql1: lacp_sm_rx_timer: EXPIRED -> DEFAULTED Jul 10 02:38:27 bsd-15 kernel: ql1: partner timeout changed Jul 10 02:38:27 bsd-15 kernel: ql1: lacp_sm_mux_timer: aggregator [(8000,00-0E-1E-04-2C-F0, 0213,0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)], pending 1 -> 0 Jul 10 02:38:27 bsd-15 kernel: ql1: collecting disabled Jul 10 02:38:27 bsd-15 kernel: lacp_aggregator_delref: lagid=[(8000,00-0E-1E-04-2C-F0,0213, 0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)], refcnt 1 -> 0 Jul 10 02:38:27 bsd-15 kernel: ql1: mux_state 1 -> 0 Jul 10 02:38:27 bsd-15 kernel: ql1: lacpdu transmit Jul 10 02:38:27 bsd-15 kernel: actor=(8000,00-0E-1E-04-2C-F0,0213,8000,000F) Jul 10 02:38:27 bsd-15 kernel: actor.state=45<ACTIVITY,AGGREGATION,DEFAULTED> Jul 10 02:38:27 bsd-15 kernel: partner=(FFFF,00-00-00-00-00-00,0000,FFFF,0000) Jul 10 02:38:27 bsd-15 kernel: partner.state=3c<AGGREGATION,SYNC,COLLECTING,DISTRIBUTING> Jul 10 02:38:27 bsd-15 kernel: maxdelay=0 Jul 10 02:38:27 bsd-15 kernel: ql0: port lagid=[(8000,00-0E-1E-04-2C-F0,0213,8000,000E),(8000,00-0E-1E-08-05-20,01D3,8000,000C)] Jul 10 02:38:27 bsd-15 kernel: ql0: aggregator created Jul 10 02:38:27 bsd-15 kernel: ql0: aggregator lagid=[(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(8000,00-0E-1E-08-05-20,01D3,0000,0000)] Jul 10 02:38:27 bsd-15 kernel: ql0: mux_state 0 -> 1 Jul 10 02:38:28 bsd-15 kernel: ql1: port lagid=[(8000,00-0E-1E-04-2C-F0,0213,8000,000F),(FFFF,00-00-00-00-00-00,0000,FFFF,0000)] Jul 10 02:38:28 bsd-15 kernel: ql1: aggregator created Jul 10 02:38:28 bsd-15 kernel: ql1: aggregator lagid=[(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)] Jul 10 02:38:28 bsd-15 kernel: ql1: mux_state 0 -> 1 Jul 10 02:38:29 bsd-15 kernel: ql0: lacp_sm_mux_timer: aggregator [(8000,00-0E-1E-04-2C-F0, 0213,0000,0000),(8000,00-0E-1E-08-05-20,01D3,0000,0000)], pending 1 -> 0 Jul 10 02:38:29 bsd-15 kernel: ql0: collecting disabled Jul 10 02:38:29 bsd-15 kernel: ql0: mux_state 1 -> 2 Jul 10 02:38:29 bsd-15 kernel: ql0: lacpdu transmit Jul 10 02:38:29 bsd-15 kernel: actor=(8000,00-0E-1E-04-2C-F0,0213,8000,000E) Jul 10 02:38:29 bsd-15 kernel: actor.state=d<ACTIVITY,AGGREGATION,SYNC> Jul 10 02:38:29 bsd-15 kernel: partner=(8000,00-0E-1E-08-05-20,01D3,8000,000C) Jul 10 02:38:29 bsd-15 kernel: partner.state=5<ACTIVITY,AGGREGATION> Jul 10 02:38:29 bsd-15 kernel: maxdelay=0 Jul 10 02:38:30 bsd-15 kernel: ql0: lacpdu receive Jul 10 02:38:30 bsd-15 kernel: actor=(8000,00-0E-1E-08-05-20,01D3,8000,000C) Jul 10 02:38:30 bsd-15 kernel: actor.state=1d<ACTIVITY,AGGREGATION,SYNC,COLLECTING> Jul 10 02:38:30 bsd-15 kernel: partner=(8000,00-0E-1E-04-2C-F0,0213,8000,000E) Jul 10 02:38:30 bsd-15 kernel: partner.state=d<ACTIVITY,AGGREGATION,SYNC> Jul 10 02:38:30 bsd-15 kernel: maxdelay=0 Jul 10 02:38:30 bsd-15 kernel: ql0: old pstate 5<ACTIVITY,AGGREGATION> Jul 10 02:38:30 bsd-15 kernel: ql0: new pstate 1d<ACTIVITY,AGGREGATION,SYNC,COLLECTING> Jul 10 02:38:30 bsd-15 kernel: ql1: lacp_sm_mux_timer: aggregator [(8000,00-0E-1E-04-2C-F0, 0213,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)], pending 1 -> 0 Jul 10 02:38:30 bsd-15 kernel: ql1: collecting disabled Jul 10 02:38:30 bsd-15 kernel: ql1: mux_state 1 -> 2 Jul 10 02:38:30 bsd-15 kernel: ql1: collecting enabled Jul 10 02:38:30 bsd-15 kernel: ql1: mux_state 2 -> 3 Jul 10 02:38:30 bsd-15 kernel: ql1: enable distributing on aggregator [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)], nports 0 -> 1 Jul 10 02:38:30 bsd-15 kernel: lacp_select_active_aggregator: Jul 10 02:38:30 bsd-15 kernel: [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)], speed=10000000000, nports=1 Jul 10 02:38:30 bsd-15 kernel: active aggregator changed Jul 10 02:38:30 bsd-15 kernel: old (none) Jul 10 02:38:30 bsd-15 kernel: new [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(FFFF,00-00-00 00-00-00,0000,0000,0000)] Jul 10 02:38:30 bsd-15 kernel: Set table 1 with 1 ports Jul 10 02:38:30 bsd-15 kernel: lacp_suppress_distributing Jul 10 02:38:30 bsd-15 kernel: ql1: marker transmit, port=15, sys=00:0e:1e:04:2c:f0, id=1 Jul 10 02:38:30 bsd-15 kernel: ql0: marker transmit, port=14, sys=00:0e:1e:04:2c:f0, id=1 Jul 10 02:38:30 bsd-15 kernel: ql1: mux_state 3 -> 4 Jul 10 02:38:30 bsd-15 kernel: ql1: lacpdu transmit Jul 10 02:38:30 bsd-15 kernel: ql0: marker response, port=14, sys=00:0e:1e:04:2c:f0, id=1 Jul 10 02:38:30 bsd-15 kernel: actor=(8000,00-0E-1E-04-2C-F0,0213,8000,000F) Jul 10 02:38:30 bsd-15 kernel: actor.state=7d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING,DEFAULTED> Jul 10 02:38:30 bsd-15 kernel: partner=(FFFF,00-00-00-00-00-00,0000,FFFF,0000) Jul 10 02:38:30 bsd-15 kernel: partner.state=3c<AGGREGATION,SYNC,COLLECTING,DISTRIBUTING> Jul 10 02:38:30 bsd-15 kernel: maxdelay=0 Jul 10 02:38:30 bsd-15 kernel: ql0: collecting enabled Jul 10 02:38:30 bsd-15 kernel: ql0: mux_state 2 -> 3 Jul 10 02:38:30 bsd-15 kernel: ql0: enable distributing on aggregator [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(8000,00-0E-1E-08-05-20,01D3,0000,0000)], nports 0 -> 1 Jul 10 02:38:30 bsd-15 kernel: lacp_select_active_aggregator: Jul 10 02:38:30 bsd-15 kernel: [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(8000,00-0E-1E-08-05-20,01D3,0000,0000)], speed=10000000000, nports=1 Jul 10 02:38:30 bsd-15 kernel: [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)], speed=10000000000, nports=1 Jul 10 02:38:30 bsd-15 kernel: active aggregator not changed Jul 10 02:38:30 bsd-15 kernel: new [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)] Jul 10 02:38:30 bsd-15 kernel: ql0: mux_state 3 -> 4 Jul 10 02:38:30 bsd-15 kernel: ql0: lacpdu transmit Jul 10 02:38:30 bsd-15 kernel: actor=(8000,00-0E-1E-04-2C-F0,0213,8000,000E) Jul 10 02:38:30 bsd-15 kernel: actor.state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING> Jul 10 02:38:30 bsd-15 kernel: partner=(8000,00-0E-1E-08-05-20,01D3,8000,000C) Jul 10 02:38:30 bsd-15 kernel: partner.state=1d<ACTIVITY,AGGREGATION,SYNC,COLLECTING> Jul 10 02:38:30 bsd-15 kernel: maxdelay=0 Jul 10 02:38:33 bsd-15 kernel: lacp_transit_expire ^C Let me know if you need more info. Thanks Adarsh -----Original Message----- From: owner-freebsd-...@freebsd.org [mailto:owner-freebsd-...@freebsd.org] On Behalf Of Adarsh Joshi Sent: Tuesday, July 10, 2012 10:08 AM To: Andrew Boyer Cc: freebsd-net@freebsd.org Subject: RE: lacp lagg port flags do not show correctly resulting in poor traffic distribution/performance Andrew, Thanks for the reply. The reason for my suspicion on the portflags is thus (extracted from the ifconfig output in my previous mail): System 1: Laggport: ql1 flags = 18 state = 7D Laggport: ql0 flags = 1c state = 3D System 2: Laggport: ql1 flags = 1c state = 7D Laggport: ql0 flags = 18 state = 3D I should have explained my setup to you before. Here it is. Both the ql0 interfaces of the 2 systems are connected using a single cable and ql1 interfaces of the 2 systems are connected using a single cable. System 1 System 2 ql0 <=======> ql0 ql1 <=======> ql1 With this setup, I don't think it is possible for ports ql0 to talk to their partners (each other) and ql1 ports not getting a response from their partner and still get the lagg configuration I have posted. I thought the portflags are dependent on the LACP state. But I see different flags for the same LACP state (For the state 7D, ql1 on system 1 shows flags = 18 and ql1 on system 2 shows flags = 1c). Or is my understanding totally wrong? I will send the LACP_DEBUG logs within the hour. Thanks Adarsh From: Andrew Boyer [mailto:abo...@averesystems.com] Sent: Tuesday, July 10, 2012 5:57 AM To: Adarsh Joshi Cc: freebsd-net@freebsd.org Subject: Re: lacp lagg port flags do not show correctly resulting in poor traffic distribution/performance On Jul 9, 2012, at 8:38 PM, Adarsh Joshi wrote: Hi, I am trying to configure lacp lagg interfaces with 2 systems connected back to back as follows: Ifconfig lagg0 create Ifconfig lagg0 laggproto lacp laggport ql0 laggport ql1 192.168.100.1 netmask 255.255.255.0 Sometimes, the lag interface comes up correctly but sometimes the laggport flags do not show properly. Instead of 1c<ACTIVE,COLLECTING,DISTRIBUTING>, it shows values of 18. I have seen similar issues reported on various forums with no solution. Looking at the lagg driver code and reading the standard, I thought the laggport flags ( defined in if_lagg.h) are based on the LACP_STATE_BITS in file ieee8023ad_lacp.h. But the following ifconfig -v output does not make any sense to me. My concern is that when all the interfaces show flags as 1c, the traffic is distributed across both the interfaces uniformly and I get aggregated throughput. If not, the traffic flows only on 1 interface. Is this a bug? How do I solve this? Or am I doing something wrong? I am using Free-BSD 9.0 release. System 1: # ifconfig -v lagg0 lag id: [(8000,00-0E-1E-08-05-20,0213,0000,0000), (8000,00-0E-1E-04-2C-F0,0213,0000,0000)] laggport: ql1 flags=18<COLLECTING,DISTRIBUTING> state=7D [(8000,00-0E-1E-08-05-20,0213,8000,000F), (FFFF,00-00-00-00-00-00,0000,FFFF,0000)] laggport: ql0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3D [(8000,00-0E-1E-08-05-20,0213,8000,000E), (8000,00-0E-1E-04-2C-F0,0213,8000,000E)] System 2: # ifconfig -v lagg0 lag id: [(8000,00-0E-1E-04-2C-F0,0213,0000,0000), (FFFF,00-00-00-00-00-00,0000,0000,0000)] laggport: ql1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=7D [(8000,00-0E-1E-04-2C-F0,0213,8000,000F), (FFFF,00-00-00-00-00-00,0000,FFFF,0000)] laggport: ql0 flags=18<COLLECTING,DISTRIBUTING> state=3D [(8000,00-0E-1E-04-2C-F0,0213,8000,000E), (8000,00-0E-1E-08-05-20,0213,8000,000E)] thanks Adarsh I don't think you have a port flags problem per se; the flags are correctly displaying the state of the lagg. Your problem is that your systems aren't negotiating the correct lagg configuration. Each tuple after the laggport represents the [(actor state),(partner state)]. Ports ql0 have been able to talk to their partners (each other). Neither ql1 port has seen a response from a partner, though. You could try restarting the state machine on one box with 'ifconfig lagg0 laggproto lacp'. To see the negotiation you'll need to rebuild your kernel with '#define LACP_DEBUG 1' added to the top of sys/net/ieee802.3ad_lacp.c. Or upgrade to a newer stable snapshot that has the net.lacp_debug sysctl and turn it on. Or just turn off LACP. What does it get you in this configuration? Hope this helps, Andrew -------------------------------------------------- Andrew Boyer abo...@averesystems.com<mailto:abo...@averesystems.com> ________________________________ This message and any attached documents contain information from QLogic Corporation or its wholly-owned subsidiaries that may be confidential. If you are not the intended recipient, you may not read, copy, distribute, or use this information. If you have received this transmission in error, please notify the sender immediately by reply e-mail and then delete this message. _______________________________________________ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org" This message and any attached documents contain information from QLogic Corporation or its wholly-owned subsidiaries that may be confidential. If you are not the intended recipient, you may not read, copy, distribute, or use this information. If you have received this transmission in error, please notify the sender immediately by reply e-mail and then delete this message. _______________________________________________ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org" This message and any attached documents contain information from QLogic Corporation or its wholly-owned subsidiaries that may be confidential. If you are not the intended recipient, you may not read, copy, distribute, or use this information. If you have received this transmission in error, please notify the sender immediately by reply e-mail and then delete this message. _______________________________________________ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
