FreeBSD 10-stable (r274577) LACP / IEEE 802.3ad with TP-Link TL-SG2008 - not working
Hey Net - In probably a poor, cheap choice, I picked up a TP-Link TL-SG2008 Desktop Smart Switch, which supports LACP/802.3ad. I’m currently running 10.1-STABLE r274577 on the machine I’m testing with. I’m testing right now with just 1 port (two ports didn’t work either.). Hardware is Supermicro X7DB8. em1: port 0x2020-0x203f mem 0xd826-0xd827,0xd824-0xd825 irq 19 at device 0.1 on pci6 em1@pci0:6:0:1: class=0x02 card=0x109615d9 chip=0x10968086 rev=0x01 hdr=0x00 vendor = 'Intel Corporation' device = '80003ES2LAN Gigabit Ethernet Controller (Copper)' class = network subclass = ethernet > ifconfig lagg0 lagg0: flags=8843 metric 0 mtu 1500 options=4219b ether 00:30:48:35:cc:25 inet 10.1.10.150 netmask 0xff00 broadcast 10.1.10.255 nd6 options=29 media: Ethernet autoselect status: active laggproto lacp lagghash l2,l3,l4 laggport: em1 flags=0<> Setting net.link.lagg.lacp.debug=2, here is the output from the the host trying to negotiate : em1: lacp_sm_rx_update_default_selected em1: lacp_sm_rx_update_selected_from_peerinfo em1: lacp_sm_rx_record_default em1: lacp_sm_mux: state= 0x1, selected= 0x0, p_sync= 0x0, p_collecting= 0x0 em1: lacpdu transmit actor=(8000,00-30-48-35-CC-25,008B,8000,0002) actor.state=45 partner=(,00-00-00-00-00-00,,,) partner.state=0 maxdelay=0 em1: lacp_sm_mux: state= 0x0, selected= 0x2, p_sync= 0x0, p_collecting= 0x0 em1: lacp_sm_mux: state= 0x1, selected= 0x2, p_sync= 0x0, p_collecting= 0x0 em1: lacp_sm_mux: state= 0x1, selected= 0x2, p_sync= 0x0, p_collecting= 0x0 em1: lacpdu transmit actor=(8000,00-30-48-35-CC-25,008B,8000,0002) actor.state=4d partner=(,00-00-00-00-00-00,,,) partner.state=0 maxdelay=0 em1: lacp_sm_mux: state= 0x2, selected= 0x2, p_sync= 0x0, p_collecting= 0x0 em1: lacp_sm_mux: state= 0x2, selected= 0x2, p_sync= 0x0, p_collecting= 0x0 em1: lacp_sm_mux: state= 0x2, selected= 0x2, p_sync= 0x0, p_collecting= 0x0 [...] Not sure if the attachment will work to the list, but here is a pcap attached em1 (sudo tcpdump -i em1 -s 0 -w lacp.pcap). Attaching to the lagg0 with tcpdump, filters out the freebsd’s LACP packets, but still sees the TP-Link’s packets. - David P. Discher http://davidpdischer.com/ AIM: DavidDPD | Y!M: daviddpdz lacp.pcap Description: Binary data lacp-lagg0.pcap Description: Binary data signature.asc Description: Message signed with OpenPGP using GPGMail
Re: FreeBSD 10-stable (r274577) LACP / IEEE 802.3ad with TP-Link TL-SG2008 - not working
On Dec 4, 2014, at 12:50 PM, Alan Somers wrote: >> Not sure if the attachment will work to the list, but here is a pcap >> attached em1 (sudo tcpdump -i em1 -s 0 -w lacp.pcap). Attaching to the lagg0 >> with tcpdump, filters out the freebsd’s LACP packets, but still sees the >> TP-Link’s packets. > > Did you remember to configure the switch to use LACP on those ports? > It isn't automatic. And did you remember to manually up em1? Those > are the two commonest LACP configuration mistakes. > > -Alan 99% sure the switch is configured correctly. I’ve actually tried all the different combination of the switch supports, and can’t seem to get them to sync up. From the pcaps, both the host and switch are sending the LACP packets … but it seems that the freebsd is just passing them through/up … almost like FreeBSD doesn’t recognized them as LACP packets. In looking a bit deeper on the pcaps in wireshark - the TP-Link’s packet correctly labels the “partner” systems to the super micro/freebsd box by port and mac address, however the FreeBSD response has all zeros for the partner systems mac and port. So without looking at the logs on the switch, my guess is the switch correctly sees the FreeBSD box, however the FreeBSD box is not correctly recognizing the packets from the switch as LACP. Feels like a single bit or flag in the switches packets that is not matching what FreeBSD is expecting. I was hoping someone would have some insight before I start looking at each bit of the packets what’s not aligned. - David P. Discher http://davidpdischer.com/ AIM: DavidDPD | Y!M: daviddpdz signature.asc Description: Message signed with OpenPGP using GPGMail
Re: FreeBSD 10-stable (r274577) LACP / IEEE 802.3ad with TP-Link TL-SG2008 - not working
bytes from 192.168.0.2: icmp_seq=243 ttl=64 time=1.482 ms Feels like some there is some glue missing. - David P. Discher http://davidpdischer.com/ AIM: DavidDPD | Y!M: daviddpdz signature.asc Description: Message signed with OpenPGP using GPGMail
EM(4) - Link flap when bridged and adding members (em bridge flap)
I’m seeing this in -head as well as 10-stable (r274577) on two different sets of hardware. When an em interface is a member of a bridge, (if_bridge(4)), the link flaps. This is reported on the console as well as can be seen with loss of connectivity. This seems to be easily replicated. ifconfig bridge0 create ifconfig bridge addm em0 ifconfig epair0 create ifconfig bridge addm epair0a < link flap > Two different systems, the behavior is the same. em0@pci0:6:0:0: class=0x02 card=0x109615d9 chip=0x10968086 rev=0x01 hdr=0x00 vendor = 'Intel Corporation' device = '80003ES2LAN Gigabit Ethernet Controller (Copper)' class = network subclass = ethernet em0@pci0:1:0:0: class=0x02 card=0x10d315d9 chip=0x10d38086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '82574L Gigabit Network Connection' class = network subclass = ethernet - David P. Discher http://davidpdischer.com/ AIM: DavidDPD | Y!M: daviddpdz signature.asc Description: Message signed with OpenPGP using GPGMail
Re: EM(4) - Link flap when bridged and adding members (em bridge flap)
Thanks ryan - this did appear work around the issue. - David P. Discher http://davidpdischer.com/ AIM: DavidDPD | Y!M: daviddpdz On Dec 10, 2014, at 10:25 AM, Ryan Stone wrote: > From a quick look at the code, whenever an interface is added to a > bridge, if that interface does not support a feature currently enabled > on the bridge then it has to disable that feature on all member > interfaces of that bridge. That would re-init the em interface, which > would cause a link flap. > > Take a look at the ifconfig output for em0 and epair0 before epair0 is > added to the bridge. You will probably see that one or more features > that em supports is not supported by epair (I would bet on something > like LRO or TSO).. The solution would be to disable those features > from em0 before adding it to the bridge (you can persist this in > rc.conf with ifconfig_em0="...") signature.asc Description: Message signed with OpenPGP using GPGMail
Re: FreeBSD 10-stable (r274577) LACP / IEEE 802.3ad with TP-Link TL-SG2008 - not working
So, I think I’ve identified the issue. In sys/net/ieee8023ad_lacp.c,
lacp_pdu_input() has a sanity check :
if (m->m_pkthdr.len != sizeof(*du)) {
goto bad;
}
I added some debugging information in if_lagg, and ran with it. The lacpdu
packet that being sent by the TP-Link switch is 4 bytes longer than the FreeBSD
"struct lacpdu du”.
em1: lacp_pdu_input-sizeof(du) bad m_pkthdr.len=128 sizeof(du)=124
My packet captures shows the packet size differing as well.
I’m still poking around. I’ve been trying to look for the official LACPDU
binary/wire format … if someone can point me to the reference for that, that
would be helpful (at least my own understanding).
Not exactly sure what these extra 4 bytes are at the end of the packet. Still
trying to figure that out.
-
David P. Discher
http://davidpdischer.com/
AIM: DavidDPD | Y!M: daviddpdz
On Dec 4, 2014, at 8:41 PM, David P. Discher wrote:
> Thanks Adam -
>
> On Dec 4, 2014, at 1:58 PM, Adam McDougall wrote:
>
>>
>> Is the switch side set to "active" for the lacp mode (instead of
>> passive)? Also, try:
>> sysctl net.link.lagg.0.lacp.lacp_strict_mode=0
>>
>> If either of those work, I'll explain more, don't have time to dig for
>> old email right this minute.
>
> Yes, the switch was in active mode. Turn strict mode off (which I thought I
> did before, but of course this sysctl clears when destroying the interface).
> So, the LACP did finally negotiated, at least in ifconfig :
signature.asc
Description: Message signed with OpenPGP using GPGMail
Re: FreeBSD 10-stable (r274577) LACP / IEEE 802.3ad with TP-Link TL-SG2008 - not working
On Dec 15, 2014, at 11:33 AM, Alan Somers wrote:
> On Sun, Dec 14, 2014 at 6:23 PM, David P. Discher wrote:
>>
>> So, I think I’ve identified the issue. In sys/net/ieee8023ad_lacp.c,
>> lacp_pdu_input() has a sanity check :
>>
>>if (m->m_pkthdr.len != sizeof(*du)) {
>>goto bad;
>>}
>>
>> I added some debugging information in if_lagg, and ran with it. The lacpdu
>> packet that being sent by the TP-Link switch is 4 bytes longer than the
>> FreeBSD "struct lacpdu du”.
>>
>>em1: lacp_pdu_input-sizeof(du) bad m_pkthdr.len=128 sizeof(du)=124
>>
>> My packet captures shows the packet size differing as well.
>>
>> I’m still poking around. I’ve been trying to look for the official LACPDU
>> binary/wire format … if someone can point me to the reference for that, that
>> would be helpful (at least my own understanding).
>
> Try here:
> http://standards.ieee.org/findstds/standard/802.1AX-2008.html
>
Thanks - I hadn’t seen the “free” version from IEEE, and may looking into that
later. However, I think my time with messing around with this switch and lagg
is just about over.
I did get FreeBSD to work with LACP in this Switch. I hacked in the 4 extra
bytes in struct lacpdu in src/sys/net/ieee8023ad_lacp.h
Index: ieee8023ad_lacp.h
===
--- ieee8023ad_lacp.h (revision 275779)
+++ ieee8023ad_lacp.h (working copy)
@@ -151,6 +151,7 @@
struct lacp_collectorinfo ldu_collector;
struct tlvhdr ldu_tlv_term;
uint8_t ldu_resv[50];
+ uint8_t tplink[4];
} __packed;
/*
This work great and without any issue. All the defaults with 10-stable
(r275778) and recent version of -head with this one line made it work. Of
course, this will likely break FreeBSD with all other switches LACP.
However, what I have also discovered this this switch is unlike FreeBSD which
lagghash includes L4, the switch only seems to hash over SRC+DST IP or SRC+DST
MAC. Which makes it pretty much just sends all the traffic down one link from
the switch. So for my particular use case with a small set of hosts, this
switch is not useful for me.
I would not recommend the TP-Link TL-SG2008 8-Port Gigabit Smart Switch for use
with FreeBSD … or for any use with LACP that is expecting increased throughput
for a small set of hosts.
-
David P. Discher
http://davidpdischer.com/
AIM: DavidDPD | Y!M: daviddpdz
___
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: FreeBSD 10-stable (r274577) LACP / IEEE 802.3ad with TP-Link TL-SG2008 - not working
On Dec 17, 2014, at 5:19 PM, Alan Somers wrote: >> >> >> This work great and without any issue. All the defaults with 10-stable >> (r275778) and recent version of -head with this one line made it work. Of >> course, this will likely break FreeBSD with all other switches LACP. > > > I'm glad that you got your problem sorted out. Please do let us know > if you find a more general solution. > Yeah, Alan - will do … if I decided to look into more. That is why I was looking for spec on LACP. One side is doing it wrong. FreeBSD is looking for a LACPDU of exactly sizeof ( struct lacpdu ) which is 124 bytes. The TP-Link is sending a PDU of 128 bytes. I was hoping someone would know off hand what the spec says, if the PDU “should be” or “must be”. I’m assuming this is an error on TP-Link’s firmware, and will try to file a bug with them if I can get to the root of the issue. I’m assuming the Linux driver isn’t strict on the PDU size. If this was really an error on the FreeSBD side … someone should have stumbled over long before I did. > >> >> However, what I have also discovered this this switch is unlike FreeBSD >> which lagghash includes L4, the switch only seems to hash over SRC+DST IP or >> SRC+DST MAC. Which makes it pretty much just sends all the traffic down one >> link from the switch. So for my particular use case with a small set of >> hosts, this switch is not useful for me. > > > Actually, that's a fairly common problem. I've even seen it on some > expensive Cisco switches. Yeah, in the end, this research and experiment has proven to me that LAGG/LACP is not likely the correct solution for this project. 10g Ethernet would work well, even back-to-back, but even used this looks to be a bit pricey for home/hobby setup. I’m now looking towards Infiniband, as the cards and parts on the use market are a great value (but this is the wrong list for talking about that.) Thanks ! - David P. Discher http://davidpdischer.com/ AIM: DavidDPD | Y!M: daviddpdz ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: FreeBSD 10-stable (r274577) LACP / IEEE 802.3ad with TP-Link TL-SG2008 - not working
On Dec 17, 2014, at 9:11 PM, Craig Rodrigues wrote: > > > And here: > > http://www-01.ibm.com/support/docview.wss?uid=isg1VM64842 > Ah, wow … that IBM article - what I’m seeing is the four-byte Frame Check Sequence (FCS). I download the IEEE Std 802.1AX-2008 PDF, Figure 5–8— LACPDU structure, page 33, shows this structure with the FCS field. (802.3ad has evolved into 802.1AX) - The 802.3ad spec PDF is $155. It’s unclear if this was added to a later version spec. And with FCS field, FreeBSD LACP just does not work. So … it would seem that big iron routers and switches either are not adding in the FCS field. Or - it is possible, with how the IBM support article is written - it is possibly this field might be getting stripped by some NIC hardware or drivers. Ok … I’m going to investigate this further. I’m working on getting/borrowing a few different vendors switches with LACP and see what I can find out. - David P. Discher http://davidpdischer.com/ AIM: DavidDPD | Y!M: daviddpdz ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: FreeBSD 10-stable (r274577) LACP / IEEE 802.3ad with TP-Link TL-SG2008 - not working
I’m going to look into other switches/routers and read the spec a bit closer, but linux seems to think the LACPDU check is packet size => sizeof(struct lacpdu) : - https://github.com/torvalds/linux/blob/70e71ca0af244f48a5dcf56dc435243792e3a495/drivers/net/bonding/bond_3ad.c#L2185 - David P. Discher http://davidpdischer.com/ AIM: DavidDPD | Y!M: daviddpdz On Dec 18, 2014, at 10:46 AM, John-Mark Gurney wrote: >> >> Good find, Craig. Also, I found the full LACPDU definition. It's in >> section 5.4.2.2, page 33, of the 802.1ax-2008 spec that I linked to. >> You can see the 4-byte FCS field at the end. Does your tp_link[4] >> field look like an FCS? If so, you need to figure out why it's >> propagating all the way up to the LACP level. > > It very well could be that the authors of the TP-Link firmware missed > the comment in 1ax that says the FCS is generated by the MAC, and > include it in their sending... > > If you could capture the original frame, and check the ether_type > field, to see if it is 124 or 128... If it's 128, they probably added > the FCS manually and the the MAC adds it a second time... > signature.asc Description: Message signed with OpenPGP using GPGMail
Is if_ipsec/ipsec - AESNI accelerated ?
I’m suspecting that IPSec in FreeBSD is not leveraging AESNI on Intel. Is this correct ? A small system, with an Atom C2758 and AESNI can hit 940-950 Mbps on a 1g copper link SCPing a file with Chiper=aes256-gcm. SSH/OpenSSL automatically uses AESNI if available. (Side Note, loading cryptodev - openSSH/SSL will grab crypto dev and cut your speed in half). Same with un-encryrpted iperf2/3, even with just a single TCP connection. Over an IPsec tunnel, this same system bottle necks at 180 Mbps. These systems are on the same vlan and subnet, same physical switch - so direct route. So, does IPSec use AESNI ? I would have at least expected 600-700 Mbps. -- David P. Discher https://davidpdischer.com/ ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: Is if_ipsec/ipsec - AESNI accelerated ?
> On Aug 8, 2018, at 10:37 PM, Andrey V. Elsukov wrote: > > On 09.08.2018 06:57, David P. Discher wrote: >> I’m suspecting that IPSec in FreeBSD is not leveraging AESNI on Intel. Is >> this correct ? > > IPsec uses crypto(9) framework that works by default without any > acceleration. You need to load aesni(4) kernel module to enable > acceleration. Also, you need to recreate security associations after > module loading to take effect. Yes. I booted with AESNI loaded … via loader.conf. Transcript below. Two endpoint are identical hardware. -- David P. Discher https://davidpdischer.com/ 408.368.3725 • d...@dpdtech.com [ pts/0 sjc2 util201:~ ] [ dpd ] > kldstat Id Refs AddressSize Name 1 32 0x8020 2081408 kernel 21 0x82283000 259e0geom_mirror.ko 31 0x822a9000 e568 if_bridge.ko 42 0x822b8000 6d28 bridgestp.ko 51 0x822bf000 7600 if_tap.ko 61 0x822c7000 f988 ipmi.ko 72 0x822d7000 2d10 smbus.ko 81 0x822da000 381130 zfs.ko 92 0x8265c000 a380 opensolaris.ko 101 0x82667000 af98 aesni.ko 111 0x82b11000 2328 ums.ko [ pts/0 sjc2 util201:~ ] [ dpd ] > sudo /usr/local/etc/rc.d/racoon stop Password: Stopping racoon. Waiting for PIDS: 1065. [ pts/0 sjc2 util201:~ ] [ dpd ] > sudo /usr/local/etc/rc.d/racoon start Starting racoon. [ pts/0 sjc2 util201:~ ] [ dpd ] > sudo setkey -f /usr/local/etc/racoon/setkey.conf [ pts/0 sjc2 util201:~ ] [ dpd ] > ifconfig ipsec12 ipsec12: flags=8151 metric 0 mtu 1350 tunnel inet 10.245.0.201 --> 10.245.0.202 inet 172.30.1.13 --> 172.30.1.14 netmask 0xfffc nd6 options=29 reqid: 12 groups: ipsec [ pts/0 sjc2 util201:~ ] [ dpd ] > ping 172.30.1.14 PING 172.30.1.14 (172.30.1.14): 56 data bytes 64 bytes from 172.30.1.14: icmp_seq=2 ttl=64 time=0.452 ms 64 bytes from 172.30.1.14: icmp_seq=3 ttl=64 time=0.368 ms 64 bytes from 172.30.1.14: icmp_seq=4 ttl=64 time=0.353 ms ^C --- 172.30.1.14 ping statistics --- 5 packets transmitted, 3 packets received, 40.0% packet loss round-trip min/avg/max/stddev = 0.353/0.391/0.452/0.044 ms [ pts/0 sjc2 util201:~ ] [ dpd ] > iperf3 -c 10.245.0.202 -i 8 -t 16 Connecting to host 10.245.0.202, port 5201 [ 5] local 10.245.0.201 port 55165 connected to 10.245.0.202 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-8.00 sec 887 MBytes 930 Mbits/sec0419 KBytes [ 5] 8.00-16.00 sec 898 MBytes 941 Mbits/sec0419 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-16.00 sec 1.74 GBytes 936 Mbits/sec0 sender [ 5] 0.00-16.01 sec 1.74 GBytes 935 Mbits/sec receiver iperf Done. [ pts/0 sjc2 util201:~ ] [ dpd ] > iperf3 -c 172.30.1.14 -i 8 -t 16 Connecting to host 172.30.1.14, port 5201 [ 5] local 172.30.1.13 port 41671 connected to 172.30.1.14 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-8.00 sec 166 MBytes 174 Mbits/sec0 64.3 KBytes [ 5] 8.00-16.00 sec 168 MBytes 176 Mbits/sec0 64.3 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-16.00 sec 334 MBytes 175 Mbits/sec0 sender [ 5] 0.00-16.01 sec 334 MBytes 175 Mbits/sec receiver iperf Done. [ pts/0 sjc2 util201:~ ] [ dpd ] > uname -a FreeBSD util201.sjc2.ixsystems.com 11.2-STABLE FreeBSD 11.2-STABLE #3: Tue Jul 24 20:57:34 UTC 2018 r...@proxima.sjc2.ixsystems.com:/usr/obj/usr/src/sys/IX amd64 [ pts/0 sjc2 util201:~ ] [ dpd ] > ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: Is if_ipsec/ipsec - AESNI accelerated ?
/unique:12 spid=7 seq=8 pid=2443 scope=ifnet ifname=ipsec12 refcnt=1 0.0.0.0/0[any] 0.0.0.0/0[any] any in ipsec esp/tunnel/10.245.0.203-10.245.0.201/unique:4 spid=13 seq=7 pid=2443 scope=ifnet ifname=ipsec4 refcnt=1 ::/0[any] ::/0[any] any in ipsec esp/tunnel/10.245.0.203-10.245.0.201/unique:4 spid=15 seq=6 pid=2443 scope=ifnet ifname=ipsec4 refcnt=1 172.30.1.12/30[any] 172.30.1.12/30[any] any out ipsec esp/tunnel/10.245.0.201-10.245.0.202/unique:12 spid=21 seq=5 pid=2443 scope=global refcnt=1 172.30.1.4/30[any] 172.30.1.4/30[any] any out ipsec esp/tunnel/10.245.0.201-10.245.0.203/unique:4 spid=23 seq=4 pid=2443 scope=global refcnt=1 0.0.0.0/0[any] 0.0.0.0/0[any] any out ipsec esp/tunnel/10.245.0.201-10.245.0.202/unique:12 spid=6 seq=3 pid=2443 scope=ifnet ifname=ipsec12 refcnt=1 ::/0[any] ::/0[any] any out ipsec esp/tunnel/10.245.0.201-10.245.0.202/unique:12 spid=8 seq=2 pid=2443 scope=ifnet ifname=ipsec12 refcnt=1 0.0.0.0/0[any] 0.0.0.0/0[any] any out ipsec esp/tunnel/10.245.0.201-10.245.0.203/unique:4 spid=14 seq=1 pid=2443 scope=ifnet ifname=ipsec4 refcnt=1 ::/0[any] ::/0[any] any out ipsec esp/tunnel/10.245.0.201-10.245.0.203/unique:4 spid=16 seq=0 pid=2443 scope=ifnet ifname=ipsec4 refcnt=1 -- David P. Discher https://davidpdischer.com/ 408.368.3725 • d...@dpdtech.com ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Chelsio 10GB PCI-e Opt Card PCI-E 110-1088-30 is a T320 supported via cxgb(4)
For the Community Documentation on the inter-webs and Google searches - For about a month or two, I was trying to figure out exactly which platform the Chelsio 10GB Opt Card, part number 110-1088-30 was built on, and if it was supported under FreeBSD. I suspected it was an N320, but could not confirm it. Chelsio's site didn’t even have any cross reference for this part number. There are various eBay auctions for these cards, with some PCB variants, running at the $25-35 range, which would seem to be a steal for a dual ported 10Gbps ethernet card. So, I broken down and purchased one of these part number 110-1088-30 PCIe 10Gb “Opt Cards”. - http://www.ebay.com/itm/351719918339 In fact, this does appears to be the Terminator 3 ASIC platform (T3). I assume this was later rebranded/rev’ed by Chelsio to the T3 Unified Wire collection under product name “N320”. But can’t find any references or documentation to confirm this. I don’t know how to probe FreeBSD to check the PCIe sync up. However, I believe it is a PCIe 1.1 x8 device. This means the PCIe x8 bus maxes out at 16 Gbps. However, it appears that the T3 version on this card maxes out at about 11 Gbps (~5.5 Gbps each port with iperf when lighting up both ports at the same time). This card - even as the N320, the marketing material lists this as a failover/HA card, not intended for a 20 Gbps LAG. I also found some new, Finisar SFP+ SR optics on eBay for about $18-20/each. Combined with some fiber from mono price. For about $50, this feels like a pretty good and cheap solution for cheap 10Gbps connectivity for home labs/NASes - with a really good and well supported brand/card. (** This should work in FreeNAS, at least by the kernel, too - the cxgb(4) support has been around for a long time ! *** ) Hopefully someone at some point down the road, finds this info useful. === pciconf -lv === cxgbc0@pci0:8:0:0: class=0x02 card=0x00011425 chip=0x00311425 rev=0x00 hdr=0x00 vendor = 'Chelsio Communications Inc' device = 'T320 10GbE Dual Port Adapter' class = network subclass = ethernet == dmesg, verbose boot === pcib8: at device 4.0 on pci0 pcib0: allocated type 3 (0xd830-0xd83f) for rid 20 of pcib8 pcib8: domain0 pcib8: secondary bus 8 pcib8: subordinate bus 8 pcib8: memory decode 0xd830-0xd83f pcib8: special decodeISA pci8: on pcib8 pcib8: allocated bus range (8-8) for rid 0 of pci8 pci8: domain=0, physical bus=8 found-> vendor=0x1425, dev=0x0031, revid=0x00 domain=0, bus=8, slot=0, func=0 class=02-00-00, hdrtype=0x00, mfdev=0 cmdreg=0x0147, statreg=0x0010, cachelnsz=8 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=a, irq=7 powerspec 3 supports D0 D3 current D0 MSI supports 32 messages, 64 bit MSI-X supports 32 messages in map 0x20 map[10]: type Memory, range 64, base 0xd8301000, size 12, enabled pcib8: allocated memory range (0xd8301000-0xd8301fff) for rid 10 of pci0:8:0:0 map[20]: type Memory, range 64, base 0xd830, size 12, enabled pcib8: allocated memory range (0xd830-0xd8300fff) for rid 20 of pci0:8:0:0 pcib8: matched entry for 8.0.INTA pcib8: slot 0 INTA hardwired to IRQ 16 cxgbc0: mem 0xd8301000-0xd8301fff,0xd830-0xd8300fff irq 16 at device 0.0 on pci8 cxgbc0: attempting to allocate 9 MSI-X vectors (32 supported) msi: routing MSI-X IRQ 258 to local APIC 0 vector 54 msi: routing MSI-X IRQ 259 to local APIC 0 vector 55 msi: routing MSI-X IRQ 260 to local APIC 0 vector 56 msi: routing MSI-X IRQ 261 to local APIC 0 vector 57 msi: routing MSI-X IRQ 262 to local APIC 0 vector 58 msi: routing MSI-X IRQ 263 to local APIC 0 vector 59 msi: routing MSI-X IRQ 264 to local APIC 0 vector 60 msi: routing MSI-X IRQ 265 to local APIC 0 vector 61 msi: routing MSI-X IRQ 266 to local APIC 0 vector 62 cxgbc0: using IRQs 258-266 for MSI-X cxgbc0: using MSI-X interrupts (9 vectors) cxgb0: on cxgbc0 cxgb0: Using defaults for TSO: 65518/35/2048 cxgb0: bpf attached cxgb0: Ethernet address: 00:07:43:0a:a0:84 cxgb1: on cxgbc0 cxgb1: Using defaults for TSO: 65518/35/2048 cxgb1: bpf attached cxgb1: Ethernet address: 00:07:43:0a:a0:85 cxgbc0: Firmware Version 7.11.0 - David P. Discher http://davidpdischer.com/ signature.asc Description: Message signed with OpenPGP using GPGMail
