Hi Sascha - I had a similar experience earlier this week. I was testing bond performance on a dual-port Mellanox NIC, LACP-bonded with layer 3+4 transmit hash. First run of iperf (8 streams), I saw a reasonable distribution across the links. Shortly after that, performance dropped to a level that suggested no distribution. I didn’t look into it any further at the time.
This was on CoreOS beta (kernel 4.3-6 IIRC). If our experiences have a common root cause, I have a different distro and NIC to you, which might eliminate those components. I should have the kit back in this configuration within a week, I’ll try to probe and report back. Best wishes, Stig > On 23 Mar 2016, at 15:45, MailingLists - EWS > <mailingli...@expresswebsystems.com> wrote: > > Sascha, > > What version of the ixgbe driver are you using? Is it the same on both > kernels? Have you tried the latest "out of tree driver" from E1000 to see if > the issue goes away? > > I follow the E1000 mailing list and I seem to recall some rather recent > posts regarding bonding and the ixgbe along with some patches being applied > to the driver, however I don't know what version of kernel these issues were > on, or even if the patches were accepted. > > https://sourceforge.net/p/e1000/mailman/e1000-devel/thread/87618083B2453E4A8 > 714035B62D67992504FB5FF%40FMSMSX105.amr.corp.intel.com/#msg34727125 > > Something about a timing issue with detecting the slave's link speed and > passing that information to the bonding driver in a timely fashion. > > Tom Walsh > ExpressHosting > https://expresshosting.net/ > >> -----Original Message----- >> From: Sascha Vogt [mailto:sascha.v...@gmail.com] >> Sent: Wednesday, March 23, 2016 5:54 AM >> To: openstack-operators >> Subject: [Openstack-operators] Intel 10 GbE / bonding issue with hash > policy >> layer3+4 >> >> Hi all, >> >> I thought it might be of interest / get feedback from the operators >> community about an oddity we experienced with Intel 10 GbE NICs and LACP >> bonding. >> >> We have Ubuntu 14.04.4 as OS and Intel 10 GbE NICs with the ixgbe Kernel >> module. We use VLANS for ceph-client, ceph-data, openstack-data, >> openstack-client networks all on a single LACP bonding of two 10 GbE > ports. >> >> As bonding hash policy we chose layer3+4 so we can use the full 20 Gb even >> if only two servers communicate with each other. Typically we check that > by >> using iperf to a single server with -P 4 and see if we exceed the 10 Gb > limit >> (just a few times to check). >> >> Due to Ubuntus default of installing the latest Kernel our new host had >> Kernel 4.2.0 instead of the Kernel 3.16 the other machines had and we >> noticed that iperf only used 10 Gb. >> >>> # cat /proc/net/bonding/bond0 >>> Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011) >>> >>> Bonding Mode: IEEE 802.3ad Dynamic link aggregation Transmit Hash >>> Policy: layer3+4 (1) >> >> This was shown on both - Kernel 3.16 and 4.2.0 >> >> After downgrading to Kernel 3.16 we got the iperf results we expected. >> >> Does anyone have a similar setup? Anyone noticed the same things? To us >> this looks like a bug in the Kernel (ixgbe module?), or are we >> misunderstanding the hash policy layer3+4? >> >> Any feedback is welcome :) I have not yet posted this to the Kernel ML or >> Ubuntus ML yet, so if no one here is having a similar setup I'll move over >> there. I just thought OpenStack ops might be the place were it is most > likely >> that someone has a similar setup :) >> >> Greetings >> -Sascha- >> >> _______________________________________________ >> OpenStack-operators mailing list >> OpenStack-operators@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators > > > _______________________________________________ > OpenStack-operators mailing list > OpenStack-operators@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators _______________________________________________ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
