Daniel Braniss wrote:
>
> > On Aug 18, 2015, at 12:49 AM, Rick Macklem <rmack...@uoguelph.ca> wrote:
> >
> > Daniel Braniss wrote:
> >>
> >>> On Aug 17, 2015, at 3:21 PM, Rick Macklem <rmack...@uoguelph.ca> wrote:
> >>>
> >>> Daniel Braniss wrote:
> >>>>
> >>>>> On Aug 17, 2015, at 1:41 PM, Christopher Forgeron
> >>>>> <csforge...@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>> FYI, I can regularly hit 9.3 Gib/s with my Intel X520-DA2's and FreeBSD
> >>>>> 10.1. Before 10.1 it was less.
> >>>>>
> >>>>
> >>>> this is NOT iperf/3 where i do get close to wire speed,
> >>>> it’s NFS writes, i.e., almost real work :-)
> >>>>
> >>>>> I used to tweak the card settings, but now it's just stock. You may
> >>>>> want
> >>>>> to
> >>>>> check your settings, the Mellanox may just have better defaults for
> >>>>> your
> >>>>> switch.
> >>>>>
> >>> Have you tried disabling TSO for the Intel? With TSO enabled, it will be
> >>> copying
> >>> every transmitted mbuf chain to a new chain of mbuf clusters via.
> >>> m_defrag() when
> >>> TSO is enabled. (Assuming you aren't an 82598 chip. Most seem to be the
> >>> 82599 chip
> >>> these days?)
> >>>
> >>
> >> hi Rick
> >>
> >> how can i check the chip?
> >>
> > Haven't a clue. Does "dmesg" tell you? (To be honest, since disabling TSO
> > helped,
> > I'll bet you don't have a 82598.)
> >
> >>> This has been fixed in the driver very recently, but those fixes won't be
> >>> in 10.1.
> >>>
> >>> rick
> >>> ps: If you could test with 10.2, it would be interesting to see how the
> >>> ix
> >>> does with
> >>> the current driver fixes in it?
> >>
> >> I new TSO was involved!
> >> ok, firstly, it’s 10.2 stable.
> >> with TSO enabled, ix is bad, around 64MGB/s.
> >> disabling TSO it’s better, around 130
> >>
> > Hmm, could you check to see of these lines are in sys/dev/ixgbe/if_ix.c at
> > around
> > line#2500?
> > /* TSO parameters */
> > 2572 ifp->if_hw_tsomax = 65518;
> > 2573 ifp->if_hw_tsomaxsegcount =
> > IXGBE_82599_SCATTER;
> > 2574 ifp->if_hw_tsomaxsegsize = 2048;
> >
> > They are in stable/10. I didn't look at releng/10.2. (And if they're in a
> > #ifdef
> > for FreeBSD11, take the #ifdef away.)
> > If they are there and not ifdef'd, I can't explain why disabling TSO would
> > help.
> > Once TSO is fixed so that it handles the 64K transmit segments without
> > copying all
> > the mbufs, I suspect you might get better perf. with it enabled?
> >
>
> this is 10.2 :
> they are on lines 2509-2511 and I don’t see any #ifdefs around it.
>
> the plot thickens :-)
>
If this is just a test machine, maybe you could test with these lines (at about
#880)
in sys/netinet/tcp_output.c commented out? (It looks to me like this will
disable TSO
for almost all the NFS writes.)
- around line #880 in sys/netinet/tcp_output.c:
/*
* In case there are too many small fragments
* don't use TSO:
*/
if (len <= max_len) {
len = max_len;
sendalot = 1;
tso = 0;
}
This was added along with the other stuff that did the if_hw_tsomaxsegcount,
etc and I
never noticed it until now (not my patch).
rick
> danny
>
> > Good luck with it, rick
> >
> >> still, mlxen0 is about 250! with and without TSO
> >>
> >>
> >>>
> >>>>> On Mon, Aug 17, 2015 at 6:41 AM, Slawa Olhovchenkov <s...@zxy.spb.ru
> >>>>> <mailto:s...@zxy.spb.ru>> wrote:
> >>>>> On Mon, Aug 17, 2015 at 10:27:41AM +0300, Daniel Braniss wrote:
> >>>>>
> >>>>>> hi,
> >>>>>> I have a host (Dell R730) with both cards, connected to an HP8200
> >>>>>> switch at 10Gb.
> >>>>>> when writing to the same storage (netapp) this is what I get:
> >>>>>> ix0: ~130MGB/s
> >>>>>> mlxen0 ~330MGB/s
> >>>>>> this is via nfs/tcpv3
> >>>>>>
> >>>>>> I can get similar (bad) performance with the mellanox if I
> >>>>>> increase
> >>>>>> the file size
> >>>>>> to 512MGB.
> >>>>>
> >>>>> Look like mellanox have internal beffer for caching and do ACK
> >>>>> acclerating.
> >>>>>
> >>>>>> so at face value, it seems the mlxen does a better use of
> >>>>>> resources
> >>>>>> than the intel.
> >>>>>> Any ideas how to improve ix/intel's performance?
> >>>>>
> >>>>> Are you sure about netapp performance?
> >>>>> _______________________________________________
> >>>>> freebsd-...@freebsd.org <mailto:freebsd-...@freebsd.org> mailing list
> >>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> >>>>> <https://lists.freebsd.org/mailman/listinfo/freebsd-net>
> >>>>> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org
> >>>>> <mailto:freebsd-net-unsubscr...@freebsd.org>"
> >>>>>
> >>>>
> >>>> _______________________________________________
> >>>> freebsd-stable@freebsd.org mailing list
> >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> >>>> To unsubscribe, send any mail to
> >>>> "freebsd-stable-unsubscr...@freebsd.org"
> >>
> >> _______________________________________________
> >> freebsd-stable@freebsd.org mailing list
> >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> >> To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
>
> _______________________________________________
> freebsd-...@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
_______________________________________________
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
