decent 40G network adapters

[Eugene M. Zheganin]

Hi.

Could someone recommend a decent 40Gbit adapter that are proven to be
working under FreeBSD ? The intended purpose - iSCSI traffic, not much
pps, but rates definitely above 10G. I've tried Supermicro-manufactured
Intel XL710 ones (two boards, different servers - same sad story:
packets loss, server unresponsive, spikes), seems like they have a
problem in a driver (or firmware), and though Intel support states this
is because the Supermicro tampered with the adapter, I'm still
suspicious about ixl(4). I've also seen in the ML a guy reported the
exact same problem with ixl(4) as I have found.

So, what would you say ? Chelsio ?

Thanks.
Eugene.

___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: decent 40G network adapters

[Hans Petter Selasky]

On 01/18/17 10:48, Eugene M. Zheganin wrote:

Hi.

Could someone recommend a decent 40Gbit adapter that are proven to be
working under FreeBSD ? The intended purpose - iSCSI traffic, not much
pps, but rates definitely above 10G. I've tried Supermicro-manufactured
Intel XL710 ones (two boards, different servers - same sad story:
packets loss, server unresponsive, spikes), seems like they have a
problem in a driver (or firmware), and though Intel support states this
is because the Supermicro tampered with the adapter, I'm still
suspicious about ixl(4). I've also seen in the ML a guy reported the
exact same problem with ixl(4) as I have found.

So, what would you say ? Chelsio ?



Hi,

I think also the Mellanox, mlx4 and mlx5 drivers will support this. Are 
you using infiniband or TCP for backend?


--HPS

___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: decent 40G network adapters

[Eugene M. Zheganin]

Hi.

On 18.01.2017 14:51, Hans Petter Selasky wrote:
> On 01/18/17 10:48, Eugene M. Zheganin wrote:
>> Hi.
>>
>> Could someone recommend a decent 40Gbit adapter that are proven to be
>> working under FreeBSD ? The intended purpose - iSCSI traffic, not much
>> pps, but rates definitely above 10G. I've tried Supermicro-manufactured
>> Intel XL710 ones (two boards, different servers - same sad story:
>> packets loss, server unresponsive, spikes), seems like they have a
>> problem in a driver (or firmware), and though Intel support states this
>> is because the Supermicro tampered with the adapter, I'm still
>> suspicious about ixl(4). I've also seen in the ML a guy reported the
>> exact same problem with ixl(4) as I have found.
>>
>> So, what would you say ? Chelsio ?
>>
>
> Hi,
>
> I think also the Mellanox, mlx4 and mlx5 drivers will support this.
> Are you using infiniband or TCP for backend?
>
>
Thanks. I'm using TCP.

Eugene.

___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: decent 40G network adapters

[Slawa Olhovchenkov]

On Wed, Jan 18, 2017 at 02:48:19PM +0500, Eugene M. Zheganin wrote:

> Hi.
> 
> Could someone recommend a decent 40Gbit adapter that are proven to be
> working under FreeBSD ? The intended purpose - iSCSI traffic, not much
> pps, but rates definitely above 10G. I've tried Supermicro-manufactured
> Intel XL710 ones (two boards, different servers - same sad story:
> packets loss, server unresponsive, spikes), seems like they have a
> problem in a driver (or firmware), and though Intel support states this
> is because the Supermicro tampered with the adapter, I'm still
> suspicious about ixl(4). I've also seen in the ML a guy reported the
> exact same problem with ixl(4) as I have found.
> 
> So, what would you say ? Chelsio ?

I am use Chelsio and Solarflare.
Not sure about you workload -- I am have 40K+ TCP connections, you
workload need different tuning.
Do you planed to utilise both ports?
For this case you need PCIe 16x card. This is Chelsio T6 and
Solarflare 9200. 
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: decent 40G network adapters

[Andrew Rybchenko]

On 01/18/2017 12:48 PM, Eugene M. Zheganin wrote:

Hi.

Could someone recommend a decent 40Gbit adapter that are proven to be
working under FreeBSD ? The intended purpose - iSCSI traffic, not much
pps, but rates definitely above 10G. I've tried Supermicro-manufactured
Intel XL710 ones (two boards, different servers - same sad story:
packets loss, server unresponsive, spikes), seems like they have a
problem in a driver (or firmware), and though Intel support states this
is because the Supermicro tampered with the adapter, I'm still
suspicious about ixl(4). I've also seen in the ML a guy reported the
exact same problem with ixl(4) as I have found.

So, what would you say ? Chelsio ?

Hi,

Solarflare SFN8542 with sfxge driver will do the job

Andrew.


Thanks.
Eugene.

___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"



___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: decent 40G network adapters

[Eugene M. Zheganin]

Hi.

On 18.01.2017 15:03, Slawa Olhovchenkov wrote:
> I am use Chelsio and Solarflare.
> Not sure about you workload -- I am have 40K+ TCP connections, you
> workload need different tuning.
> Do you planed to utilise both ports?
> For this case you need PCIe 16x card. This is Chelsio T6 and
> Solarflare 9200. 
Thanks. No, the number of connections in my case will be small -
hundreds, and right now target servers are utilizing 4-5 Gbit/sec
bandwidth, so I'm looking forward to something more performing, thats
all. The pps number is also way below Mpps - at this time it's about 200
kpps, so I really hope I won't be facing a situation with millions of
pps, - though seems like it will be slightly above 1 Mpps. Hopefully
with the help of the community I'll be able to tune the servers to
handle this ! :)

Thanks.
Eugene.

___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: sosend returning ERESTART

[Konstantin Belousov]

On Wed, Jan 18, 2017 at 04:37:40AM +, Colin Percival wrote:
> On 01/17/17 02:06, Konstantin Belousov wrote:
> > On Tue, Jan 17, 2017 at 04:57:23AM +, Colin Percival wrote:
> >> I think I've tracked an NFS problem down to sosend returning ERESTART; it
> >> looks like it's easy to work around this, but I'm not sure *why* sosend is
> >> returning ERESTART... or for that matter *how* since I can't find anywhere
> >> in relevant code where that gets returned.
> > ERESTART is most likely returned by msleep(9) or similar call down the
> > path when unblocked signal is pending with the restart disposition.
> 
> Thanks, looks like that was exactly it -- if the TCP send buffer was full
> we would call sbwait, and if a signal arrived it would return ERESTART.
> It looks like setting the SB_NOINTR flag will prevent this; I'm testing a
> patch right now.
> 
> (Google bait in case anyone else trips over this: FreeBSD 11.0 NFS client
> dropping TCP connections under concurrent I/O load.)

Note that passing SB_NOINTR unconditionally or even only for mounts
with nointr (default) option is wrong. You make the socket operation
uninterruptible, process terminate-ability becomes depended on the
external factor, the behaviour of the remote system.
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: decent 40G network adapters

[Scott Larson]

 I can vouch for the Mellanox ConnectX-4 cards as working well, along
with the Chelsio T5 stuff, both under heavy production usage in both
routing and endpoint cases. In my testing the T5 was a slightly better
option in situations demanding PPS with a higher ceiling and lower cpu
utilization, but they came out essentially in a dead heat when testing for
raw throughput.


*[image: userimage]Scott Larson[image: los angeles]
Lead
Systems Administrator[image: wdlogo]  [image:
linkedin]  [image: facebook]
 [image: twitter]
 [image: instagram]
T 310 823 8238
<310%20823%208238%20x1106>  |  M 310 904 8818 <310%20904%208818>*

On Wed, Jan 18, 2017 at 1:48 AM, Eugene M. Zheganin 
wrote:

> Hi.
>
> Could someone recommend a decent 40Gbit adapter that are proven to be
> working under FreeBSD ? The intended purpose - iSCSI traffic, not much
> pps, but rates definitely above 10G. I've tried Supermicro-manufactured
> Intel XL710 ones (two boards, different servers - same sad story:
> packets loss, server unresponsive, spikes), seems like they have a
> problem in a driver (or firmware), and though Intel support states this
> is because the Supermicro tampered with the adapter, I'm still
> suspicious about ixl(4). I've also seen in the ML a guy reported the
> exact same problem with ixl(4) as I have found.
>
> So, what would you say ? Chelsio ?
>
> Thanks.
> Eugene.
>
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: sosend returning ERESTART

[Colin Percival]

On 01/18/17 02:36, Konstantin Belousov wrote:
> On Wed, Jan 18, 2017 at 04:37:40AM +, Colin Percival wrote:
>> Thanks, looks like that was exactly it -- if the TCP send buffer was full
>> we would call sbwait, and if a signal arrived it would return ERESTART.
>> It looks like setting the SB_NOINTR flag will prevent this; I'm testing a
>> patch right now.
> 
> Note that passing SB_NOINTR unconditionally or even only for mounts
> with nointr (default) option is wrong. You make the socket operation
> uninterruptible, process terminate-ability becomes depended on the
> external factor, the behaviour of the remote system.

I'm not sure what you're getting at here.  The fact that "NFS mounted without
the intr flag" + "unresponsive NFS server" = "unkillable processes" has been
a (mis)feature of NFS for decades.

-- 
Colin Percival
Security Officer Emeritus, FreeBSD | The power to serve
Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: sosend returning ERESTART

[Rick Macklem]

Colin Percival wrote:
>On 01/18/17 02:36, Konstantin Belousov wrote:
>> On Wed, Jan 18, 2017 at 04:37:40AM +, Colin Percival wrote:
>>> Thanks, looks like that was exactly it -- if the TCP send buffer was full
>>> we would call sbwait, and if a signal arrived it would return ERESTART.
>>> It looks like setting the SB_NOINTR flag will prevent this; I'm testing a
>>> patch right now.
>>
>> Note that passing SB_NOINTR unconditionally or even only for mounts
>> with nointr (default) option is wrong. You make the socket operation
>> uninterruptible, process terminate-ability becomes depended on the
>> external factor, the behaviour of the remote system.
>
>I'm not sure what you're getting at here.  The fact that "NFS mounted without
>the intr flag" + "unresponsive NFS server" = "unkillable processes" has been
>a (mis)feature of NFS for decades.
The case I would like to see work is the forced dismount. I need to go look at
what it does and see if SB_NOINTR would break it worse than it is broken now.
(It is currently broken when something like "umount" without -f is done, which
 locks up the mounted on vnode so "umount -f" never gets to the umount(2) 
syscall.
 I do plan on a "straight ot NFS" option for umount(8) to avoid this problem, 
but
 haven't gotten around to it.)

The alternative to SB_NOINTR is looping and doing the sosend() again for the
case where it returns ERESTART and "intr" wasn't set on the mount.
--> For this to be ok, we must be sure that when sosend() returns ERESTART,
  it has not queued the data for sending so it is safe to send it all again.
  I think this is true for this case?

rick
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: sosend returning ERESTART

[Rick Macklem]

Colin Percival wrote:
>On 01/18/17 02:36, Konstantin Belousov wrote:
>> On Wed, Jan 18, 2017 at 04:37:40AM +, Colin Percival wrote:
>>> Thanks, looks like that was exactly it -- if the TCP send buffer was full
>>> we would call sbwait, and if a signal arrived it would return ERESTART.
>>> It looks like setting the SB_NOINTR flag will prevent this; I'm testing a
>>> patch right now.
>>
>> Note that passing SB_NOINTR unconditionally or even only for mounts
>> with nointr (default) option is wrong. You make the socket operation
>> uninterruptible, process terminate-ability becomes depended on the
>> external factor, the behaviour of the remote system.
I looked and I think forced dismounts are broken when the thread is stuck in
sosend(). (It assumes that the threads doing RPCs are waiting for replies.)
--> I think this can be fixed by posting a signal to the threads, but only if
  SB_NOINTR isn't used.
--> As such, I think looping on ERESTART when PCATCH isn't set in ct_waitflag
 is the better way to go. (At a glance, I think sosend_generic() will fail 
with
 ERESTART before putting any data in the send queue. For NFS, it uses the
 mbuf list "top" and is always "atomic".)

Kostik, Colin has already been testing the looping case.

>I'm not sure what you're getting at here.  The fact that "NFS mounted without
>the intr flag" + "unresponsive NFS server" = "unkillable processes" has been
>a (mis)feature of NFS for decades.
As I already mentioned, I'd like to at least get forced dismounts to work, rick

___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Netmap TX with no impact to host

[David Belle-Isle]

Hi,

I'm trying to open a netmap descriptor to an interface to send packets.
However, I don't want to prevent the host to send and receive data
(transparent). I don't think this should be hard but I can't figure out how
to do this.

I tried to run the bridge example in the FreeBSD distribution but even that
I can't get to run without stopping the host's networking completely. I
tried running ./bridge em0 em0 which, if I understand correctly should open
the NIC and host rings and pass the traffic through. As soon as I start it
all the networking stops.

I tried testing in a VMware VM and on bare-metal with em cards and got the
same results with both.

Can someone help me?

Thanks,

David
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: sosend returning ERESTART

[Konstantin Belousov]

On Wed, Jan 18, 2017 at 10:52:02PM +, Rick Macklem wrote:
> Colin Percival wrote:
> >On 01/18/17 02:36, Konstantin Belousov wrote:
> >> On Wed, Jan 18, 2017 at 04:37:40AM +, Colin Percival wrote:
> >>> Thanks, looks like that was exactly it -- if the TCP send buffer was full
> >>> we would call sbwait, and if a signal arrived it would return ERESTART.
> >>> It looks like setting the SB_NOINTR flag will prevent this; I'm testing a
> >>> patch right now.
> >>
> >> Note that passing SB_NOINTR unconditionally or even only for mounts
> >> with nointr (default) option is wrong. You make the socket operation
> >> uninterruptible, process terminate-ability becomes depended on the
> >> external factor, the behaviour of the remote system.
> >
> >I'm not sure what you're getting at here.  The fact that "NFS mounted without
> >the intr flag" + "unresponsive NFS server" = "unkillable processes" has been
> >a (mis)feature of NFS for decades.
> The case I would like to see work is the forced dismount. I need to go look at
> what it does and see if SB_NOINTR would break it worse than it is broken now.
> (It is currently broken when something like "umount" without -f is done, which
>  locks up the mounted on vnode so "umount -f" never gets to the umount(2) 
> syscall.
>  I do plan on a "straight ot NFS" option for umount(8) to avoid this problem, 
> but
>  haven't gotten around to it.)
> 
> The alternative to SB_NOINTR is looping and doing the sosend() again for the
> case where it returns ERESTART and "intr" wasn't set on the mount.
Note that the condition of pending signal which triggered ERESTART is
permanent until the signal is delivered or blocked. In other words, or
future PCATCH sleeps will fail with ERESTART/EINTR.

> --> For this to be ok, we must be sure that when sosend() returns ERESTART,
>   it has not queued the data for sending so it is safe to send it all 
> again.
>   I think this is true for this case?
> 
> rick
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

[Bug 208205] re0 watchdog timeout

[bugzilla-noreply]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=208205

--- Comment #12 from m...@netfence.it ---
I disabled powerd, but the problem showed up again.

It only happened twice in some months, but it's still a critical problem for
us.

-- 
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"