Re: Ryzen lockup on bhyve was (Re: new Ryzen lockup issue ?)

2018-03-17 Thread Nimrod Levy 
Looks like I got almost 4 full weeks before it locked up this morning
:(



On Fri, Feb 23, 2018 at 3:33 PM Nimrod Levy  wrote:

> After a couple of hours of running the iperf commands you were testing
> with, I'm unable to duplicate this so far.
>
> I'm running with FreeBSD stable from 17-Feb with the commits noted in
> https://reviews.freebsd.org/D14347 pulled in.
>
> I've also lowered the memory clock and disabled c-states in the bios.
>
> The bhyve VM is running CentOS.
>
> The system has been up for over 6 days and has been running the iperf3
> loop for over 2 hours.
>
> The hardware is an Asus prime B350-Plus with a Ryzen 5 1600 and 32G of RAM.
>
> --
> Nimrod
>
>
> On Fri, Feb 23, 2018 at 3:22 PM, Mike Tancsa  wrote:
>
>> Actually I can confirm the same sort of hard lockup happens on my Epyc
>> board with RELENG11.  It also happens in current. I will file a PR and
>> post on freebsd-current in case someone has any suggestions on how to
>> try and figure out whats going on.
>>
>> I upgraded the box to
>> 12.0-CURRENT #0 r329866
>> in order to see if it could avoid the lockup, but same deal.  The vmm
>> driver does seem different when loaded, but the same lock up under load
>>
>> CPU: AMD Ryzen 5 1600X Six-Core Processor(3593.35-MHz
>> K8-class CPU)
>>   Origin="AuthenticAMD"  Id=0x800f11  Family=0x17  Model=0x1  Stepping=1
>>
>>
>> Features=0x178bfbff
>>
>>
>> Features2=0x7ed8320b
>>   AMD Features=0x2e500800
>>   AMD
>>
>> Features2=0x35c233ff
>>   Structured Extended
>>
>> Features=0x209c01a9
>>   XSAVE Features=0xf
>>   AMD Extended Feature Extensions ID EBX=0x7
>>   SVM: NP,NRIP,VClean,AFlush,DAssist,NAsids=32768
>>   TSC: P-state invariant, performance statistics
>>
>>
>> AMD-Vi: IVRS Info VAsize = 64 PAsize = 48 GVAsize = 2 flags:0
>> driver bug: Unable to set devclass (class: ppc devname: (unknown))
>> ivhd0:  on acpi0
>> ivhd0: Flag:b0
>> ivhd0: Features(type:0x11) MsiNumPPR = 0 PNBanks= 2 PNCounters= 0
>> ivhd0: Extended features[31:0]:22294ada HATS =
>> 0x2 GATS = 0x0 GLXSup = 0x1 SmiFSup = 0x1 SmiFRC = 0x2 GAMSup = 0x1
>> DualPortLogSup = 0x2 DualEventLogSup = 0x2
>> ivhd0: Extended features[62:32]:f77ef Max PASID: 0x2f
>> DevTblSegSup = 0x3 MarcSup = 0x1
>> ivhd0: supported paging level:7, will use only: 4
>> ivhd0: device range: 0x0 - 0x
>> ivhd0: PCI cap 0x190b640f@0x40 feature:19
>>
>>
>>
>> On 2/23/2018 12:35 PM, Nimrod Levy wrote:
>> > Now that is a fascinating data point. My machine that I've been having
>> > issues with has been running a bhyve vm from the beginning.  I never
>> > made the connection. I'll try throwing some network traffic at the VM
>> > and see if I can make it lock up.
>> >
>> > On Fri, Feb 23, 2018 at 10:14 AM, Mike Tancsa > > > wrote:
>> >
>> > On 2/22/2018 3:41 PM, Mike Tancsa wrote:
>> > > On 2/21/2018 3:04 PM, Mike Tancsa wrote:
>> > >> Not sure if I have found another issue specific to Ryzen, or a
>> bug that
>> > >> manifests itself on Ryzen systems easier.  I installed the latest
>> > >> virtualbox from the ports and was doing some network performance
>> tests
>> > >> between a vm and the hypervisor using iperf3.  The guest is just
>> a
>> > >> RELENG11 image and the network is an em nic bridged to epair1b
>> > >
>> > > This looks possibly related to VirtualBox. Doing the same tests
>> and more
>> > > using bhyve, I dont get any lockup.  Not to mention, network IO
>> is MUCH
>> > > faster.
>> >
>> >
>> > Actually, it just took a little bit longer to lock up the box with
>> bhyve
>> > on RELENG_11 as the hypervisor.   Would be great if anyone can
>> confirm
>> > this locks up their Ryzen boxes ? I tried 2 different boxes to
>> eliminate
>> > a hardware issue.  Also tried a similar test on Ubuntu and I can
>> spin up
>> > 4 instances and run without lockups.
>> >
>> > Just grab a copy of
>> >
>> >
>> https://download.freebsd.org/ftp/releases/VM-IMAGES/11.1-RELEASE/amd64/Latest/FreeBSD-11.1-RELEASE-amd64.raw.xz
>> > <
>> https://download.freebsd.org/ftp/releases/VM-IMAGES/11.1-RELEASE/amd64/Latest/FreeBSD-11.1-RELEASE-amd64.raw.xz
>> >
>> >
>> > and make 2 copies. tmp.raw and tmp2.raw
>> >
>> >
>> > kldload vmm
>> > ifconfig tap0 create
>> > ifconfig tap1 create
>> > ifconfig tap1 up
>> > ifconfig tap0 up
>> > ifconfig bridge0 create addm tap0 addm tap1
>> > ifconfig bridge0 192.168.99.1/24 
>> >
>> > screen -d -m sh /usr/share/examples/bhyve/vmrun.sh -c 4 -m 6144M -t
>> tap0
>> > -d tmp.raw BSD11a
>> > screen -d -m sh /usr/share/examples/bhyve/vmrun.sh -c 4 -m 6144M -t
>> tap1
>> > -d tmp2.raw BSD11b
>> >
>> > Install netperf on the 2 vms and give the vtnet interface
>> > 192.168.99.2/24  and 192.168.99.3/24
>> > 
>> >
>> > In both VMs pkg install iperf3 and start it up as
>> > iperf -s
>> >
>> > In the hy

Re: [HEADS UP] - OFED/RDMA stack update

2018-03-17 Thread Navdeep Parhar 
Hold your horses.  Do you have confirmation from the affected party that
the shims are adequate for them?  I have been waiting for that before
looking at this branch.

Is the iw_cxgbe breakage a simple merge conflict as previously discussed
or do the shims require driver changes?  If they don't then it should be
possible to drop the iw_cxgbe from head into this branch and it should
just work, is that correct?

Regards,
Navdeep

On Fri, Mar 16, 2018 at 05:44:41PM +0100, Hans Petter Selasky wrote:
> Hi,
> 
> The bsd_rdma_4_9_stable_11 projects branch is close to being merged into
> FreeBSD 11-stable. Mellanox plans to merge no later than 12:00 CEST TUE 20th
> of March 2018, unless objections are received.
> 
> A compatibility header file has been created, ib_verbs_compat.h, which
> offers full source compatibility to existing OFED kernel applications, as a
> response to the raised conserns. User-space compatibility is maintained
> through library symbol versioning.
> 
> https://svnweb.freebsd.org/base/projects/bsd_rdma_4_9_stable_11/sys/ofed/include/rdma/ib_verbs_compat.h
> 
> An example client for this header file can be found here:
> 
> https://svnweb.freebsd.org/base/projects/bsd_rdma_4_9_stable_11/sys/contrib/rdma/krping_compat/
> 
> Currently the cxgb and cxgbe i-Warp drivers are not building, and will be
> stubbed from the kernel build before the branch is merged, unless Chelsio
> can add patches for these.
> 
> Here is a quick and dirty patch to make the bsd_rdma_4_9_stable_11 branch
> build:
> 
> >diff --git a/sys/modules/Makefile b/sys/modules/Makefile
> >index 6b005c854d7..b918a208f21 100644
> >--- a/sys/modules/Makefile
> >+++ b/sys/modules/Makefile
> >@@ -530,7 +530,7 @@ _txp=   txp
> > .if ${MK_SOURCELESS_UCODE} != "no" && ${MACHINE_CPUARCH} != "arm" && \
> >${MACHINE_ARCH:C/mips(el)?/mips/} != "mips" && \
> >${MACHINE_ARCH} != "powerpc" && ${MACHINE_CPUARCH} != "riscv"
> >-_cxgbe=cxgbe
> >+#_cxgbe=   cxgbe
> > .endif
> > .if ${MK_TESTS} != "no" || defined(ALL_MODULES)
> >@@ -554,7 +554,7 @@ _vpo=   vpo
> > _sym=  sym
> > # intr_disable() is a macro, causes problems
> > .if ${MK_SOURCELESS_UCODE} != "no"
> >-_cxgb= cxgb
> >+#_cxgb=cxgb
> > .endif
> > .endif
> 
> --HPS
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: [HEADS UP] - OFED/RDMA stack update

2018-03-17 Thread Hans Petter Selasky 

On 03/17/18 20:52, Navdeep Parhar wrote:

Hold your horses.  Do you have confirmation from the affected party that
the shims are adequate for them?  I have been waiting for that before
looking at this branch.


Hi Navdeep,

Mellanox has received an API list from at least one party, and has taken 
the action to support all the required APIs.



Is the iw_cxgbe breakage a simple merge conflict as previously discussed
or do the shims require driver changes?  


It is a merge conflict. The code already compiles in 12-current.


If they don't then it should be
possible to drop the iw_cxgbe from head into this branch and it should
just work, is that correct?


Yes, it shouldn't be too hard to do and I would appreciate if Chelsio 
could throw some resources at this ASAP. I believe the new code will 
benefit everyone using Infiniband/RoCE and iWarp with FreeBSD.


--HPS
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"