Here are the steps that we use. If you just want to use bare metal, I think you’ll need to use passthrough : false; in the .conf. · The only supported output from virtual machines is serial console (i.e. you don't get a nice GUI if you use a Linux guest) · You should start your VM in a tmux session, so you can disconnect from the VM without having to shut it down. · To use SR-IOV, it's required that your host's FreeBSD version is 11 or newer, but it's recommended that your guest is 11 or newer, too. · A listing of currently active VMs can be found in /dev/vmm · VM names can only be up to 31 characters long Steps to start a VM 1. If you want to attach VF's to the virtual machine, you'll need to run these two commands first: kenv hw.vmm.force_iommu=1 kldload vmm 2. Then you will want to set up a tap interface so that your VM has network connectivity (either to the outside or your host, but in this case, we'll use a bridge so it has outside connectivity). ifconfig tap0 create sysctl net.link.tap.up_on_open=1 ifconfig bridge0 create ifconfig bridge0 addm igb0 addm tap0 ifconfig bridge0 up 3. Create a disk image that will serve as your VM's hard disk truncate -s 40g freebsdvm-fs.img 4. Booting up a VM is a two-step process: start the bootloader using bhyveload (only if you're using a FreeBSD guest), and then boot the kernel using the bhyve command. · Example bhyveload+bhyve commands I used to start a FreeBSD guest installation (where disc1.iso is a FreeBSD install disc .iso):
bhyveload -m 4096M -d disc1.iso fbsdvm bhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s 3:0,virtio-blk,freebsdvm-fs.img -s 4:0,ahci-cd,disc1.iso -l com1,stdio -c 4 -m 4096M fbsdvm · The -m arguments given to both commands need to match! · Specify the xterm option when prompted during the install process to get color. · Letting the installer configure partitions for you is fine. · Then, the following will run your new, installed FreeBSD VM: bhyveload -m 4096M -d freebsdvm-fs.img fbsdvm bhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s 3:0,virtio-blk,freebsdvm-fs.img -l com1,stdio -c 4 -m 4096M fbsdvm Using VFs (SR-IOV) 1. kldload your SR-IOV enabled PF driver if it isn't already loaded 2. Create a .conf (for iovctl to use; the file name doesn't matter, but it's recommended to use the interface name) for each PF interface that you want to add VFs to. · An example for an ixl interface: · PF { · device : "ixl0"; · num_vfs : 3; · } · DEFAULT { · passthrough : true; · } · VF-0 { · mac-addr : "02:01:02:03:04:05"; } · The only required fields are the PF and DEFAULT fields. · Though it's generally a good idea to specify a VF's MAC address from the PF · iovctl -d <interface> -S will show all of the parameters supported by the driver 3. iovctl -f <interface>.conf -C will create the VFs using the parameters in the .conf. 4. To attach the newly-spawned VFs to a VM: a. Run "pciconf -v -l", and find the device bus/slot/function for the VF that you want to attach: ppt0@pci0:130:0:16: class=0x020000 card=0x00008086 chip=0x154c8086 rev=0x01 hdr=0x00 b. Run the bhyveload and bhyve commands as above, but add -s 4:0,passthru,130/0/16 (replace with your b/s/f from above) to the bhyve command, and add -S to both commands. i. The -S part is new for FreeBSD 11.0-CURRENT 5. After the VM boots, load the PF's corresponding VF driver in the VM. Shutting down a VM · Shutdown or reboot through the guest · Or, if that's not possible, use the bhyvectl command: bhyvectl --force-off --vm=<vm-name> · To completely delete a VM, use this bhyvectl command, instead: bhyvectl --destroy --vm=<vm-name> Links · Handbook page on bhyve<https://www.freebsd.org/doc/handbook/virtualization-host-bhyve.html> · Wiki page on bhyve<https://wiki.freebsd.org/bhyve> · Note: This page has a three-line quickstart if you just want to boot up a VM with minimal customization · Man page for bhyve(8)<https://www.freebsd.org/cgi/man.cgi?query=bhyve&sektion=8> · Man page for bhyveload(8)<https://www.freebsd.org/cgi/man.cgi?query=bhyveload&sektion=8&apropos=0&manpath=FreeBSD+10.1-RELEASE> · UEFI & VNC graphics on bhyve<https://wiki.freebsd.org/bhyve/UEFI> Jeff On 5/29/18, 11:19 AM, "Sean Bruno" <owner-freebsd-...@freebsd.org on behalf of sbr...@freebsd.org> wrote: On 05/29/18 12:15, Kevin Bowling wrote: > iovctl worked at one point with ixgbe. I don’t recall if there is any > way to attach the vf directly if you do that but you can use ppt driver > to pass them into bhyve. > > On Tue, May 29, 2018 at 9:58 AM Sean Bruno <sbr...@freebsd.org > <mailto:sbr...@freebsd.org>> wrote: > > Does anyone have a process for testing the VF drivers (ixgbe igb, etc) > in FreeBSD without actually firing up linux to instantiate a VM or using > EC2? > > sean > I'm using iovctl with a different vendor h/w on ARM64, so I think its still working. Do you have a reference to a "how to" or whatever that someone like Dexter is maintaining? sean _______________________________________________ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"