On Wed, Sep 11, 2024 at 9:36 PM Sahil <icegambi...@gmail.com> wrote:
>
> Hi,
>
> On Monday, September 9, 2024 6:04:45 PM GMT+5:30 Eugenio Perez Martin wrote:
> > On Sun, Sep 8, 2024 at 9:47 PM Sahil <icegambi...@gmail.com> wrote:
> > > On Friday, August 30, 2024 4:18:31 PM GMT+5:30 Eugenio Perez Martin wrote:
> > > > On Fri, Aug 30, 2024 at 12:20 PM Sahil <icegambi...@gmail.com> wrote:
> > > > [...]
> > > > vdpa_sim does not support packed vq at the moment. You need to build
> > > > the use case #3 of the second part of that blog [1]. It's good that
> > > > you build the vdpa_sim earlier as it is a simpler setup.
> > > >
> > > > If you have problems with the vp_vdpa environment please let me know
> > > > so we can find alternative setups.
> > >
> > > Thank you for the clarification. I tried setting up the vp_vdpa
> > > environment (scenario 3) but I ended up running into a problem
> > > in the L1 VM.
> > >
> > > I verified that nesting is enabled in KVM (L0):
> > >
> > > $ grep -oE "(vmx|svm)" /proc/cpuinfo | sort | uniq
> > > vmx
> > >
> > > $ cat /sys/module/kvm_intel/parameters/nested
> > > Y
> > >
> > > There are no issues when booting L1. I start the VM by running:
> > >
> > > $ sudo ./qemu/build/qemu-system-x86_64 \
> > > -enable-kvm \
> > > -drive file=//home/ig91/fedora_qemu_test_vm/L1.qcow2,media=disk,if=virtio
> > > \
> > > -net nic,model=virtio \
> > > -net user,hostfwd=tcp::2222-:22 \
> > > -device intel-iommu,snoop-control=on \
> > > -device
> > > virtio-net-pci,netdev=net0,disable-legacy=on,disable-modern=off,iommu_pla
> > > tform=on,event_idx=off,packed=on,bus=pcie.0,addr=0x4 \ -netdev
> > > tap,id=net0,script=no,downscript=no \
> > > -nographic \
> > > -m 2G \
> > > -smp 2 \
> > > -M q35 \
> > > -cpu host \
> > > 2>&1 | tee vm.log
> > >
> > > Kernel version in L1:
> > >
> > > # uname -a
> > > Linux fedora 6.8.5-201.fc39.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Apr 11
> > > 18:25:26 UTC 2024 x86_64 GNU/Linux
> > Did you run the kernels with the arguments "iommu=pt intel_iommu=on"?
> > You can print them with cat /proc/cmdline.
>
> I missed this while setting up the environment. After setting the kernel
> params I managed to move past this issue but my environment in virtualbox
> was very unstable and it kept crashing.
>
I've no experience with virtualbox+vdpa, sorry :). Why not use QEMU also for L1?
> I managed to get L1 to run on my host OS, so scenario 3 is now up and
> running. However, the packed bit seems to be disabled in this scenario too.
>
> L0 (host machine) specs:
> - kernel version:
> 6.6.46-1-lts
>
> - QEMU version:
> 9.0.50 (v8.2.0-5536-g16514611dc)
>
> - vDPA version:
> iproute2-6.10.0
>
> L1 specs:
>
> - kernel version:
> 6.8.5-201.fc39.x86_64
>
> - QEMU version:
> 9.0.91
>
> - vDPA version:
> iproute2-6.10.0
>
> L2 specs:
> - kernel version
> 6.8.7-200.fc39.x86_64
>
> I followed the following steps to set up scenario 3:
>
> ==== In L0 ====
>
> $ grep -oE "(vmx|svm)" /proc/cpuinfo | sort | uniq
> vmx
>
> $ cat /sys/module/kvm_intel/parameters/nested
> Y
>
> $ sudo ./qemu/build/qemu-system-x86_64 \
> -enable-kvm \
> -drive
> file=//home/valdaarhun/valdaarhun/qcow2_img/L1.qcow2,media=disk,if=virtio \
> -net nic,model=virtio \
> -net user,hostfwd=tcp::2222-:22 \
> -device intel-iommu,snoop-control=on \
> -device
> virtio-net-pci,netdev=net0,disable-legacy=on,disable-modern=off,iommu_platform=on,event_idx=off,packed=on,bus=pcie.0,addr=0x4
> \
> -netdev tap,id=net0,script=no,downscript=no \
> -nographic \
> -m 8G \
> -smp 4 \
> -M q35 \
> -cpu host \
> 2>&1 | tee vm.log
>
> ==== In L1 ====
>
> I verified that the following config variables are set as decribed in the
> blog [1].
>
> CONFIG_VIRTIO_VDPA=m
> CONFIG_VDPA=m
> CONFIG_VP_VDPA=m
> CONFIG_VHOST_VDPA=m
>
> # modprobe vdpa
> # modprobe vhost_vdpa
> # modprobe vp_vdpa
>
> # lsmod | grep -i vdpa
> vp_vdpa 20480 0
> vhost_vdpa 32768 0
> vhost 65536 1 vhost_vdpa
> vhost_iotlb 16384 2 vhost_vdpa,vhost
> vdpa 36864 2 vp_vdpa,vhost_vdpa
> irqbypass 12288 2 vhost_vdpa,kvm
>
> # lspci | grep -i ethernet
> 00:04.0 Ethernet controller: Red Hat, Inc. Virtio 1.0 network device (rev 01)
>
> # lspci -nn | grep 00:04.0
> 00:04.0 Ethernet controller [0200]: Red Hat, Inc. Virtio 1.0 network device
> [1af4:1041] (rev 01)
>
> # echo 0000:00:04.0 > /sys/bus/pci/drivers/virtio-pci/unbind
> # echo 1af4 1041 > /sys/bus/pci/drivers/vp-vdpa/new_id
>
> # vdpa mgmtdev show
> pci/0000:00:04.0:
> supported_classes net < unknown class >
> max_supported_vqs 3
> dev_features CSUM GUEST_CSUM CTRL_GUEST_OFFLOADS MAC GUEST_TSO4
> GUEST_TSO6 GUEST_ECN GUEST_UFO HOST_TSO4 HOST_TSO6 HOST_ECN
> HOST_UFO MRG_RXBUF STATUS CTRL_VQ CTRL_RX CTRL_VLAN CTRL_RX_EXTRA
> GUEST_ANNOUNCE CTRL_MAC_ADDR RING_INDIRECT_DESC RING_EVENT_IDX
> VERSION_1 ACCESS_PLATFORM bit_40 bit_54 bit_55 bit_56
>
> # vdpa dev add name vdpa0 mgmtdev pci/0000:00:04.0
> # vdpa dev show -jp
> {
> "dev": {
> "vdpa0": {
> "type": "network",
> "mgmtdev": "pci/0000:00:04.0",
> "vendor_id": 6900,
> "max_vqs": 3,
> "max_vq_size": 256
> }
> }
> }
>
> # ls -l /sys/bus/vdpa/devices/vdpa0/driver
> lrwxrwxrwx. 1 root root 0 Sep 11 17:07 /sys/bus/vdpa/devices/vdpa0/driver ->
> ../../../../bus/vdpa/drivers/vhost_vdpa
>
> # ls -l /dev/ |grep vdpa
> crw-------. 1 root root 239, 0 Sep 11 17:07 vhost-vdpa-0
>
> # driverctl -b vdpa
> vdpa0 vhost_vdpa [*]
>
> Booting L2:
>
> # ./qemu/build/qemu-system-x86_64 \
> -nographic \
> -m 4G \
> -enable-kvm \
> -M q35 \
> -drive file=//root/L2.qcow2,media=disk,if=virtio \
> -netdev type=vhost-vdpa,vhostdev=/dev/vhost-vdpa-0,id=vhost-vdpa0 \
> -device
> virtio-net-pci,netdev=vhost-vdpa0,disable-legacy=on,disable-modern=off,event_idx=off,packed=on,bus=pcie.0,addr=0x7
> \
> -smp 4 \
> -cpu host \
> 2>&1 | tee vm.log
>
> ==== In L2 ====
>
> # cut -c 35 /sys/devices/pci0000:00/0000:00:07.0/virtio1/features
> 0
>
> Based on what I have understood from the kernel's source, vhost-vdpa and
> vp-vdpa both support packed vqs and v6.6.46 is more recent than the
> minimum required kernel version. I am not sure where I am going wrong.
>
> > > However, I managed to set up scenario 4 successfully
> > > and I see that packed vq is enabled in this case.
> > >
> > > # cut -c 35 /sys/devices/pci0000:00/0000:00:04.0/virtio1/features
> > > 1
>
> So far scenario 4 is the only scenario where the virtio-net device has the
> packed
> feature enabled. The difference between scenarios 3 and 4 in terms of the
> kernel
> modules loaded is that scenario 4 uses virtio_vdpa while scenario 3 uses vdpa
> and
> vhost_vdpa.
>
We're pretty close then :). But I don't see anything obviously wrong here.
I guess it is the same L1 VM in both cases, isn't it?
The function that gets the features from vhost-vdpa in QEMU is
hw/virtio/vhost-vdpa.c:vhost_vdpa_get_features. Can you check that it
returns bit 34 (offset starts with 0 here)? If it returns it, can you
keep debugging until you see what clears it?
If it comes clear, then we need to check the kernel.
If you don't see anything wrong there, we can keep debugging the
function that handles getting the features through L2 guest PCI read,
hw/virtio/virtio-pci.c:virtio_pci_common_read. VIRTIO_PCI_COMMON_DF
case when proxy->dfselect==1.
Let me know what you find.
Thanks!
