On Fri, Aug 18, 2023 at 5:08 AM Akihiko Odaki <akihiko.od...@gmail.com> wrote:
> On 2023/08/18 8:47, Gurchetan Singh wrote: > > > > > > On Wed, Aug 16, 2023 at 10:28 PM Akihiko Odaki <akihiko.od...@gmail.com > > <mailto:akihiko.od...@gmail.com>> wrote: > > > > On 2023/08/17 11:23, Gurchetan Singh wrote: > > > From: Gurchetan Singh <gurchetansi...@chromium.org > > <mailto:gurchetansi...@chromium.org>> > > > > > > This adds basic documentation for virtio-gpu. > > > > > > Suggested-by: Akihiko Odaki <akihiko.od...@daynix.com > > <mailto:akihiko.od...@daynix.com>> > > > Signed-off-by: Gurchetan Singh <gurchetansi...@chromium.org > > <mailto:gurchetansi...@chromium.org>> > > > Tested-by: Alyssa Ross <h...@alyssa.is <mailto:h...@alyssa.is>> > > > Tested-by: Emmanouil Pitsidianakis > > <manos.pitsidiana...@linaro.org <mailto: > manos.pitsidiana...@linaro.org>> > > > Reviewed-by: Emmanouil Pitsidianakis > > <manos.pitsidiana...@linaro.org <mailto: > manos.pitsidiana...@linaro.org>> > > > --- > > > v2: - Incorporated suggestions by Akihiko Odaki > > > - Listed the currently supported capset_names (Bernard) > > > > > > v3: - Incorporated suggestions by Akihiko Odaki and Alyssa Ross > > > > > > v4: - Incorporated suggestions by Akihiko Odaki > > > > > > v5: - Removed pci suffix from examples > > > - Verified that -device virtio-gpu-rutabaga works. Strangely > > > enough, I don't remember changing anything, and I remember > > > it not working. I did rebase to top of tree though. > > > - Fixed meson examples in crosvm docs > > > > > > docs/system/device-emulation.rst | 1 + > > > docs/system/devices/virtio-gpu.rst | 113 > > +++++++++++++++++++++++++++++ > > > 2 files changed, 114 insertions(+) > > > create mode 100644 docs/system/devices/virtio-gpu.rst > > > > > > diff --git a/docs/system/device-emulation.rst > > b/docs/system/device-emulation.rst > > > index 4491c4cbf7..1167f3a9f2 100644 > > > --- a/docs/system/device-emulation.rst > > > +++ b/docs/system/device-emulation.rst > > > @@ -91,6 +91,7 @@ Emulated Devices > > > devices/nvme.rst > > > devices/usb.rst > > > devices/vhost-user.rst > > > + devices/virtio-gpu.rst > > > devices/virtio-pmem.rst > > > devices/vhost-user-rng.rst > > > devices/canokey.rst > > > diff --git a/docs/system/devices/virtio-gpu.rst > > b/docs/system/devices/virtio-gpu.rst > > > new file mode 100644 > > > index 0000000000..8c5c708272 > > > --- /dev/null > > > +++ b/docs/system/devices/virtio-gpu.rst > > > @@ -0,0 +1,113 @@ > > > +.. > > > + SPDX-License-Identifier: GPL-2.0 > > > + > > > +virtio-gpu > > > +========== > > > + > > > +This document explains the setup and usage of the virtio-gpu > device. > > > +The virtio-gpu device paravirtualizes the GPU and display > > controller. > > > + > > > +Linux kernel support > > > +-------------------- > > > + > > > +virtio-gpu requires a guest Linux kernel built with the > > > +``CONFIG_DRM_VIRTIO_GPU`` option. > > > + > > > +QEMU virtio-gpu variants > > > +------------------------ > > > + > > > +QEMU virtio-gpu device variants come in the following form: > > > + > > > + * ``virtio-vga[-BACKEND]`` > > > + * ``virtio-gpu[-BACKEND][-INTERFACE]`` > > > + * ``vhost-user-vga`` > > > + * ``vhost-user-pci`` > > > + > > > +**Backends:** QEMU provides a 2D virtio-gpu backend, and two > > accelerated > > > +backends: virglrenderer ('gl' device label) and rutabaga_gfx > > ('rutabaga' > > > +device label). There is a vhost-user backend that runs the > > graphics stack > > > +in a separate process for improved isolation. > > > + > > > +**Interfaces:** QEMU further categorizes virtio-gpu device > > variants based > > > +on the interface exposed to the guest. The interfaces can be > > classified > > > +into VGA and non-VGA variants. The VGA ones are prefixed with > > virtio-vga > > > +or vhost-user-vga while the non-VGA ones are prefixed with > > virtio-gpu or > > > +vhost-user-gpu. > > > + > > > +The VGA ones always use the PCI interface, but for the non-VGA > > ones, the > > > +user can further pick between MMIO or PCI. For MMIO, the user > > can suffix > > > +the device name with -device, though vhost-user-gpu does not > > support MMIO. > > > +For PCI, the user can suffix it with -pci. Without these > > suffixes, the > > > +platform default will be chosen. > > > + > > > +virtio-gpu 2d > > > +------------- > > > + > > > +The default 2D backend only performs 2D operations. The guest > > needs to > > > +employ a software renderer for 3D graphics. > > > + > > > +Typically, the software renderer is provided by `Mesa`_ or > > `SwiftShader`_. > > > +Mesa's implementations (LLVMpipe, Lavapipe and virgl below) work > > out of box > > > +on typical modern Linux distributions. > > > + > > > +.. parsed-literal:: > > > + -device virtio-gpu > > > + > > > +.. _Mesa: https://www.mesa3d.org/ <https://www.mesa3d.org/> > > > +.. _SwiftShader: https://github.com/google/swiftshader > > <https://github.com/google/swiftshader> > > > + > > > +virtio-gpu virglrenderer > > > +------------------------ > > > + > > > +When using virgl accelerated graphics mode in the guest, OpenGL > > API calls > > > +are translated into an intermediate representation (see > > `Gallium3D`_). The > > > +intermediate representation is communicated to the host and the > > > +`virglrenderer`_ library on the host translates the intermediate > > > +representation back to OpenGL API calls. > > > + > > > +.. parsed-literal:: > > > + -device virtio-gpu-gl > > > + > > > +.. _Gallium3D: > > https://www.freedesktop.org/wiki/Software/gallium/ > > <https://www.freedesktop.org/wiki/Software/gallium/> > > > +.. _virglrenderer: > > https://gitlab.freedesktop.org/virgl/virglrenderer/ > > <https://gitlab.freedesktop.org/virgl/virglrenderer/> > > > + > > > +virtio-gpu rutabaga > > > +------------------- > > > + > > > +virtio-gpu can also leverage `rutabaga_gfx`_ to provide > `gfxstream`_ > > > +rendering and `Wayland display passthrough`_. With the > > gfxstream rendering > > > +mode, GLES and Vulkan calls are forwarded to the host with > minimal > > > +modification. > > > + > > > +The crosvm book provides directions on how to build a > > `gfxstream-enabled > > > +rutabaga`_ and launch a `guest Wayland proxy`_. > > > + > > > +This device does require host blob support (``hostmem`` field > > below). The > > > +``hostmem`` field specifies the size of virtio-gpu host memory > > window. > > > +This is typically between 256M and 8G. > > > + > > > +At least one capset (see colon separated ``capset_names`` below) > > must be > > > +specified when starting the device. The currently supported > > > +``capset_names`` are ``gfxstream-vulkan`` and ``cross-domain`` > > on Linux > > > +guests. For Android guests, ``gfxstream-gles`` is also supported. > > > + > > > +The device will try to auto-detect the wayland socket path if the > > > +``cross-domain`` capset name is set. The user may optionally > > specify > > > +``wayland_socket_path`` for non-standard paths. > > > + > > > +The ``wsi`` option can be set to ``surfaceless`` or ``headless``. > > > +Surfaceless doesn't create a native window surface, but does > > copy from the > > > +render target to the Pixman buffer if a virtio-gpu 2D hypercall > > is issued. > > > +Headless is like surfaceless, but doesn't copy to the Pixman > buffer. > > > +Surfaceless is the default if ``wsi`` is not specified. > > > + > > > +.. parsed-literal:: > > > + -device > > virtio-gpu-rutabaga,capset_names=gfxstream-vulkan:cross-domain, > > > + > > hostmem=8G,wayland_socket_path=/tmp/nonstandard/mock_wayland.sock, > > > + wsi=headless > > > + > > > +.. _rutabaga_gfx: > > > https://github.com/google/crosvm/blob/main/rutabaga_gfx/ffi/src/include/rutabaga_gfx_ffi.h > < > https://github.com/google/crosvm/blob/main/rutabaga_gfx/ffi/src/include/rutabaga_gfx_ffi.h > > > > > +.. _gfxstream: > > https://android.googlesource.com/platform/hardware/google/gfxstream/ > > < > https://android.googlesource.com/platform/hardware/google/gfxstream/> > > > +.. _Wayland display passthrough: > > https://www.youtube.com/watch?v=OZJiHMtIQ2M > > <https://www.youtube.com/watch?v=OZJiHMtIQ2M> > > > +.. _gfxstream-enabled rutabaga: > > https://crosvm.dev/book/appendix/rutabaga_gfx.html > > <https://crosvm.dev/book/appendix/rutabaga_gfx.html> > > > > You have different links for "rutabaga_gfx" and "gfxstream-enabled > > rutabaga", but I think you only need one. > > > > > > Done. Didn't resend the entire patch series (to avoid spamming list), > > just did "in-reply-to". The change is also available at: > > > > > https://gitlab.freedesktop.org/gurchetansingh/qemu-gfxstream/-/commits/qemu-gfxstream-v8 > < > https://gitlab.freedesktop.org/gurchetansingh/qemu-gfxstream/-/commits/qemu-gfxstream-v8 > > > > The patch series now looks good so I finally decided to try this patch > series, but I couldn't get it work. > > I noticed gfxstream has page size hardcoded as 4 KiB, which broke my > setup on M2 MacBook Air (running Asahi Linux) that has 16 KiB page. You > may add code to check host page size and to report an error if it's not > 4 KiB to virtio-gpu-rutabaga, but I think it's trivial to fix gfxstream > to query page size at runtime as QEMU and Rutabaga does so I hope you to > do so. For testing purpose, I have replaced it with 16 KiB on my setup. > Good catch, the fix to the 16KiB was merged today and is in gfxstream ToT right now. > > I also found some bugs on QEMU side so I added comments to respective > patches. > > Below is the logs from my last attempt of running vkgears: > > $ VK_LOADER_DEBUG=all demos/build/src/vulkan/vkgears > INFO: Vulkan Loader Version 1.3.243 > LAYER: Searching for layer manifest files > LAYER: In following locations: > LAYER: /var/home/person/.config/vulkan/implicit_layer.d > LAYER: /etc/xdg/vulkan/implicit_layer.d > LAYER: /etc/vulkan/implicit_layer.d > LAYER: > /var/home/person/.local/share/vulkan/implicit_layer.d > LAYER: > /var/home/person/.local/share/flatpak/exports/share/vulkan/implicit_layer.d > LAYER: > /var/lib/flatpak/exports/share/vulkan/implicit_layer.d > LAYER: /usr/local/share/vulkan/implicit_layer.d > LAYER: /usr/share/vulkan/implicit_layer.d > LAYER: Found the following files: > LAYER: > /usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json > INFO: Found manifest file > /usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json (file > version "1.0.0") > LAYER: Searching for layer manifest files > LAYER: In following locations: > LAYER: /var/home/person/.config/vulkan/explicit_layer.d > LAYER: /etc/xdg/vulkan/explicit_layer.d > LAYER: /etc/vulkan/explicit_layer.d > LAYER: > /var/home/person/.local/share/vulkan/explicit_layer.d > LAYER: > /var/home/person/.local/share/flatpak/exports/share/vulkan/explicit_layer.d > LAYER: > /var/lib/flatpak/exports/share/vulkan/explicit_layer.d > LAYER: /usr/local/share/vulkan/explicit_layer.d > LAYER: /usr/share/vulkan/explicit_layer.d > LAYER: Found no files > DRIVER: Searching for driver manifest files > DRIVER: In following locations: > DRIVER: /var/home/person/.config/vulkan/icd.d > DRIVER: /etc/xdg/vulkan/icd.d > DRIVER: /etc/vulkan/icd.d > DRIVER: /var/home/person/.local/share/vulkan/icd.d > DRIVER: > /var/home/person/.local/share/flatpak/exports/share/vulkan/icd.d > DRIVER: /var/lib/flatpak/exports/share/vulkan/icd.d > DRIVER: /usr/local/share/vulkan/icd.d > DRIVER: /usr/share/vulkan/icd.d > DRIVER: Found the following files: > DRIVER: > /usr/local/share/vulkan/icd.d/cereal_icd.aarch64.json > DRIVER: /usr/share/vulkan/icd.d/broadcom_icd.aarch64.json > DRIVER: /usr/share/vulkan/icd.d/freedreno_icd.aarch64.json > DRIVER: /usr/share/vulkan/icd.d/lvp_icd.aarch64.json > DRIVER: /usr/share/vulkan/icd.d/panfrost_icd.aarch64.json > DRIVER: /usr/share/vulkan/icd.d/radeon_icd.aarch64.json > DRIVER: Found ICD manifest file > /usr/local/share/vulkan/icd.d/cereal_icd.aarch64.json, version "1.0.0" > DEBUG | DRIVER: Searching for ICD drivers named > /usr/local/lib64/libvulkan_cereal.so > [VirtGpuDevice.cpp(71)] > [prio 6] virtgpu backend not enabling > VIRTGPU_PARAM_CREATE_GUEST_HANDLE[AndroidHealthMonitor.cpp(275)] > [prio 4] HealthMonitor disabled. Returning nullptrI0818 21:00:07.980257 > 154451 IntelDrmDecoder.cpp:38] IntelDrmDecoder created for context 2 > DRIVER: Found ICD manifest file > /usr/share/vulkan/icd.d/broadcom_icd.aarch64.json, version "1.0.0" > DEBUG | DRIVER: Searching for ICD drivers named > /usr/lib64/libvulkan_broadcom.so > DRIVER: Found ICD manifest file > /usr/share/vulkan/icd.d/freedreno_icd.aarch64.json, version "1.0.0" > DEBUG | DRIVER: Searching for ICD drivers named > /usr/lib64/libvulkan_freedreno.so > DRIVER: Found ICD manifest file > /usr/share/vulkan/icd.d/lvp_icd.aarch64.json, version "1.0.0" > DEBUG | DRIVER: Searching for ICD drivers named > /usr/lib64/libvulkan_lvp.so > DRIVER: Found ICD manifest file > /usr/share/vulkan/icd.d/panfrost_icd.aarch64.json, version "1.0.0" > DEBUG | DRIVER: Searching for ICD drivers named > /usr/lib64/libvulkan_panfrost.so > DRIVER: Found ICD manifest file > /usr/share/vulkan/icd.d/radeon_icd.aarch64.json, version "1.0.0" > DEBUG | DRIVER: Searching for ICD drivers named > /usr/lib64/libvulkan_radeon.so > DEBUG | LAYER: Loading layer library libVkLayer_MESA_device_select.so > INFO | LAYER: Insert instance layer "VK_LAYER_MESA_device_select" > (libVkLayer_MESA_device_select.so) > LAYER: vkCreateInstance layer callstack setup to: > LAYER: <Application> > LAYER: || > LAYER: <Loader> > LAYER: || > LAYER: VK_LAYER_MESA_device_select > LAYER: Type: Implicit > LAYER: Disable Env Var: NODEVICE_SELECT > LAYER: Manifest: > /usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json > LAYER: Library: libVkLayer_MESA_device_select.so > LAYER: || > LAYER: <Drivers> > I0818 21:00:08.014896 154451 VkDecoderGlobalState.cpp:443] Creating > Vulkan instance for app: vkgears > INFO | DRIVER: linux_read_sorted_physical_devices: > INFO | DRIVER: Original order: > INFO | DRIVER: [0] llvmpipe (LLVM 16.0.6, 128 bits) > INFO | DRIVER: [1] llvmpipe (LLVM 15.0.7, 128 bits) > INFO | DRIVER: Sorted order: > INFO | DRIVER: [0] llvmpipe (LLVM 15.0.7, 128 bits) > INFO | DRIVER: [1] llvmpipe (LLVM 16.0.6, 128 bits) > INFO | DRIVER: linux_read_sorted_physical_devices: > INFO | DRIVER: Original order: > INFO | DRIVER: [0] llvmpipe (LLVM 16.0.6, 128 bits) > INFO | DRIVER: [1] llvmpipe (LLVM 15.0.7, 128 bits) > INFO | DRIVER: Sorted order: > INFO | DRIVER: [0] llvmpipe (LLVM 15.0.7, 128 bits) > INFO | DRIVER: [1] llvmpipe (LLVM 16.0.6, 128 bits) > DEBUG | DRIVER: Copying old device 0 into new device 0 > DEBUG | DRIVER: Copying old device 1 into new device 1 > INFO | DRIVER: linux_read_sorted_physical_devices: > INFO | DRIVER: Original order: > INFO | DRIVER: [0] llvmpipe (LLVM 16.0.6, 128 bits) > INFO | DRIVER: [1] llvmpipe (LLVM 15.0.7, 128 bits) > INFO | DRIVER: Sorted order: > INFO | DRIVER: [0] llvmpipe (LLVM 15.0.7, 128 bits) > INFO | DRIVER: [1] llvmpipe (LLVM 16.0.6, 128 bits) > DEBUG | DRIVER: Copying old device 0 into new device 0 > DEBUG | DRIVER: Copying old device 1 into new device 1 > INFO | DRIVER: linux_read_sorted_physical_devices: > INFO | DRIVER: Original order: > INFO | DRIVER: [0] llvmpipe (LLVM 16.0.6, 128 bits) > INFO | DRIVER: [1] llvmpipe (LLVM 15.0.7, 128 bits) > INFO | DRIVER: Sorted order: > INFO | DRIVER: [0] llvmpipe (LLVM 15.0.7, 128 bits) > INFO | DRIVER: [1] llvmpipe (LLVM 16.0.6, 128 bits) > DEBUG | DRIVER: Copying old device 0 into new device 0 > DEBUG | DRIVER: Copying old device 1 into new device 1 > ERROR: loader_validate_device_extensions: Device extension > VK_KHR_swapchain not supported by selected physical device or enabled Yeah, any non-headless Linux tests are unlikely to work. Maybe in a future gfxstream release, since our focus is of course on Android and getting Vulkan in QEMU in general. > > layers. > ERROR: vkCreateDevice: Failed to validate extensions in list > ERROR: vkGetDeviceQueue2: Invalid device > [VUID-vkGetDeviceQueue2-device-parameter] > Aborted (core dumped) >
