When checking for possible causes, I noticed that the framebuffer is now (in Debian11) associated with the first gpu. This seems wrong as this is a headless machine, and the nvidia-gpu's are used only as a computational accelerator for cuda workloads, and not for visualization.
This is shown with the command lshw -C display which has this entry logical name: /dev/fb0 The full log is attached. When running Debian10, this entry is not shown, and vulkaninfo was working. Is there an (easy?) way to disable /dev/fb0 ?
*-display description: VGA compatible controller product: GP102 [GeForce GTX 1080 Ti] vendor: NVIDIA Corporation physical id: 0 bus info: pci@0000:05:00.0 logical name: /dev/fb0 version: a1 width: 64 bits clock: 33MHz capabilities: pm msi pciexpress vga_controller bus_master cap_list rom fb configuration: depth=32 driver=nvidia latency=0 mode=1024x768 visual=truecolor xres=1024 yres=768 resources: iomemory:27f0-27ef iomemory:27f0-27ef irq:27 memory:c4000000-c4ffffff memory:27fe0000000-27fefffffff memory:27ff0000000-27ff1ffffff ioport:b000(size=128) memory:c5000000-c507ffff *-display description: VGA compatible controller product: GP102 [GeForce GTX 1080 Ti] vendor: NVIDIA Corporation physical id: 0 bus info: pci@0000:06:00.0 version: a1 width: 64 bits clock: 33MHz capabilities: pm msi pciexpress vga_controller bus_master cap_list rom configuration: driver=nvidia latency=0 resources: iomemory:27f0-27ef iomemory:27f0-27ef irq:27 memory:c2000000-c2ffffff memory:27fc0000000-27fcfffffff memory:27fd0000000-27fd1ffffff ioport:a000(size=128) memory:c3000000-c307ffff *-display description: VGA compatible controller product: GP102 [GeForce GTX 1080 Ti] vendor: NVIDIA Corporation physical id: 0 bus info: pci@0000:07:00.0 version: a1 width: 64 bits clock: 33MHz capabilities: pm msi pciexpress vga_controller bus_master cap_list rom configuration: driver=nvidia latency=0 resources: iomemory:27f0-27ef iomemory:27f0-27ef irq:27 memory:c0000000-c0ffffff memory:27fa0000000-27fafffffff memory:27fb0000000-27fb1ffffff ioport:9000(size=128) memory:c1000000-c107ffff *-display description: VGA compatible controller product: GP102 [GeForce GTX 1080 Ti] vendor: NVIDIA Corporation physical id: 0 bus info: pci@0000:08:00.0 version: a1 width: 64 bits clock: 33MHz capabilities: pm msi pciexpress vga_controller bus_master cap_list rom configuration: driver=nvidia latency=0 resources: iomemory:27f0-27ef iomemory:27f0-27ef irq:27 memory:be000000-beffffff memory:27f80000000-27f8fffffff memory:27f90000000-27f91ffffff ioport:8000(size=128) memory:bf000000-bf07ffff *-display description: VGA compatible controller product: GP102 [GeForce GTX 1080 Ti] vendor: NVIDIA Corporation physical id: 0 bus info: pci@0000:0b:00.0 version: a1 width: 64 bits clock: 33MHz capabilities: pm msi pciexpress vga_controller bus_master cap_list rom configuration: driver=nvidia latency=0 resources: iomemory:27f0-27ef iomemory:27f0-27ef irq:29 memory:bc000000-bcffffff memory:27f60000000-27f6fffffff memory:27f70000000-27f71ffffff ioport:7000(size=128) memory:bd000000-bd07ffff *-display description: VGA compatible controller product: GP102 [GeForce GTX 1080 Ti] vendor: NVIDIA Corporation physical id: 0 bus info: pci@0000:0c:00.0 version: a1 width: 64 bits clock: 33MHz capabilities: pm msi pciexpress vga_controller bus_master cap_list rom configuration: driver=nvidia latency=0 resources: iomemory:27f0-27ef iomemory:27f0-27ef irq:29 memory:ba000000-baffffff memory:27f40000000-27f4fffffff memory:27f50000000-27f51ffffff ioport:6000(size=128) memory:bb000000-bb07ffff *-display description: VGA compatible controller product: GP102 [GeForce GTX 1080 Ti] vendor: NVIDIA Corporation physical id: 0 bus info: pci@0000:0d:00.0 version: a1 width: 64 bits clock: 33MHz capabilities: pm msi pciexpress vga_controller bus_master cap_list rom configuration: driver=nvidia latency=0 resources: iomemory:27f0-27ef iomemory:27f0-27ef irq:29 memory:b8000000-b8ffffff memory:27f20000000-27f2fffffff memory:27f30000000-27f31ffffff ioport:5000(size=128) memory:b9000000-b907ffff *-display description: VGA compatible controller product: GP102 [GeForce GTX 1080 Ti] vendor: NVIDIA Corporation physical id: 0 bus info: pci@0000:0e:00.0 version: a1 width: 64 bits clock: 33MHz capabilities: pm msi pciexpress vga_controller bus_master cap_list rom configuration: driver=nvidia latency=0 resources: iomemory:27f0-27ef iomemory:27f0-27ef irq:29 memory:b6000000-b6ffffff memory:27f00000000-27f0fffffff memory:27f10000000-27f11ffffff ioport:4000(size=128) memory:b7000000-b707ffff *-display description: VGA compatible controller product: GP102 [GeForce GTX 1080 Ti] vendor: NVIDIA Corporation physical id: 0 bus info: pci@0000:0f:00.0 version: a1 width: 64 bits clock: 33MHz capabilities: pm msi pciexpress vga_controller bus_master cap_list rom configuration: driver=nvidia latency=0 resources: iomemory:27e0-27df iomemory:27e0-27df irq:29 memory:b4000000-b4ffffff memory:27ee0000000-27eefffffff memory:27ef0000000-27ef1ffffff ioport:3000(size=128) memory:b5000000-b507ffff *-display description: VGA compatible controller product: ASPEED Graphics Family vendor: ASPEED Technology, Inc. physical id: 0 bus info: pci@0000:12:00.0 version: 30 width: 32 bits clock: 33MHz capabilities: pm msi vga_controller bus_master cap_list rom configuration: driver=ast latency=0 resources: irq:16 memory:c6000000-c6ffffff memory:c7000000-c701ffff ioport:c000(size=128) memory:c0000-dffff