Re: [PATCH kernel 3/3] vfio_pci: Add NVIDIA GV100GL [Tesla V100 SXM2] [10de:1db1] subdriver
On 10/18/18 9:55 AM, Alex Williamson wrote: On Thu, 18 Oct 2018 11:31:33 +1100 Alexey Kardashevskiy wrote: On 18/10/2018 08:52, Alex Williamson wrote: On Wed, 17 Oct 2018 12:19:20 +1100 Alexey Kardashevskiy wrote: On 17/10/2018 06:08, Alex Williamson wrote: On Mon, 15 Oct 2018 20:42:33 +1100 Alexey Kardashevskiy wrote: POWER9 Witherspoon machines come with 4 or 6 V100 GPUs which are not pluggable PCIe devices but implement PCIe links for config space and MMIO. In addition to that the GPUs are interconnected to each other and also have direct links to the P9 CPU. The links are NVLink2 and provide direct access to the system RAM for GPUs via NPU (an NVLink2 "proxy" on P9 chip). These systems also support ATS (address translation services) which is a part of the NVLink2 prototol. Such GPUs also share on-board RAM (16GB in tested config) to the system via the same NVLink2 so a CPU has cache-coherent access to a GPU RAM. This exports GPU RAM to the userspace as a new PCI region. This preregisters the new memory as device memory as it might be used for DMA. This inserts pfns from the fault handler as the GPU memory is not onlined until the NVIDIA driver is loaded and trained the links so doing this earlier produces low level errors which we fence in the firmware so it does not hurt the host system but still better to avoid. This exports ATSD (Address Translation Shootdown) register of NPU which allows the guest to invalidate TLB. The register conviniently occupies a single 64k page. Since NPU maps the GPU memory, it has a "tgt" property (which is an abbreviated host system bus address). This exports the "tgt" as a capability so the guest can program it into the GPU so the GPU can know how to route DMA trafic. I'm not really following what "tgt" is and why it's needed. Is the GPU memory here different than the GPU RAM region above? Why does the user need the host system bus address of this "tgt" thing? Are we not able to relocate it in guest physical address space, does this shootdown only work in the host physical address space and therefore we need this offset? Please explain, I'm confused. This "tgt" is made of: - "memory select" (bits 45, 46) - "group select" (bits 43, 44) - "chip select" (bit 42) - chip internal address (bits 0..41) These are internal to GPU and this is where GPU RAM is mapped into the GPU's real space, this fits 46 bits. On POWER9 CPU the bits are different and higher so the same memory is mapped higher on P9 CPU. Just because we can map it higher, I guess. So it is not exactly the address but this provides the exact physical location of the memory. We have a group of 3 interconnected GPUs, they got their own memory/group/chip numbers. The GPUs use ATS service to translate userspace to physical (host or guest) addresses. Now a GPU needs to know which specific link to use for a specific physical address, in other words what this physical address belongs to - a CPU or one of GPUs. This is when "tgt" is used by the GPU hardware. Clear as mud ;) /me is sad. I hope Piotr explained it better... It's starting to be a bit more clear, and Piotr anticipated the security questions I was mulling over. Great. So tgt, provided by the npu2 capability of the ATSD region of the NPU tells the GPU (a completely separate device) how to route it its own RAM via its NVLink interface? How can one tgt indicate the routing for multiple interfaces? This NVLink DMA is using direct host physical addresses (no IOMMU, no filtering) which come from ATS. So unless we tell the GPU its own address range on the host CPU, it will route trafic via CPU. And the driver can also discover the NVLink topology and tell each GPU physical addresses of peer GPUs. I think this is a key point too, the tgt seems to only identify the routing for the GPU local RAM, but the guest driver is able to conglomerate this information so each GPU knows how to get directly to the other GPUs. Yes. A GPU could run all the DMA trafic via the system bus indeed, just not as fast. I am also struggling here and adding an Nvidia person in cc: (I should have done that when I posted the patches, my bad) to correct when/if I am wrong. For ATS to work, the nest MMU (an NVIDIA block in a P9 CPU) needs to know LPID (a logical partition ID or a KVM guest hardware ID in other words) and PID (a memory context ID of an userspace process, not to be confused with a linux pid). This assigns a GPU to LPID in the NPU and this is why this adds a listener for KVM on an IOMMU group. A PID comes via NVLink from a GPU and NPU uses a PID wildcard to pass it through. This requires coherent memory and ATSD to be available on the host as the GPU vendor only supports configurations with both features enabled and other configurations are known not to work. Because of this and because of the ways the features are advertised to the host system (which is a device tree with very platform specific properties), this
Re: [PATCH kernel 3/3] vfio_pci: Add NVIDIA GV100GL [Tesla V100 SXM2] [10de:1db1] subdriver
On 10/18/18 11:05 AM, Alex Williamson wrote:
On Thu, 18 Oct 2018 10:37:46 -0700
Piotr Jaroszynski wrote:
On 10/18/18 9:55 AM, Alex Williamson wrote:
On Thu, 18 Oct 2018 11:31:33 +1100
Alexey Kardashevskiy wrote:
On 18/10/2018 08:52, Alex Williamson wrote:
On Wed, 17 Oct 2018 12:19:20 +1100
Alexey Kardashevskiy wrote:
On 17/10/2018 06:08, Alex Williamson wrote:
On Mon, 15 Oct 2018 20:42:33 +1100
Alexey Kardashevskiy wrote:
+
+ if (pdev->vendor == PCI_VENDOR_ID_IBM &&
+ pdev->device == 0x04ea) {
+ ret = vfio_pci_ibm_npu2_init(vdev);
+ if (ret) {
+ dev_warn(&vdev->pdev->dev,
+ "Failed to setup NVIDIA NV2 ATSD
region\n");
+ goto disable_exit;
}
So the NPU is also actually owned by vfio-pci and assigned to the VM?
Yes. On a running system it looks like:
0007:00:00.0 Bridge: IBM Device 04ea (rev 01)
0007:00:00.1 Bridge: IBM Device 04ea (rev 01)
0007:00:01.0 Bridge: IBM Device 04ea (rev 01)
0007:00:01.1 Bridge: IBM Device 04ea (rev 01)
0007:00:02.0 Bridge: IBM Device 04ea (rev 01)
0007:00:02.1 Bridge: IBM Device 04ea (rev 01)
0035:00:00.0 PCI bridge: IBM Device 04c1
0035:01:00.0 PCI bridge: PLX Technology, Inc. Device 8725 (rev ca)
0035:02:04.0 PCI bridge: PLX Technology, Inc. Device 8725 (rev ca)
0035:02:05.0 PCI bridge: PLX Technology, Inc. Device 8725 (rev ca)
0035:02:0d.0 PCI bridge: PLX Technology, Inc. Device 8725 (rev ca)
0035:03:00.0 3D controller: NVIDIA Corporation GV100GL [Tesla V100 SXM2]
(rev a1
0035:04:00.0 3D controller: NVIDIA Corporation GV100GL [Tesla V100 SXM2]
(rev a1)
0035:05:00.0 3D controller: NVIDIA Corporation GV100GL [Tesla V100 SXM2]
(rev a1)
One "IBM Device" bridge represents one NVLink2, i.e. a piece of NPU.
They all and 3 GPUs go to the same IOMMU group and get passed through to
a guest.
The entire NPU does not have representation via sysfs as a whole though.
So the NPU is a bridge, but it uses a normal header type so vfio-pci
will bind to it?
An NPU is a NVLink bridge, it is not PCI in any sense. We (the host
powerpc firmware known as "skiboot" or "opal") have chosen to emulate a
virtual bridge per 1 NVLink on the firmware level. So for each physical
NPU there are 6 virtual bridges. So the NVIDIA driver does not need to
know much about NPUs.
And the ATSD register that we need on it is not
accessible through these PCI representations of the sub-pieces of the
NPU? Thanks,
No, only via the device tree. The skiboot puts the ATSD register address
to the PHB's DT property called 'ibm,mmio-atsd' of these virtual bridges.
Ok, so the NPU is essential a virtual device already, mostly just a
stub. But it seems that each NPU is associated to a specific GPU, how
is that association done? In the use case here it seems like it's just
a vehicle to provide this ibm,mmio-atsd property to guest DT and the tgt
routing information to the GPU. So if both of those were attached to
the GPU, there'd be no purpose in assigning the NPU other than it's in
the same IOMMU group with a type 0 header, so something needs to be
done with it. If it's a virtual device, perhaps it could have a type 1
header so vfio wouldn't care about it, then we would only assign the
GPU with these extra properties, which seems easier for management
tools and users. If the guest driver needs a visible NPU device, QEMU
could possibly emulate one to make the GPU association work
automatically. Maybe this isn't really a problem, but I wonder if
you've looked up the management stack to see what tools need to know to
assign these NPU devices and whether specific configurations are
required to make the NPU to GPU association work. Thanks,
I'm not that familiar with how this was originally set up, but note that
Alexey is just making it work exactly like baremetal does. The baremetal
GPU driver works as-is in the VM and expects the same properties in the
device-tree. Obviously it doesn't have to be that way, but there is
value in keeping it identical.
Another probably bigger point is that the NPU device also implements the
nvlink HW interface and is required for actually training and
maintaining the link up. The driver in the guest trains the links by
programming both the GPU end and the NPU end of each link so the NPU
device needs to be exposed to the guest.
Ok, so there is functionality in assigning the NPU device itself, it's
not just an attachment point for meta data. But it still seems there
must be some association of NPU to GPU, the tgt address seems to pair
the NPU with a specific GPU, they're not simply a fungible set of NPUs
and GPUs. Is that association explicit anywhere or is it related to
the topology or device numbering that needs to match between the host
and guest? Thanks,
GPUs
