Re: [PATCH v2 8/8] habanalabs: enable 64-bit DMA mask in POWER9

2019-06-23 Thread Christoph Hellwig
Because as a matter of policy the driver has no business knowing the actual bridge. Even if we'd agree that a driver workaround would be the right thing it has to be discoverable by an actual interface and not a system type or root port PCI ID.

Re: [PATCH v2 8/8] habanalabs: enable 64-bit DMA mask in POWER9

2019-06-23 Thread Oded Gabbay
On Mon, Jun 17, 2019 at 11:35 AM Oded Gabbay wrote: > > On Mon, Jun 17, 2019 at 11:19 AM Christoph Hellwig wrote: > > > > On Sun, Jun 16, 2019 at 02:24:08PM +0300, Oded Gabbay wrote: > > > So the alternative is that my device won't work on POWER9. > > > > The alternative is that we fix the powerp

Re: [PATCH v2 8/8] habanalabs: enable 64-bit DMA mask in POWER9

2019-06-17 Thread Oded Gabbay
On Mon, Jun 17, 2019 at 11:19 AM Christoph Hellwig wrote: > > On Sun, Jun 16, 2019 at 02:24:08PM +0300, Oded Gabbay wrote: > > So the alternative is that my device won't work on POWER9. > > The alternative is that we fix the powerpc code to do the right > thing, which already is in progress. Great

Re: [PATCH v2 8/8] habanalabs: enable 64-bit DMA mask in POWER9

2019-06-17 Thread Christoph Hellwig
On Sun, Jun 16, 2019 at 02:24:08PM +0300, Oded Gabbay wrote: > So the alternative is that my device won't work on POWER9. The alternative is that we fix the powerpc code to do the right thing, which already is in progress.

Re: [PATCH v2 8/8] habanalabs: enable 64-bit DMA mask in POWER9

2019-06-16 Thread Oded Gabbay
On Sun, Jun 16, 2019 at 12:55 PM Christoph Hellwig wrote: > > On Sat, Jun 15, 2019 at 03:12:36PM +0300, Oded Gabbay wrote: > > So after the dust has settled a bit, do you think it is reasonable to > > add this patch upstream ? > > I'm not Greg, but the answer is a very clear no. drivers have absl

Re: [PATCH v2 8/8] habanalabs: enable 64-bit DMA mask in POWER9

2019-06-16 Thread Christoph Hellwig
On Sat, Jun 15, 2019 at 03:12:36PM +0300, Oded Gabbay wrote: > So after the dust has settled a bit, do you think it is reasonable to > add this patch upstream ? I'm not Greg, but the answer is a very clear no. drivers have abslutely no business adding these hacks.

Re: [PATCH v2 8/8] habanalabs: enable 64-bit DMA mask in POWER9

2019-06-15 Thread Oded Gabbay
On Tue, Jun 11, 2019 at 6:26 PM Greg KH wrote: > > On Tue, Jun 11, 2019 at 08:17:53AM -0700, Christoph Hellwig wrote: > > On Tue, Jun 11, 2019 at 11:58:57AM +0200, Greg KH wrote: > > > That feels like a big hack. ppc doesn't have any "what arch am I > > > running on?" runtime call? Did you ask o

Re: [PATCH v2 8/8] habanalabs: enable 64-bit DMA mask in POWER9

2019-06-12 Thread Oliver O'Halloran
On Wed, Jun 12, 2019 at 4:53 PM Christoph Hellwig wrote: > > On Wed, Jun 12, 2019 at 04:35:22PM +1000, Oliver O'Halloran wrote: > > Setting a 48 bit DMA mask doesn't work today because we only allocate > > IOMMU tables to cover the 0..2GB range of PCI bus addresses. > > I don't think that is true

Re: [PATCH v2 8/8] habanalabs: enable 64-bit DMA mask in POWER9

2019-06-12 Thread Benjamin Herrenschmidt
On Wed, 2019-06-12 at 09:25 +0300, Oded Gabbay wrote: > > > You can't. Your device is broken. Devices that don't support DMAing to > > the full 64-bit deserve to be added to the trash pile. > > > > Hmm... right know they are added to customers data-centers but what do I know > ;) Well, some cu

Re: [PATCH v2 8/8] habanalabs: enable 64-bit DMA mask in POWER9

2019-06-12 Thread Benjamin Herrenschmidt
On Wed, 2019-06-12 at 15:45 +1000, Oliver O'Halloran wrote: > > Also, are you sure about the MSI thing? The IODA3 spec says the only > important bits for a 64bit MSI are bits 61:60 (to hit the window) and > the lower bits that determine what IVE to use. Everything in between > is ignored so ORing

Re: [PATCH v2 8/8] habanalabs: enable 64-bit DMA mask in POWER9

2019-06-11 Thread Christoph Hellwig
On Wed, Jun 12, 2019 at 04:35:22PM +1000, Oliver O'Halloran wrote: > Setting a 48 bit DMA mask doesn't work today because we only allocate > IOMMU tables to cover the 0..2GB range of PCI bus addresses. I don't think that is true upstream, and if it is we need to fix bug in the powerpc code. power

Re: [PATCH v2 8/8] habanalabs: enable 64-bit DMA mask in POWER9

2019-06-11 Thread Oliver O'Halloran
On Wed, Jun 12, 2019 at 3:25 AM Oded Gabbay wrote: > > On Tue, Jun 11, 2019 at 8:03 PM Oded Gabbay wrote: > > > > On Tue, Jun 11, 2019 at 6:26 PM Greg KH wrote: > > > *snip* > > > > Now, when I tried to integrate Goya into a POWER9 machine, I got a > > reject from the call to pci_set_dma_mask(pd

Re: [PATCH v2 8/8] habanalabs: enable 64-bit DMA mask in POWER9

2019-06-11 Thread Oded Gabbay
On Wed, Jun 12, 2019 at 1:53 AM Benjamin Herrenschmidt wrote: > > On Tue, 2019-06-11 at 20:22 +0300, Oded Gabbay wrote: > > > > > So, to summarize: > > > If I call pci_set_dma_mask with 48, then it fails on POWER9. However, > > > in runtime, I don't know if its POWER9 or not, so upon failure I wil

Re: [PATCH v2 8/8] habanalabs: enable 64-bit DMA mask in POWER9

2019-06-11 Thread Oliver O'Halloran
On Wed, Jun 12, 2019 at 8:54 AM Benjamin Herrenschmidt wrote: > > On Tue, 2019-06-11 at 20:22 +0300, Oded Gabbay wrote: > > > > > So, to summarize: > > > If I call pci_set_dma_mask with 48, then it fails on POWER9. However, > > > in runtime, I don't know if its POWER9 or not, so upon failure I wil

Re: [PATCH v2 8/8] habanalabs: enable 64-bit DMA mask in POWER9

2019-06-11 Thread Benjamin Herrenschmidt
On Tue, 2019-06-11 at 20:22 +0300, Oded Gabbay wrote: > > > So, to summarize: > > If I call pci_set_dma_mask with 48, then it fails on POWER9. However, > > in runtime, I don't know if its POWER9 or not, so upon failure I will > > call it again with 32, which makes our device pretty much unusable.

Re: [PATCH v2 8/8] habanalabs: enable 64-bit DMA mask in POWER9

2019-06-11 Thread Oded Gabbay
On Tue, Jun 11, 2019 at 8:03 PM Oded Gabbay wrote: > > On Tue, Jun 11, 2019 at 6:26 PM Greg KH wrote: > > > > On Tue, Jun 11, 2019 at 08:17:53AM -0700, Christoph Hellwig wrote: > > > On Tue, Jun 11, 2019 at 11:58:57AM +0200, Greg KH wrote: > > > > That feels like a big hack. ppc doesn't have any

Re: [PATCH v2 8/8] habanalabs: enable 64-bit DMA mask in POWER9

2019-06-11 Thread Oded Gabbay
On Tue, Jun 11, 2019 at 6:26 PM Greg KH wrote: > > On Tue, Jun 11, 2019 at 08:17:53AM -0700, Christoph Hellwig wrote: > > On Tue, Jun 11, 2019 at 11:58:57AM +0200, Greg KH wrote: > > > That feels like a big hack. ppc doesn't have any "what arch am I > > > running on?" runtime call? Did you ask o

Re: [PATCH v2 8/8] habanalabs: enable 64-bit DMA mask in POWER9

2019-06-11 Thread Greg KH
On Tue, Jun 11, 2019 at 08:17:53AM -0700, Christoph Hellwig wrote: > On Tue, Jun 11, 2019 at 11:58:57AM +0200, Greg KH wrote: > > That feels like a big hack. ppc doesn't have any "what arch am I > > running on?" runtime call? Did you ask on the ppc64 mailing list? I'm > > ok to take this for now

Re: [PATCH v2 8/8] habanalabs: enable 64-bit DMA mask in POWER9

2019-06-11 Thread Christoph Hellwig
On Tue, Jun 11, 2019 at 11:58:57AM +0200, Greg KH wrote: > That feels like a big hack. ppc doesn't have any "what arch am I > running on?" runtime call? Did you ask on the ppc64 mailing list? I'm > ok to take this for now, but odds are you need a better fix for this > sometime... That isn't the

Re: [PATCH v2 8/8] habanalabs: enable 64-bit DMA mask in POWER9

2019-06-11 Thread Oded Gabbay
On Tue, Jun 11, 2019 at 12:59 PM Greg KH wrote: > > On Tue, Jun 11, 2019 at 12:21:44PM +0300, Oded Gabbay wrote: > > +bool hl_pci_parent_is_phb4(struct hl_device *hdev) > > +{ > > + struct pci_dev *parent_port = hdev->pdev->bus->self; > > + > > + if ((parent_port->vendor == PCI_VENDOR_ID_I

Re: [PATCH v2 8/8] habanalabs: enable 64-bit DMA mask in POWER9

2019-06-11 Thread Greg KH
On Tue, Jun 11, 2019 at 12:21:44PM +0300, Oded Gabbay wrote: > +bool hl_pci_parent_is_phb4(struct hl_device *hdev) > +{ > + struct pci_dev *parent_port = hdev->pdev->bus->self; > + > + if ((parent_port->vendor == PCI_VENDOR_ID_IBM) && > + (parent_port->device == PCI_DEVI

[PATCH v2 8/8] habanalabs: enable 64-bit DMA mask in POWER9

2019-06-11 Thread Oded Gabbay
This patch enables support in the driver for 64-bit DMA mask when running in a POWER9 machine. POWER9 supports either 32-bit or 64-bit DMA mask. However, our ASICs support 48-bit DMA mask. To support 64-bit, the driver needs to add a special configuration to the ASIC's PCIe controller. The activa