Re: [PATCH 02/12] intel-ipu3: mmu: implement driver

[Tomasz Figa]

Hi Yong, Tuukka,

+CC IOMMU ML and Joerg. (Technically you should resend this patch
including them.)

On Tue, Jun 6, 2017 at 5:39 AM, Yong Zhi  wrote:
> From: Tuukka Toivonen 
>
> This driver translates Intel IPU3 internal virtual
> address to physical address.

Please see my comments inline.

>
> Signed-off-by: Yong Zhi 

Tuukka needs to sign off this patch as well (above the sender's
sign-off to maintain the patch flow order) to consent his approval for
the submission.

> ---
>  drivers/media/pci/intel/ipu3/Kconfig|  11 +
>  drivers/media/pci/intel/ipu3/Makefile   |   1 +
>  drivers/media/pci/intel/ipu3/ipu3-mmu.c | 423 
> 
>  drivers/media/pci/intel/ipu3/ipu3-mmu.h |  73 ++
>  4 files changed, 508 insertions(+)
>  create mode 100644 drivers/media/pci/intel/ipu3/ipu3-mmu.c
>  create mode 100644 drivers/media/pci/intel/ipu3/ipu3-mmu.h
>
> diff --git a/drivers/media/pci/intel/ipu3/Kconfig 
> b/drivers/media/pci/intel/ipu3/Kconfig
> index 2a895d6..ab2edcb 100644
> --- a/drivers/media/pci/intel/ipu3/Kconfig
> +++ b/drivers/media/pci/intel/ipu3/Kconfig
> @@ -15,3 +15,14 @@ config VIDEO_IPU3_CIO2
> Say Y or M here if you have a Skylake/Kaby Lake SoC with MIPI CSI-2
> connected camera.
> The module will be called ipu3-cio2.
> +
> +config INTEL_IPU3_MMU
> +   tristate "Intel ipu3-mmu driver"
> +   select IOMMU_API
> +   select IOMMU_IOVA
> +   ---help---
> + For IPU3, this option enables its MMU driver to translate its 
> internal
> + virtual address to 39 bits wide physical address for 64GBytes space 
> access.
> +
> + Say Y here if you have Skylake/Kaby Lake SoC with IPU3.
> + Say N if un-sure.

Is the MMU optional? I.e. can you still use the IPU3 without the MMU
driver? If no, then it doesn't make sense to flood the user with
meaningless choice and the driver could simply be selected by other
IPU3 drivers.

And the other way around, is the IPU3 MMU driver useful for anything
else than IPU3? If no (but yes for the above), then it should depend
on some other IPU3 drivers being enabled, as otherwise it would just
confuse the user.

> diff --git a/drivers/media/pci/intel/ipu3/Makefile 
> b/drivers/media/pci/intel/ipu3/Makefile
> index 20186e3..2b669df 100644
> --- a/drivers/media/pci/intel/ipu3/Makefile
> +++ b/drivers/media/pci/intel/ipu3/Makefile
> @@ -1 +1,2 @@
>  obj-$(CONFIG_VIDEO_IPU3_CIO2) += ipu3-cio2.o
> +obj-$(CONFIG_INTEL_IPU3_MMU) += ipu3-mmu.o
> diff --git a/drivers/media/pci/intel/ipu3/ipu3-mmu.c 
> b/drivers/media/pci/intel/ipu3/ipu3-mmu.c
> new file mode 100644
> index 000..a9fb116
> --- /dev/null
> +++ b/drivers/media/pci/intel/ipu3/ipu3-mmu.c
> @@ -0,0 +1,423 @@
> +/*
> + * Copyright (c) 2017 Intel Corporation.
> + *
> + * This program is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU General Public License version
> + * 2 as published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + */
> +
> +#include 
> +#include 
> +#include 
> +
> +#include "ipu3-mmu.h"
> +
> +/**
> + * ipu3_mmu_tlb_invalidate - invalidate translation look-aside buffer
> + * @addr: base address to access REG_TLB_INVALIDATE
> + *
> + * This function must be called when MMU has power
> + */
> +static void ipu3_mmu_tlb_invalidate(void __iomem *addr)

Passing a void* argument is kind of shaky. It would make much more
sense to instead pass a const struct ipu3_mmu *pointer to this
function.

> +{
> +   writel(TLB_INVALIDATE, addr + REG_TLB_INVALIDATE);
> +}
> +
> +static int ipu3_mmu_add_device(struct device *dev)
> +{
> +   struct ipu3_mmu *mmu = dev_get_drvdata(dev);

The drvdata of dev belongs only to its driver. Other drivers shouldn't
expect anything in particular to be there.

> +
> +   /* mapping domain must be prepared */
> +   if (!mmu->domain)
> +   return 0;
> +
> +   return iommu_attach_device(mmu->domain, dev);

I believe this should be handled by the IOMMU core. Maybe your
iommu_group somehow ends up without a default_domain?

> +}
> +
> +/**
> + * ipu3_mmu_alloc_page_table - get page to fill entries with dummy defaults
> + * @d: mapping domain to be worked on
> + * @l1: True for L1 page table, false for L2 page table.
> + *
> + * Index of L1 page table points to L2 tbl
> + *
> + * Return: Pointer to allocated page table
> + * or NULL on failure.
> + */
> +static uint32_t *ipu3_mmu_alloc_page_table(struct ipu3_mmu_domain *d, bool 
> l1)
> +{
> +   uint32_t *pt = (uint32_t *)__get_free_page(GFP_KERNEL);

Style: I believe u32 is preferred in the kernel.

> +   int i;
> +
> +   if (!pt)
> +   return NULL;
> +
> +   for (i = 0; i < IPU3_MMU_L1PT_PTES; i+

Re: [PATCH 02/12] intel-ipu3: mmu: implement driver

[Tomasz Figa]

Hi Yong, Tuukka,

Continuing from yesterday. Please see comments inline.

> On Tue, Jun 6, 2017 at 5:39 AM, Yong Zhi  wrote:
[snip]
>> +   ptr = ipu3_mmu_alloc_page_table(mmu_dom, false);
>> +   if (!ptr)
>> +   goto fail_page_table;
>> +
>> +   /*
>> +* We always map the L1 page table (a single page as well as
>> +* the L2 page tables).
>> +*/
>> +   mmu_dom->dummy_l2_tbl = virt_to_phys(ptr) >> IPU3_MMU_PAGE_SHIFT;
>> +   mmu_dom->pgtbl = ipu3_mmu_alloc_page_table(mmu_dom, true);
>> +   if (!mmu_dom->pgtbl)
>> +   goto fail_page_table;
>> +
>> +   spin_lock_init(&mmu_dom->lock);
>> +   return &mmu_dom->domain;
>> +
>> +fail_page_table:
>> +   free_page((unsigned long)TBL_VIRT_ADDR(mmu_dom->dummy_page));
>> +   free_page((unsigned long)TBL_VIRT_ADDR(mmu_dom->dummy_l2_tbl));
>> +fail_get_page:
>> +   kfree(mmu_dom);
>> +   return NULL;
>> +}
>> +
>> +static void ipu3_mmu_domain_free(struct iommu_domain *dom)
>> +{
>> +   struct ipu3_mmu_domain *mmu_dom =
>> +   container_of(dom, struct ipu3_mmu_domain, domain);
>> +   uint32_t l1_idx;
>> +
>> +   for (l1_idx = 0; l1_idx < IPU3_MMU_L1PT_PTES; l1_idx++)
>> +   if (mmu_dom->pgtbl[l1_idx] != mmu_dom->dummy_l2_tbl)
>> +   free_page((unsigned long)
>> + TBL_VIRT_ADDR(mmu_dom->pgtbl[l1_idx]));
>> +
>> +   free_page((unsigned long)TBL_VIRT_ADDR(mmu_dom->dummy_page));
>> +   free_page((unsigned long)TBL_VIRT_ADDR(mmu_dom->dummy_l2_tbl));

I might be overly paranoid, but reading back kernel virtual pointers
from device accessible memory doesn't seem safe to me. Other drivers
keep kernel pointers of page tables in a dedicated array (it's only 8K
of memory, but much better safety).

>> +   free_page((unsigned long)mmu_dom->pgtbl);
>> +   kfree(mmu_dom);
>> +}
>> +
>> +/**
>> + * ipu3_mmu_map - mapping iova allocated cache to phy addr
>> + * @domain: iommu domain
>> + * @iova: virtual address
>> + * @paddr: physical address
>> + * @size: size to be mapped
>> + * Allocate L2 pgt if needed and establish the mapping between
>> + * iova address space and pfn
>> + *
>> + * Return: 0 for success
>> + * or negative on failure.
>> + */
>> +static int ipu3_mmu_map(struct iommu_domain *domain, unsigned long iova,
>> +   phys_addr_t paddr, size_t size, int prot)
>> +{
>> +   struct ipu3_mmu_domain *mmu_dom =
>> +   container_of(domain, struct ipu3_mmu_domain, domain);

Please add a static inline function for this conversion.

>> +   uint32_t iova_start = round_down(iova, IPU3_MMU_PAGE_SIZE);
>> +   uint32_t iova_end = ALIGN(iova + size, IPU3_MMU_PAGE_SIZE);
>> +   uint32_t l1_idx = iova >> IPU3_MMU_L1PT_SHIFT;
>> +   uint32_t l1_entry = mmu_dom->pgtbl[l1_idx];
>> +   uint32_t *l2_pt;
>> +   uint32_t l2_idx;
>> +   unsigned long flags;
>> +
>> +   /* map to single PAGE */
>> +   WARN_ON(size != IPU3_MMU_PAGE_SIZE);

If we already check this, we could fail as well, i.e.

if (WARN_ON(size != IPU3_MMU_PAGE_SIZE))
return -EINVAL;

>> +
>> +   dev_dbg(mmu_dom->mmu->dev,
>> +   "mapping iova 0x%8.8x--0x%8.8x, size %zu at paddr 0x%pa\n",
>> +   iova_start, iova_end, size, &paddr);
>> +   dev_dbg(mmu_dom->mmu->dev,
>> +   "mapping l2 page table for l1 index %u (iova 0x%8.8lx)\n",
>> +   l1_idx, iova);
>> +
>> +   if (l1_entry == mmu_dom->dummy_l2_tbl) {
>> +   uint32_t *l2_virt = ipu3_mmu_alloc_page_table(mmu_dom, 
>> false);
>> +
>> +   if (!l2_virt)
>> +   return -ENOMEM;
>> +
>> +   l1_entry = virt_to_phys(l2_virt) >> IPU3_MMU_PAGE_SHIFT;
>> +   dev_dbg(mmu_dom->mmu->dev,
>> +   "allocated page for l1_idx %u\n", l1_idx);
>> +
>> +   spin_lock_irqsave(&mmu_dom->lock, flags);
>> +   if (mmu_dom->pgtbl[l1_idx] == mmu_dom->dummy_l2_tbl) {
>> +   mmu_dom->pgtbl[l1_idx] = l1_entry;
>> +   clflush_cache_range(&mmu_dom->pgtbl[l1_idx],
>> +   sizeof(mmu_dom->pgtbl[l1_idx]));
>> +   } else {
>> +   spin_unlock_irqrestore(&mmu_dom->lock, flags);
>> +   free_page((unsigned long)TBL_VIRT_ADDR(l1_entry));
>> +   spin_lock_irqsave(&mmu_dom->lock, flags);
>> +   }
>> +   } else {
>> +   spin_lock_irqsave(&mmu_dom->lock, flags);
>> +   }
>> +
>> +   l2_pt = TBL_VIRT_ADDR(mmu_dom->pgtbl[l1_idx]);
>> +
>> +   dev_dbg(mmu_dom->mmu->dev, "l2_pt at %p\n", l2_pt);
>> +
>> +   paddr = ALIGN(paddr, IPU3_MMU_PAGE_SIZE);
>> +
>> +   l2_idx = (iova_start & IPU3_MMU_L2PT_MASK) >> IPU3_MMU_L2PT_SHIFT;
>> +
>> +   dev_dbg(mmu_dom->mmu->dev,
>> + 

Re: [PATCH 03/12] intel-ipu3: Add DMA API implementation

[Tomasz Figa]

Hi Yong,

+Robin, Joerg, IOMMU ML

Please see my comments inline.

On Tue, Jun 6, 2017 at 5:39 AM, Yong Zhi  wrote:
> IPU3 mmu based DMA mapping driver
>
> Signed-off-by: Yong Zhi 
> ---
>  drivers/media/pci/intel/ipu3/Kconfig   |   6 +
>  drivers/media/pci/intel/ipu3/Makefile  |   1 +
>  drivers/media/pci/intel/ipu3/ipu3-dmamap.c | 408 
> +
>  drivers/media/pci/intel/ipu3/ipu3-dmamap.h |  20 ++
>  4 files changed, 435 insertions(+)
>  create mode 100644 drivers/media/pci/intel/ipu3/ipu3-dmamap.c
>  create mode 100644 drivers/media/pci/intel/ipu3/ipu3-dmamap.h
>
> diff --git a/drivers/media/pci/intel/ipu3/Kconfig 
> b/drivers/media/pci/intel/ipu3/Kconfig
> index ab2edcb..2030be7 100644
> --- a/drivers/media/pci/intel/ipu3/Kconfig
> +++ b/drivers/media/pci/intel/ipu3/Kconfig
> @@ -26,3 +26,9 @@ config INTEL_IPU3_MMU
>
>   Say Y here if you have Skylake/Kaby Lake SoC with IPU3.
>   Say N if un-sure.
> +
> +config INTEL_IPU3_DMAMAP
> +   bool "Intel ipu3 DMA mapping driver"
> +   select IOMMU_IOVA
> +   ---help---
> + This is IPU3 IOMMU domain specific DMA driver.
> diff --git a/drivers/media/pci/intel/ipu3/Makefile 
> b/drivers/media/pci/intel/ipu3/Makefile
> index 2b669df..2c2a035 100644
> --- a/drivers/media/pci/intel/ipu3/Makefile
> +++ b/drivers/media/pci/intel/ipu3/Makefile
> @@ -1,2 +1,3 @@
>  obj-$(CONFIG_VIDEO_IPU3_CIO2) += ipu3-cio2.o
>  obj-$(CONFIG_INTEL_IPU3_MMU) += ipu3-mmu.o
> +obj-$(CONFIG_INTEL_IPU3_DMAMAP) += ipu3-dmamap.o
> diff --git a/drivers/media/pci/intel/ipu3/ipu3-dmamap.c 
> b/drivers/media/pci/intel/ipu3/ipu3-dmamap.c
> new file mode 100644
> index 000..74704d9
> --- /dev/null
> +++ b/drivers/media/pci/intel/ipu3/ipu3-dmamap.c
> @@ -0,0 +1,408 @@
> +/*
> + * Copyright (c) 2017 Intel Corporation.
> + *
> + * This program is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU General Public License version
> + * 2 as published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + */
> +#include 
> +#include 
> +#include 
> +#include 
> +#include "ipu3-mmu.h"
> +
> +/* Begin of things adapted from arch/arm/mm/dma-mapping.c */

ARM's DMA ops are not a good example of today's coding standards.
There are already generic DMA mapping helpers available in
drivers/iommu/dma-iommu.c and drivers/base/dma-*. (Hmm, I remember
writing this already, déjà vu maybe...)

> +static void ipu3_dmamap_clear_buffer(struct page *page, size_t size,
> +unsigned long attrs)
> +{
> +   /*
> +* Ensure that the allocated pages are zeroed, and that any data
> +* lurking in the kernel direct-mapped region is invalidated.
> +*/
> +   if (PageHighMem(page)) {
> +   while (size > 0) {
> +   void *ptr = kmap_atomic(page);
> +
> +   memset(ptr, 0, PAGE_SIZE);
> +   if ((attrs & DMA_ATTR_SKIP_CPU_SYNC) == 0)
> +   clflush_cache_range(ptr, PAGE_SIZE);
> +   kunmap_atomic(ptr);
> +   page++;
> +   size -= PAGE_SIZE;
> +   }
> +   } else {
> +   void *ptr = page_address(page);
> +
> +   memset(ptr, 0, size);
> +   if ((attrs & DMA_ATTR_SKIP_CPU_SYNC) == 0)
> +   clflush_cache_range(ptr, size);
> +   }
> +}
> +
> +/**
> + * ipu3_dmamap_alloc_buffer - allocate buffer based on attributes
> + * @dev: struct device pointer
> + * @size: size of buffer in bytes
> + * @gfp: specify the free page type
> + * @attrs: defined in linux/dma-attrs.h
> + *
> + * This is a helper function for physical page allocation
> + *
> + * Return array representing buffer from alloc_pages() on success
> + * or NULL on failure
> + *
> + * Must be freed with ipu3_dmamap_free_buffer.
> + */
> +static struct page **ipu3_dmamap_alloc_buffer(struct device *dev, size_t 
> size,
> + gfp_t gfp, unsigned long attrs)
> +{
> +   struct page **pages;
> +   int count = size >> PAGE_SHIFT;
> +   int array_size = count * sizeof(struct page *);
> +   int i = 0;
> +
> +   /* Allocate mem for array of page ptrs */
> +   if (array_size <= PAGE_SIZE)
> +   pages = kzalloc(array_size, GFP_KERNEL);
> +   else
> +   pages = vzalloc(array_size);
> +   if (!pages)
> +   return NULL;
> +
> +   gfp |= __GFP_NOWARN;
> +
> +   while (count) {
> +   int j, order = __fls(count);
> +
> +   pages[i] = alloc_pages(gfp, order);
> +   while (!pages[i] && order)
> + 

Re: [PATCH 03/12] intel-ipu3: Add DMA API implementation

[Tomasz Figa]

Hi Alan,

On Thu, Jun 8, 2017 at 2:45 AM, Alan Cox  wrote:
>> > +   struct ipu3_mmu *mmu = to_ipu3_mmu(dev);
>> > +   dma_addr_t daddr = iommu_iova_to_phys(mmu->domain, dma_handle);
>> > +
>> > +   clflush_cache_range(phys_to_virt(daddr), size);
>>
>> You might need to consider another IOMMU on the way here. Generally,
>> given that daddr is your MMU DMA address (not necessarily CPU physical
>> address), you should be able to call
>>
>> dma_sync_single_for_cpu(, daddr, size, dir)
>
> Te system IOMMU (if enabled) may be cache coherent - and on x86 would be,
> so it doesn't think it needs to do anything for cache synchronization
> and the dma_sync won't actually do any work.

I'm not very familiar with x86, but typically I found coherency to be
an attribute of the DMA master (i.e. if it is connected to a coherent
memory port).

Looking at all the IPU3 code, it looks like the whole PCI device is
non-coherent for some reason (e.g. you can see implicit cache flushes
for page tables). So I would have expected that a non-coherent variant
of x86 dma_ops is used for the PCI struct device, which would do cache
maintenance in its dma_sync_* ops.

Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH 02/12] intel-ipu3: mmu: implement driver

[Tomasz Figa]

Hi Sakari,

On Thu, Jun 8, 2017 at 6:59 AM, Sakari Ailus
 wrote:
> Hi Tomasz,
>
> On Tue, Jun 06, 2017 at 07:13:19PM +0900, Tomasz Figa wrote:
>> Hi Yong, Tuukka,
>>
>> +CC IOMMU ML and Joerg. (Technically you should resend this patch
>> including them.)
>
> Thanks!
>
>>
>> On Tue, Jun 6, 2017 at 5:39 AM, Yong Zhi  wrote:
> ...
>> > diff --git a/drivers/media/pci/intel/ipu3/Kconfig 
>> > b/drivers/media/pci/intel/ipu3/Kconfig
>> > index 2a895d6..ab2edcb 100644
>> > --- a/drivers/media/pci/intel/ipu3/Kconfig
>> > +++ b/drivers/media/pci/intel/ipu3/Kconfig
>> > @@ -15,3 +15,14 @@ config VIDEO_IPU3_CIO2
>> > Say Y or M here if you have a Skylake/Kaby Lake SoC with MIPI CSI-2
>> > connected camera.
>> > The module will be called ipu3-cio2.
>> > +
>> > +config INTEL_IPU3_MMU
>> > +   tristate "Intel ipu3-mmu driver"
>> > +   select IOMMU_API
>> > +   select IOMMU_IOVA
>> > +   ---help---
>> > + For IPU3, this option enables its MMU driver to translate its 
>> > internal
>> > + virtual address to 39 bits wide physical address for 64GBytes 
>> > space access.
>> > +
>> > + Say Y here if you have Skylake/Kaby Lake SoC with IPU3.
>> > + Say N if un-sure.
>>
>> Is the MMU optional? I.e. can you still use the IPU3 without the MMU
>> driver? If no, then it doesn't make sense to flood the user with
>> meaningless choice and the driver could simply be selected by other
>> IPU3 drivers.
>
> There are other IPUs that contain the same hardware, so they would
> presumably use the same driver.

My question was slightly different. I was wondering if one can use the
IPU3 without the MMU, i.e. if the implication "if IPU3 then IPU3_MMU"
exists.

>
>>
>> And the other way around, is the IPU3 MMU driver useful for anything
>> else than IPU3? If no (but yes for the above), then it should depend
>> on some other IPU3 drivers being enabled, as otherwise it would just
>> confuse the user.
>
> Very likely not.
>
> For now I think it'd be fine to have the driver separate from the rest of
> the IPU3 but without a separate Kconfig option.

Yeah, I'm not questioning the need for this to be a separate driver.
:) I just want to avoid adding Kconfig option, in case there is no
practical choice (i.e. IPU3 requires IPU3_MMU and IPU3_MMU is useful
without IPU3).

>
>>
>> > diff --git a/drivers/media/pci/intel/ipu3/Makefile 
>> > b/drivers/media/pci/intel/ipu3/Makefile
>> > index 20186e3..2b669df 100644
>> > --- a/drivers/media/pci/intel/ipu3/Makefile
>> > +++ b/drivers/media/pci/intel/ipu3/Makefile
>> > @@ -1 +1,2 @@
>> >  obj-$(CONFIG_VIDEO_IPU3_CIO2) += ipu3-cio2.o
>> > +obj-$(CONFIG_INTEL_IPU3_MMU) += ipu3-mmu.o
>> > diff --git a/drivers/media/pci/intel/ipu3/ipu3-mmu.c 
>> > b/drivers/media/pci/intel/ipu3/ipu3-mmu.c
>> > new file mode 100644
>> > index 000..a9fb116
>> > --- /dev/null
>> > +++ b/drivers/media/pci/intel/ipu3/ipu3-mmu.c
>
> ...
>
>> > +/**
>> > + * ipu3_mmu_alloc_page_table - get page to fill entries with dummy 
>> > defaults
>> > + * @d: mapping domain to be worked on
>> > + * @l1: True for L1 page table, false for L2 page table.
>> > + *
>> > + * Index of L1 page table points to L2 tbl
>> > + *
>> > + * Return: Pointer to allocated page table
>> > + * or NULL on failure.
>> > + */
>> > +static uint32_t *ipu3_mmu_alloc_page_table(struct ipu3_mmu_domain *d, 
>> > bool l1)
>> > +{
>> > +   uint32_t *pt = (uint32_t *)__get_free_page(GFP_KERNEL);
>>
>> Style: I believe u32 is preferred in the kernel.
>
> There are some 3 users of uint32_t alone in the kernel. I'd say it
> should be fine. (I'm not trying saying it'd be more common than u32
> though.)

Okay, checked the CodingStyle again and they are generally okay,
except userspace headers, where __u* ones should be used.

>
>> > +   DMA_BIT_MASK(IPU3_MMU_ADDRESS_BITS);
>> > +   mmu_dom->domain.geometry.force_aperture = true;
>> > +
>> > +   ptr = (void *)__get_free_page(GFP_KERNEL);
>> > +   if (!ptr)
>> > +   goto fail_get_page;
>> > +   mmu_dom->dummy_page = virt_to_phys(ptr) >> IPU3_MMU_PAGE_SHIFT;
>>
>> Is virt_to_phys() correct here? I'm not an expert on x86 systems, but
>>

Re: [PATCH 03/12] intel-ipu3: Add DMA API implementation

[Tomasz Figa]

On Thu, Jun 8, 2017 at 10:22 PM, Robin Murphy  wrote:
> On 07/06/17 10:47, Tomasz Figa wrote:
>> Hi Yong,
>>
>> +Robin, Joerg, IOMMU ML
>>
>> Please see my comments inline.
>>
>> On Tue, Jun 6, 2017 at 5:39 AM, Yong Zhi  wrote:
[snip]
>>> +
>>> +/* End of things adapted from arch/arm/mm/dma-mapping.c */
>>> +static void ipu3_dmamap_sync_single_for_cpu(struct device *dev,
>>> +   dma_addr_t dma_handle, size_t 
>>> size,
>>> +   enum dma_data_direction dir)
>>> +{
>>> +   struct ipu3_mmu *mmu = to_ipu3_mmu(dev);
>>> +   dma_addr_t daddr = iommu_iova_to_phys(mmu->domain, dma_handle);
>>> +
>>> +   clflush_cache_range(phys_to_virt(daddr), size);
>>
>> You might need to consider another IOMMU on the way here. Generally,
>> given that daddr is your MMU DMA address (not necessarily CPU physical
>> address), you should be able to call
>>
>> dma_sync_single_for_cpu(, daddr, size, dir)
>
> I'd hope that this IPU complex is some kind of embedded endpoint thing
> that bypasses the VT-d IOMMU or is always using its own local RAM,
> because it would be pretty much unworkable otherwise.

It uses system RAM and, as far as my understanding goes, by default it
operates without the VT-d IOMMU and that's how it's used right now.
I'm suggesting VT-d IOMMU as a way to further strengthen the security
and error resilience in future (due to the IPU complex being
non-coherent and also running a closed source firmware).

> The whole
> infrastructure isn't really capable of dealing with nested IOMMUs, and
> nested DMA ops would be an equally horrible idea.

Could you elaborate a bit more on this? I think we should be able to
deal with this in a way I suggested before:

a) the PCI device would use the system DMA ops,
b) the PCI device would implement a secondary bus for which it would
provide its own DMA and IOMMU ops.
c) a secondary device would be registered on the secondary bus,
d) all memory for the IPU would be managed on behalf of the secondary device.

In fact, the driver already is designed in a way that all the points
above are true. If I'm not missing something, the only significant
missing point is calling into system DMA ops from IPU DMA ops.

Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH 02/12] intel-ipu3: mmu: implement driver

[Tomasz Figa]

On Fri, Jun 9, 2017 at 1:43 AM, Sakari Ailus  wrote:
> Hi Tomasz,
>
> On Wed, Jun 07, 2017 at 05:35:13PM +0900, Tomasz Figa wrote:
>> Hi Yong, Tuukka,
>>
>> Continuing from yesterday. Please see comments inline.
>>
>> > On Tue, Jun 6, 2017 at 5:39 AM, Yong Zhi  wrote:
>> [snip]
>> >> +   ptr = ipu3_mmu_alloc_page_table(mmu_dom, false);
>> >> +   if (!ptr)
>> >> +   goto fail_page_table;
>> >> +
>> >> +   /*
>> >> +* We always map the L1 page table (a single page as well as
>> >> +* the L2 page tables).
>> >> +*/
>> >> +   mmu_dom->dummy_l2_tbl = virt_to_phys(ptr) >> IPU3_MMU_PAGE_SHIFT;
>> >> +   mmu_dom->pgtbl = ipu3_mmu_alloc_page_table(mmu_dom, true);
>> >> +   if (!mmu_dom->pgtbl)
>> >> +   goto fail_page_table;
>> >> +
>> >> +   spin_lock_init(&mmu_dom->lock);
>> >> +   return &mmu_dom->domain;
>> >> +
>> >> +fail_page_table:
>> >> +   free_page((unsigned long)TBL_VIRT_ADDR(mmu_dom->dummy_page));
>> >> +   free_page((unsigned long)TBL_VIRT_ADDR(mmu_dom->dummy_l2_tbl));
>> >> +fail_get_page:
>> >> +   kfree(mmu_dom);
>> >> +   return NULL;
>> >> +}
>> >> +
>> >> +static void ipu3_mmu_domain_free(struct iommu_domain *dom)
>> >> +{
>> >> +   struct ipu3_mmu_domain *mmu_dom =
>> >> +   container_of(dom, struct ipu3_mmu_domain, domain);
>> >> +   uint32_t l1_idx;
>> >> +
>> >> +   for (l1_idx = 0; l1_idx < IPU3_MMU_L1PT_PTES; l1_idx++)
>> >> +   if (mmu_dom->pgtbl[l1_idx] != mmu_dom->dummy_l2_tbl)
>> >> +   free_page((unsigned long)
>> >> + TBL_VIRT_ADDR(mmu_dom->pgtbl[l1_idx]));
>> >> +
>> >> +   free_page((unsigned long)TBL_VIRT_ADDR(mmu_dom->dummy_page));
>> >> +   free_page((unsigned long)TBL_VIRT_ADDR(mmu_dom->dummy_l2_tbl));
>>
>> I might be overly paranoid, but reading back kernel virtual pointers
>> from device accessible memory doesn't seem safe to me. Other drivers
>> keep kernel pointers of page tables in a dedicated array (it's only 8K
>> of memory, but much better safety).
>
> Do you happen to have an example of that?

Hmm, looks like I misread rockchip-iommu code. Let me quietly back off
from this claim, sorry.

>
> All system memory typically is accessible for devices, I think you wanted to
> say that the device is intended to access that memory. Albeit for reading
> only.

Unless you activate DMAR and make only the memory you want to be
accessible to your devices. I know DMAR is a device too, but there is
a difference between a system level fixed function IOMMU and a PCI
device running a closed source firmware. Still, given Robin's reply,
current DMA and IOMMU frameworks might not be able to handle this
easily, so let's temporarily forget about this setup. We might revisit
it later, with incremental patches, anyway.

>
> ...
>
>> >> +static int ipu3_mmu_bus_remove(struct device *dev)
>> >> +{
>> >> +   struct ipu3_mmu *mmu = dev_get_drvdata(dev);
>> >> +
>> >> +   put_iova_domain(&mmu->iova_domain);
>> >> +   iova_cache_put();
>>
>> Don't we need to set the L1 table address to something invalid and
>> invalidate the TLB, so that the IOMMU doesn't reference the page freed
>> below anymore?
>
> I think the expectation is that if a device gets removed, its memory is
> unmapped by that time. Unmapping that memory will cause explicit TLB flush.

Right, but that will only mark the L2 entries as invalid. The L1 table
will still point to the L2 tables installed earlier and the MMU page
directory register will still point to the L1 table, despite the call
below freeing all the associated memory.

Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH 03/12] intel-ipu3: Add DMA API implementation

[Tomasz Figa]

On Fri, Jun 9, 2017 at 3:07 AM, Robin Murphy  wrote:
> On 08/06/17 15:35, Tomasz Figa wrote:
>> On Thu, Jun 8, 2017 at 10:22 PM, Robin Murphy  wrote:
>>> On 07/06/17 10:47, Tomasz Figa wrote:
>>>> Hi Yong,
>>>>
>>>> +Robin, Joerg, IOMMU ML
>>>>
>>>> Please see my comments inline.
>>>>
>>>> On Tue, Jun 6, 2017 at 5:39 AM, Yong Zhi  wrote:
>> [snip]
>>>>> +
>>>>> +/* End of things adapted from arch/arm/mm/dma-mapping.c */
>>>>> +static void ipu3_dmamap_sync_single_for_cpu(struct device *dev,
>>>>> +   dma_addr_t dma_handle, size_t 
>>>>> size,
>>>>> +   enum dma_data_direction dir)
>>>>> +{
>>>>> +   struct ipu3_mmu *mmu = to_ipu3_mmu(dev);
>>>>> +   dma_addr_t daddr = iommu_iova_to_phys(mmu->domain, dma_handle);
>>>>> +
>>>>> +   clflush_cache_range(phys_to_virt(daddr), size);
>>>>
>>>> You might need to consider another IOMMU on the way here. Generally,
>>>> given that daddr is your MMU DMA address (not necessarily CPU physical
>>>> address), you should be able to call
>>>>
>>>> dma_sync_single_for_cpu(, daddr, size, dir)
>>>
>>> I'd hope that this IPU complex is some kind of embedded endpoint thing
>>> that bypasses the VT-d IOMMU or is always using its own local RAM,
>>> because it would be pretty much unworkable otherwise.
>>
>> It uses system RAM and, as far as my understanding goes, by default it
>> operates without the VT-d IOMMU and that's how it's used right now.
>
> OK, if it *is* behind a DMAR unit then booting with "iommu=force" (or
> whatever the exact incantation for intel-iommu is) should be fun...
>
>> I'm suggesting VT-d IOMMU as a way to further strengthen the security
>> and error resilience in future (due to the IPU complex being
>> non-coherent and also running a closed source firmware).
>
> TBH, doing DMA remapping through *two* IOMMUS will add horrible hardware
> overhead,

Not necessarily, if done right and with right hardware (I lack the
details about Intel hardware unfortunately). One can for example
notice the fact that the IOVA ranges from the parent IOMMU are going
to be contiguous for the child IOMMU, so one could use huge pages in
the child IOMMU and essentially make a selective 1:1 mapping.

> increase the scope for kernel-side bugs, and not much more. If
> we don't trust this IOMMU to behave, why are we trying to drive it in
> the first place? If we do, then a second IOMMU behind it won't protect
> anything that the first one doesn't already.

That's a valid point, right. But on the other hand, I lack the
hardware details on whether we can just disable the internal IOMMU and
use DMAR alone instead.

>
>>> The whole
>>> infrastructure isn't really capable of dealing with nested IOMMUs, and
>>> nested DMA ops would be an equally horrible idea.
>>
>> Could you elaborate a bit more on this? I think we should be able to
>> deal with this in a way I suggested before:
>>
>> a) the PCI device would use the system DMA ops,
>> b) the PCI device would implement a secondary bus for which it would
>> provide its own DMA and IOMMU ops.
>> c) a secondary device would be registered on the secondary bus,
>> d) all memory for the IPU would be managed on behalf of the secondary device.
>>
>> In fact, the driver already is designed in a way that all the points
>> above are true. If I'm not missing something, the only significant
>> missing point is calling into system DMA ops from IPU DMA ops.
>
> I don't believe x86 has any non-coherent DMA ops, therefore the IPU DMA
> ops would still probably have to do all their own cache maintenance.

I'd argue that it means that we need to add non-coherent DMA ops on
x86, as we have on other archs, which can have both coherent and
non-coherent devices in the same system.

> Allocation/mapping, though, would have to be done with the parent DMA
> ops first (in case DMA address != physical address), *then* mapped at
> the IPU MMU, which is the real killer - if the PCI DMA ops are from
> intel-iommu, then there's little need for the IPU MMU mapping to be
> anything other than 1:1, so you may as well not bother.

Okay, I think I can agree with you on this. It indeed makes little
sense to use both MMUs at the same time, if there is a way to disable
one of them.

Let's just keep this unaware of DMAR at this point of t

Re: [PATCH 02/12] intel-ipu3: mmu: implement driver

[Tomasz Figa]

On Fri, Jun 9, 2017 at 8:16 PM, Sakari Ailus  wrote:
> Hi Tomasz,
>
> On Fri, Jun 09, 2017 at 02:59:10PM +0900, Tomasz Figa wrote:
>> On Fri, Jun 9, 2017 at 1:43 AM, Sakari Ailus  wrote:
>> >> >> +static void ipu3_mmu_domain_free(struct iommu_domain *dom)
>> >> >> +{
>> >> >> +   struct ipu3_mmu_domain *mmu_dom =
>> >> >> +   container_of(dom, struct ipu3_mmu_domain, domain);
>> >> >> +   uint32_t l1_idx;
>> >> >> +
>> >> >> +   for (l1_idx = 0; l1_idx < IPU3_MMU_L1PT_PTES; l1_idx++)
>> >> >> +   if (mmu_dom->pgtbl[l1_idx] != mmu_dom->dummy_l2_tbl)
>> >> >> +   free_page((unsigned long)
>> >> >> + 
>> >> >> TBL_VIRT_ADDR(mmu_dom->pgtbl[l1_idx]));
>> >> >> +
>> >> >> +   free_page((unsigned long)TBL_VIRT_ADDR(mmu_dom->dummy_page));
>> >> >> +   free_page((unsigned long)TBL_VIRT_ADDR(mmu_dom->dummy_l2_tbl));
>> >>
>> >> I might be overly paranoid, but reading back kernel virtual pointers
>> >> from device accessible memory doesn't seem safe to me. Other drivers
>> >> keep kernel pointers of page tables in a dedicated array (it's only 8K
>> >> of memory, but much better safety).
>> >
>> > Do you happen to have an example of that?
>>
>> Hmm, looks like I misread rockchip-iommu code. Let me quietly back off
>> from this claim, sorry.
>>
>> >
>> > All system memory typically is accessible for devices, I think you wanted 
>> > to
>> > say that the device is intended to access that memory. Albeit for reading
>> > only.
>>
>> Unless you activate DMAR and make only the memory you want to be
>> accessible to your devices. I know DMAR is a device too, but there is
>> a difference between a system level fixed function IOMMU and a PCI
>> device running a closed source firmware. Still, given Robin's reply,
>> current DMA and IOMMU frameworks might not be able to handle this
>> easily, so let's temporarily forget about this setup. We might revisit
>> it later, with incremental patches, anyway.
>
> I don't think it's needed because
>
> 1) The firmware running on the device only has access to memory mapped to it
> by the MMU and
>
> 2) the MMU L1 page table base address is not writable to the firmware.

If that's the case, then I guess it's fine indeed.

>> >> >> +static int ipu3_mmu_bus_remove(struct device *dev)
>> >> >> +{
>> >> >> +   struct ipu3_mmu *mmu = dev_get_drvdata(dev);
>> >> >> +
>> >> >> +   put_iova_domain(&mmu->iova_domain);
>> >> >> +   iova_cache_put();
>> >>
>> >> Don't we need to set the L1 table address to something invalid and
>> >> invalidate the TLB, so that the IOMMU doesn't reference the page freed
>> >> below anymore?
>> >
>> > I think the expectation is that if a device gets removed, its memory is
>> > unmapped by that time. Unmapping that memory will cause explicit TLB flush.
>>
>> Right, but that will only mark the L2 entries as invalid. The L1 table
>> will still point to the L2 tables installed earlier and the MMU page
>> directory register will still point to the L1 table, despite the call
>> below freeing all the associated memory.
>
> Ah. I think I misunderstood what I read the previous time.
>
> An alternative would be not to release the L1 page table and the dummy pages
> --- the MMU does not have a concept of an invalid page. Now that the domain
> is gone, you can't really use the MMU anyway, can you?

Yes, given the hardware design, that would make sense indeed.

Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH 02/12] intel-ipu3: mmu: implement driver

[Tomasz Figa]

On Fri, Jun 9, 2017 at 5:26 PM, Tuukka Toivonen
 wrote:
> Hi Tomasz,
>
> Couple of small comments below.
>
> On Wednesday, June 07, 2017 17:35:13 Tomasz Figa wrote:
>> >> +static void ipu3_mmu_domain_free(struct iommu_domain *dom)
>> >> +{
>> >> +   struct ipu3_mmu_domain *mmu_dom =
>> >> +   container_of(dom, struct ipu3_mmu_domain, domain);
>> >> +   uint32_t l1_idx;
>> >> +
>> >> +   for (l1_idx = 0; l1_idx < IPU3_MMU_L1PT_PTES; l1_idx++)
>> >> +   if (mmu_dom->pgtbl[l1_idx] != mmu_dom-
>>dummy_l2_tbl)
>> >> +   free_page((unsigned long)
>> >> + TBL_VIRT_ADDR(mmu_dom-
>>pgtbl[l1_idx]));
>> >> +
>> >> +   free_page((unsigned long)TBL_VIRT_ADDR(mmu_dom-
>>dummy_page));
>> >> +   free_page((unsigned long)TBL_VIRT_ADDR(mmu_dom-
>>dummy_l2_tbl));
>>
>> I might be overly paranoid, but reading back kernel virtual pointers
>> from device accessible memory doesn't seem safe to me. Other drivers
>> keep kernel pointers of page tables in a dedicated array (it's only 8K
>> of memory, but much better safety).
>
> They are accessible only to the IPU3 IOMMU, which can access whole
> system memory anyway and always does a read-only access to the MMU
> tables. So, I wouldn't worry too much, although extra copy for safety
> wouldn't necessarily harm too much.

Fair enough. Thanks for explanation.

>
> <...>
>
>> >> +   ipu3_mmu_tlb_invalidate(mmu_dom->mmu->base);
>> >> +
>> >> +   return unmapped << IPU3_MMU_PAGE_SHIFT;
>> >> +}
>> >> +
>> >> +static phys_addr_t ipu3_mmu_iova_to_phys(struct iommu_domain
> *domain,
>> >> +dma_addr_t iova)
>> >> +{
>> >> +   struct ipu3_mmu_domain *d =
>> >> +   container_of(domain, struct ipu3_mmu_domain,
> domain);
>> >> +   uint32_t *l2_pt = TBL_VIRT_ADDR(d->pgtbl[iova >>
> IPU3_MMU_L1PT_SHIFT]);
>> >> +
>> >> +   return (phys_addr_t)l2_pt[(iova & IPU3_MMU_L2PT_MASK)
>> >> +   >> IPU3_MMU_L2PT_SHIFT] <<
> IPU3_MMU_PAGE_SHIFT;
>>
>> Could we avoid this TBL_VIRT_ADDR() here too? The memory cost to store
>> the page table CPU pointers is really small, but safety seems much
>> better. Moreover, it should make it possible to use the VT-d IOMMU to
>> further secure the system.
>
> IPU3 doesn't support VT-d and can't be enabled while VT-d is on.

Got it. Thanks.

Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[PATCH 1/2] iommu/dma: Respect __GFP_DMA and __GFP_DMA32 in incoming GFP flags

[Tomasz Figa]

Current implementation of __iommu_dma_alloc_pages() keeps adding
__GFP_HIGHMEM to GFP flags regardless of whether other zone flags are
already included in the incoming flags. If __GFP_DMA or __GFP_DMA32 is
set at the same time as __GFP_HIGHMEM, the allocation fails due to
invalid zone flag combination.

Fix this by checking for __GFP_DMA and __GFP_DMA32 in incoming GFP flags
and adding __GFP_HIGHMEM only if they are not present.

Signed-off-by: Tomasz Figa 
---
 drivers/iommu/dma-iommu.c | 10 --
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
index 9d1cebe7f6cb..29965a092a69 100644
--- a/drivers/iommu/dma-iommu.c
+++ b/drivers/iommu/dma-iommu.c
@@ -445,8 +445,14 @@ static struct page **__iommu_dma_alloc_pages(unsigned int 
count,
if (!pages)
return NULL;
 
-   /* IOMMU can map any pages, so himem can also be used here */
-   gfp |= __GFP_NOWARN | __GFP_HIGHMEM;
+   /*
+* IOMMU can map any pages, so himem can also be used here,
+* unless another DMA zone is explicitly requested.
+*/
+   if (!(gfp & (__GFP_DMA | __GFP_DMA32)))
+   gfp |= __GFP_HIGHMEM;
+
+   gfp |= __GFP_NOWARN;
 
while (count) {
struct page *page = NULL;
-- 
2.13.1.611.g7e3b11ae1-goog

___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[PATCH 2/2] iommu/dma: use __GFP_NOWARN only for high-order allocations

[Tomasz Figa]

Memory allocation routines are expected to report allocation errors to
kernel log. However, current implementation of __iommu_dma_alloc_pages()
adds __GFP_NOWARN for all calls to alloc_pages(), which completely
disables any logging.

Fix it by adding __GFP_NOWARN only to high order allocation attempts,
which are not critical.

Signed-off-by: Tomasz Figa 
---
 drivers/iommu/dma-iommu.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
index 29965a092a69..8507987eed90 100644
--- a/drivers/iommu/dma-iommu.c
+++ b/drivers/iommu/dma-iommu.c
@@ -433,6 +433,7 @@ static struct page **__iommu_dma_alloc_pages(unsigned int 
count,
 {
struct page **pages;
unsigned int i = 0, array_size = count * sizeof(*pages);
+   const gfp_t high_order_gfp = __GFP_NOWARN | __GFP_NORETRY;
 
order_mask &= (2U << MAX_ORDER) - 1;
if (!order_mask)
@@ -452,8 +453,6 @@ static struct page **__iommu_dma_alloc_pages(unsigned int 
count,
if (!(gfp & (__GFP_DMA | __GFP_DMA32)))
gfp |= __GFP_HIGHMEM;
 
-   gfp |= __GFP_NOWARN;
-
while (count) {
struct page *page = NULL;
unsigned int order_size;
@@ -469,7 +468,7 @@ static struct page **__iommu_dma_alloc_pages(unsigned int 
count,
 
order_size = 1U << order;
page = alloc_pages((order_mask - order_size) ?
-  gfp | __GFP_NORETRY : gfp, order);
+  gfp | high_order_gfp : gfp, order);
if (!page)
continue;
if (!order)
-- 
2.13.1.611.g7e3b11ae1-goog

___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH 1/2] iommu/dma: Respect __GFP_DMA and __GFP_DMA32 in incoming GFP flags

[Tomasz Figa]

On Tue, Jun 27, 2017 at 8:01 PM, Robin Murphy  wrote:
> On 27/06/17 08:28, Tomasz Figa wrote:
>> Current implementation of __iommu_dma_alloc_pages() keeps adding
>> __GFP_HIGHMEM to GFP flags regardless of whether other zone flags are
>> already included in the incoming flags. If __GFP_DMA or __GFP_DMA32 is
>> set at the same time as __GFP_HIGHMEM, the allocation fails due to
>> invalid zone flag combination.
>>
>> Fix this by checking for __GFP_DMA and __GFP_DMA32 in incoming GFP flags
>> and adding __GFP_HIGHMEM only if they are not present.
>
> Wouldn't it make more sense to strip off the ZONE_DMA* related flags,
> since the whole point here is that we don't care where the pages come from?

I guess for my use case it wouldn't break things, as I strip them in
my DMA mapping implementation right now (+/- one minor detail below).

However I recall existing IOMMUs that could only use pages from within
the 32-bit address space (e.g. Tegra X1). Also the IOMMU I'm working
on is a part of a PCI device and it might actually prefer 32-bit
addressable memory as well (to avoid DAC addressing; I still need to
evaluate this). With this said, maybe it could actually make sense to
leave the choice to the DMA mapping implementation?

Best regards,
Tomasz

>
> Robin.
>
>> Signed-off-by: Tomasz Figa 
>> ---
>>  drivers/iommu/dma-iommu.c | 10 --
>>  1 file changed, 8 insertions(+), 2 deletions(-)
>>
>> diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
>> index 9d1cebe7f6cb..29965a092a69 100644
>> --- a/drivers/iommu/dma-iommu.c
>> +++ b/drivers/iommu/dma-iommu.c
>> @@ -445,8 +445,14 @@ static struct page **__iommu_dma_alloc_pages(unsigned 
>> int count,
>>   if (!pages)
>>   return NULL;
>>
>> - /* IOMMU can map any pages, so himem can also be used here */
>> - gfp |= __GFP_NOWARN | __GFP_HIGHMEM;
>> + /*
>> +  * IOMMU can map any pages, so himem can also be used here,
>> +  * unless another DMA zone is explicitly requested.
>> +  */
>> + if (!(gfp & (__GFP_DMA | __GFP_DMA32)))
>> + gfp |= __GFP_HIGHMEM;
>> +
>> + gfp |= __GFP_NOWARN;
>>
>>   while (count) {
>>   struct page *page = NULL;
>>
>
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH 1/2] iommu/dma: Respect __GFP_DMA and __GFP_DMA32 in incoming GFP flags

[Tomasz Figa]

On Tue, Jun 27, 2017 at 9:26 PM, Robin Murphy  wrote:
> On 27/06/17 12:17, Tomasz Figa wrote:
>> On Tue, Jun 27, 2017 at 8:01 PM, Robin Murphy  wrote:
>>> On 27/06/17 08:28, Tomasz Figa wrote:
>>>> Current implementation of __iommu_dma_alloc_pages() keeps adding
>>>> __GFP_HIGHMEM to GFP flags regardless of whether other zone flags are
>>>> already included in the incoming flags. If __GFP_DMA or __GFP_DMA32 is
>>>> set at the same time as __GFP_HIGHMEM, the allocation fails due to
>>>> invalid zone flag combination.
>>>>
>>>> Fix this by checking for __GFP_DMA and __GFP_DMA32 in incoming GFP flags
>>>> and adding __GFP_HIGHMEM only if they are not present.
>>>
>>> Wouldn't it make more sense to strip off the ZONE_DMA* related flags,
>>> since the whole point here is that we don't care where the pages come from?
>>
>> I guess for my use case it wouldn't break things, as I strip them in
>> my DMA mapping implementation right now (+/- one minor detail below).
>>
>> However I recall existing IOMMUs that could only use pages from within
>> the 32-bit address space (e.g. Tegra X1).
>
> In general, iommu-dma can't really support IOMMUs which can't reach the
> entirety of kernel memory - there's no easy way to determine what such a
> limit is if it exists, nor necessarily enforce it, and either way the
> streaming API callbacks are pretty much dead in the water.

Right. Especially with various user pointer sources it doesn't sound
very realistic to enforce that all memory comes from __GFP_DMA(32)
zone.

On the other hand, support for user pointer is optional in subsystems
such as V4L2 and drivers for such disabled hardware could opt out.
Then at least dma_alloc can be made working, giving some level of
usability, IMHO higher than no IOMMU and contiguous memory allocated
from CMA.

>
>> Also the IOMMU I'm working
>> on is a part of a PCI device and it might actually prefer 32-bit
>> addressable memory as well (to avoid DAC addressing; I still need to
>> evaluate this). With this said, maybe it could actually make sense to
>> leave the choice to the DMA mapping implementation?
>
> I think you're right - we're just not in a position to make any decision
> at this level, so we probably do have to do this for robustness. I would
> like to fix the longstanding dodgy comment, though, to clarify that
> "IOMMU can map any pages" is only an assumption, and particularly one
> which is invalidated by the presence of GFP_DMA flags.

Just to make sure, should I resent with the commit updated? If so,
what would be your preference on the wording?

Best regards,
Tomasz

>
> Robin.
>
>>
>> Best regards,
>> Tomasz
>>
>>>
>>> Robin.
>>>
>>>> Signed-off-by: Tomasz Figa 
>>>> ---
>>>>  drivers/iommu/dma-iommu.c | 10 --
>>>>  1 file changed, 8 insertions(+), 2 deletions(-)
>>>>
>>>> diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
>>>> index 9d1cebe7f6cb..29965a092a69 100644
>>>> --- a/drivers/iommu/dma-iommu.c
>>>> +++ b/drivers/iommu/dma-iommu.c
>>>> @@ -445,8 +445,14 @@ static struct page **__iommu_dma_alloc_pages(unsigned 
>>>> int count,
>>>>   if (!pages)
>>>>   return NULL;
>>>>
>>>> - /* IOMMU can map any pages, so himem can also be used here */
>>>> - gfp |= __GFP_NOWARN | __GFP_HIGHMEM;
>>>> + /*
>>>> +  * IOMMU can map any pages, so himem can also be used here,
>>>> +  * unless another DMA zone is explicitly requested.
>>>> +  */
>>>> + if (!(gfp & (__GFP_DMA | __GFP_DMA32)))
>>>> + gfp |= __GFP_HIGHMEM;
>>>> +
>>>> + gfp |= __GFP_NOWARN;
>>>>
>>>>   while (count) {
>>>>   struct page *page = NULL;
>>>>
>>>
>
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[PATCH v2 1/2] iommu/dma: Respect __GFP_DMA and __GFP_DMA32 in incoming GFP flags

[Tomasz Figa]

Current implementation of __iommu_dma_alloc_pages() keeps adding
__GFP_HIGHMEM to GFP flags regardless of whether other zone flags are
already included in the incoming flags. If __GFP_DMA or __GFP_DMA32 is
set at the same time as __GFP_HIGHMEM, the allocation fails due to
invalid zone flag combination.

Fix this by checking for __GFP_DMA and __GFP_DMA32 in incoming GFP flags
and adding __GFP_HIGHMEM only if they are not present.

Signed-off-by: Tomasz Figa 
---
 drivers/iommu/dma-iommu.c | 10 --
 1 file changed, 8 insertions(+), 2 deletions(-)

Changes from v1:
 - Update the comment as per Robin's suggestion.

diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
index 9d1cebe7f6cb..bf23989b5158 100644
--- a/drivers/iommu/dma-iommu.c
+++ b/drivers/iommu/dma-iommu.c
@@ -445,8 +445,14 @@ static struct page **__iommu_dma_alloc_pages(unsigned int 
count,
if (!pages)
return NULL;
 
-   /* IOMMU can map any pages, so himem can also be used here */
-   gfp |= __GFP_NOWARN | __GFP_HIGHMEM;
+   /*
+* Unless we have some addressing limitation implied by GFP_DMA flags,
+* assume the IOMMU can map all of RAM and we can allocate anywhere.
+*/
+   if (!(gfp & (__GFP_DMA | __GFP_DMA32)))
+   gfp |= __GFP_HIGHMEM;
+
+   gfp |= __GFP_NOWARN;
 
while (count) {
struct page *page = NULL;
-- 
2.13.2.725.g09c95d1e9-goog

___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[PATCH v2 2/2] iommu/dma: Use __GFP_NOWARN only for high-order allocations

[Tomasz Figa]

Memory allocation routines are expected to report allocation errors to
kernel log. However, current implementation of __iommu_dma_alloc_pages()
adds __GFP_NOWARN for all calls to alloc_pages(), which completely
disables any logging.

Fix it by adding __GFP_NOWARN only to high order allocation attempts,
which are not critical.

Signed-off-by: Tomasz Figa 
Reviewed-by: Robin Murphy 
---
 drivers/iommu/dma-iommu.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

Changes from v1:
 - Fix typo in subject.
 - Add Robin's Reviewed-by.

diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
index bf23989b5158..6ed8c8f941d8 100644
--- a/drivers/iommu/dma-iommu.c
+++ b/drivers/iommu/dma-iommu.c
@@ -433,6 +433,7 @@ static struct page **__iommu_dma_alloc_pages(unsigned int 
count,
 {
struct page **pages;
unsigned int i = 0, array_size = count * sizeof(*pages);
+   const gfp_t high_order_gfp = __GFP_NOWARN | __GFP_NORETRY;
 
order_mask &= (2U << MAX_ORDER) - 1;
if (!order_mask)
@@ -452,8 +453,6 @@ static struct page **__iommu_dma_alloc_pages(unsigned int 
count,
if (!(gfp & (__GFP_DMA | __GFP_DMA32)))
gfp |= __GFP_HIGHMEM;
 
-   gfp |= __GFP_NOWARN;
-
while (count) {
struct page *page = NULL;
unsigned int order_size;
@@ -469,7 +468,7 @@ static struct page **__iommu_dma_alloc_pages(unsigned int 
count,
 
order_size = 1U << order;
page = alloc_pages((order_mask - order_size) ?
-  gfp | __GFP_NORETRY : gfp, order);
+  gfp | high_order_gfp : gfp, order);
if (!page)
continue;
if (!order)
-- 
2.13.2.725.g09c95d1e9-goog

___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[RFC PATCH 1/5] base: dma-mapping: Export commonly used symbols

[Tomasz Figa]

There is nothing wrong in having a loadable module implementing DMA API,
for example to be used for sub-devices registered by the module.
However, most of the functions from dma-mapping do not have their
symbols exported, making it impossible to use them from loadable modules.

Export the remaining non-static functions in the file, so that loadable
modules can benefit from them. Use EXPORT_SYMBOL() for consistency with
other exports in the file.

Signed-off-by: Tomasz Figa 
---
 drivers/base/dma-mapping.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
index 5096755d185e..1fda8df3d849 100644
--- a/drivers/base/dma-mapping.c
+++ b/drivers/base/dma-mapping.c
@@ -287,6 +287,7 @@ void *dma_common_pages_remap(struct page **pages, size_t 
size,
 
return area->addr;
 }
+EXPORT_SYMBOL(dma_common_pages_remap);
 
 /*
  * remaps an allocated contiguous region into another vm_area.
@@ -316,6 +317,7 @@ void *dma_common_contiguous_remap(struct page *page, size_t 
size,
return NULL;
return area->addr;
 }
+EXPORT_SYMBOL(dma_common_contiguous_remap);
 
 /*
  * unmaps a range previously mapped by dma_common_*_remap
@@ -332,6 +334,7 @@ void dma_common_free_remap(void *cpu_addr, size_t size, 
unsigned long vm_flags)
unmap_kernel_range((unsigned long)cpu_addr, PAGE_ALIGN(size));
vunmap(cpu_addr);
 }
+EXPORT_SYMBOL(dma_common_free_remap);
 #endif
 
 /*
-- 
2.13.2.725.g09c95d1e9-goog

___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[RFC PATCH 3/5] iommu: Export non-static functions to use in modules

[Tomasz Figa]

There are some non-static functions potentially useful in IOMMU drivers
that do not have their symbols exported. Export them too, so that
loadable modules can benefit from them. Use EXPORT_SYMBOL_GPL() for
consistency with other exports in the file.

Signed-off-by: Tomasz Figa 
---
 drivers/iommu/iommu.c | 12 
 1 file changed, 12 insertions(+)

diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index 3f6ea160afed..41dd6b435ae3 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -96,6 +96,7 @@ int iommu_device_register(struct iommu_device *iommu)
 
return 0;
 }
+EXPORT_SYMBOL_GPL(iommu_device_register);
 
 void iommu_device_unregister(struct iommu_device *iommu)
 {
@@ -103,6 +104,7 @@ void iommu_device_unregister(struct iommu_device *iommu)
list_del(&iommu->list);
spin_unlock(&iommu_device_lock);
 }
+EXPORT_SYMBOL_GPL(iommu_device_unregister);
 
 static struct iommu_domain *__iommu_domain_alloc(struct bus_type *bus,
 unsigned type);
@@ -744,6 +746,7 @@ struct iommu_group *iommu_group_ref_get(struct iommu_group 
*group)
kobject_get(group->devices_kobj);
return group;
 }
+EXPORT_SYMBOL_GPL(iommu_group_ref_get);
 
 /**
  * iommu_group_put - Decrement group reference
@@ -917,6 +920,7 @@ struct iommu_group *generic_device_group(struct device *dev)
 {
return iommu_group_alloc();
 }
+EXPORT_SYMBOL_GPL(generic_device_group);
 
 /*
  * Use standard PCI bus topology, isolation features, and DMA alias quirks
@@ -984,6 +988,7 @@ struct iommu_group *pci_device_group(struct device *dev)
/* No shared group found, allocate new */
return iommu_group_alloc();
 }
+EXPORT_SYMBOL_GPL(pci_device_group);
 
 /**
  * iommu_group_get_for_dev - Find or create the IOMMU group for a device
@@ -1044,11 +1049,13 @@ struct iommu_group *iommu_group_get_for_dev(struct 
device *dev)
 
return group;
 }
+EXPORT_SYMBOL_GPL(iommu_group_get_for_dev);
 
 struct iommu_domain *iommu_group_default_domain(struct iommu_group *group)
 {
return group->default_domain;
 }
+EXPORT_SYMBOL_GPL(iommu_group_default_domain);
 
 static int add_iommu_group(struct device *dev, void *data)
 {
@@ -1795,6 +1802,7 @@ void iommu_get_resv_regions(struct device *dev, struct 
list_head *list)
if (ops && ops->get_resv_regions)
ops->get_resv_regions(dev, list);
 }
+EXPORT_SYMBOL_GPL(iommu_get_resv_regions);
 
 void iommu_put_resv_regions(struct device *dev, struct list_head *list)
 {
@@ -1803,6 +1811,7 @@ void iommu_put_resv_regions(struct device *dev, struct 
list_head *list)
if (ops && ops->put_resv_regions)
ops->put_resv_regions(dev, list);
 }
+EXPORT_SYMBOL_GPL(iommu_put_resv_regions);
 
 struct iommu_resv_region *iommu_alloc_resv_region(phys_addr_t start,
  size_t length, int prot,
@@ -1821,6 +1830,7 @@ struct iommu_resv_region 
*iommu_alloc_resv_region(phys_addr_t start,
region->type = type;
return region;
 }
+EXPORT_SYMBOL_GPL(iommu_alloc_resv_region);
 
 /* Request that a device is direct mapped by the IOMMU */
 int iommu_request_dm_for_dev(struct device *dev)
@@ -1874,6 +1884,7 @@ int iommu_request_dm_for_dev(struct device *dev)
 
return ret;
 }
+EXPORT_SYMBOL_GPL(iommu_request_dm_for_dev);
 
 const struct iommu_ops *iommu_ops_from_fwnode(struct fwnode_handle *fwnode)
 {
@@ -1889,6 +1900,7 @@ const struct iommu_ops *iommu_ops_from_fwnode(struct 
fwnode_handle *fwnode)
spin_unlock(&iommu_device_lock);
return ops;
 }
+EXPORT_SYMBOL_GPL(iommu_ops_from_fwnode);
 
 int iommu_fwspec_init(struct device *dev, struct fwnode_handle *iommu_fwnode,
  const struct iommu_ops *ops)
-- 
2.13.2.725.g09c95d1e9-goog

___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[RFC PATCH 4/5] iommu/dma: Export non-static functions to use in modules

[Tomasz Figa]

There is nothing wrong in having a loadable module implementing DMA API,
for example to be used for sub-devices registered by the module. However,
most of the functions from dma-iommu do not have their symbols exported,
making it impossible to use them from loadable modules.

Export all the non-static functions in the file, so that loadable modules
can benefit from them. Use EXPORT_SYMBOL() for consistency with other
exports in the file.

Signed-off-by: Tomasz Figa 
---
 drivers/iommu/dma-iommu.c | 13 +
 1 file changed, 13 insertions(+)

diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
index 9d1cebe7f6cb..7cdeaf930106 100644
--- a/drivers/iommu/dma-iommu.c
+++ b/drivers/iommu/dma-iommu.c
@@ -80,6 +80,7 @@ int iommu_dma_init(void)
 {
return iova_cache_get();
 }
+EXPORT_SYMBOL(iommu_dma_init);
 
 /**
  * iommu_get_dma_cookie - Acquire DMA-API resources for a domain
@@ -357,6 +358,7 @@ int dma_info_to_prot(enum dma_data_direction dir, bool 
coherent,
return 0;
}
 }
+EXPORT_SYMBOL(dma_info_to_prot);
 
 static dma_addr_t iommu_dma_alloc_iova(struct iommu_domain *domain,
size_t size, dma_addr_t dma_limit, struct device *dev)
@@ -504,6 +506,7 @@ void iommu_dma_free(struct device *dev, struct page 
**pages, size_t size,
__iommu_dma_free_pages(pages, PAGE_ALIGN(size) >> PAGE_SHIFT);
*handle = IOMMU_MAPPING_ERROR;
 }
+EXPORT_SYMBOL(iommu_dma_free);
 
 /**
  * iommu_dma_alloc - Allocate and map a buffer contiguous in IOVA space
@@ -588,6 +591,7 @@ struct page **iommu_dma_alloc(struct device *dev, size_t 
size, gfp_t gfp,
__iommu_dma_free_pages(pages, count);
return NULL;
 }
+EXPORT_SYMBOL(iommu_dma_alloc);
 
 /**
  * iommu_dma_mmap - Map a buffer into provided user VMA
@@ -613,6 +617,7 @@ int iommu_dma_mmap(struct page **pages, size_t size, struct 
vm_area_struct *vma)
}
return ret;
 }
+EXPORT_SYMBOL(iommu_dma_mmap);
 
 static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys,
size_t size, int prot)
@@ -643,12 +648,14 @@ dma_addr_t iommu_dma_map_page(struct device *dev, struct 
page *page,
 {
return __iommu_dma_map(dev, page_to_phys(page) + offset, size, prot);
 }
+EXPORT_SYMBOL(iommu_dma_map_page);
 
 void iommu_dma_unmap_page(struct device *dev, dma_addr_t handle, size_t size,
enum dma_data_direction dir, unsigned long attrs)
 {
__iommu_dma_unmap(iommu_get_domain_for_dev(dev), handle, size);
 }
+EXPORT_SYMBOL(iommu_dma_unmap_page);
 
 /*
  * Prepare a successfully-mapped scatterlist to give back to the caller.
@@ -802,6 +809,7 @@ int iommu_dma_map_sg(struct device *dev, struct scatterlist 
*sg,
__invalidate_sg(sg, nents);
return 0;
 }
+EXPORT_SYMBOL(iommu_dma_map_sg);
 
 void iommu_dma_unmap_sg(struct device *dev, struct scatterlist *sg, int nents,
enum dma_data_direction dir, unsigned long attrs)
@@ -822,6 +830,7 @@ void iommu_dma_unmap_sg(struct device *dev, struct 
scatterlist *sg, int nents,
end = sg_dma_address(sg) + sg_dma_len(sg);
__iommu_dma_unmap(iommu_get_domain_for_dev(dev), start, end - start);
 }
+EXPORT_SYMBOL(iommu_dma_unmap_sg);
 
 dma_addr_t iommu_dma_map_resource(struct device *dev, phys_addr_t phys,
size_t size, enum dma_data_direction dir, unsigned long attrs)
@@ -829,17 +838,20 @@ dma_addr_t iommu_dma_map_resource(struct device *dev, 
phys_addr_t phys,
return __iommu_dma_map(dev, phys, size,
dma_info_to_prot(dir, false, attrs) | IOMMU_MMIO);
 }
+EXPORT_SYMBOL(iommu_dma_map_resource);
 
 void iommu_dma_unmap_resource(struct device *dev, dma_addr_t handle,
size_t size, enum dma_data_direction dir, unsigned long attrs)
 {
__iommu_dma_unmap(iommu_get_domain_for_dev(dev), handle, size);
 }
+EXPORT_SYMBOL(iommu_dma_unmap_resource);
 
 int iommu_dma_mapping_error(struct device *dev, dma_addr_t dma_addr)
 {
return dma_addr == IOMMU_MAPPING_ERROR;
 }
+EXPORT_SYMBOL(iommu_dma_mapping_error);
 
 static struct iommu_dma_msi_page *iommu_dma_get_msi_page(struct device *dev,
phys_addr_t msi_addr, struct iommu_domain *domain)
@@ -913,3 +925,4 @@ void iommu_dma_map_msi_msg(int irq, struct msi_msg *msg)
msg->address_lo += lower_32_bits(msi_page->iova);
}
 }
+EXPORT_SYMBOL(iommu_dma_map_msi_msg);
-- 
2.13.2.725.g09c95d1e9-goog

___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[RFC PATCH 0/5] Fixes for loadable modules implementing DMA/IOMMU APIs

[Tomasz Figa]

This series attempts to prepare the common DMA mapping helpers, IOMMU
subsystem and IOMMU DMA helpers to be used from within loadable modules.

It does not introduce any functional changes to the code itself. The
only things done are:
 - exporting related non-static functions,
 - adding a common DMA mapping helper that removes the need for the
   module to import find_vm_area() and rely on implementation detail
   of other DMA mapping helpers,
 - adding missing cleanup function in IOMMU DMA framework.

Obviously there is no mainline user that could benefit from the above,
as for now, but there is a work in progress on mainlining the Intel IPU3
camera subsystem driver, which requires its own DMA mapping code and
IOMMU driver. It can be found on ChromiumOS gerrit at

https://chromium-review.googlesource.com/c/548626/4

or checked out directly with git with following commands

git fetch https://chromium.googlesource.com/chromiumos/third_party/kernel 
refs/changes/26/548626/4
git checkout FETCH_HEAD

The above is based on ChromeOS 4.4 kernel branch and has been used for
testing this series with code using it on real devices.

Tomasz Figa (5):
  base: dma-mapping: Export commonly used symbols
  base: dma-mapping: Provide a function to look up remapped pages
  iommu: Export non-static functions to use in modules
  iommu/dma: Export non-static functions to use in modules
  iommu/dma: Add iommu_dma_cleanup()

 drivers/base/dma-mapping.c  | 18 ++
 drivers/iommu/dma-iommu.c   | 19 +++
 drivers/iommu/iommu.c   | 12 
 include/linux/dma-iommu.h   |  6 ++
 include/linux/dma-mapping.h |  2 ++
 5 files changed, 57 insertions(+)

-- 
2.13.2.725.g09c95d1e9-goog

___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[RFC PATCH 2/5] base: dma-mapping: Provide a function to look up remapped pages

[Tomasz Figa]

DMA API implementations, which use the dma_common_*() helpers, typically
use them in pair with other helpers, such as iommu_dma_*(). For example,
a typical .free() callback needs to retrieve the pages remapped earlier
by dma_common_remap() and call iommu_dma_unmap() on them. Currently it
is done by calling find_vm_area() manually, however it relies on
implementation details of dma_common_remap() and is also difficult to
expose to loadable modules, due to find_vm_area() being quite a low
level function without its symbol exported.

Improve this by providing a function to look-up the pages previously
remapped. It hides implementation details, can do more sanity checks
than find_vm_area() and can be exported for use in loadable modules.

Signed-off-by: Tomasz Figa 
---
 drivers/base/dma-mapping.c  | 15 +++
 include/linux/dma-mapping.h |  2 ++
 2 files changed, 17 insertions(+)

diff --git a/drivers/base/dma-mapping.c b/drivers/base/dma-mapping.c
index 1fda8df3d849..9add50dd7a08 100644
--- a/drivers/base/dma-mapping.c
+++ b/drivers/base/dma-mapping.c
@@ -335,6 +335,21 @@ void dma_common_free_remap(void *cpu_addr, size_t size, 
unsigned long vm_flags)
vunmap(cpu_addr);
 }
 EXPORT_SYMBOL(dma_common_free_remap);
+
+struct page **dma_common_get_mapped_pages(void *cpu_addr,
+ unsigned long vm_flags)
+{
+   struct vm_struct *area = find_vm_area(cpu_addr);
+
+   if (!area || (area->flags & vm_flags) != vm_flags) {
+   WARN(1, "trying to get pages for invalid coherent area: %p\n",
+cpu_addr);
+   return NULL;
+   }
+
+   return area->pages;
+}
+EXPORT_SYMBOL(dma_common_get_mapped_pages);
 #endif
 
 /*
diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h
index 843ab866e0f4..bd20435dac16 100644
--- a/include/linux/dma-mapping.h
+++ b/include/linux/dma-mapping.h
@@ -422,6 +422,8 @@ void *dma_common_pages_remap(struct page **pages, size_t 
size,
unsigned long vm_flags, pgprot_t prot,
const void *caller);
 void dma_common_free_remap(void *cpu_addr, size_t size, unsigned long 
vm_flags);
+struct page **dma_common_get_mapped_pages(void *cpu_addr,
+ unsigned long vm_flags);
 
 /**
  * dma_mmap_attrs - map a coherent DMA allocation into user space
-- 
2.13.2.725.g09c95d1e9-goog

___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[RFC PATCH 5/5] iommu/dma: Add iommu_dma_cleanup()

[Tomasz Figa]

In case of loadable modules using dma-iommu helpers, it makes sense to
drop the reference to the iova cache on module exit. Add a helper called
iommu_dma_cleanup() that undoes the effects of iommu_dma_init(), so that
modules can be unloaded cleanly.

Signed-off-by: Tomasz Figa 
---
 drivers/iommu/dma-iommu.c | 6 ++
 include/linux/dma-iommu.h | 6 ++
 2 files changed, 12 insertions(+)

diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
index 7cdeaf930106..51afa18a5de1 100644
--- a/drivers/iommu/dma-iommu.c
+++ b/drivers/iommu/dma-iommu.c
@@ -82,6 +82,12 @@ int iommu_dma_init(void)
 }
 EXPORT_SYMBOL(iommu_dma_init);
 
+void iommu_dma_cleanup(void)
+{
+   iova_cache_put();
+}
+EXPORT_SYMBOL(iommu_dma_cleanup);
+
 /**
  * iommu_get_dma_cookie - Acquire DMA-API resources for a domain
  * @domain: IOMMU domain to prepare for DMA-API usage
diff --git a/include/linux/dma-iommu.h b/include/linux/dma-iommu.h
index 92f20832fd28..b12d9207a87a 100644
--- a/include/linux/dma-iommu.h
+++ b/include/linux/dma-iommu.h
@@ -24,7 +24,9 @@
 #include 
 #include 
 
+/* Framework initialization - reference counted */
 int iommu_dma_init(void);
+void iommu_dma_cleanup(void);
 
 /* Domain management interface for IOMMU drivers */
 int iommu_get_dma_cookie(struct iommu_domain *domain);
@@ -85,6 +87,10 @@ static inline int iommu_dma_init(void)
return 0;
 }
 
+static inline void iommu_dma_cleanup(void)
+{
+}
+
 static inline int iommu_get_dma_cookie(struct iommu_domain *domain)
 {
return -ENODEV;
-- 
2.13.2.725.g09c95d1e9-goog

___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [RFC PATCH 1/5] base: dma-mapping: Export commonly used symbols

[Tomasz Figa]

Hi Christoph,

Thanks for comments!

On Thu, Jul 6, 2017 at 12:17 AM, Christoph Hellwig  wrote:
> Please use EXPORT_SYMBOL_GPL for any of these exports, as they are
> internal linux implementration details by any definition of it.

Right. I typically lean towards EXPORT_SYMBOL_GPL(), but was misled by
existing EXPORT_SYMBOL()s in the file. Will fix.

>
> On Wed, Jul 05, 2017 at 04:12:11PM +0900, Tomasz Figa wrote:
>> There is nothing wrong in having a loadable module implementing DMA API,
>> for example to be used for sub-devices registered by the module.
>> However, most of the functions from dma-mapping do not have their
>> symbols exported, making it impossible to use them from loadable modules.
>
> I'd like to see the patches for this use case as well.  We don't
> generally export symbols without seeing the users.

Generally the user is a work in progress that should be posted in a
very near future. You can find a reference to our downstream tree at
chromium.org in the cover letter. Obviously I don't mind including
patches from this series in the driver series later and that's one of
the reasons for this series being RFC.

Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [RFC PATCH 1/5] base: dma-mapping: Export commonly used symbols

[Tomasz Figa]

On Thu, Jul 6, 2017 at 2:20 AM, Christoph Hellwig  wrote:
> On Thu, Jul 06, 2017 at 12:22:35AM +0900, Tomasz Figa wrote:
>> Generally the user is a work in progress that should be posted in a
>> very near future. You can find a reference to our downstream tree at
>> chromium.org in the cover letter. Obviously I don't mind including
>> patches from this series in the driver series later and that's one of
>> the reasons for this series being RFC.
>
> Please post and explain them here.

That's the intention. I will talk with people to try to get it posted earlier.

You can also refer to the old series, which we are reworking to
address review comments:
https://www.mail-archive.com/linux-media@vger.kernel.org/msg113505.html

The old code essentially reimplements all the DMA mapping operations
on its own, without using the generic helpers (and not even following
the API properly, see my review comments to the original patch 2 and
3), which IMHO is really ugly (but avoids exporting symbols of the
helpers, which I assumed to be a no-problem).

> In general I think moving dma
> ops and iommu implementations into modules is a bad idea

Could you elaborate on this? I'd be interested in seeing the reasoning
behind this.

> but I
> don't want to reject the idea before seeing the code.  Or maybe
> by looking at the user we can come up with an even better idea
> to solve the original issue you're trying to solve, so please also
> explain your rationale.

Basically we have an x86 platform with a camera subsystem that is a
PCI device, has its own MMU and needs cache maintenance. Moreover, the
V4L2 subsystem, which is the right place for camera drivers, heavily
relies on DMA mapping as a way to abstract memory allocation, mapping
and cache maintenance. So it feels natural to me to hide the hardware
details (additional cache maintenance, mapping into the built-in
IOMMU) in the DMA mapping ops for this camera subsystem and simply
make V4L2 just work without knowing those details.

Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [RFC PATCH 4/5] iommu/dma: Export non-static functions to use in modules

[Tomasz Figa]

On Thu, Jul 6, 2017 at 1:22 AM, Robin Murphy  wrote:
> On 05/07/17 08:12, Tomasz Figa wrote:
>> There is nothing wrong in having a loadable module implementing DMA API,
>> for example to be used for sub-devices registered by the module. However,
>> most of the functions from dma-iommu do not have their symbols exported,
>> making it impossible to use them from loadable modules.
>>
>> Export all the non-static functions in the file, so that loadable modules
>> can benefit from them. Use EXPORT_SYMBOL() for consistency with other
>> exports in the file.
>
> To echo what Christoph said, everything not already exported here
> shouldn't in any way be considered a driver-facing API in the general
> sense, it's horrible glue code to sit behind an arch-specific DMA
> mapping implementation (and frankly I'd consider even the current
> exports more of an unfortunate abstraction leakage).

Well, if I remember correctly, we agreed that the IPU3 driver would
benefit from using all the iommu_dma_*() helpers in its DMA ops,
similarly to ARM64. This is IMHO much better than re-implementing them
again internally just for this driver. However almost none of
necessary helpers are currently exported...

>
>> Signed-off-by: Tomasz Figa 
>> ---
>
> [...]
>
>> @@ -829,17 +838,20 @@ dma_addr_t iommu_dma_map_resource(struct device *dev, 
>> phys_addr_t phys,
>>   return __iommu_dma_map(dev, phys, size,
>>   dma_info_to_prot(dir, false, attrs) | IOMMU_MMIO);
>>  }
>> +EXPORT_SYMBOL(iommu_dma_map_resource);
>>
>>  void iommu_dma_unmap_resource(struct device *dev, dma_addr_t handle,
>>   size_t size, enum dma_data_direction dir, unsigned long attrs)
>>  {
>>   __iommu_dma_unmap(iommu_get_domain_for_dev(dev), handle, size);
>>  }
>> +EXPORT_SYMBOL(iommu_dma_unmap_resource);
>
> Do you need these two? Unless your custom DMA ops really have to support
> slave DMA or other peer-to-peer traffic through their IOMMU, I'd be more
> inclined to implement dma_map_resource as "return 0;" and ignore
> dma_unmap_resource.

I don't need them. Getting an idea what is desirable to export and
what not is actually one of the goals of this RFC.

>
>> @@ -913,3 +925,4 @@ void iommu_dma_map_msi_msg(int irq, struct msi_msg *msg)
>>   msg->address_lo += lower_32_bits(msi_page->iova);
>>   }
>>  }
>> +EXPORT_SYMBOL(iommu_dma_map_msi_msg);
>
> Given the nature of the kind of irqchip drivers this exists for, the
> chances of one ever being modular seem vanishingly small.

Agreed. The IPU3 driver does not need it either.

Let me list the (not yet exported) helpers it requires:

dma-iommu.c
 - iommu_dma_init,
 - dma_info_to_prot,
 - iommu_dma_free,
 - iommu_dma_alloc,
 - iommu_dma_mmap,
 - iommu_dma_map_page,
 - iommu_dma_unmap_page,
 - iommu_dma_map_sg,
 - iommu_dma_unmap_sg,
 - iommu_dma_mapping_error,
(added by my patch) iommu_dma_cleanup,

iommu.c
 - iommu_group_get_for_dev,

base/dma-mapping.c
 - dma_common_pages_remap,
 - dma_common_free_remap,
(added by my patch) dma_common_get_mapped_pages (OR find_vm_area),

Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [RFC PATCH 1/5] base: dma-mapping: Export commonly used symbols

[Tomasz Figa]

On Thu, Jul 6, 2017 at 5:26 PM, Arnd Bergmann  wrote:
> On Thu, Jul 6, 2017 at 3:44 AM, Tomasz Figa  wrote:
>> On Thu, Jul 6, 2017 at 2:20 AM, Christoph Hellwig  wrote:
>>> On Thu, Jul 06, 2017 at 12:22:35AM +0900, Tomasz Figa wrote:
>
>>> In general I think moving dma
>>> ops and iommu implementations into modules is a bad idea
>>
>> Could you elaborate on this? I'd be interested in seeing the reasoning
>> behind this.
>>
>>> but I
>>> don't want to reject the idea before seeing the code.  Or maybe
>>> by looking at the user we can come up with an even better idea
>>> to solve the original issue you're trying to solve, so please also
>>> explain your rationale.
>
> I had pretty much the same thoughts here.
>
>> Basically we have an x86 platform with a camera subsystem that is a
>> PCI device, has its own MMU and needs cache maintenance. Moreover, the
>> V4L2 subsystem, which is the right place for camera drivers, heavily
>> relies on DMA mapping as a way to abstract memory allocation, mapping
>> and cache maintenance. So it feels natural to me to hide the hardware
>> details (additional cache maintenance, mapping into the built-in
>> IOMMU) in the DMA mapping ops for this camera subsystem and simply
>> make V4L2 just work without knowing those details.
>
> I can understand your reasoning here, but I'm also not convinced
> that this is the best approach. There may be a middle ground somewhere
> though.
>
> Generally speaking I don't want to have to deal with the horrors of
> deciding whether an IOMMU is going to be there eventually or not
> at probe() time. At some point, we had decided that IOMMUs need to
> be initialized (almost) as early as irqchips and clocksources so we can
> rely on them to be there at device discovery time. That got pushed
> back already, and now we may have to deal with -EPROBE_DEFER
> when an IOMMU has not been fully initialized at device probe time,
> but at least we can reliably see if one is there or not. Making IOMMUs
> modular will add further uncertainty here. Obviously we cannot attach
> an IOMMU to a device once we have started using DMA mapping
> calls on it.

The hardware can only work with IOMMU and so the main module is highly
tied with the IOMMU module and it initialized it directly. There is no
separate struct driver or device associated with the IOMMU, as it's a
part of the one and only one PCI device (as visible from the system
PCI bus point of view) and technically handled by one pci_driver.

>
> For your particular use case, I would instead leave the knowledge
> about the IOMMU in the driver itself, like we do for the IOMMUs
> that are integrated in desktop GPUs, and have the code use the
> DMA mapping API with the system-provided dma_map_ops to
> get dma_addr_t tokens which you then program into the device
> IOMMU.
>
> An open question however would be whether to use the IOMMU
> API without the DMA mapping API here, or whether to completely
> leave the knowledge of the IOMMU inside of the driver itself.
> I don't have a strong opinion on that part, and I guess this mostly
> depends on what the hardware looks like.

+ linux-media and some media folks

I'd say that this is something that has been consistently tried to be
avoided by V4L2 and that's why it's so tightly integrated with DMA
mapping. IMHO re-implementing the code that's already there in
videobuf2 again in the driver, only because, for no good reason
mentioned as for now, having a loadable module providing DMA ops was
disliked.

Similarly with IOMMU API. It provides a lot of help in managing the
mappings and re-implementing this would be IMHO backwards.

Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [RFC PATCH 1/5] base: dma-mapping: Export commonly used symbols

[Tomasz Figa]

On Thu, Jul 6, 2017 at 5:34 PM, Tomasz Figa  wrote:
> On Thu, Jul 6, 2017 at 5:26 PM, Arnd Bergmann  wrote:
>> On Thu, Jul 6, 2017 at 3:44 AM, Tomasz Figa  wrote:
>>> On Thu, Jul 6, 2017 at 2:20 AM, Christoph Hellwig  wrote:
>>>> On Thu, Jul 06, 2017 at 12:22:35AM +0900, Tomasz Figa wrote:
>>
>>>> In general I think moving dma
>>>> ops and iommu implementations into modules is a bad idea
>>>
>>> Could you elaborate on this? I'd be interested in seeing the reasoning
>>> behind this.
>>>
>>>> but I
>>>> don't want to reject the idea before seeing the code.  Or maybe
>>>> by looking at the user we can come up with an even better idea
>>>> to solve the original issue you're trying to solve, so please also
>>>> explain your rationale.
>>
>> I had pretty much the same thoughts here.
>>
>>> Basically we have an x86 platform with a camera subsystem that is a
>>> PCI device, has its own MMU and needs cache maintenance. Moreover, the
>>> V4L2 subsystem, which is the right place for camera drivers, heavily
>>> relies on DMA mapping as a way to abstract memory allocation, mapping
>>> and cache maintenance. So it feels natural to me to hide the hardware
>>> details (additional cache maintenance, mapping into the built-in
>>> IOMMU) in the DMA mapping ops for this camera subsystem and simply
>>> make V4L2 just work without knowing those details.
>>
>> I can understand your reasoning here, but I'm also not convinced
>> that this is the best approach. There may be a middle ground somewhere
>> though.
>>
>> Generally speaking I don't want to have to deal with the horrors of
>> deciding whether an IOMMU is going to be there eventually or not
>> at probe() time. At some point, we had decided that IOMMUs need to
>> be initialized (almost) as early as irqchips and clocksources so we can
>> rely on them to be there at device discovery time. That got pushed
>> back already, and now we may have to deal with -EPROBE_DEFER
>> when an IOMMU has not been fully initialized at device probe time,
>> but at least we can reliably see if one is there or not. Making IOMMUs
>> modular will add further uncertainty here. Obviously we cannot attach
>> an IOMMU to a device once we have started using DMA mapping
>> calls on it.
>
> The hardware can only work with IOMMU and so the main module is highly
> tied with the IOMMU module and it initialized it directly. There is no
> separate struct driver or device associated with the IOMMU, as it's a
> part of the one and only one PCI device (as visible from the system
> PCI bus point of view) and technically handled by one pci_driver.
>
>>
>> For your particular use case, I would instead leave the knowledge
>> about the IOMMU in the driver itself, like we do for the IOMMUs
>> that are integrated in desktop GPUs, and have the code use the
>> DMA mapping API with the system-provided dma_map_ops to
>> get dma_addr_t tokens which you then program into the device
>> IOMMU.
>>
>> An open question however would be whether to use the IOMMU
>> API without the DMA mapping API here, or whether to completely
>> leave the knowledge of the IOMMU inside of the driver itself.
>> I don't have a strong opinion on that part, and I guess this mostly
>> depends on what the hardware looks like.
>
> + linux-media and some media folks
>
> I'd say that this is something that has been consistently tried to be
> avoided by V4L2 and that's why it's so tightly integrated with DMA
> mapping. IMHO re-implementing the code that's already there in
> videobuf2 again in the driver, only because, for no good reason
> mentioned as for now, having a loadable module providing DMA ops was
> disliked.

Sorry, I intended to mean:

IMHO re-implementing the code that's already there in videobuf2 again
in the driver, only because, for no good reason mentioned as for now,
having a loadable module providing DMA ops was disliked, would make no
sense.

>
> Similarly with IOMMU API. It provides a lot of help in managing the
> mappings and re-implementing this would be IMHO backwards.
>
> Best regards,
> Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [RFC PATCH 1/5] base: dma-mapping: Export commonly used symbols

[Tomasz Figa]

On Thu, Jul 6, 2017 at 9:23 PM, Arnd Bergmann  wrote:
> On Thu, Jul 6, 2017 at 10:36 AM, Tomasz Figa  wrote:
>> On Thu, Jul 6, 2017 at 5:34 PM, Tomasz Figa  wrote:
>>> On Thu, Jul 6, 2017 at 5:26 PM, Arnd Bergmann  wrote:
>>>> On Thu, Jul 6, 2017 at 3:44 AM, Tomasz Figa  wrote:
>
>>>
>>> I'd say that this is something that has been consistently tried to be
>>> avoided by V4L2 and that's why it's so tightly integrated with DMA
>>> mapping. IMHO re-implementing the code that's already there in
>>> videobuf2 again in the driver, only because, for no good reason
>>> mentioned as for now, having a loadable module providing DMA ops was
>>> disliked.
>>
>> Sorry, I intended to mean:
>>
>> IMHO re-implementing the code that's already there in videobuf2 again
>> in the driver, only because, for no good reason mentioned as for now,
>> having a loadable module providing DMA ops was disliked, would make no
>> sense.
>
> Why would we need to duplicate that code? I would expect that the videobuf2
> core can simply call the regular dma_mapping interfaces, and you handle the
> IOPTE generation at the point when the buffer is handed off from the core
> code to the device driver. Am I missing something?

Well, for example, the iommu-dma helpers already implement all the
IOVA management, SG iterations, IOMMU API calls, sanity checks and so
on. There is a significant amount of common code.

On the other hand, if it's strictly about base/dma-mapping, we might
not need it indeed. The driver could call iommu-dma helpers directly,
without the need to provide its own DMA ops. One caveat, though, we
are not able to obtain coherent (i.e. uncached) memory with this
approach, which might have some performance effects and complicates
the code, that would now need to flush caches even for some small
internal buffers.

Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [RFC PATCH 1/5] base: dma-mapping: Export commonly used symbols

[Tomasz Figa]

On Thu, Jul 6, 2017 at 10:31 PM, Tomasz Figa  wrote:
> On Thu, Jul 6, 2017 at 9:23 PM, Arnd Bergmann  wrote:
>> On Thu, Jul 6, 2017 at 10:36 AM, Tomasz Figa  wrote:
>>> On Thu, Jul 6, 2017 at 5:34 PM, Tomasz Figa  wrote:
>>>> On Thu, Jul 6, 2017 at 5:26 PM, Arnd Bergmann  wrote:
>>>>> On Thu, Jul 6, 2017 at 3:44 AM, Tomasz Figa  wrote:
>>
>>>>
>>>> I'd say that this is something that has been consistently tried to be
>>>> avoided by V4L2 and that's why it's so tightly integrated with DMA
>>>> mapping. IMHO re-implementing the code that's already there in
>>>> videobuf2 again in the driver, only because, for no good reason
>>>> mentioned as for now, having a loadable module providing DMA ops was
>>>> disliked.
>>>
>>> Sorry, I intended to mean:
>>>
>>> IMHO re-implementing the code that's already there in videobuf2 again
>>> in the driver, only because, for no good reason mentioned as for now,
>>> having a loadable module providing DMA ops was disliked, would make no
>>> sense.
>>
>> Why would we need to duplicate that code? I would expect that the videobuf2
>> core can simply call the regular dma_mapping interfaces, and you handle the
>> IOPTE generation at the point when the buffer is handed off from the core
>> code to the device driver. Am I missing something?
>
> Well, for example, the iommu-dma helpers already implement all the
> IOVA management, SG iterations, IOMMU API calls, sanity checks and so
> on. There is a significant amount of common code.
>
> On the other hand, if it's strictly about base/dma-mapping, we might
> not need it indeed. The driver could call iommu-dma helpers directly,
> without the need to provide its own DMA ops. One caveat, though, we
> are not able to obtain coherent (i.e. uncached) memory with this
> approach, which might have some performance effects and complicates
> the code, that would now need to flush caches even for some small
> internal buffers.

I think I should add a bit of explanation here:
 1) the device is non-coherent with CPU caches, even on x86,
 2) it looks like x86 does not have non-coherent DMA ops, (but it
might be something that could be fixed)
 3) one technically could still use __get_vm_area() and map_vm_area(),
which _are_ exported, to create an uncached mapping. I'll leave it to
you to judge if it would be better than using the already available
generic helpers.

Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [RFC PATCH 1/5] base: dma-mapping: Export commonly used symbols

[Tomasz Figa]

On Thu, Jul 6, 2017 at 11:02 PM, Arnd Bergmann  wrote:
> On Thu, Jul 6, 2017 at 3:49 PM, Tomasz Figa  wrote:
>> On Thu, Jul 6, 2017 at 10:31 PM, Tomasz Figa  wrote:
>
>>> On the other hand, if it's strictly about base/dma-mapping, we might
>>> not need it indeed. The driver could call iommu-dma helpers directly,
>>> without the need to provide its own DMA ops. One caveat, though, we
>>> are not able to obtain coherent (i.e. uncached) memory with this
>>> approach, which might have some performance effects and complicates
>>> the code, that would now need to flush caches even for some small
>>> internal buffers.
>>
>> I think I should add a bit of explanation here:
>>  1) the device is non-coherent with CPU caches, even on x86,
>>  2) it looks like x86 does not have non-coherent DMA ops, (but it
>> might be something that could be fixed)
>
> I don't understand what this means here. The PCI on x86 is always
> cache-coherent, so why is the device not?
>
> Do you mean that the device has its own caches that may need
> flushing to make the device cache coherent with the CPU cache,
> rather than flushing the CPU caches?

Sakari might be able to explain this with more technical details, but
generally the device is not a standard PCI device one might find on
existing x86 systems.

It is some kind of embedded subsystem that behaves mostly like a PCI
device, with certain exceptions, one being the lack of coherency with
CPU caches, at least for certain parts of the subsystem. The reference
vendor code disables the coherency completely, for reasons not known
to me, but AFAICT this is the preferred operating mode, possibly due
to performance effects (this is a memory-heavy image processing
subsystem).

Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [RFC PATCH 4/5] iommu/dma: Export non-static functions to use in modules

[Tomasz Figa]

On Thu, Jul 6, 2017 at 11:10 PM, Christoph Hellwig  wrote:
> On Thu, Jul 06, 2017 at 12:09:45PM +0100, Robin Murphy wrote:
>> I suppose another option is to just make the IOMMU and DMA ops a
>> self-contained non-modular driver mirroring the VT-d/AMD-Vi IOMMUs -
>> AFAICS it shouldn't have to be all that tightly coupled to the IPU bus
>> code, the latter more or less just needs to create the appropriate IOMMU
>> device for the driver to find.
>
> I still haven't seen the driver code, but this seems to be best
> solution so far.  Given that it's not a plug in device but part of
> an SOC that seems perfectly acceptable to me.

I guess that's something that could work. With its caveats of not
being able to avoid including the very platform specific code in a
generic kernel image or do any quick testing of code changes without a
restart, but I guess that's something one could quickly hack in their
own downstream (i.e. export the symbols and turn the Kconfig entry
into tristate).

On the other hand, I'm yet to see any real reasons why not to export
those symbols. Personally I don't see anything that one wouldn't be
able to do in their downstream without the symbols exported in
mainline (one can add the exports any time or if the kernel source
can't be modified can just load a wrapper module that exports its own
symbols...)

Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [RFC PATCH 4/5] iommu/dma: Export non-static functions to use in modules

[Tomasz Figa]

On Thu, Jul 6, 2017 at 11:17 PM, Tomasz Figa  wrote:
> On Thu, Jul 6, 2017 at 11:10 PM, Christoph Hellwig  wrote:
>> On Thu, Jul 06, 2017 at 12:09:45PM +0100, Robin Murphy wrote:
>>> I suppose another option is to just make the IOMMU and DMA ops a
>>> self-contained non-modular driver mirroring the VT-d/AMD-Vi IOMMUs -
>>> AFAICS it shouldn't have to be all that tightly coupled to the IPU bus
>>> code, the latter more or less just needs to create the appropriate IOMMU
>>> device for the driver to find.
>>
>> I still haven't seen the driver code, but this seems to be best
>> solution so far.  Given that it's not a plug in device but part of
>> an SOC that seems perfectly acceptable to me.
>
> I guess that's something that could work. With its caveats of not
> being able to avoid including the very platform specific code in a
> generic kernel image or do any quick testing of code changes without a
> restart, but I guess that's something one could quickly hack in their
> own downstream (i.e. export the symbols and turn the Kconfig entry
> into tristate).
>
> On the other hand, I'm yet to see any real reasons why not to export
> those symbols. Personally I don't see anything that one wouldn't be
> able to do in their downstream without the symbols exported in
> mainline (one can add the exports any time or if the kernel source
> can't be modified can just load a wrapper module that exports its own
> symbols...)

Sorry, I just realized that last sentence might sound nonsense. By
wrapper I meant reimplementing the missing functions using some
already exported functions I mentioned in my post to another patch
from this series, such as __get_vm_area() and map_vm_area().
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [RFC PATCH 1/5] base: dma-mapping: Export commonly used symbols

[Tomasz Figa]

On Thu, Jul 6, 2017 at 11:27 PM, Arnd Bergmann  wrote:
> On Thu, Jul 6, 2017 at 4:06 PM, Tomasz Figa  wrote:
>> On Thu, Jul 6, 2017 at 11:02 PM, Arnd Bergmann  wrote:
>>> On Thu, Jul 6, 2017 at 3:49 PM, Tomasz Figa  wrote:
>>>> On Thu, Jul 6, 2017 at 10:31 PM, Tomasz Figa  wrote:
>>>
>>>>> On the other hand, if it's strictly about base/dma-mapping, we might
>>>>> not need it indeed. The driver could call iommu-dma helpers directly,
>>>>> without the need to provide its own DMA ops. One caveat, though, we
>>>>> are not able to obtain coherent (i.e. uncached) memory with this
>>>>> approach, which might have some performance effects and complicates
>>>>> the code, that would now need to flush caches even for some small
>>>>> internal buffers.
>>>>
>>>> I think I should add a bit of explanation here:
>>>>  1) the device is non-coherent with CPU caches, even on x86,
>>>>  2) it looks like x86 does not have non-coherent DMA ops, (but it
>>>> might be something that could be fixed)
>>>
>>> I don't understand what this means here. The PCI on x86 is always
>>> cache-coherent, so why is the device not?
>>>
>>> Do you mean that the device has its own caches that may need
>>> flushing to make the device cache coherent with the CPU cache,
>>> rather than flushing the CPU caches?
>>
>> Sakari might be able to explain this with more technical details, but
>> generally the device is not a standard PCI device one might find on
>> existing x86 systems.
>>
>> It is some kind of embedded subsystem that behaves mostly like a PCI
>> device, with certain exceptions, one being the lack of coherency with
>> CPU caches, at least for certain parts of the subsystem. The reference
>> vendor code disables the coherency completely, for reasons not known
>> to me, but AFAICT this is the preferred operating mode, possibly due
>> to performance effects (this is a memory-heavy image processing
>
> Ok, got it. I think something similar happens on integrated GPUs for
> a certain CPU family. The DRM code has its own ways of dealing with
> this kind of device. If you find that the hardware to be closely
> related (either the implementation, or the location on the internal
> buses) to the GPU on this machine, I'd recommend having a look
> in drivers/gpu/drm to see how it's handled there, and if that code could
> be shared.

I think it's not closely related, but might be a very similar case.

Still, DRM is very liberal in terms of not using common code for doing
things, while V4L2 tries to makes things generic as much as possible.
There is already the vb2_dma_contig backend, which allocates coherent
memory (in case of V4L2-allocated buffers), manages caches (in case of
userptr or DMA-buf buffers) and so on for you. If we can't have the
DMA ops do the right thing, the code there is essentially useless and
you are left with vb2_dma_sg that uses a page allocator and gives the
driver sg tables (it actually can also do cache management for you,
but since dma_sync_sg_*() is essentially a no-op on x86, the driver
would have to do it on its own).

Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [RFC PATCH 4/5] iommu/dma: Export non-static functions to use in modules

[Tomasz Figa]

On Thu, Jul 6, 2017 at 11:35 PM, Arnd Bergmann  wrote:
> On Thu, Jul 6, 2017 at 4:24 PM, Tomasz Figa  wrote:
>> On Thu, Jul 6, 2017 at 11:17 PM, Tomasz Figa  wrote:
>>> On Thu, Jul 6, 2017 at 11:10 PM, Christoph Hellwig  wrote:
>>>> On Thu, Jul 06, 2017 at 12:09:45PM +0100, Robin Murphy wrote:
>>>>> I suppose another option is to just make the IOMMU and DMA ops a
>>>>> self-contained non-modular driver mirroring the VT-d/AMD-Vi IOMMUs -
>>>>> AFAICS it shouldn't have to be all that tightly coupled to the IPU bus
>>>>> code, the latter more or less just needs to create the appropriate IOMMU
>>>>> device for the driver to find.
>>>>
>>>> I still haven't seen the driver code, but this seems to be best
>>>> solution so far.  Given that it's not a plug in device but part of
>>>> an SOC that seems perfectly acceptable to me.
>>>
>>> I guess that's something that could work. With its caveats of not
>>> being able to avoid including the very platform specific code in a
>>> generic kernel image or do any quick testing of code changes without a
>>> restart, but I guess that's something one could quickly hack in their
>>> own downstream (i.e. export the symbols and turn the Kconfig entry
>>> into tristate).
>>>
>>> On the other hand, I'm yet to see any real reasons why not to export
>>> those symbols. Personally I don't see anything that one wouldn't be
>>> able to do in their downstream without the symbols exported in
>>> mainline (one can add the exports any time or if the kernel source
>>> can't be modified can just load a wrapper module that exports its own
>>> symbols...)
>>
>> Sorry, I just realized that last sentence might sound nonsense. By
>> wrapper I meant reimplementing the missing functions using some
>> already exported functions I mentioned in my post to another patch
>> from this series, such as __get_vm_area() and map_vm_area().
>
> We should look at the two aspects separately: one is how to drive
> the IOMMU as part of a loadable driver, the other is how to handle
> DMA to uncached memory on x86. You are in an unfortunate
> position of needing both, but aside from that they seem unrelated.

So generally I'm not in such a desperate need to have this code as a
loadable module. I can just hack around it in my own working tree if I
need it. However I'm trying to understand, why this is such a bad
idea, if I already have it working.

Uncached DMA memory on x86 is actually more complicated. We could
still work around this by not relying on the memory being uncached
(with caveats obviously). But here too I'd like to understand why
having my own DMA ops is bad.

Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v2 1/2] iommu/dma: Respect __GFP_DMA and __GFP_DMA32 in incoming GFP flags

[Tomasz Figa]

Hi Joerg,

On Wed, Jul 26, 2017 at 6:15 PM, Joerg Roedel  wrote:
> On Tue, Jul 04, 2017 at 10:55:55PM +0900, Tomasz Figa wrote:
>> Current implementation of __iommu_dma_alloc_pages() keeps adding
>> __GFP_HIGHMEM to GFP flags regardless of whether other zone flags are
>> already included in the incoming flags. If __GFP_DMA or __GFP_DMA32 is
>> set at the same time as __GFP_HIGHMEM, the allocation fails due to
>> invalid zone flag combination.
>>
>> Fix this by checking for __GFP_DMA and __GFP_DMA32 in incoming GFP flags
>> and adding __GFP_HIGHMEM only if they are not present.
>>
>> Signed-off-by: Tomasz Figa 
>
> Isn't it better to mask out __GFP_DMA and __GFP_DMA32 in the allocation
> flags and only take them into account for iova allocation?
>
> When the IOMMU re-maps the DMA to this memory it doesn't matter where it
> is in system memory.

It's a platform specific knowledge and I'd say the generic helper is
not where it should be decided. Please see my reply to Robin for v1
[1].

[1] https://patchwork.kernel.org/patch/9810921/

Best regards.
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v2 2/2] iommu/dma: Use __GFP_NOWARN only for high-order allocations

[Tomasz Figa]

Hi Joerg,

On Wed, Jul 26, 2017 at 6:24 PM, Joerg Roedel  wrote:
> On Tue, Jul 04, 2017 at 10:55:56PM +0900, Tomasz Figa wrote:
>> diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
>> index bf23989b5158..6ed8c8f941d8 100644
>> --- a/drivers/iommu/dma-iommu.c
>> +++ b/drivers/iommu/dma-iommu.c
>> @@ -433,6 +433,7 @@ static struct page **__iommu_dma_alloc_pages(unsigned 
>> int count,
>>  {
>>   struct page **pages;
>>   unsigned int i = 0, array_size = count * sizeof(*pages);
>> + const gfp_t high_order_gfp = __GFP_NOWARN | __GFP_NORETRY;
>>
>>   order_mask &= (2U << MAX_ORDER) - 1;
>>   if (!order_mask)
>> @@ -452,8 +453,6 @@ static struct page **__iommu_dma_alloc_pages(unsigned 
>> int count,
>>   if (!(gfp & (__GFP_DMA | __GFP_DMA32)))
>>   gfp |= __GFP_HIGHMEM;
>>
>> - gfp |= __GFP_NOWARN;
>> -
>
> Okay, so a warning should be there if allocation fails, independent of
> what the allocation order is.

The allocation fails only if the order drops to the lowest possible
fallback order.

> So either this function prints a message
> in case of allocation failure or we remove __GFP_NOWARN for all
> allocation attempts.
>
> Adding __GFP_NOWARN only makes sense when there is another fall-back in
> case allocation fails anyway, which is not the case here.

There is fall back here. The loop tries allocating with higher order
and then falls back to a lower order if it fails and so on, until the
lowest acceptable order is reached.

>
>>   while (count) {
>>   struct page *page = NULL;
>>   unsigned int order_size;
>> @@ -469,7 +468,7 @@ static struct page **__iommu_dma_alloc_pages(unsigned 
>> int count,
>>
>>   order_size = 1U << order;
>>   page = alloc_pages((order_mask - order_size) ?
>> -gfp | __GFP_NORETRY : gfp, order);
>> +gfp | high_order_gfp : gfp, order);
>
> Why does it specify __GFP_NORETRY at all? The alloc-routines in the
> DMA-API don't need to be atomic.

This doesn't have anything to do with being atomic. __GFP_NORETRY here
is to avoid the page allocator retrying indefinitely and actually
triggering OOM for high order allocation, if we can safely fall back
to lower order.

Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v3 02/12] intel-ipu3: mmu: implement driver

[Tomasz Figa]

Hi Robin,

On Wed, Jul 19, 2017 at 10:37 PM, Robin Murphy  wrote:
> On 19/07/17 04:12, Yong Zhi wrote:
>> From: Tomasz Figa 
>>
>> This driver translates Intel IPU3 internal virtual
>> address to physical address.
>>
>> Signed-off-by: Tomasz Figa 
>> Signed-off-by: Yong Zhi 
>> ---
>>  drivers/media/pci/intel/ipu3/Kconfig|   9 +
>>  drivers/media/pci/intel/ipu3/Makefile   |  15 +
>>  drivers/media/pci/intel/ipu3/ipu3-mmu.c | 639 
>> 
>>  drivers/media/pci/intel/ipu3/ipu3-mmu.h |  27 ++
>>  4 files changed, 690 insertions(+)
>>  create mode 100644 drivers/media/pci/intel/ipu3/ipu3-mmu.c
>>  create mode 100644 drivers/media/pci/intel/ipu3/ipu3-mmu.h
>>
>> diff --git a/drivers/media/pci/intel/ipu3/Kconfig 
>> b/drivers/media/pci/intel/ipu3/Kconfig
>> index 2a895d6..7bcdfa5 100644
>> --- a/drivers/media/pci/intel/ipu3/Kconfig
>> +++ b/drivers/media/pci/intel/ipu3/Kconfig
>> @@ -15,3 +15,12 @@ config VIDEO_IPU3_CIO2
>>   Say Y or M here if you have a Skylake/Kaby Lake SoC with MIPI CSI-2
>>   connected camera.
>>   The module will be called ipu3-cio2.
>> +
>> +config INTEL_IPU3_MMU
>> + tristate
>
> Shouldn't this be bool now?

Well, depends on what we expect it to be. I still didn't see any good
reason not to make it a loadable module.

>
>> + default n
>> + select IOMMU_API
>> + select IOMMU_IOVA
>> + ---help---
>> +   For IPU3, this option enables its MMU driver to translate its 
>> internal
>> +   virtual address to 39 bits wide physical address for 64GBytes space 
>> access.
>> diff --git a/drivers/media/pci/intel/ipu3/Makefile 
>> b/drivers/media/pci/intel/ipu3/Makefile
>> index 20186e3..91cac9c 100644
>> --- a/drivers/media/pci/intel/ipu3/Makefile
>> +++ b/drivers/media/pci/intel/ipu3/Makefile
>> @@ -1 +1,16 @@
>> +#
>> +#  Copyright (c) 2017, Intel Corporation.
>> +#
>> +#  This program is free software; you can redistribute it and/or modify it
>> +#  under the terms and conditions of the GNU General Public License,
>> +#  version 2, as published by the Free Software Foundation.
>> +#
>> +#  This program is distributed in the hope it will be useful, but WITHOUT
>> +#  ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
>> +#  FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
>> +#  more details.
>> +#
>> +
>>  obj-$(CONFIG_VIDEO_IPU3_CIO2) += ipu3-cio2.o
>> +obj-$(CONFIG_INTEL_IPU3_MMU) += ipu3-mmu.o
>> +
>> diff --git a/drivers/media/pci/intel/ipu3/ipu3-mmu.c 
>> b/drivers/media/pci/intel/ipu3/ipu3-mmu.c
>> new file mode 100644
>> index 000..093b821
>> --- /dev/null
>> +++ b/drivers/media/pci/intel/ipu3/ipu3-mmu.c
>> @@ -0,0 +1,639 @@
>> +/*
>> + * Copyright (c) 2017 Intel Corporation.
>> + * Copyright (C) 2017 Google, Inc.
>> + *
>> + * This program is free software; you can redistribute it and/or
>> + * modify it under the terms of the GNU General Public License version
>> + * 2 as published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + *
>> + */
>> +
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +
>> +#include 
>> +
>> +#include "ipu3-mmu.h"
>> +
>> +#define IPU3_PAGE_SHIFT  12
>> +#define IPU3_PAGE_SIZE   (1UL << IPU3_PAGE_SHIFT)
>> +
>> +#define IPU3_PT_BITS 10
>> +#define IPU3_PT_PTES (1UL << IPU3_PT_BITS)
>> +
>> +#define IPU3_ADDR2PTE(addr)  ((addr) >> IPU3_PAGE_SHIFT)
>> +#define IPU3_PTE2ADDR(pte)   ((phys_addr_t)(pte) << IPU3_PAGE_SHIFT)
>> +
>> +#define IPU3_L2PT_SHIFT  IPU3_PT_BITS
>> +#define IPU3_L2PT_MASK   ((1UL << IPU3_L2PT_SHIFT) - 1)
>> +
>> +#define IPU3_L1PT_SHIFT  IPU3_PT_BITS
>> +#define IPU3_L1PT_MASK   ((1UL << IPU3_L1PT_SHIFT) - 1)
>> +
>> +#define IPU3_MMU_ADDRESS_BITS(IPU3_PAGE_SHIFT + \
>> +  IPU3_L2PT_SHIFT + \
>> +  IPU3_L1PT_SHIFT)
>> +
>&g

Re: [PATCH v3 03/12] intel-ipu3: Add DMA API implementation

[Tomasz Figa]

On Fri, Jul 21, 2017 at 7:09 AM, Sakari Ailus  wrote:
> Hi Arnd,
>
> On Wed, Jul 19, 2017 at 09:24:41AM +0200, Arnd Bergmann wrote:
>> On Wed, Jul 19, 2017 at 5:12 AM, Yong Zhi  wrote:
>> > From: Tomasz Figa 
>> >
>> > This patch adds support for the IPU3 DMA mapping API.
>> >
>> > Signed-off-by: Tomasz Figa 
>> > Signed-off-by: Yong Zhi 
>>
>> This needs some explanation on why you decided to go down the
>> route of adding your own dma_map_ops. It's not obvious at all,
>> and and I'm still concerned that this complicates things more than
>> it helps.
>
> There are a few considerations here --- they could be documented in the
> patch commit message
>
> - The device has its own MMU. The default x86 DMA ops assume there isn't.
>
> - As this is an image signal processor device, the buffers are typically
>   large (often in the range of tens of MB) and they do not need to be
>   physically contiguous. The current implementation of e.g.
>   drivers/iommu/intel-iommu.c allocate memory using alloc_pages() which is
>   unfeasible for such single allocations. Neither CMA is needed.
>
>   Also other IOMMU implementations have their own DMA ops currently.
>
> I agree it'd be nice to unify these in the long run but I don't think this
> stands apart from the rest currently --- except that the MMU is only used
> by a single PCI device, the same which it is contained in.

On top of what Sakari said, it just perfectly matches what V4L2
videobuf2 framework expects. It does all the buffer mapping and
synchronization using DMA mapping and given the x86 DMA ops being
useless for this device, it makes everything that videobuf2 does using
them useless too.

Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v9 06/16] ARM: dts: Add description of System MMU of Exynos SoCs

[Tomasz Figa]

On Friday 09 of August 2013 15:15:57 Cho KyongHo wrote:
> On Fri, 09 Aug 2013 00:26:51 +0200, Tomasz Figa wrote:
> > Hi KyongHo,
> > 
> > On Thursday 08 of August 2013 18:38:35 Cho KyongHo wrote:
> > > Signed-off-by: Cho KyongHo 
> > > ---
> > > 
> > >  .../bindings/iommu/samsung,exynos4210-sysmmu.txt   |  103 +++
> > >  arch/arm/boot/dts/exynos4.dtsi |  122 
> > >  arch/arm/boot/dts/exynos4210.dtsi  |   25 ++
> > >  arch/arm/boot/dts/exynos4x12.dtsi  |   82 ++
> > >  arch/arm/boot/dts/exynos5250.dtsi  |  290
> > > 
> > >  5 files changed, 622 insertions(+), 0
> > > deletions(-)
> > > 
> > >  create mode 100644
> > > 
> > > Documentation/devicetree/bindings/iommu/samsung,exynos4210-sysmmu.tx
> > > t
> > > 
> > > diff --git
> > > a/Documentation/devicetree/bindings/iommu/samsung,exynos4210-sysmmu.
> > > txt
> > > b/Documentation/devicetree/bindings/iommu/samsung,exynos4210-sysmmu.
> > > txt
> > > new file mode 100644
> > > index 000..92f0a33
> > > --- /dev/null
> > > +++
> > > b/Documentation/devicetree/bindings/iommu/samsung,exynos4210-sysmmu.
> > > txt
> > > @@ -0,0 +1,103 @@
> > > +Samsung Exynos4210 IOMMU H/W, System MMU (System Memory Management
> > > Unit) +
> > > +Samsung's Exynos architecture contains System MMU that enables
> > > scattered +physical memory chunks visible as a contiguous region to
> > > DMA-capable peripheral +devices like MFC, FIMC, FIMD, GScaler,
> > > FIMC-IS
> > > and so forth.
> > > +
> > > +System MMU is a sort of IOMMU and support identical translation
> > > table
> > > format to +ARMv7 translation tables with minimum set of page
> > > properties
> > > including access +permissions, shareability and security protection.
> > > In
> > > addition, System MMU has +another capabilities like L2 TLB or
> > > block-fetch buffers to minimize translation +latency.
> > > +
> > > +A System MMU is dedicated to a single master peripheral device. 
> > > Thus,
> > > it is +important to specify the correct System MMU in the device
> > > node
> > > of its master +device. Whereas a System MMU is dedicated to a master
> > > device, the master device +may have more than one System MMU.
> > 
> > This paragraph is still not clear. What about something among these 
lines:
> Yes. It is my fault.
> It is still unchanged... even though Grant sugessted clear and simple
> sentence. It must be changed :)
> 
> > System MMUs are in many to one relation with peripheral devices, i.e.
> > single peripheral device might have multiple System MMUs (usually one
> > for each bus master), but one System MMU can handle only one
> > peripheral device. The relation between a System MMU and the
> > peripheral device it handles needs to be defined in device node of
> > this peripheral device.
> That looks good to me.
> Please let me use your sentences.

OK.

> > > +Required properties:
> > > +- compatible: Should be "samsung,exynos4210-sysmmu"
> > > +- reg: A tuple of base address and size of System MMU registers.
> > > +- interrupt-parent: The phandle of the interrupt controller of
> > > System
> > > MMU +- interrupts: A tuple of numbers that indicates the interrupt
> > > source.
> > 
> > interrupts: An interrupt specifier for interrupt signal of System MMU,
> > according to format defined for particular interrupt parent.
> 
> Yes.. it is not changed..
> 
> > > +- clock-names: Should be "sysmmu" if the System MMU is needed
> > > to gate its clock. +   Please refer to the following
> > > documents:
> > > +Documentation/devicetree/bindings/clock/clock-bindings.txt
> > > +Documentation/devicetree/bindings/clock/exynos4-clock.txt
> > > +Documentation/devicetree/bindings/clock/exynos5250-
> > 
> > clock.txt
> > 
> > > +Optional "master" if the clock to the System MMU is gated
> > 
> > by
> > 
> > > +another gate clock other than "sysmmu". The System MMU
> > 
> > driver
> > 
> > > +sets "master" the parent of "sysmmu".
> > > +Exynos4 SoCs, there needs no "master" clocks.
> > > +Exynos

Re: [PATCH] iommu/exynos: Remove driver

[Tomasz Figa]

Hi KyongHo,

On 09.03.2014 14:54, KyongHo Cho wrote:

On Thu, Mar 6, 2014 at 8:48 AM, Kyungmin Park  wrote:

On Fri, Feb 14, 2014 at 9:17 AM, Cho KyongHo  wrote:

-Original Message-
From: Olof Johansson [mailto:o...@lixom.net]
Sent: Friday, February 14, 2014 4:34 AM

On Mon, Feb 10, 2014 at 10:21 PM, Kukjin Kim  wrote:


Just adding KyongHo Cho.

If he can fixup for this time, it would be best solution because he knows
well than others, I think.


It's not so much a matter of "fixup for this time", it's a about
having ownership of the driver, making sure it works (and keeps
working if there is related development). The posted patches have not
been followed through on and the result is a broken driver. :(

I definitely appreciate his expertise, and we should make sure that he
gets to review the code, but if someone else is able to spend time on
reworking the driver (or rewriting a newer one) and maintaining it
longer-term, then we should not stop them from doing so. And there is
no reason to keep broken stale code in the kernel meanwhile.



Thank you for your concerning.
I also definitely agree with you that the driver must work.
I am always concerning about it but it was not easy to make some time
for the patches.

I will continue to post the next version of patches, of course.
I think it is not far from now to show it.


Lots of time is going from last reply. there are two options.
1. just waiting more
2. remove it as patch and start it again by someone.

what's the opinions?



Please be patient until the patches are OK enough to post.

The patches have lots of improvements and workarounds.

Sorry for delaying the patches.

One thing is certain, starting from scratch is not a good idea.
Why not discuss with me to make exynos-iommu driver better?


It is not a matter of starting from scratch or not, but rather of 
getting a reasonable IOMMU driver for Exynos in mainline in reasonable 
period of time. Usually it is much easier to just remove a stale driver 
and quickly refactor all the code at once and then add a "new" driver, 
than wasting time on submitting heaps of patches for a driver that is 
not used and does not work anyway.


In fact, we already have a working driver, based on heavily refactored 
original one in our internal tree anyway and we are just wasting time on 
waiting for next version of the original refactor series to come, while 
we could simply drop current version from the kernel, squash all of any 
changes to it into a one patch adding the new driver and submit it 
instead. Of course care would be taken to preserve any authorship 
information.


Do you have some time frame in which you may post next version of the 
series (and guaranteed time to work on it)?


Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v11 11/27] clk: exynos: add gate clock descriptions of System MMU

[Tomasz Figa]

Hi KyongHo,

On 14.03.2014 06:06, Cho KyongHo wrote:

This adds gate clocks of all System MMUs and their master IPs
that are not apeared in clk-exynos5250.c and clk-exynos5420.c
Also fixes GATE_IP_ACP to 0x18800 and changed GATE_DA to GATE
for System MMU clocks in clk-exynos4.c

Signed-off-by: Cho KyongHo 
---
  .../devicetree/bindings/clock/exynos5250-clock.txt |3 +++
  .../devicetree/bindings/clock/exynos5420-clock.txt |6 +-
  drivers/clk/samsung/clk-exynos5250.c   |5 +
  drivers/clk/samsung/clk-exynos5420.c   |   13 +++--
  include/dt-bindings/clock/exynos5250.h |4 
  include/dt-bindings/clock/exynos5420.h |6 +-
  6 files changed, 33 insertions(+), 4 deletions(-)

diff --git a/Documentation/devicetree/bindings/clock/exynos5250-clock.txt 
b/Documentation/devicetree/bindings/clock/exynos5250-clock.txt
index 72ce617..67e50ba 100644
--- a/Documentation/devicetree/bindings/clock/exynos5250-clock.txt
+++ b/Documentation/devicetree/bindings/clock/exynos5250-clock.txt
@@ -162,6 +162,9 @@ clock which they consume.
g2d 345
mdma0   346
smmu_mdma0  347
+  smmu_tv  348
+  smmu_fimd1   349
+  smmu_2d  350



This patch should be rebased on top of Andrzej Hajda's patches removing 
these clock ID listings and reworking dts files to use defined macros. 
They are present in v3.15-next/dt-clk-exynos branch of linux-samsung 
tree, but I have asked Kukjin to merge them to his for-next branch, so 
they could show up in linux-next tree.


Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v11 12/27] ARM: dts: Add description of System MMU of Exynos SoCs

[Tomasz Figa]

Hi KyongHo,

On 14.03.2014 06:06, Cho KyongHo wrote:

This patch adds dts entries for the System MMU devices found on
Exynos4 and Exynos5 SoC series and the System MMU binding
documentation.

CC: Rob Herring 
CC: Sylwester Nawrocki 
Signed-off-by: Cho KyongHo 
---
  .../bindings/iommu/samsung,exynos4210-sysmmu.txt   |   86 +++
  arch/arm/boot/dts/exynos4.dtsi |  107 
  arch/arm/boot/dts/exynos4210.dtsi  |   23 +-
  arch/arm/boot/dts/exynos4x12.dtsi  |   77 +-
  arch/arm/boot/dts/exynos5250.dtsi  |  266 +++-
  arch/arm/boot/dts/exynos5420.dtsi  |  205 ++-
  6 files changed, 758 insertions(+), 6 deletions(-)
  create mode 100644 
Documentation/devicetree/bindings/iommu/samsung,exynos4210-sysmmu.txt

diff --git 
a/Documentation/devicetree/bindings/iommu/samsung,exynos4210-sysmmu.txt 
b/Documentation/devicetree/bindings/iommu/samsung,exynos4210-sysmmu.txt
new file mode 100644
index 000..e4417bb
--- /dev/null
+++ b/Documentation/devicetree/bindings/iommu/samsung,exynos4210-sysmmu.txt
@@ -0,0 +1,86 @@
+Samsung Exynos IOMMU H/W, System MMU (System Memory Management Unit)
+
+Samsung's Exynos architecture contains System MMUs that enables scattered
+physical memory chunks visible as a contiguous region to DMA-capable peripheral
+devices like MFC, FIMC, FIMD, GScaler, FIMC-IS and so forth.
+
+System MMU is an IOMMU and supports identical translation table format to
+ARMv7 translation tables with minimum set of page properties including access
+permissions, shareability and security protection. In addition, System MMU has
+another capabilities like L2 TLB or block-fetch buffers to minimize translation
+latency.
+
+System MMUs are in many to one relation with peripheral devices, i.e. single
+peripheral device might have multiple System MMUs (usually one for each bus
+master), but one System MMU can handle transactions from only one peripheral
+device. The relation between a System MMU and the peripheral device needs to be
+defined in device node of the peripheral device.
+
+MFC in all Exynos SoCs and FIMD, M2M Scalers and G2D in Exynos5420 has 2 System
+MMUs.
+* MFC has one System MMU on its left and right bus.
+* FIMD in Exynos5420 has one System MMU for window 0 and 4, the other system 
MMU
+  for window 1, 2 and 3.
+* M2M Scalers and G2D in Exynos5420 has one System MMU on the read channel and
+  the other System MMU on the write channel.
+The drivers must consider how to handle those System MMUs. One of the idea is
+to implement child devices or sub-devices which are the client devices of the
+System MMU.
+
+Required properties:
+- compatible: Should be one of:
+   "samsung,sysmmu-v1"
+   "samsung,sysmmu-v2"
+   "samsung,sysmmu-v3.1"
+   "samsung,sysmmu-v3.2"
+   "samsung,sysmmu-v3.3"
+
+- reg: A tuple of base address and size of System MMU registers.
+- interrupt-parent: The phandle of the interrupt controller of System MMU
+- interrupts: An interrupt specifier for interrupt signal of System MMU,
+ according to the format defined by a particular interrupt
+ controller.
+- clock-names: Should be "sysmmu" if the System MMU is needed to gate its 
clock.
+   Please refer to the following documents:
+  Documentation/devicetree/bindings/clock/clock-bindings.txt
+  Documentation/devicetree/bindings/clock/exynos4-clock.txt
+  Documentation/devicetree/bindings/clock/exynos5250-clock.txt
+  Documentation/devicetree/bindings/clock/exynos5420-clock.txt
+  Optional "master" if the clock to the System MMU is gated by
+  another gate clock other than "sysmmu". The System MMU driver
+  sets "master" the parent of "sysmmu".
+  Exynos4 SoCs, there needs no "master" clockj.
+  Exynos5 SoCs, some System MMUs must have "master" clocks.
+- clocks: Required if the System MMU is needed to gate its clock.
+ Please refer to the documents listed above.
+- samsung,power-domain: Required if the System MMU is needed to gate its power.
+ Please refer to the following document:
+ Documentation/devicetree/bindings/arm/exynos/power_domain.txt
+- mmu-masters: A phandle to device nodes representing the master for which
+   the System MMU can provide a translation. Any additional values
+  after the phandle will be ignored because a System MMU never
+  have two or more masters. "#stream-id-cells" specified in the
+  master's node will be also ignored.
+  If more than one phandle is specified, only the first phandle
+  will be treated.
+
+Examples:
+   gsc_0: gsc@13e0 {
+   compatible = "samsung,exynos5-gsc";
+   reg = <0x13e0 0x1000>;
+   interrupts = <0 85 0>;
+   

Re: [PATCH v11 17/27] iommu/exynos: remove calls to Runtime PM API functions

[Tomasz Figa]

Hi KyongHo,

On 14.03.2014 06:08, Cho KyongHo wrote:

Runtime power management by exynos-iommu driver independently from
master H/W's runtime pm is not useful for power saving since attaching
master H/W in probing time turns on its local power endlessly.
Thus this removes runtime pm API calls.
Runtime PM support is added in the following commits to exynos-iommu
driver.


The patch seems to be doing something completely different than the 
commit description suggests. Please rewrite the description to describe 
the patch correctly.




Signed-off-by: Cho KyongHo 
---
  drivers/iommu/exynos-iommu.c |  369 +++---
  1 file changed, 238 insertions(+), 131 deletions(-)

diff --git a/drivers/iommu/exynos-iommu.c b/drivers/iommu/exynos-iommu.c
index 3458349..6834556 100644
--- a/drivers/iommu/exynos-iommu.c
+++ b/drivers/iommu/exynos-iommu.c
@@ -27,6 +27,8 @@
  #include 
  #include 
  #include 
+#include 
+#include 

  #include 
  #include 
@@ -111,6 +113,8 @@
  #define __master_clk_enable(data) __clk_gate_ctrl(data, clk_master, en)
  #define __master_clk_disable(data)__clk_gate_ctrl(data, clk_master, dis)

+#define has_sysmmu(dev)(dev->archdata.iommu != NULL)
+
  static struct kmem_cache *lv2table_kmem_cache;

  static unsigned long *section_entry(unsigned long *pgtable, unsigned long 
iova)
@@ -159,6 +163,16 @@ static char *sysmmu_fault_name[SYSMMU_FAULTS_NUM] = {
"UNKNOWN FAULT"
  };

+/* attached to dev.archdata.iommu of the master device */
+struct exynos_iommu_owner {
+   struct list_head client; /* entry of exynos_iommu_domain.clients */
+   struct device *dev;
+   struct device *sysmmu;
+   struct iommu_domain *domain;
+   void *vmm_data; /* IO virtual memory manager's data */
+   spinlock_t lock;/* Lock to preserve consistency of System MMU */


Please don't use spaces for alignment.


+};
+
  struct exynos_iommu_domain {
struct list_head clients; /* list of sysmmu_drvdata.node */
unsigned long *pgtable; /* lv1 page table, 16KB */
@@ -168,9 +182,8 @@ struct exynos_iommu_domain {
  };

  struct sysmmu_drvdata {
-   struct list_head node; /* entry of exynos_iommu_domain.clients */
struct device *sysmmu;  /* System MMU's device descriptor */
-   struct device *dev; /* Owner of system MMU */
+   struct device *master;  /* Owner of system MMU */
void __iomem *sfrbase;
struct clk *clk;
struct clk *clk_master;
@@ -239,7 +252,6 @@ static void __sysmmu_tlb_invalidate_entry(void __iomem 
*sfrbase,
  static void __sysmmu_set_ptbase(void __iomem *sfrbase,
   unsigned long pgd)
  {
-   __raw_writel(0x1, sfrbase + REG_MMU_CFG); /* 16KB LV1, LRU */
__raw_writel(pgd, sfrbase + REG_PT_BASE_ADDR);

__sysmmu_tlb_invalidate(sfrbase);
@@ -299,7 +311,7 @@ static irqreturn_t exynos_sysmmu_irq(int irq, void *dev_id)
itype, base, addr);
if (data->domain)
ret = report_iommu_fault(data->domain,
-   data->dev, addr, itype);
+   data->master, addr, itype);
}

/* fault is not recovered by fault handler */
@@ -316,116 +328,148 @@ static irqreturn_t exynos_sysmmu_irq(int irq, void 
*dev_id)
return IRQ_HANDLED;
  }

-static bool __exynos_sysmmu_disable(struct sysmmu_drvdata *data)
+static void __sysmmu_disable_nocount(struct sysmmu_drvdata *data)


If you are changing the names anyway, it would be probably a good idea 
to reduce code obfuscation a bit and drop the underscores from 
beginnings of function names. Also I'd suggest keeping the "exynos_" prefix.



  {
-   unsigned long flags;
-   bool disabled = false;
-
-   write_lock_irqsave(&data->lock, flags);
-
-   if (!set_sysmmu_inactive(data))
-   goto finish;
-
-   __master_clk_enable(data);
+   clk_enable(data->clk_master);

__raw_writel(CTRL_DISABLE, data->sfrbase + REG_MMU_CTRL);
+   __raw_writel(0, data->sfrbase + REG_MMU_CFG);

__sysmmu_clk_disable(data);
__master_clk_disable(data);
+}

-   disabled = true;
-   data->pgtable = 0;
-   data->domain = NULL;
-finish:
-   write_unlock_irqrestore(&data->lock, flags);
+static bool __sysmmu_disable(struct sysmmu_drvdata *data)
+{
+   bool disabled;
+   unsigned long flags;
+
+   write_lock_irqsave(&data->lock, flags);
+
+   disabled = set_sysmmu_inactive(data);
+
+   if (disabled) {
+   data->pgtable = 0;
+   data->domain = NULL;
+
+   __sysmmu_disable_nocount(data);

-   if (disabled)
dev_dbg(data->sysmmu, "Disabled\n");
-   else
-   dev_dbg(data->sysmmu, "%d times left to be disabled\n",
+   } else  {
+   dev_dbg(data->sysmmu, "%d times left to disable

Re: [PATCH v11 08/27] iommu/exynos: always use a single clock descriptor

[Tomasz Figa]

Hi KyongHo,

On 14.03.2014 06:05, Cho KyongHo wrote:

System MMU driver is changed to control only a single instance of
System MMU at a time. Since a single instance of System MMU has only
a single clock descriptor for its clock gating, there is no need to
obtain two or more clock descriptors.



This patch does much more than just making the driver use a single clock 
descriptor. Please update the subject and description accordingly.



Signed-off-by: Cho KyongHo 
---
  drivers/iommu/exynos-iommu.c |  223 ++
  1 file changed, 72 insertions(+), 151 deletions(-)

diff --git a/drivers/iommu/exynos-iommu.c b/drivers/iommu/exynos-iommu.c
index 8dc7031..a4499b2 100644
--- a/drivers/iommu/exynos-iommu.c
+++ b/drivers/iommu/exynos-iommu.c
@@ -171,9 +171,8 @@ struct sysmmu_drvdata {
struct device *sysmmu;  /* System MMU's device descriptor */
struct device *dev; /* Owner of system MMU */
char *dbgname;
-   int nsfrs;
-   void __iomem **sfrbases;
-   struct clk *clk[2];
+   void __iomem *sfrbase;
+   struct clk *clk;
int activations;
rwlock_t lock;
struct iommu_domain *domain;
@@ -294,56 +293,39 @@ static irqreturn_t exynos_sysmmu_irq(int irq, void 
*dev_id)
  {
/* SYSMMU is in blocked when interrupt occurred. */
struct sysmmu_drvdata *data = dev_id;
-   struct resource *irqres;
-   struct platform_device *pdev;
enum exynos_sysmmu_inttype itype;
unsigned long addr = -1;
-
-   int i, ret = -ENOSYS;
+   int ret = -ENOSYS;

read_lock(&data->lock);

WARN_ON(!is_sysmmu_active(data));

-   pdev = to_platform_device(data->sysmmu);
-   for (i = 0; i < (pdev->num_resources / 2); i++) {
-   irqres = platform_get_resource(pdev, IORESOURCE_IRQ, i);
-   if (irqres && ((int)irqres->start == irq))
-   break;
-   }
-
-   if (i == pdev->num_resources) {
+   itype = (enum exynos_sysmmu_inttype)
+   __ffs(__raw_readl(data->sfrbase + REG_INT_STATUS));
+   if (WARN_ON(!((itype >= 0) && (itype < SYSMMU_FAULT_UNKNOWN
itype = SYSMMU_FAULT_UNKNOWN;
-   } else {
-   itype = (enum exynos_sysmmu_inttype)
-   __ffs(__raw_readl(data->sfrbases[i] + REG_INT_STATUS));
-   if (WARN_ON(!((itype >= 0) && (itype < SYSMMU_FAULT_UNKNOWN
-   itype = SYSMMU_FAULT_UNKNOWN;
-   else
-   addr = __raw_readl(
-   data->sfrbases[i] + fault_reg_offset[itype]);
-   }
+   else
+   addr = __raw_readl(data->sfrbase + fault_reg_offset[itype]);

if (data->domain)
-   ret = report_iommu_fault(data->domain, data->dev,
-   addr, itype);
+   ret = report_iommu_fault(data->domain, data->dev, addr, itype);

if ((ret == -ENOSYS) && data->fault_handler) {
unsigned long base = data->pgtable;
if (itype != SYSMMU_FAULT_UNKNOWN)
-   base = __raw_readl(
-   data->sfrbases[i] + REG_PT_BASE_ADDR);
+   base = __raw_readl(data->sfrbase + REG_PT_BASE_ADDR);
ret = data->fault_handler(itype, base, addr);
}

if (!ret && (itype != SYSMMU_FAULT_UNKNOWN))
-   __raw_writel(1 << itype, data->sfrbases[i] + REG_INT_CLEAR);
+   __raw_writel(1 << itype, data->sfrbase + REG_INT_CLEAR);
else
dev_dbg(data->sysmmu, "(%s) %s is not handled.\n",
data->dbgname, sysmmu_fault_name[itype]);

if (itype != SYSMMU_FAULT_UNKNOWN)
-   sysmmu_unblock(data->sfrbases[i]);
+   sysmmu_unblock(data->sfrbase);

read_unlock(&data->lock);

@@ -354,20 +336,16 @@ static bool __exynos_sysmmu_disable(struct sysmmu_drvdata 
*data)
  {
unsigned long flags;
bool disabled = false;
-   int i;

write_lock_irqsave(&data->lock, flags);

if (!set_sysmmu_inactive(data))
goto finish;

-   for (i = 0; i < data->nsfrs; i++)
-   __raw_writel(CTRL_DISABLE, data->sfrbases[i] + REG_MMU_CTRL);
+   __raw_writel(CTRL_DISABLE, data->sfrbase + REG_MMU_CTRL);

-   if (data->clk[1])
-   clk_disable(data->clk[1]);
-   if (data->clk[0])
-   clk_disable(data->clk[0]);
+   if (data->clk)


I know this is already in the driver, but checking (struct clk *) for 
NULL is incorrect. NULL is a valid pointer for dummy clocks on platforms 
which do not provide particular clocks, to make this transparent to 
drivers. IS_ERR() should be used to check whether a clock pointer is valid.


This patch is changing all the clock code anyway, so this change could 
be squashed into it to fix this.



+   clk_di

Re: [PATCH v11 10/27] iommu/exynos: use managed device helper functions

[Tomasz Figa]

Hi KyongHo,

On 14.03.2014 06:05, Cho KyongHo wrote:

This patch uses managed device helper functions in the probe().

Signed-off-by: Cho KyongHo 
---
  drivers/iommu/exynos-iommu.c |   64 +-
  1 file changed, 26 insertions(+), 38 deletions(-)

diff --git a/drivers/iommu/exynos-iommu.c b/drivers/iommu/exynos-iommu.c
index 36e6b73..33b424d 100644
--- a/drivers/iommu/exynos-iommu.c
+++ b/drivers/iommu/exynos-iommu.c
@@ -499,51 +499,48 @@ void exynos_sysmmu_tlb_invalidate(struct device *dev)

  static int exynos_sysmmu_probe(struct platform_device *pdev)
  {
-   int ret;
+   int irq, ret;
struct device *dev = &pdev->dev;
struct sysmmu_drvdata *data;
struct resource *res;

-   data = kzalloc(sizeof(*data), GFP_KERNEL);
-   if (!data) {
-   dev_dbg(dev, "Not enough memory\n");
-   ret = -ENOMEM;
-   goto err_alloc;
-   }
+   data = devm_kzalloc(dev, sizeof(*data), GFP_KERNEL);
+   if (!data)
+   return -ENOMEM;

res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
if (!res) {
-   dev_dbg(dev, "Unable to find IOMEM region\n");
-   ret = -ENOENT;
-   goto err_init;
+   dev_err(dev, "Unable to find IOMEM region\n");
+   return -ENOENT;
}


No need to check for error and print message, because 
devm_ioremap_resource() already checks the passed resource and handles 
error cases.


Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v11 13/27] iommu/exynos: support for device tree

[Tomasz Figa]

Hi KyongHo,

On 14.03.2014 06:09, Cho KyongHo wrote:

This commit adds device tree support for System MMU.

Signed-off-by: Cho KyongHo 
---
  drivers/iommu/Kconfig|5 ++---
  drivers/iommu/exynos-iommu.c |   21 +
  2 files changed, 19 insertions(+), 7 deletions(-)

diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig
index df56e4c..22af807 100644
--- a/drivers/iommu/Kconfig
+++ b/drivers/iommu/Kconfig
@@ -178,16 +178,15 @@ config TEGRA_IOMMU_SMMU

  config EXYNOS_IOMMU
bool "Exynos IOMMU Support"
-   depends on ARCH_EXYNOS && EXYNOS_DEV_SYSMMU
+   depends on ARCH_EXYNOS
select IOMMU_API
+   default n
help
  Support for the IOMMU(System MMU) of Samsung Exynos application


nit: There should be a white space before the opening parenthesis.


  processor family. This enables H/W multimedia accellerators to see


typo: s/accellerators/accelerators/


  non-linear physical memory chunks as a linear memory in their
  address spaces

- If unsure, say N here.
-
  config EXYNOS_IOMMU_DEBUG
bool "Debugging log for Exynos IOMMU"
depends on EXYNOS_IOMMU
diff --git a/drivers/iommu/exynos-iommu.c b/drivers/iommu/exynos-iommu.c
index 33b424d..34feb04 100644
--- a/drivers/iommu/exynos-iommu.c
+++ b/drivers/iommu/exynos-iommu.c
@@ -26,6 +26,7 @@
  #include 
  #include 
  #include 
+#include 

  #include 
  #include 
@@ -497,7 +498,7 @@ void exynos_sysmmu_tlb_invalidate(struct device *dev)
read_unlock_irqrestore(&data->lock, flags);
  }

-static int exynos_sysmmu_probe(struct platform_device *pdev)
+static int __init exynos_sysmmu_probe(struct platform_device *pdev)
  {
int irq, ret;
struct device *dev = &pdev->dev;
@@ -557,11 +558,23 @@ static int exynos_sysmmu_probe(struct platform_device 
*pdev)
return 0;
  }

-static struct platform_driver exynos_sysmmu_driver = {
-   .probe  = exynos_sysmmu_probe,
-   .driver = {
+#ifdef CONFIG_OF
+static struct of_device_id sysmmu_of_match[] __initconst = {
+   { .compatible   = "samsung,sysmmu-v1", },
+   { .compatible   = "samsung,sysmmu-v2", },
+   { .compatible   = "samsung,sysmmu-v3.1", },
+   { .compatible   = "samsung,sysmmu-v3.2", },
+   { .compatible   = "samsung,sysmmu-v3.3", },


Do you need all these compatible strings? I mean, are there any 
implementation differences that can't be identified by reading IP 
registers, such as REG_MMU_VERSION?


Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v11 15/27] iommu/exynos: use convenient macro to handle gate clocks

[Tomasz Figa]

Hi KyongHo,

On 14.03.2014 06:09, Cho KyongHo wrote:

exynos-iommu driver must care about master H/W's gate clock as well as
System MMU's gate clock. To enhance readability of the source code,
macros to gate/ungate those clocks are defined.

Signed-off-by: Cho KyongHo 
---
  drivers/iommu/exynos-iommu.c |   34 ++
  1 file changed, 22 insertions(+), 12 deletions(-)

diff --git a/drivers/iommu/exynos-iommu.c b/drivers/iommu/exynos-iommu.c
index 71e77f1..cef62d0 100644
--- a/drivers/iommu/exynos-iommu.c
+++ b/drivers/iommu/exynos-iommu.c
@@ -101,6 +101,16 @@
  #define REG_PB1_SADDR 0x054
  #define REG_PB1_EADDR 0x058

+#define __clk_gate_ctrl(data, clk, en) do {\
+   if (data->clk)   \
+   clk_##en##able(data->clk);   \
+   } while (0)
+
+#define __sysmmu_clk_enable(data)  __clk_gate_ctrl(data, clk, en)
+#define __sysmmu_clk_disable(data) __clk_gate_ctrl(data, clk, dis)
+#define __master_clk_enable(data)  __clk_gate_ctrl(data, clk_master, en)
+#define __master_clk_disable(data) __clk_gate_ctrl(data, clk_master, dis)
+


I'd say that such macros only obfuscate code, without any gains, as you 
can see in diffstat - this patch adds more lines than it removes.


Please drop this change.

Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v11 19/27] iommu/exynos: add support for power management subsystems.

[Tomasz Figa]

Hi KyongHo,

On 14.03.2014 06:10, Cho KyongHo wrote:

This adds support for Suspend to RAM and Runtime Power Management.

Since System MMU is located in the same local power domain of its
master H/W, System MMU must be initialized before it is working if
its power domain was ever turned off. TLB invalidation according to
unmapping on page tables must also be performed while power domain is
turned on.

This patch ensures that resume and runtime_resume(restore_state)
functions in this driver is called before the calls to resume and
runtime_resume callback functions in the drivers of master H/Ws.
Likewise, suspend and runtime_suspend(save_state) functions in this
driver is called after the calls to suspend and runtime_suspend in the
drivers of master H/Ws.

In order to get benefit of this support, the master H/W and its System
MMU must resides in the same power domain in terms of Linux kernel. If
a master H/W does not use generic I/O power domain, its driver must
call iommu_attach_device() after its local power domain is turned on,
iommu_detach_device before turned off.

Signed-off-by: Cho KyongHo 
---
  drivers/iommu/exynos-iommu.c |  220 ++
  1 file changed, 201 insertions(+), 19 deletions(-)

diff --git a/drivers/iommu/exynos-iommu.c b/drivers/iommu/exynos-iommu.c
index 9037da0..84ba29a 100644
--- a/drivers/iommu/exynos-iommu.c
+++ b/drivers/iommu/exynos-iommu.c
@@ -28,6 +28,7 @@
  #include 
  #include 
  #include 
+#include 
  #include 

  #include 
@@ -203,6 +204,7 @@ struct sysmmu_drvdata {
int activations;
rwlock_t lock;
struct iommu_domain *domain;
+   bool runtime_active;
unsigned long pgtable;
  };

@@ -388,7 +390,8 @@ static bool __sysmmu_disable(struct sysmmu_drvdata *data)
data->pgtable = 0;
data->domain = NULL;

-   __sysmmu_disable_nocount(data);
+   if (data->runtime_active)
+   __sysmmu_disable_nocount(data);

dev_dbg(data->sysmmu, "Disabled\n");
} else  {
@@ -449,7 +452,8 @@ static int __sysmmu_enable(struct sysmmu_drvdata *data,
data->pgtable = pgtable;
data->domain = domain;

-   __sysmmu_enable_nocount(data);
+   if (data->runtime_active)
+   __sysmmu_enable_nocount(data);

dev_dbg(data->sysmmu, "Enabled\n");
} else {
@@ -534,13 +538,11 @@ static void sysmmu_tlb_invalidate_entry(struct device 
*dev, unsigned long iova,
data = dev_get_drvdata(owner->sysmmu);

read_lock_irqsave(&data->lock, flags);
-   if (is_sysmmu_active(data)) {
-   unsigned int maj;
+   if (is_sysmmu_active(data) && data->runtime_active) {
unsigned int num_inv = 1;

__master_clk_enable(data);

-   maj = __raw_readl(data->sfrbase + REG_MMU_VERSION);
/*
 * L2TLB invalidation required
 * 4KB page: 1 invalidation
@@ -551,7 +553,7 @@ static void sysmmu_tlb_invalidate_entry(struct device *dev, 
unsigned long iova,
 * 1MB page can be cached in one of all sets.
 * 64KB page can be one of 16 consecutive sets.
 */
-   if ((maj >> 28) == 2) /* major version number */
+   if (__sysmmu_version(data, NULL) == 2) /* major version number 
*/
num_inv = min_t(unsigned int, size / PAGE_SIZE, 64);

if (sysmmu_block(data->sfrbase)) {
@@ -576,7 +578,7 @@ void exynos_sysmmu_tlb_invalidate(struct device *dev)
data = dev_get_drvdata(owner->sysmmu);

read_lock_irqsave(&data->lock, flags);
-   if (is_sysmmu_active(data)) {
+   if (is_sysmmu_active(data) && data->runtime_active) {
__master_clk_enable(data);
if (sysmmu_block(data->sfrbase)) {
__sysmmu_tlb_invalidate(data->sfrbase);
@@ -677,11 +679,40 @@ static int __init exynos_sysmmu_probe(struct 
platform_device *pdev)
platform_set_drvdata(pdev, data);

pm_runtime_enable(dev);
+   data->runtime_active = !pm_runtime_enabled(dev);


Hmm, this seems to be a bit misleading. The field is named 
runtime_active, but the assignment makes it true if PM runtime is _not_ 
enabled (i.e. inactive). Is this correct?




dev_dbg(dev, "Probed and initialized\n");
return 0;
  }

+#ifdef CONFIG_PM_SLEEP
+static int sysmmu_suspend(struct device *dev)
+{
+   struct sysmmu_drvdata *data = dev_get_drvdata(dev);
+   unsigned long flags;
+   read_lock_irqsave(&data->lock, flags);
+   if (is_sysmmu_active(data) &&
+   (!pm_runtime_enabled(dev) || data->runtime_active))
+   __sysmmu_disable_nocount(data);
+   read_unlock_irqrestore(&data->lock, flags);
+   return 0;
+}
+
+static int sysmmu_resume(struct device *dev)
+{
+   struct sysmmu_drvdata *data

Re: [PATCH v11 20/27] iommu/exynos: allow having multiple System MMUs for a master H/W

[Tomasz Figa]

Hi KyongHo,

On 14.03.2014 06:10, Cho KyongHo wrote:

Some master device descriptor like fimc-is which is an abstraction
of very complex H/W may have multiple System MMUs. For those devices,
the design of the link between System MMU and its master H/W is needed
to be reconsidered.

A link structure, sysmmu_list_data is introduced that provides a link
to master H/W and that has a pointer to the device descriptor of a
System MMU. Given a device descriptor of a master H/W, it is possible
to traverse all System MMUs that must be controlled along with the
master H/W.


NAK.

A device driver should handle particular hardware instances separately, 
without abstracting a virtual hardware instance consisting of multiple 
physical ones.


If such abstraction is needed, it should be done above the exynos-iommu 
driver, e.g. by something like iommu-composite driver that would 
aggregate several IOMMUs. Keep in mind that such IOMMUs in a group could 
be different, e.g. different Exynos SysMMU versions or even completely 
different IPs handled by different drivers.


Still, I don't think there is a real need for such abstraction. Instead, 
related drivers shall be fixed to properly handle multiple memory 
masters and their IOMMUs.


Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v11 15/27] iommu/exynos: use convenient macro to handle gate clocks

[Tomasz Figa]

On 18.03.2014 12:18, Sachin Kamat wrote:

On 18 March 2014 16:33, Cho KyongHo  wrote:

On Fri, 14 Mar 2014 22:27:59 +0530, Sachin Kamat wrote:

  Hi KyongHo,

On 14 March 2014 19:13, Tomasz Figa  wrote:

Hi KyongHo,


On 14.03.2014 06:09, Cho KyongHo wrote:


exynos-iommu driver must care about master H/W's gate clock as well as
System MMU's gate clock. To enhance readability of the source code,
macros to gate/ungate those clocks are defined.

Signed-off-by: Cho KyongHo 
---
   drivers/iommu/exynos-iommu.c |   34 ++
   1 file changed, 22 insertions(+), 12 deletions(-)

diff --git a/drivers/iommu/exynos-iommu.c b/drivers/iommu/exynos-iommu.c
index 71e77f1..cef62d0 100644
--- a/drivers/iommu/exynos-iommu.c
+++ b/drivers/iommu/exynos-iommu.c
@@ -101,6 +101,16 @@
   #define REG_PB1_SADDR 0x054
   #define REG_PB1_EADDR 0x058

+#define __clk_gate_ctrl(data, clk, en) do {\
+   if (data->clk)  \
+   clk_##en##able(data->clk);  \
+   } while (0)
+
+#define __sysmmu_clk_enable(data)  __clk_gate_ctrl(data, clk, en)
+#define __sysmmu_clk_disable(data) __clk_gate_ctrl(data, clk, dis)
+#define __master_clk_enable(data)  __clk_gate_ctrl(data, clk_master,
en)
+#define __master_clk_disable(data) __clk_gate_ctrl(data, clk_master,
dis)
+



I'd say that such macros only obfuscate code, without any gains, as you can
see in diffstat - this patch adds more lines than it removes.

Please drop this change.


  I agree with Tomasz here.



Are you concerning about using macros or more insertions than deletions?


It is just making the code more difficult to read and understand.


Especially when hiding accesses to struct fields inside and doing fancy 
stuff like concatenations.


Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v11 20/27] iommu/exynos: allow having multiple System MMUs for a master H/W

[Tomasz Figa]

On 18.03.2014 14:01, Cho KyongHo wrote:

On Fri, 14 Mar 2014 17:12:03 +0100, Tomasz Figa wrote:

Hi KyongHo,

On 14.03.2014 06:10, Cho KyongHo wrote:

Some master device descriptor like fimc-is which is an abstraction
of very complex H/W may have multiple System MMUs. For those devices,
the design of the link between System MMU and its master H/W is needed
to be reconsidered.

A link structure, sysmmu_list_data is introduced that provides a link
to master H/W and that has a pointer to the device descriptor of a
System MMU. Given a device descriptor of a master H/W, it is possible
to traverse all System MMUs that must be controlled along with the
master H/W.


NAK.

A device driver should handle particular hardware instances separately,
without abstracting a virtual hardware instance consisting of multiple
physical ones.

If such abstraction is needed, it should be done above the exynos-iommu
driver, e.g. by something like iommu-composite driver that would
aggregate several IOMMUs. Keep in mind that such IOMMUs in a group could
be different, e.g. different Exynos SysMMU versions or even completely
different IPs handled by different drivers.

Still, I don't think there is a real need for such abstraction. Instead,
related drivers shall be fixed to properly handle multiple memory
masters and their IOMMUs.



G2D, Scalers and FIMD of Exynos5420 has 2 System MMUs while aother SoC like
Exynos5250 does not.

I don't understand why you are negative to this approach.
This is the simplest than the others.

Let me show you an example.
FIMC-IS driver just controls MCU in FIMC-IS subsystem and the firmware of
the MCU controls all other peripherals in the subsystem. Each peripherals
have their own System MMU. Moreover, the configuration of the peripherals
varies according to the SoCs.

If System MMU driver accepts multiple masters, everything is done in DT.
But I worry that it is not easy if System MMU driver does not support
multiple masters.


I believe I have stated enough reasons why this kind of implementation 
is bad. I'm not going to waste time repeating myself.


Your concerns presented above are valid, however they are not related to 
what is wrong with this patch. I have given you two proper ways to 
handle this, none should be forced upon particular IOMMU master drivers 
- their authors should have the chance to select the method that works 
best for them.


Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v11 17/27] iommu/exynos: remove calls to Runtime PM API functions

[Tomasz Figa]

On 18.03.2014 10:56, Cho KyongHo wrote:

On Fri, 14 Mar 2014 13:59:00 +0100, Tomasz Figa wrote:

Hi KyongHo,

On 14.03.2014 06:08, Cho KyongHo wrote:


[snip]


-static bool __exynos_sysmmu_disable(struct sysmmu_drvdata *data)
+static void __sysmmu_disable_nocount(struct sysmmu_drvdata *data)


If you are changing the names anyway, it would be probably a good idea
to reduce code obfuscation a bit and drop the underscores from
beginnings of function names. Also I'd suggest keeping the "exynos_" prefix.


Thanks for the suggestion.
__exynos_sysmmu_disable is splitted into 2 functions: __sysmmu_disable
and __sysmmu_disable_nocount.
I agree with you that it is good idea to reduce code obfuscation but
I don't think dropping beginning underscores of function names reduces
obfuscation.



Well, if you are ending up with a function like 
__sysmmu_enable_nocount() below with every line starting with two 
underscores, do you think this improves code readability?


Of course this is a minor issue, but let's keep some code quality level 
in Linux kernel.





   {
-   unsigned long flags;
-   bool disabled = false;
-
-   write_lock_irqsave(&data->lock, flags);


[snip]

Here's the function mentioned above:


+
+static void __sysmmu_enable_nocount(struct sysmmu_drvdata *data)
+{
+   __master_clk_enable(data);
+   __sysmmu_clk_enable(data);
+
+   __raw_writel(CTRL_BLOCK, data->sfrbase + REG_MMU_CTRL);
+
+   __sysmmu_init_config(data);
+
+   __sysmmu_set_ptbase(data->sfrbase, data->pgtable);
+
+   __raw_writel(CTRL_ENABLE, data->sfrbase + REG_MMU_CTRL);
+
+   __master_clk_disable(data);
+}
+


[snip]



@@ -629,7 +700,7 @@ err_pgtable:
   static void exynos_iommu_domain_destroy(struct iommu_domain *domain)
   {
struct exynos_iommu_domain *priv = domain->priv;
-   struct sysmmu_drvdata *data;
+   struct exynos_iommu_owner *owner;
unsigned long flags;
int i;

@@ -637,11 +708,14 @@ static void exynos_iommu_domain_destroy(struct 
iommu_domain *domain)

spin_lock_irqsave(&priv->lock, flags);

-   list_for_each_entry(data, &priv->clients, node) {
-   while (!exynos_sysmmu_disable(data->dev))
+   list_for_each_entry(owner, &priv->clients, client) {
+   while (!exynos_sysmmu_disable(owner->dev))
; /* until System MMU is actually disabled */


What about using list_for_each_entry_safe() and calling list_del_init()
here directly?



That require another variable to be defined.


Is it a problem?


I just wanted to avoid that because I think it is prettier.
Moreover, list_del_init() below the empty while() clause may make
the source code readers misunderstood..


This raises another question, why the loop above is even needed. 
exynos_sysmmu_disable() should make sure that SYSMMU is actually 
disabled, without any need for looping like this.



}

+   while (!list_empty(&priv->clients))
+   list_del_init(priv->clients.next);
+
spin_unlock_irqrestore(&priv->lock, flags);

for (i = 0; i < NUM_LV1ENTRIES; i++)


[snip]


+static int sysmmu_hook_driver_register(struct notifier_block *nb,
+   unsigned long val,
+   void *p)
+{
+   struct device *dev = p;
+
+   switch (val) {
+   case BUS_NOTIFY_BIND_DRIVER:
+   {
+   struct exynos_iommu_owner *owner;


Please move this variable to the top of the function and drop the braces
around case blocks.


I don't think it is required because this function is modified
by the following patches.


OK, if so, and similar issue is not present after further patches.






+
+   /* No System MMU assigned. See exynos_sysmmu_probe(). */
+   if (dev->archdata.iommu == NULL)
+   break;


This looks strange... (see below)

Also this looks racy. There are no guarantees about device probing
order, so you may end up with master devices being probed before the
IOMMUs. Deferred probing should be used to handle this correctly.


System MMU driver must be probed earlier than the drivers of master devices
because the drivers may want to use System MMU for their initial task.


As I said, there are no guarantees about platform device probe order in 
Linux kernel. Code must be designed to check whether required 
dependencies are met and if not, deferred probing must be used.





+
+   owner = devm_kzalloc(dev, sizeof(*owner), GFP_KERNEL);
+   if (!owner) {
+   dev_err(dev, "No Memory for exynos_iommu_owner\n");
+   return -ENOMEM;
+   }
+
+   owner->dev = dev;
+   INIT_LIST_HEAD(&owner->client);
+   owner->sysmmu = dev->archdata.iommu;
+
+   dev-&g

Re: [PATCH v11 10/27] iommu/exynos: use managed device helper functions

[Tomasz Figa]

On 18.03.2014 11:38, Cho KyongHo wrote:

On Fri, 14 Mar 2014 14:28:36 +0100, Tomasz Figa wrote:

Hi KyongHo,

On 14.03.2014 06:05, Cho KyongHo wrote:

This patch uses managed device helper functions in the probe().

Signed-off-by: Cho KyongHo 
---
   drivers/iommu/exynos-iommu.c |   64 
+-
   1 file changed, 26 insertions(+), 38 deletions(-)

diff --git a/drivers/iommu/exynos-iommu.c b/drivers/iommu/exynos-iommu.c
index 36e6b73..33b424d 100644
--- a/drivers/iommu/exynos-iommu.c
+++ b/drivers/iommu/exynos-iommu.c
@@ -499,51 +499,48 @@ void exynos_sysmmu_tlb_invalidate(struct device *dev)

   static int exynos_sysmmu_probe(struct platform_device *pdev)
   {
-   int ret;
+   int irq, ret;
struct device *dev = &pdev->dev;
struct sysmmu_drvdata *data;
struct resource *res;

-   data = kzalloc(sizeof(*data), GFP_KERNEL);
-   if (!data) {
-   dev_dbg(dev, "Not enough memory\n");
-   ret = -ENOMEM;
-   goto err_alloc;
-   }
+   data = devm_kzalloc(dev, sizeof(*data), GFP_KERNEL);
+   if (!data)
+   return -ENOMEM;

res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
if (!res) {
-   dev_dbg(dev, "Unable to find IOMEM region\n");
-   ret = -ENOENT;
-   goto err_init;
+   dev_err(dev, "Unable to find IOMEM region\n");
+   return -ENOENT;
}


No need to check for error and print message, because
devm_ioremap_resource() already checks the passed resource and handles
error cases.



Yes but devm_ioremap_resource() just tells that the given 'res' is not
correct. I think the message in the driver is more informative.


The common practice used in Linux kernel is to not duplicate such 
messages. It is obvious that devm_ioremap_resource() printing such 
message is related to an IOMEM resource anyway, as you can't used it 
with other types of resources.


Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v11 10/27] iommu/exynos: use managed device helper functions

[Tomasz Figa]

On 18.03.2014 12:09, Cho KyongHo wrote:

On Fri, 14 Mar 2014 20:52:43 +0530, Sachin Kamat wrote:

Hi KyongHo,

On 14 March 2014 10:35, Cho KyongHo  wrote:

This patch uses managed device helper functions in the probe().

Signed-off-by: Cho KyongHo 
---

[snip]


+   data->clk = devm_clk_get(dev, "sysmmu");
+   if (IS_ERR(data->clk)) {
+   dev_info(dev, "No gate clock found!\n");
+   data->clk = NULL;
+   }


Why aren't you returning from here upon error?


It is for the case of a System MMU which does not need clock gating.



Are there really such cases?

Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v11 13/27] iommu/exynos: support for device tree

[Tomasz Figa]

On 18.03.2014 11:52, Cho KyongHo wrote:

On Fri, 14 Mar 2014 14:39:33 +0100, Tomasz Figa wrote:

@@ -557,11 +558,23 @@ static int exynos_sysmmu_probe(struct platform_device 
*pdev)
return 0;
   }

-static struct platform_driver exynos_sysmmu_driver = {
-   .probe  = exynos_sysmmu_probe,
-   .driver = {
+#ifdef CONFIG_OF
+static struct of_device_id sysmmu_of_match[] __initconst = {
+   { .compatible   = "samsung,sysmmu-v1", },
+   { .compatible   = "samsung,sysmmu-v2", },
+   { .compatible   = "samsung,sysmmu-v3.1", },
+   { .compatible   = "samsung,sysmmu-v3.2", },
+   { .compatible   = "samsung,sysmmu-v3.3", },


Do you need all these compatible strings? I mean, are there any
implementation differences that can't be identified by reading IP
registers, such as REG_MMU_VERSION?



Unfortunately, there is a SoC which overrides REG_MMU_VERSION with
a value for RTL designers and it is not related to System MMU
versions.


OK.

What about having a generic compatible string for Samsung SysMMU then, 
but an additional property that can override the version to account for 
such brokenness? If not provided, the version would be read from 
REG_MMU_VERSION.


Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v11 19/27] iommu/exynos: add support for power management subsystems.

[Tomasz Figa]

On 18.03.2014 12:23, Cho KyongHo wrote:

On Fri, 14 Mar 2014 17:07:53 +0100, Tomasz Figa wrote:

Hi KyongHo,

On 14.03.2014 06:10, Cho KyongHo wrote:


[snip]


@@ -677,11 +679,40 @@ static int __init exynos_sysmmu_probe(struct 
platform_device *pdev)
platform_set_drvdata(pdev, data);

pm_runtime_enable(dev);
+   data->runtime_active = !pm_runtime_enabled(dev);


Hmm, this seems to be a bit misleading. The field is named
runtime_active, but the assignment makes it true if PM runtime is _not_
enabled (i.e. inactive). Is this correct?



I agree that it may lead misunderstood.
data->runtime_active actually indicates if electric power is asserted
to the System MMU. pm_runtime_enable() call must enable runtime pm
for the given device. If runtime pm is not enabled although pm_runtime_enable()
is called, CONFIG_PM_RUNTIME is not configured.

Actually, it is replacible with
if (IS_ENABLED(CONFIG_PM_RUNTIME))
 data->runtime_active = true;


I would keep it as !pm_runtime_enabled(dev), but rename the field to 
something more meaningful, like data->is_powered_on.


Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v11 10/27] iommu/exynos: use managed device helper functions

[Tomasz Figa]

On 19.03.2014 10:01, Sachin Kamat wrote:

On 19 March 2014 14:29, Cho KyongHo  wrote:

On Tue, 18 Mar 2014 16:14:53 +0100, Tomasz Figa wrote:

On 18.03.2014 12:09, Cho KyongHo wrote:

On Fri, 14 Mar 2014 20:52:43 +0530, Sachin Kamat wrote:

Hi KyongHo,

On 14 March 2014 10:35, Cho KyongHo  wrote:

This patch uses managed device helper functions in the probe().

Signed-off-by: Cho KyongHo 
---

[snip]


+   data->clk = devm_clk_get(dev, "sysmmu");
+   if (IS_ERR(data->clk)) {
+   dev_info(dev, "No gate clock found!\n");
+   data->clk = NULL;
+   }


Why aren't you returning from here upon error?


It is for the case of a System MMU which does not need clock gating.



Are there really such cases?



Yes.
Especially in the case of initial stage of new SoC development.

I have experianced some software workaround for H/W restriction
needs prevention of clock gating for some devices.


So aren't these basically some exceptions/hacks rather than the usual way
of functioning of the device?



This actually raises a good question, whether we really need to support 
such early development SoC versions in mainline.


Another thing is that if you need to assure that a clock is ungated, you 
must acquire it and prepare_enable explicitly, so I don't think this 
kind of handling is correct.


Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v11 17/27] iommu/exynos: remove calls to Runtime PM API functions

[Tomasz Figa]

On 19.03.2014 02:03, Cho KyongHo wrote:

On Tue, 18 Mar 2014 16:09:50 +0100, Tomasz Figa wrote:

On 18.03.2014 10:56, Cho KyongHo wrote:

On Fri, 14 Mar 2014 13:59:00 +0100, Tomasz Figa wrote:

Hi KyongHo,

On 14.03.2014 06:08, Cho KyongHo wrote:


[snip]


@@ -637,11 +708,14 @@ static void exynos_iommu_domain_destroy(struct 
iommu_domain *domain)

spin_lock_irqsave(&priv->lock, flags);

-   list_for_each_entry(data, &priv->clients, node) {
-   while (!exynos_sysmmu_disable(data->dev))
+   list_for_each_entry(owner, &priv->clients, client) {
+   while (!exynos_sysmmu_disable(owner->dev))
; /* until System MMU is actually disabled */


What about using list_for_each_entry_safe() and calling list_del_init()
here directly?



That require another variable to be defined.


Is it a problem?


That is not a problem.
But I think using list_for_each_entry() is not a problem likewise.


I just wanted to avoid that because I think it is prettier.
Moreover, list_del_init() below the empty while() clause may make
the source code readers misunderstood..


This raises another question, why the loop above is even needed.
exynos_sysmmu_disable() should make sure that SYSMMU is actually
disabled, without any need for looping like this.


Some driver needs enabling sysmmu to be counted due to its complex structure.
It can be also removed by the driver with an extra effort
but the reality is important.


My comment was not about the need to have reference counting, as this is 
a common design pattern, but rather that there should be a low level 
power control function, which simply disables the IOMMU, without the 
need to loop through existing references.


Actually there is already such function - __sysmmu_disable_nocount() - 
so the code could look like:


list_for_each_entry_safe(owner, n, &priv->clients, client) {
__sysmmu_disable_nocount(owner->dev);
list_del_init(&owner->client);
}

As for reference counting in this use case in general, as far as I'm 
looking through the whole driver code, there is no other usage of this 
reference counting than exynos_iommu_attach_device() and 
exynos_iommu_detach_device(). Assuming that there is just a single 
master for each SYSMMU, I don't see why reference counting is even 
needed, as you can't bind a device to IOMMU multiple times.



Device driver is not only for the scholarship but also for the real use.


Huh? I'm not sure what kind of comment is this.




}

+   while (!list_empty(&priv->clients))
+   list_del_init(priv->clients.next);
+
spin_unlock_irqrestore(&priv->lock, flags);

for (i = 0; i < NUM_LV1ENTRIES; i++)


[snip]


+static int sysmmu_hook_driver_register(struct notifier_block *nb,
+   unsigned long val,
+   void *p)
+{
+   struct device *dev = p;
+
+   switch (val) {
+   case BUS_NOTIFY_BIND_DRIVER:
+   {
+   struct exynos_iommu_owner *owner;


Please move this variable to the top of the function and drop the braces
around case blocks.


I don't think it is required because this function is modified
by the following patches.


OK, if so, and similar issue is not present after further patches.






+
+   /* No System MMU assigned. See exynos_sysmmu_probe(). */
+   if (dev->archdata.iommu == NULL)
+   break;


This looks strange... (see below)

Also this looks racy. There are no guarantees about device probing
order, so you may end up with master devices being probed before the
IOMMUs. Deferred probing should be used to handle this correctly.


System MMU driver must be probed earlier than the drivers of master devices
because the drivers may want to use System MMU for their initial task.


As I said, there are no guarantees about platform device probe order in
Linux kernel. Code must be designed to check whether required
dependencies are met and if not, deferred probing must be used.



I told that System MMU driver must be probed earlier.
That's why exynos_iommu_init() is called by subsys_initcall level.
I also noticed that it must be promoted to arch_initcall for some driver.



No. Proper Linux drivers must support deferred probing mechanism and 
there should be no assumptions about probing orders. Using other 
initcall level than module_initcall for particular drivers is strongly 
discouraged.





+
+   owner = devm_kzalloc(dev, sizeof(*owner), GFP_KERNEL);
+   if (!owner) {
+   dev_err(dev, "No Memory for exynos_iommu_owner\n");
+   return -ENOMEM;
+   }
+
+   owner->dev = dev;
+   INIT_LIST_HEAD(&owner->client);
+   owner->sysmmu =

Re: [PATCH v11 20/27] iommu/exynos: allow having multiple System MMUs for a master H/W

[Tomasz Figa]

On 19.03.2014 01:39, Cho KyongHo wrote:

On Tue, 18 Mar 2014 15:26:48 +0100, Tomasz Figa wrote:



On 18.03.2014 14:01, Cho KyongHo wrote:

On Fri, 14 Mar 2014 17:12:03 +0100, Tomasz Figa wrote:

Hi KyongHo,

On 14.03.2014 06:10, Cho KyongHo wrote:

Some master device descriptor like fimc-is which is an abstraction
of very complex H/W may have multiple System MMUs. For those devices,
the design of the link between System MMU and its master H/W is needed
to be reconsidered.

A link structure, sysmmu_list_data is introduced that provides a link
to master H/W and that has a pointer to the device descriptor of a
System MMU. Given a device descriptor of a master H/W, it is possible
to traverse all System MMUs that must be controlled along with the
master H/W.


NAK.

A device driver should handle particular hardware instances separately,
without abstracting a virtual hardware instance consisting of multiple
physical ones.

If such abstraction is needed, it should be done above the exynos-iommu
driver, e.g. by something like iommu-composite driver that would
aggregate several IOMMUs. Keep in mind that such IOMMUs in a group could
be different, e.g. different Exynos SysMMU versions or even completely
different IPs handled by different drivers.

Still, I don't think there is a real need for such abstraction. Instead,
related drivers shall be fixed to properly handle multiple memory
masters and their IOMMUs.



G2D, Scalers and FIMD of Exynos5420 has 2 System MMUs while aother SoC like
Exynos5250 does not.

I don't understand why you are negative to this approach.
This is the simplest than the others.

Let me show you an example.
FIMC-IS driver just controls MCU in FIMC-IS subsystem and the firmware of
the MCU controls all other peripherals in the subsystem. Each peripherals
have their own System MMU. Moreover, the configuration of the peripherals
varies according to the SoCs.

If System MMU driver accepts multiple masters, everything is done in DT.
But I worry that it is not easy if System MMU driver does not support
multiple masters.


I believe I have stated enough reasons why this kind of implementation
is bad. I'm not going to waste time repeating myself.

Your concerns presented above are valid, however they are not related to
what is wrong with this patch. I have given you two proper ways to
handle this, none should be forced upon particular IOMMU master drivers
- their authors should have the chance to select the method that works
best for them.



I don't still understand why you think this patch is wrong.
I think this is the best way not to think for all the driver developers
about other things than their business logic.


I agree, but one of the ways I proposed (an iommu-composite layer above 
the IOMMU low level drivers) doesn't add any extra responsibility of 
driver developers.


Moreover, it's this kind of business logic in low level drivers that is 
adding more responsibility, because it introduces additional complexity 
and makes the driver harder to read, maintain and extend in future.




This does not hurt anyone and I think this is good enough.



Well, it is barely good enough. It is a good practice to make a low 
level driver handle a single device instance and this is how Linux 
driver model is designed.


Moreover, a single device tree node _must_ represent a single hardware 
block, so you can't group multiple SysMMUs into a single device tree node.


Furthermore, if you force grouping of SysMMUs into a single virtual one, 
you enforce using the same address space for all masters of some 
particular hardware blocks, while potentially driver developers would 
like to separate them.


A good interface design should not enforce any particular implementation 
and this is what we should stick to in this case as well.



If you want to provide another layer between master device and system mmu
as you mentioned, you do that. This patch does not restrict it.


It does, as mentioned above. With a device tree listing multiple SysMMUs 
as one, you can't separate them.


Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v11 20/27] iommu/exynos: allow having multiple System MMUs for a master H/W

[Tomasz Figa]

On 19.03.2014 14:20, Tomasz Figa wrote:

On 19.03.2014 01:39, Cho KyongHo wrote:

On Tue, 18 Mar 2014 15:26:48 +0100, Tomasz Figa wrote:



On 18.03.2014 14:01, Cho KyongHo wrote:

On Fri, 14 Mar 2014 17:12:03 +0100, Tomasz Figa wrote:

Hi KyongHo,

On 14.03.2014 06:10, Cho KyongHo wrote:

Some master device descriptor like fimc-is which is an abstraction
of very complex H/W may have multiple System MMUs. For those devices,
the design of the link between System MMU and its master H/W is
needed
to be reconsidered.

A link structure, sysmmu_list_data is introduced that provides a link
to master H/W and that has a pointer to the device descriptor of a
System MMU. Given a device descriptor of a master H/W, it is possible
to traverse all System MMUs that must be controlled along with the
master H/W.


NAK.

A device driver should handle particular hardware instances
separately,
without abstracting a virtual hardware instance consisting of multiple
physical ones.

If such abstraction is needed, it should be done above the
exynos-iommu
driver, e.g. by something like iommu-composite driver that would
aggregate several IOMMUs. Keep in mind that such IOMMUs in a group
could
be different, e.g. different Exynos SysMMU versions or even completely
different IPs handled by different drivers.

Still, I don't think there is a real need for such abstraction.
Instead,
related drivers shall be fixed to properly handle multiple memory
masters and their IOMMUs.



G2D, Scalers and FIMD of Exynos5420 has 2 System MMUs while aother
SoC like
Exynos5250 does not.

I don't understand why you are negative to this approach.
This is the simplest than the others.

Let me show you an example.
FIMC-IS driver just controls MCU in FIMC-IS subsystem and the
firmware of
the MCU controls all other peripherals in the subsystem. Each
peripherals
have their own System MMU. Moreover, the configuration of the
peripherals
varies according to the SoCs.

If System MMU driver accepts multiple masters, everything is done in
DT.
But I worry that it is not easy if System MMU driver does not support
multiple masters.


I believe I have stated enough reasons why this kind of implementation
is bad. I'm not going to waste time repeating myself.

Your concerns presented above are valid, however they are not related to
what is wrong with this patch. I have given you two proper ways to
handle this, none should be forced upon particular IOMMU master drivers
- their authors should have the chance to select the method that works
best for them.



I don't still understand why you think this patch is wrong.
I think this is the best way not to think for all the driver developers
about other things than their business logic.


I agree, but one of the ways I proposed (an iommu-composite layer above
the IOMMU low level drivers) doesn't add any extra responsibility of
driver developers.

Moreover, it's this kind of business logic in low level drivers that is
adding more responsibility, because it introduces additional complexity
and makes the driver harder to read, maintain and extend in future.



This does not hurt anyone and I think this is good enough.



Well, it is barely good enough. It is a good practice to make a low
level driver handle a single device instance and this is how Linux
driver model is designed.

Moreover, a single device tree node _must_ represent a single hardware
block, so you can't group multiple SysMMUs into a single device tree node.



OK, you add nodes for single SysMMUs devices which is fine, sorry. I was 
under impression that one kernel device (struct device) corresponds to 
multiple SysMMUs, but this was before your patches, sorry. So one issue 
less, but it's still not good.



Furthermore, if you force grouping of SysMMUs into a single virtual one,
you enforce using the same address space for all masters of some
particular hardware blocks, while potentially driver developers would
like to separate them.


Probably some clarification is needed. Your other patch adds:

sysmmu_fimd0w04: sysmmu@1464 {
compatible = "samsung,sysmmu-v3.3";
reg = <0x1464 0x1000>;
interrupt-parent = <&combiner>;
interrupts = <3 2>;
clock-names = "sysmmu", "master";
clocks = <&clock 422>, <&clock 421>;
samsung,power-domain = <&disp_pd>;
mmu-masters = <&fimd>;
};

sysmmu_fimd0w123: sysmmu@1468 {
compatible = "samsung,sysmmu-v3.3";
reg = <0x1468 0x1000>;
interrupt-parent = <&combiner>;
interrupts = <3 0>;
clock-names = "sysmmu", "master";
clocks = <&clock 423>, <&clock 421>;
samsung,p

Re: [PATCH v11 17/27] iommu/exynos: remove calls to Runtime PM API functions

[Tomasz Figa]

Hi Grant,

On 19.03.2014 18:03, Grant Grundler wrote:

On Wed, Mar 19, 2014 at 6:12 AM, Tomasz Figa  wrote:
...

No. Proper Linux drivers must support deferred probing mechanism and there
should be no assumptions about probing orders. Using other initcall level
than module_initcall for particular drivers is strongly discouraged.


That's true for "end-point" devices. It's not true for
"infrastructure": Memory, CPU, DMA, Interrupt handling, etc. Those
need to be in place before "normal" drivers get called. This SysMMU
driver provides DMA services for "normal" device drivers. Or do I see
that wrong?


Of course using an early initcall level would give you some kind of 
guarantees, but it wouldn't guarantee that someone couldn't lower 
initcall level for some MMU client driver and break the ordering anyway.


As I said, AFAIK the trend is to get rid of ordering by initcalls and 
make sure that drivers can handle missing dependencies properly, even 
for "services" such as DMA, GPIO, clocks and so on, which after all are 
provided by normal drivers like other.




thanks,
grant

ps. I've written IOMMU support for four different IOMMUs on three
operating systems (See drivers/parisc for two linux examples). But I
still feel like I at best have 80% understanding of how this one is
organized/works. Abstract descriptions and convoluted code have been
handicapping me (and lack of time to dig further).


Well, this is one of my concerns with this driver. It isn't easy to read 
(and so review, maintain, extend and debug found issues).


Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v11 17/27] iommu/exynos: remove calls to Runtime PM API functions

[Tomasz Figa]

On 19.03.2014 19:37, Grant Grundler wrote:

On Wed, Mar 19, 2014 at 10:30 AM, Tomasz Figa  wrote:
...

As I said, AFAIK the trend is to get rid of ordering by initcalls and make
sure that drivers can handle missing dependencies properly, even for
"services" such as DMA, GPIO, clocks and so on, which after all are provided
by normal drivers like other.


Ok - I'm not following the general kernel dev trends. initcall()
levels are easy to understand and implement. So I would not be in a
hurry to replace them.



Well, initcall level is still a way to satisfy most of dependencies, 
i.e. all client devices with higher initcall levels will probe 
successfully. However the other case needs to be handled as well - in 
this case the IOMMU binding code needs to defer probe of client driver 
if respective IOMMU is not yet available.



ps. I've written IOMMU support for four different IOMMUs on three
operating systems (See drivers/parisc for two linux examples). But I
still feel like I at best have 80% understanding of how this one is
organized/works. Abstract descriptions and convoluted code have been
handicapping me (and lack of time to dig further).



Well, this is one of my concerns with this driver. It isn't easy to read
(and so review, maintain, extend and debug found issues).


My postscript comment was more to explain why I'm not confident in my
opinion - not a reason to reject the patch series.  I still consider
the whole series as a step forward. But I'm not the expert here.


I fully agree with you. Other than the issues mentioned in review, the 
patches are definitely a step forward. I'd even say that all the patches 
that have nothing to do with device tree could be merged in their 
current form and the code refined later. It doesn't mean that patches 
shouldn't be reviewed now and issues spotted reported, even if they 
could be fixed later - this is for the IOMMU subsystem maintainer to decide.


As for patches related to DT support, more care needs to be taken, as 
bindings should be designed with stability in mind, so the refining 
process should happen at review stage.



Right now, with ~30 patches posted by the exynos iommu (official?)
maintainer, no one else who has a clue will attempt to fix or clean up
those kinds of problems.  i.e. it's useful to enable others to fix
what are essentially unspecified "design pattern" issues.


Agreed.

Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v11 10/27] iommu/exynos: use managed device helper functions

[Tomasz Figa]

On 20.03.2014 11:03, Cho KyongHo wrote:

On Wed, 19 Mar 2014 13:08:42 +0100, Tomasz Figa wrote:

On 19.03.2014 10:01, Sachin Kamat wrote:

On 19 March 2014 14:29, Cho KyongHo  wrote:

On Tue, 18 Mar 2014 16:14:53 +0100, Tomasz Figa wrote:

On 18.03.2014 12:09, Cho KyongHo wrote:

On Fri, 14 Mar 2014 20:52:43 +0530, Sachin Kamat wrote:

Hi KyongHo,

On 14 March 2014 10:35, Cho KyongHo  wrote:

This patch uses managed device helper functions in the probe().

Signed-off-by: Cho KyongHo 
---

[snip]


+   data->clk = devm_clk_get(dev, "sysmmu");
+   if (IS_ERR(data->clk)) {
+   dev_info(dev, "No gate clock found!\n");
+   data->clk = NULL;
+   }


Why aren't you returning from here upon error?


It is for the case of a System MMU which does not need clock gating.



Are there really such cases?



Yes.
Especially in the case of initial stage of new SoC development.

I have experianced some software workaround for H/W restriction
needs prevention of clock gating for some devices.


So aren't these basically some exceptions/hacks rather than the usual way
of functioning of the device?



This actually raises a good question, whether we really need to support
such early development SoC versions in mainline.

Another thing is that if you need to assure that a clock is ungated, you
must acquire it and prepare_enable explicitly, so I don't think this
kind of handling is correct.


On early development step of a new SoC, clock related stuffs and
some device drivers like display controller are usually developed in parallel.

In that case, -ENOENT from clk_get() must not treated as an error.
"[PATCH v11 20/17] iommu/exynos: allow having multiple System MMUs for a master 
H/W"
patch distinguishes -ENOENT from other error values returned by devm_clk_get().


I still don't think upstream is right place for such development hacks 
and such assumption will mask potential errors caused by clocks 
unspecified in DT.


If such thing is needed for development, an extra patch might be kept in 
development tree, until clock driver is implemented or a dummy 
fixed-rate clock might be specified in DT.


Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v11 20/27] iommu/exynos: allow having multiple System MMUs for a master H/W

[Tomasz Figa]

On 20.03.2014 11:22, Cho KyongHo wrote:

On Wed, 19 Mar 2014 16:14:57 +0100, Tomasz Figa wrote:

On 19.03.2014 14:20, Tomasz Figa wrote:

On 19.03.2014 01:39, Cho KyongHo wrote:

On Tue, 18 Mar 2014 15:26:48 +0100, Tomasz Figa wrote:



On 18.03.2014 14:01, Cho KyongHo wrote:

On Fri, 14 Mar 2014 17:12:03 +0100, Tomasz Figa wrote:

Hi KyongHo,

On 14.03.2014 06:10, Cho KyongHo wrote:

Some master device descriptor like fimc-is which is an abstraction
of very complex H/W may have multiple System MMUs. For those devices,
the design of the link between System MMU and its master H/W is
needed
to be reconsidered.

A link structure, sysmmu_list_data is introduced that provides a link
to master H/W and that has a pointer to the device descriptor of a
System MMU. Given a device descriptor of a master H/W, it is possible
to traverse all System MMUs that must be controlled along with the
master H/W.


NAK.

A device driver should handle particular hardware instances
separately,
without abstracting a virtual hardware instance consisting of multiple
physical ones.

If such abstraction is needed, it should be done above the
exynos-iommu
driver, e.g. by something like iommu-composite driver that would
aggregate several IOMMUs. Keep in mind that such IOMMUs in a group
could
be different, e.g. different Exynos SysMMU versions or even completely
different IPs handled by different drivers.

Still, I don't think there is a real need for such abstraction.
Instead,
related drivers shall be fixed to properly handle multiple memory
masters and their IOMMUs.



G2D, Scalers and FIMD of Exynos5420 has 2 System MMUs while aother
SoC like
Exynos5250 does not.

I don't understand why you are negative to this approach.
This is the simplest than the others.

Let me show you an example.
FIMC-IS driver just controls MCU in FIMC-IS subsystem and the
firmware of
the MCU controls all other peripherals in the subsystem. Each
peripherals
have their own System MMU. Moreover, the configuration of the
peripherals
varies according to the SoCs.

If System MMU driver accepts multiple masters, everything is done in
DT.
But I worry that it is not easy if System MMU driver does not support
multiple masters.


I believe I have stated enough reasons why this kind of implementation
is bad. I'm not going to waste time repeating myself.

Your concerns presented above are valid, however they are not related to
what is wrong with this patch. I have given you two proper ways to
handle this, none should be forced upon particular IOMMU master drivers
- their authors should have the chance to select the method that works
best for them.



I don't still understand why you think this patch is wrong.
I think this is the best way not to think for all the driver developers
about other things than their business logic.


I agree, but one of the ways I proposed (an iommu-composite layer above
the IOMMU low level drivers) doesn't add any extra responsibility of
driver developers.

Moreover, it's this kind of business logic in low level drivers that is
adding more responsibility, because it introduces additional complexity
and makes the driver harder to read, maintain and extend in future.



This does not hurt anyone and I think this is good enough.



Well, it is barely good enough. It is a good practice to make a low
level driver handle a single device instance and this is how Linux
driver model is designed.

Moreover, a single device tree node _must_ represent a single hardware
block, so you can't group multiple SysMMUs into a single device tree node.



OK, you add nodes for single SysMMUs devices which is fine, sorry. I was
under impression that one kernel device (struct device) corresponds to
multiple SysMMUs, but this was before your patches, sorry. So one issue
less, but it's still not good.



Ok. Understood why you have mentioned such.


Furthermore, if you force grouping of SysMMUs into a single virtual one,
you enforce using the same address space for all masters of some
particular hardware blocks, while potentially driver developers would
like to separate them.


Probably some clarification is needed. Your other patch adds:

sysmmu_fimd0w04: sysmmu@1464 {
compatible = "samsung,sysmmu-v3.3";
reg = <0x1464 0x1000>;
interrupt-parent = <&combiner>;
interrupts = <3 2>;
clock-names = "sysmmu", "master";
clocks = <&clock 422>, <&clock 421>;
samsung,power-domain = <&disp_pd>;
mmu-masters = <&fimd>;
};

sysmmu_fimd0w123: sysmmu@1468 {
compatible = "samsung,sysmmu-v3.3";
reg = <0x1468 0x1000>;
interrupt-parent = <&combiner>;
interrupts = <3 0>;
clock-name

Re: [PATCH v11 20/27] iommu/exynos: allow having multiple System MMUs for a master H/W

[Tomasz Figa]

On 21.03.2014 06:21, Cho KyongHo wrote:

On Thu, 20 Mar 2014 11:54:58 +0100, Tomasz Figa wrote:

On 20.03.2014 11:22, Cho KyongHo wrote:

On Wed, 19 Mar 2014 16:14:57 +0100, Tomasz Figa wrote:

On 19.03.2014 14:20, Tomasz Figa wrote:

On 19.03.2014 01:39, Cho KyongHo wrote:

On Tue, 18 Mar 2014 15:26:48 +0100, Tomasz Figa wrote:



On 18.03.2014 14:01, Cho KyongHo wrote:

On Fri, 14 Mar 2014 17:12:03 +0100, Tomasz Figa wrote:

Hi KyongHo,

On 14.03.2014 06:10, Cho KyongHo wrote:

Some master device descriptor like fimc-is which is an abstraction
of very complex H/W may have multiple System MMUs. For those devices,
the design of the link between System MMU and its master H/W is
needed
to be reconsidered.

A link structure, sysmmu_list_data is introduced that provides a link
to master H/W and that has a pointer to the device descriptor of a
System MMU. Given a device descriptor of a master H/W, it is possible
to traverse all System MMUs that must be controlled along with the
master H/W.


NAK.

A device driver should handle particular hardware instances
separately,
without abstracting a virtual hardware instance consisting of multiple
physical ones.

If such abstraction is needed, it should be done above the
exynos-iommu
driver, e.g. by something like iommu-composite driver that would
aggregate several IOMMUs. Keep in mind that such IOMMUs in a group
could
be different, e.g. different Exynos SysMMU versions or even completely
different IPs handled by different drivers.

Still, I don't think there is a real need for such abstraction.
Instead,
related drivers shall be fixed to properly handle multiple memory
masters and their IOMMUs.



G2D, Scalers and FIMD of Exynos5420 has 2 System MMUs while aother
SoC like
Exynos5250 does not.

I don't understand why you are negative to this approach.
This is the simplest than the others.

Let me show you an example.
FIMC-IS driver just controls MCU in FIMC-IS subsystem and the
firmware of
the MCU controls all other peripherals in the subsystem. Each
peripherals
have their own System MMU. Moreover, the configuration of the
peripherals
varies according to the SoCs.

If System MMU driver accepts multiple masters, everything is done in
DT.
But I worry that it is not easy if System MMU driver does not support
multiple masters.


I believe I have stated enough reasons why this kind of implementation
is bad. I'm not going to waste time repeating myself.

Your concerns presented above are valid, however they are not related to
what is wrong with this patch. I have given you two proper ways to
handle this, none should be forced upon particular IOMMU master drivers
- their authors should have the chance to select the method that works
best for them.



I don't still understand why you think this patch is wrong.
I think this is the best way not to think for all the driver developers
about other things than their business logic.


I agree, but one of the ways I proposed (an iommu-composite layer above
the IOMMU low level drivers) doesn't add any extra responsibility of
driver developers.

Moreover, it's this kind of business logic in low level drivers that is
adding more responsibility, because it introduces additional complexity
and makes the driver harder to read, maintain and extend in future.



This does not hurt anyone and I think this is good enough.



Well, it is barely good enough. It is a good practice to make a low
level driver handle a single device instance and this is how Linux
driver model is designed.

Moreover, a single device tree node _must_ represent a single hardware
block, so you can't group multiple SysMMUs into a single device tree node.



OK, you add nodes for single SysMMUs devices which is fine, sorry. I was
under impression that one kernel device (struct device) corresponds to
multiple SysMMUs, but this was before your patches, sorry. So one issue
less, but it's still not good.



Ok. Understood why you have mentioned such.


Furthermore, if you force grouping of SysMMUs into a single virtual one,
you enforce using the same address space for all masters of some
particular hardware blocks, while potentially driver developers would
like to separate them.


Probably some clarification is needed. Your other patch adds:

sysmmu_fimd0w04: sysmmu@1464 {
compatible = "samsung,sysmmu-v3.3";
reg = <0x1464 0x1000>;
interrupt-parent = <&combiner>;
interrupts = <3 2>;
clock-names = "sysmmu", "master";
clocks = <&clock 422>, <&clock 421>;
samsung,power-domain = <&disp_pd>;
mmu-masters = <&fimd>;
};

sysmmu_fimd0w123: sysmmu@1468 {
compatible = "samsung,sysmmu-v3.3";
reg = <0x1468 0x1000>;
interrupt-parent

Re: [PATCH v12 18/31] iommu/exynos: allow having multiple System MMUs for a master H/W

[Tomasz Figa]

On 27.04.2014 09:37, Shaik Ameer Basha wrote:

From: Cho KyongHo 

Some master device descriptor like fimc-is which is an abstraction
of very complex H/W may have multiple System MMUs. For those devices,
the design of the link between System MMU and its master H/W is needed
to be reconsidered.

A link structure, sysmmu_list_data is introduced that provides a link
to master H/W and that has a pointer to the device descriptor of a
System MMU. Given a device descriptor of a master H/W, it is possible
to traverse all System MMUs that must be controlled along with the
master H/W.

Signed-off-by: Cho KyongHo 
---
  drivers/iommu/exynos-iommu.c |  545 ++
  1 file changed, 335 insertions(+), 210 deletions(-)

diff --git a/drivers/iommu/exynos-iommu.c b/drivers/iommu/exynos-iommu.c
index fefedec3..c2e6365 100755
--- a/drivers/iommu/exynos-iommu.c
+++ b/drivers/iommu/exynos-iommu.c
@@ -117,6 +117,10 @@
  #define REG_PB1_EADDR 0x058

  #define has_sysmmu(dev)   (dev->archdata.iommu != NULL)
+#define for_each_sysmmu_list(dev, list_data)   \
+   list_for_each_entry(list_data,  \
+   &((struct exynos_iommu_owner *)dev->archdata.iommu)->mmu_list, \
+   entry)


Sorry, NAK.

Please don't add this kind of complexity and business logic to low level 
code. We want the configuration functions to be simple, easy to read, 
maintain and extend.


The proper way to do it is to let the IOMMUs be grouped together on 
IOMMU subsystem level, so that each IOMMU consumer driver would see just 
one IOMMU, but then IOMMU driver callbacks would handle just particular 
instances of the IOMMU IP blocks, without any loops, lists and other 
crazy code...


Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v12 00/31] iommu/exynos: Fixes and Enhancements of System MMU driver with DT

[Tomasz Figa]

On 06.05.2014 19:59, Joerg Roedel wrote:

On Wed, Apr 30, 2014 at 04:27:10PM +0530, Shaik Ameer Basha wrote:

This series is going on for quite a long time and most of the patches here
doesn't depend on dt bindings. As Exynos IOMMU h/w is introducing new versions
very frequently, maintaining and reviewing all these patches again and
again is quite a hard job.

If it is acceptable, I can post one more series with the subset of
above patches,
which doesn't depend on dt-bindings. As all the patches which doesn't depend on
DT bindings are already tested,  I hope merging these subset of patches may help
in reducing the rework and review effort every time.

Once we finalize the generic DT bindings for the IOMMU devices, the driver
can be updated with the proposed DT bindings in mind.


Sounds reasonable to me if the patch subset is self-contained. When
Arnd is OK with that and no one else has objections I can take the
dt-independend parts.


+1.

I would also suggest dropping patch 18/31, for the reasons I mentioned 
in my reply to it.


Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH 1/1] iommu/arm-smmu: Add support to use Last level cache

[Tomasz Figa]

Hi Vivek,

On Fri, Jun 15, 2018 at 7:53 PM Vivek Gautam
 wrote:
>
> Qualcomm SoCs have an additional level of cache called as
> System cache or Last level cache[1]. This cache sits right
> before the DDR, and is tightly coupled with the memory
> controller.
> The cache is available to all the clients present in the
> SoC system. The clients request their slices from this system
> cache, make it active, and can then start using it. For these
> clients with smmu, to start using the system cache for
> dma buffers and related page tables [2], few of the memory
> attributes need to be set accordingly.
> This change makes the related memory Outer-Shareable, and
> updates the MAIR with necessary protection.
>
> The MAIR attribute requirements are:
> Inner Cacheablity = 0
> Outer Cacheablity = 1, Write-Back Write Allocate
> Outer Shareablity = 1
>
> This change is a realisation of following changes
> from downstream msm-4.9:
> iommu: io-pgtable-arm: Support DOMAIN_ATTRIBUTE_USE_UPSTREAM_HINT
> iommu: io-pgtable-arm: Implement IOMMU_USE_UPSTREAM_HINT

Would you be able to provide links to those 2 downstream changes?

>
> [1] https://patchwork.kernel.org/patch/10422531/
> [2] https://patchwork.kernel.org/patch/10302791/
>
> Signed-off-by: Vivek Gautam 
> ---
>  drivers/iommu/arm-smmu.c   | 14 ++
>  drivers/iommu/io-pgtable-arm.c | 24 +++-
>  drivers/iommu/io-pgtable.h |  4 
>  include/linux/iommu.h  |  4 
>  4 files changed, 41 insertions(+), 5 deletions(-)
>
> diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
> index f7a96bcf94a6..8058e7205034 100644
> --- a/drivers/iommu/arm-smmu.c
> +++ b/drivers/iommu/arm-smmu.c
> @@ -249,6 +249,7 @@ struct arm_smmu_domain {
> struct mutexinit_mutex; /* Protects smmu pointer 
> */
> spinlock_t  cb_lock; /* Serialises ATS1* ops and 
> TLB syncs */
> struct iommu_domain domain;
> +   boolhas_sys_cache;
>  };
>
>  struct arm_smmu_option_prop {
> @@ -862,6 +863,8 @@ static int arm_smmu_init_domain_context(struct 
> iommu_domain *domain,
>
> if (smmu->features & ARM_SMMU_FEAT_COHERENT_WALK)
> pgtbl_cfg.quirks = IO_PGTABLE_QUIRK_NO_DMA;
> +   if (smmu_domain->has_sys_cache)
> +   pgtbl_cfg.quirks |= IO_PGTABLE_QUIRK_SYS_CACHE;
>
> smmu_domain->smmu = smmu;
> pgtbl_ops = alloc_io_pgtable_ops(fmt, &pgtbl_cfg, smmu_domain);
> @@ -1477,6 +1480,9 @@ static int arm_smmu_domain_get_attr(struct iommu_domain 
> *domain,
> case DOMAIN_ATTR_NESTING:
> *(int *)data = (smmu_domain->stage == ARM_SMMU_DOMAIN_NESTED);
> return 0;
> +   case DOMAIN_ATTR_USE_SYS_CACHE:
> +   *((int *)data) = smmu_domain->has_sys_cache;
> +   return 0;
> default:
> return -ENODEV;
> }
> @@ -1506,6 +1512,14 @@ static int arm_smmu_domain_set_attr(struct 
> iommu_domain *domain,
> smmu_domain->stage = ARM_SMMU_DOMAIN_S1;
>
> break;
> +   case DOMAIN_ATTR_USE_SYS_CACHE:
> +   if (smmu_domain->smmu) {
> +   ret = -EPERM;
> +   goto out_unlock;
> +   }
> +   if (*((int *)data))
> +   smmu_domain->has_sys_cache = true;
> +   break;
> default:
> ret = -ENODEV;
> }
> diff --git a/drivers/iommu/io-pgtable-arm.c b/drivers/iommu/io-pgtable-arm.c
> index 010a254305dd..b2aee1828524 100644
> --- a/drivers/iommu/io-pgtable-arm.c
> +++ b/drivers/iommu/io-pgtable-arm.c
> @@ -169,9 +169,11 @@
>  #define ARM_LPAE_MAIR_ATTR_DEVICE  0x04
>  #define ARM_LPAE_MAIR_ATTR_NC  0x44
>  #define ARM_LPAE_MAIR_ATTR_WBRWA   0xff
> +#define ARM_LPAE_MAIR_ATTR_SYS_CACHE   0xf4
>  #define ARM_LPAE_MAIR_ATTR_IDX_NC  0
>  #define ARM_LPAE_MAIR_ATTR_IDX_CACHE   1
>  #define ARM_LPAE_MAIR_ATTR_IDX_DEV 2
> +#define ARM_LPAE_MAIR_ATTR_IDX_SYS_CACHE   3
>
>  /* IOPTE accessors */
>  #define iopte_deref(pte,d) __va(iopte_to_paddr(pte, d))
> @@ -442,6 +444,10 @@ static arm_lpae_iopte arm_lpae_prot_to_pte(struct 
> arm_lpae_io_pgtable *data,
> else if (prot & IOMMU_CACHE)
> pte |= (ARM_LPAE_MAIR_ATTR_IDX_CACHE
> << ARM_LPAE_PTE_ATTRINDX_SHIFT);
> +   else if (prot & IOMMU_SYS_CACHE)
> +   pte |= (ARM_LPAE_MAIR_ATTR_IDX_SYS_CACHE
> +   << ARM_LPAE_PTE_ATTRINDX_SHIFT);
> +

Okay, so we favor the full caching (IC WBRWA, OC WBRWA, OS) first if
requested or otherwise try to use system cache (IC NC, OC WBWA?, OS)?
Sounds fine.

nit: Unnecessary blank line.

> } else {
> pte = ARM_LPAE_PTE_HAP_FAULT;
> if (prot & IOMMU_READ)
> @@ -771,7 +777,8 @@ arm_64_l

Re: [PATCH v3] drm/rockchip: update cursors asynchronously through atomic.

[Tomasz Figa]

Hi Helen,

On Fri, Nov 23, 2018 at 8:31 AM Helen Koike  wrote:
>
> Hi Tomasz,
>
> On 11/20/18 4:48 AM, Tomasz Figa wrote:
> > Hi Helen,
> >
> > On Tue, Nov 20, 2018 at 4:08 AM Helen Koike  
> > wrote:
> >>
> >> From: Enric Balletbo i Serra 
> >>
> >> Add support to async updates of cursors by using the new atomic
> >> interface for that.
> >>
> >> Signed-off-by: Enric Balletbo i Serra 
> >> [updated for upstream]
> >> Signed-off-by: Helen Koike 
> >>
> >> ---
> >> Hello,
> >>
> >> This is the third version of the async-plane update suport to the
> >> Rockchip driver.
> >>
> >
> > Thanks for a quick respin. Please see my comments inline. (I'll try to
> > be better at responding from now on...)
> >
> >> I tested running igt kms_cursor_legacy and kms_atomic tests using a 
> >> 96Boards Ficus.
> >>
> >> Note that before the patch, the following igt tests failed:
> >>
> >> basic-flip-before-cursor-atomic
> >> basic-flip-before-cursor-legacy
> >> cursor-vs-flip-atomic
> >> cursor-vs-flip-legacy
> >> cursor-vs-flip-toggle
> >> flip-vs-cursor-atomic
> >> flip-vs-cursor-busy-crc-atomic
> >> flip-vs-cursor-busy-crc-legacy
> >> flip-vs-cursor-crc-atomic
> >> flip-vs-cursor-crc-legacy
> >> flip-vs-cursor-legacy
> >>
> >> Full log: https://people.collabora.com/~koike/results-4.20/html/
> >>
> >> Now with the patch applied the following were fixed:
> >> basic-flip-before-cursor-atomic
> >> basic-flip-before-cursor-legacy
> >> flip-vs-cursor-atomic
> >> flip-vs-cursor-legacy
> >>
> >> Full log: https://people.collabora.com/~koike/results-4.20-async/html/
> >
> > Could you also test modetest, with the -C switch to test the legacy
> > cursor API? I remember it triggering crashes due to synchronization
> > issues easily.
>
> Sure. I tested with
> $ modetest -M rockchip -s 37:1920x1080 -C
>
> I also vary the mode but I couldn't trigger any crashes.
>
> >
> >>
> >> Tomasz, as you mentined in v2 about waiting the hardware before updating
> >> the framebuffer, now I call the loop you pointed out in the async path,
> >> was that what you had in mind? Or do you think I would make sense to
> >> call the vop_crtc_atomic_flush() instead of just exposing that loop?
> >>
> >> Thanks
> >> Helen
> >>
> >> Changes in v3:
> >> - Rebased on top of drm-misc
> >> - Fix missing include in rockchip_drm_vop.c
> >> - New function vop_crtc_atomic_commit_flush
> >>
> >> Changes in v2:
> >> - v2: https://patchwork.freedesktop.org/patch/254180/
> >> - Change the framebuffer as well to cover jumpy cursor when hovering
> >>   text boxes or hyperlink. (Tomasz)
> >> - Use the PSR inhibit mechanism when accessing VOP hardware instead of
> >>   PSR flushing (Tomasz)
> >>
> >> Changes in v1:
> >> - Rebased on top of drm-misc
> >> - In async_check call drm_atomic_helper_check_plane_state to check that
> >>   the desired plane is valid and update various bits of derived state
> >>   (clipped coordinates etc.)
> >> - In async_check allow to configure new scaling in the fast path.
> >> - In async_update force to flush all registered PSR encoders.
> >> - In async_update call atomic_update directly.
> >> - In async_update call vop_cfg_done needed to set the vop registers and 
> >> take effect.
> >>
> >>  drivers/gpu/drm/rockchip/rockchip_drm_fb.c  |  36 ---
> >>  drivers/gpu/drm/rockchip/rockchip_drm_psr.c |  37 +++
> >>  drivers/gpu/drm/rockchip/rockchip_drm_psr.h |   3 +
> >>  drivers/gpu/drm/rockchip/rockchip_drm_vop.c | 108 +---
> >>  4 files changed, 131 insertions(+), 53 deletions(-)
> >>
> >> diff --git a/drivers/gpu/drm/rockchip/rockchip_drm_fb.c 
> >> b/drivers/gpu/drm/rockchip/rockchip_drm_fb.c
> >> index ea18cb2a76c0..08bec50d9c5d 100644
> >> --- a/drivers/gpu/drm/rockchip/rockchip_drm_fb.c
> >> +++ b/drivers/gpu/drm/rockchip/rockchip_drm_fb.c
> >> @@ -127,42 +127,6 @@ rockchip_user_fb_create(struct drm_device *dev, 
> >> struct drm_file *file_priv,
> >> return ERR_PTR(ret);
> >>  }
> >>
>

Re: [PATCH v3] drm/rockchip: update cursors asynchronously through atomic.

[Tomasz Figa]

Hi Michael,

On Fri, Nov 23, 2018 at 1:58 PM Michael Zoran  wrote:
>
> On Fri, 2018-11-23 at 11:27 +0900, Tomasz Figa wrote:
> >
> > The point here is not about setting and resetting the plane->fb
> > pointer. It's about what happens inside
> > drm_atomic_set_fb_for_plane().
> >
> > It calls drm_framebuffer_get() for the new fb and
> > drm_framebuffer_put() for the old fb. In result, if the fb changes,
> > the old fb, which had its reference count incremented in the atomic
> > commit that set it to the plane before, has its reference count
> > decremented. Moreover, if the new reference count becomes 0,
> > drm_framebuffer_put() will immediately free the buffer.
> >
> > Freeing a buffer when the hardware is still scanning out of it isn't
> > a
> > good idea, is it?
>
> No, it's not.  But the board I submitted the patch for doesn't have
> anything like hot swapable ram.  The ram access is still going to work,
> just it might display something it shouldn't. Say for example if that
> frame buffer got reused by somethig else and filled with new data in
> the very small window.

Thanks for a quick reply!

To clarify, on the Rockchip platform this patch is for (and many other
arm/arm64 SoCs) the display controller is behind an IOMMU. Freeing the
buffer would mean unmapping the related IOVAs from the IOMMU. If the
hardware is still scanning out from the unmapped addresses, it would
cause IOMMU page faults. We don't have any good IOMMU page fault
handling in the kernel, so on most platforms that would likely end up
stalling the display controller completely (on Rockchip it does).

>
> But yes, I agree the best solution would be to not release the buffer
> until the next vblank.
>
> Perhaps a good solution would be for the DRM api to have the concept of
> a deferred release?  Meaning if the put() call just added the frame
> buffer to a list that DRM core could walk during the vblank.  That
> might be better then every single driver trying to work up a custom
> solution.

Agreed.

Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [RESEND PATCH v17 5/5] iommu/arm-smmu: Add support for qcom,smmu-v2 variant

[Tomasz Figa]

Hi Vivek, Will,

On Fri, Nov 23, 2018 at 6:13 PM Vivek Gautam
 wrote:
>
> Hi Will,
>
> On Wed, Nov 21, 2018 at 11:09 PM Will Deacon  wrote:
> >
> > [+Thor]
> >
> > On Fri, Nov 16, 2018 at 04:54:30PM +0530, Vivek Gautam wrote:
> > > qcom,smmu-v2 is an arm,smmu-v2 implementation with specific
> > > clock and power requirements.
> > > On msm8996, multiple cores, viz. mdss, video, etc. use this
> > > smmu. On sdm845, this smmu is used with gpu.
> > > Add bindings for the same.
> > >
> > > Signed-off-by: Vivek Gautam 
> > > Reviewed-by: Rob Herring 
> > > Reviewed-by: Tomasz Figa 
> > > Tested-by: Srinivas Kandagatla 
> > > Reviewed-by: Robin Murphy 
> > > ---
> > >  drivers/iommu/arm-smmu.c | 13 +
> > >  1 file changed, 13 insertions(+)
> > >
> > > diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
> > > index 2098c3141f5f..d315ca637097 100644
> > > --- a/drivers/iommu/arm-smmu.c
> > > +++ b/drivers/iommu/arm-smmu.c
> > > @@ -120,6 +120,7 @@ enum arm_smmu_implementation {
> > >   GENERIC_SMMU,
> > >   ARM_MMU500,
> > >   CAVIUM_SMMUV2,
> > > + QCOM_SMMUV2,
> > >  };
> > >
> > >  struct arm_smmu_s2cr {
> > > @@ -2026,6 +2027,17 @@ ARM_SMMU_MATCH_DATA(arm_mmu401, ARM_SMMU_V1_64K, 
> > > GENERIC_SMMU);
> > >  ARM_SMMU_MATCH_DATA(arm_mmu500, ARM_SMMU_V2, ARM_MMU500);
> > >  ARM_SMMU_MATCH_DATA(cavium_smmuv2, ARM_SMMU_V2, CAVIUM_SMMUV2);
> > >
> > > +static const char * const qcom_smmuv2_clks[] = {
> > > + "bus", "iface",
> > > +};
> > > +
> > > +static const struct arm_smmu_match_data qcom_smmuv2 = {
> > > + .version = ARM_SMMU_V2,
> > > + .model = QCOM_SMMUV2,
> > > + .clks = qcom_smmuv2_clks,
> > > + .num_clks = ARRAY_SIZE(qcom_smmuv2_clks),
> > > +};
> >
> > These seems redundant if we go down the route proposed by Thor, where we
> > just pull all of the clocks out of the device-tree. In which case, why
> > do we need this match_data at all?
>
> Which is better? Driver relying solely on the device tree to tell
> which all clocks
> are required to be enabled,
> or, the driver deciding itself based on the platform's match data,
> that it should
> have X, Y, & Z clocks that should be supplied from the device tree.

The former would simplify the driver, but would also make it
impossible to spot mistakes in DT, which would ultimately surface out
as very hard to debug bugs (likely complete system lockups).

For qcom_smmuv2, I believe we're eventually going to end up with
platform-specific quirks anyway, so specifying the clocks too wouldn't
hurt. Given that, I'd recommend sticking to the latter, i.e. what this
patch does.

Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [RESEND PATCH v17 5/5] iommu/arm-smmu: Add support for qcom,smmu-v2 variant

[Tomasz Figa]

On Sat, Nov 24, 2018 at 3:34 AM Will Deacon  wrote:
>
> On Fri, Nov 23, 2018 at 03:06:29PM +0530, Vivek Gautam wrote:
> > On Fri, Nov 23, 2018 at 2:52 PM Tomasz Figa  wrote:
> > > On Fri, Nov 23, 2018 at 6:13 PM Vivek Gautam
> > >  wrote:
> > > > On Wed, Nov 21, 2018 at 11:09 PM Will Deacon  
> > > > wrote:
> > > > > On Fri, Nov 16, 2018 at 04:54:30PM +0530, Vivek Gautam wrote:
> > > > > > @@ -2026,6 +2027,17 @@ ARM_SMMU_MATCH_DATA(arm_mmu401, 
> > > > > > ARM_SMMU_V1_64K, GENERIC_SMMU);
> > > > > >  ARM_SMMU_MATCH_DATA(arm_mmu500, ARM_SMMU_V2, ARM_MMU500);
> > > > > >  ARM_SMMU_MATCH_DATA(cavium_smmuv2, ARM_SMMU_V2, CAVIUM_SMMUV2);
> > > > > >
> > > > > > +static const char * const qcom_smmuv2_clks[] = {
> > > > > > + "bus", "iface",
> > > > > > +};
> > > > > > +
> > > > > > +static const struct arm_smmu_match_data qcom_smmuv2 = {
> > > > > > + .version = ARM_SMMU_V2,
> > > > > > + .model = QCOM_SMMUV2,
> > > > > > + .clks = qcom_smmuv2_clks,
> > > > > > + .num_clks = ARRAY_SIZE(qcom_smmuv2_clks),
> > > > > > +};
> > > > >
> > > > > These seems redundant if we go down the route proposed by Thor, where 
> > > > > we
> > > > > just pull all of the clocks out of the device-tree. In which case, why
> > > > > do we need this match_data at all?
> > > >
> > > > Which is better? Driver relying solely on the device tree to tell
> > > > which all clocks
> > > > are required to be enabled,
> > > > or, the driver deciding itself based on the platform's match data,
> > > > that it should
> > > > have X, Y, & Z clocks that should be supplied from the device tree.
> > >
> > > The former would simplify the driver, but would also make it
> > > impossible to spot mistakes in DT, which would ultimately surface out
> > > as very hard to debug bugs (likely complete system lockups).
> >
> > Thanks.
> > Yea, this is how I understand things presently. Relying on device tree
> > puts the things out of driver's control.
>
> But it also has the undesirable effect of having to update the driver
> code whenever we want to add support for a new SMMU implementation. If
> we do this all in the DT, as Thor is trying to do, then older kernels
> will work well with new hardware.

Fair enough, if you're okay with that. Obviously one would still have
to change the DT bindings to list the exact set of clocks for the new
hardware variant, unless the convention changed recently.

Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v3] drm/rockchip: update cursors asynchronously through atomic.

[Tomasz Figa]

Hi Gustavo,

On Tue, Nov 27, 2018 at 8:54 AM Gustavo Padovan
 wrote:
>
> Hi Tomasz,
>
> On 11/23/18 12:27 AM, Tomasz Figa wrote:
> > Hi Helen,
> >
> > On Fri, Nov 23, 2018 at 8:31 AM Helen Koike  
> > wrote:
> >> Hi Tomasz,
> >>
> >> On 11/20/18 4:48 AM, Tomasz Figa wrote:
> >>> Hi Helen,
> >>>
> >>> On Tue, Nov 20, 2018 at 4:08 AM Helen Koike  
> >>> wrote:
> >>>> From: Enric Balletbo i Serra 
> >>>>
> >>>> Add support to async updates of cursors by using the new atomic
> >>>> interface for that.
> >>>>
> >>>> Signed-off-by: Enric Balletbo i Serra 
> >>>> [updated for upstream]
> >>>> Signed-off-by: Helen Koike 
> >>>>
> >>>> ---
> >>>> Hello,
> >>>>
> >>>> This is the third version of the async-plane update suport to the
> >>>> Rockchip driver.
> >>>>
> >>> Thanks for a quick respin. Please see my comments inline. (I'll try to
> >>> be better at responding from now on...)
> >>>
> >>>> I tested running igt kms_cursor_legacy and kms_atomic tests using a 
> >>>> 96Boards Ficus.
> >>>>
> >>>> Note that before the patch, the following igt tests failed:
> >>>>
> >>>>  basic-flip-before-cursor-atomic
> >>>>  basic-flip-before-cursor-legacy
> >>>>  cursor-vs-flip-atomic
> >>>>  cursor-vs-flip-legacy
> >>>>  cursor-vs-flip-toggle
> >>>>  flip-vs-cursor-atomic
> >>>>  flip-vs-cursor-busy-crc-atomic
> >>>>  flip-vs-cursor-busy-crc-legacy
> >>>>  flip-vs-cursor-crc-atomic
> >>>>  flip-vs-cursor-crc-legacy
> >>>>  flip-vs-cursor-legacy
> >>>>
> >>>> Full log: https://people.collabora.com/~koike/results-4.20/html/
> >>>>
> >>>> Now with the patch applied the following were fixed:
> >>>>  basic-flip-before-cursor-atomic
> >>>>  basic-flip-before-cursor-legacy
> >>>>  flip-vs-cursor-atomic
> >>>>  flip-vs-cursor-legacy
> >>>>
> >>>> Full log: https://people.collabora.com/~koike/results-4.20-async/html/
> >>> Could you also test modetest, with the -C switch to test the legacy
> >>> cursor API? I remember it triggering crashes due to synchronization
> >>> issues easily.
> >> Sure. I tested with
> >> $ modetest -M rockchip -s 37:1920x1080 -C
> >>
> >> I also vary the mode but I couldn't trigger any crashes.
> >>
> >>>> Tomasz, as you mentined in v2 about waiting the hardware before updating
> >>>> the framebuffer, now I call the loop you pointed out in the async path,
> >>>> was that what you had in mind? Or do you think I would make sense to
> >>>> call the vop_crtc_atomic_flush() instead of just exposing that loop?
> >>>>
> >>>> Thanks
> >>>> Helen
> >>>>
> >>>> Changes in v3:
> >>>> - Rebased on top of drm-misc
> >>>> - Fix missing include in rockchip_drm_vop.c
> >>>> - New function vop_crtc_atomic_commit_flush
> >>>>
> >>>> Changes in v2:
> >>>> - v2: https://patchwork.freedesktop.org/patch/254180/
> >>>> - Change the framebuffer as well to cover jumpy cursor when hovering
> >>>>text boxes or hyperlink. (Tomasz)
> >>>> - Use the PSR inhibit mechanism when accessing VOP hardware instead of
> >>>>PSR flushing (Tomasz)
> >>>>
> >>>> Changes in v1:
> >>>> - Rebased on top of drm-misc
> >>>> - In async_check call drm_atomic_helper_check_plane_state to check that
> >>>>the desired plane is valid and update various bits of derived state
> >>>>(clipped coordinates etc.)
> >>>> - In async_check allow to configure new scaling in the fast path.
> >>>> - In async_update force to flush all registered PSR encoders.
> >>>> - In async_update call atomic_update directly.
> >>>> - In async_update call vop_cfg_done needed to set the vop registers and 
> >>>> take eff

Re: [PATCH] of/device: add blacklist for iommu dma_ops

[Tomasz Figa]

device is%sbehind an iommu\n",
> iommu ? " " : " not ");
>
> +   /*
> +* There is at least one case where the driver wants to directly
> +* manage the IOMMU, but if we end up with iommu dma_ops, that
> +* interferes with the drivers ability to use dma_map_sg() for
> +* cache operations.  Since we don't currently have a better
> +* solution, and this code runs before the driver is probed and
> +* has a chance to intervene, use a simple blacklist to avoid
> +* ending up with iommu dma_ops:
> +*/
> +   if (of_match_device(iommu_blacklist, dev)) {
> +   dev_dbg(dev, "skipping iommu hookup\n");
> +   iommu = NULL;
> +   }
> +
> arch_setup_dma_ops(dev, dma_addr, size, iommu, coherent);
>
> return 0;
> --
> 2.19.2
>

+Marek Szyprowski who I believe had a similar problem with Exynos DRM before.

Reviewed-by: Tomasz Figa 

Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH 1/1] iommu/arm-smmu: Add support to use Last level cache

[Tomasz Figa]

On Fri, Dec 7, 2018 at 6:25 PM Vivek Gautam  wrote:
>
> Hi Robin,
>
> On Tue, Dec 4, 2018 at 8:51 PM Robin Murphy  wrote:
> >
> > On 04/12/2018 11:01, Vivek Gautam wrote:
> > > Qualcomm SoCs have an additional level of cache called as
> > > System cache, aka. Last level cache (LLC). This cache sits right
> > > before the DDR, and is tightly coupled with the memory controller.
> > > The cache is available to all the clients present in the SoC system.
> > > The clients request their slices from this system cache, make it
> > > active, and can then start using it.
> > > For these clients with smmu, to start using the system cache for
> > > buffers and, related page tables [1], memory attributes need to be
> > > set accordingly.
> > > This change updates the MAIR and TCR configurations with correct
> > > attributes to use this system cache.
> > >
> > > To explain a little about memory attribute requirements here:
> > >
> > > Non-coherent I/O devices can't look-up into inner caches. However,
> > > coherent I/O devices can. But both can allocate in the system cache
> > > based on system policy and configured memory attributes in page
> > > tables.
> > > CPUs can access both inner and outer caches (including system cache,
> > > aka. Last level cache), and can allocate into system cache too
> > > based on memory attributes, and system policy.
> > >
> > > Further looking at memory types, we have following -
> > > a) Normal uncached :- MAIR 0x44, inner non-cacheable,
> > >outer non-cacheable;
> > > b) Normal cached :-   MAIR 0xff, inner read write-back non-transient,
> > >outer read write-back non-transient;
> > >attribute setting for coherenet I/O devices.
> > >
> > > and, for non-coherent i/o devices that can allocate in system cache
> > > another type gets added -
> > > c) Normal sys-cached/non-inner-cached :-
> > >MAIR 0xf4, inner non-cacheable,
> > >outer read write-back non-transient
> > >
> > > So, CPU will automatically use the system cache for memory marked as
> > > normal cached. The normal sys-cached is downgraded to normal non-cached
> > > memory for CPUs.
> > > Coherent I/O devices can use system cache by marking the memory as
> > > normal cached.
> > > Non-coherent I/O devices, to use system cache, should mark the memory as
> > > normal sys-cached in page tables.
> > >
> > > This change is a realisation of following changes
> > > from downstream msm-4.9:
> > > iommu: io-pgtable-arm: Support DOMAIN_ATTRIBUTE_USE_UPSTREAM_HINT[2]
> > > iommu: io-pgtable-arm: Implement IOMMU_USE_UPSTREAM_HINT[3]
> > >
> > > [1] https://patchwork.kernel.org/patch/10302791/
> > > [2] 
> > > https://source.codeaurora.org/quic/la/kernel/msm-4.9/commit/?h=msm-4.9&id=bf762276796e79ca90014992f4d9da5593fa7d51
> > > [3] 
> > > https://source.codeaurora.org/quic/la/kernel/msm-4.9/commit/?h=msm-4.9&id=d4c72c413ea27c43f60825193d4de9cb8ffd9602
> > >
> > > Signed-off-by: Vivek Gautam 
> > > ---
> > >
> > > Changes since v1:
> > >   - Addressed Tomasz's comments for basing the change on
> > > "NO_INNER_CACHE" concept for non-coherent I/O devices
> > > rather than capturing "SYS_CACHE". This is to indicate
> > > clearly the intent of non-coherent I/O devices that
> > > can't access inner caches.
> >
> > That seems backwards to me - there is already a fundamental assumption
> > that non-coherent devices can't access caches. What we're adding here is
> > a weird exception where they *can* use some level of cache despite still
> > being non-coherent overall.
> >
> > In other words, it's not a case of downgrading coherent devices'
> > accesses to bypass inner caches, it's upgrading non-coherent devices'
> > accesses to hit the outer cache. That's certainly the understanding I
> > got from talking with Pratik at Plumbers, and it does appear to fit with
> > your explanation above despite the final conclusion you draw being
> > different.
>
> Thanks for the thorough review of the change.
> Right, I guess it's rather an upgrade for non-coherent devices to use
> an outer cache than a downgrade for coherent devices.
>

Note that it was not my suggestion to use "NO_INNER_CACHE" for
enabling the system cache, sorry for not being clear. What I was
asking for in my comment was regarding the previous patch disabling
inner cache if system cache is requested, which may not make for
coherent devices, which could benefit from using both inner and system
cache.

So note that there are several cases here:
 - coherent, IC, system cache alloc,
 - coherent. non-IC, system cache alloc,
 - coherent, IC, system cache look-up,
 - noncoherent device, non-IC, system cache alloc,
 - noncoherent device, non-IC, system cache look-up.

Given the presence or lack of coherency for the device, which of the
2/3 options is the best depends on the use case, e.g. DMA/CPU access
pattern, sharing memory between multiple devices, etc.

Best regard

Re: [PATCH 03/29] clk: exynos: add missing smmu_g2d clock and update comments

[Tomasz Figa]

On 05.08.2014 12:47, Marek Szyprowski wrote:
> This patch adds missing smmu_g2d clock implementation and updates
> comment about Exynos4 clocks from 278-282 range. Those clocks are
> available on all Exynos4 SoC series, so the misleading comment has been
> removed.
> 
> Signed-off-by: Marek Szyprowski 
> ---
>  drivers/clk/samsung/clk-exynos4.c   |  1 +
>  include/dt-bindings/clock/exynos4.h | 10 +-
>  2 files changed, 6 insertions(+), 5 deletions(-)

Acked-by: Tomasz Figa 

Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH v2 04/18] clk: exynos: add missing smmu_g2d clock and update comments

[Tomasz Figa]

On 16.09.2014 13:54, Marek Szyprowski wrote:
> This patch adds missing smmu_g2d clock implementation and updates
> comment about Exynos4 clocks from 278-282 range. Those clocks are
> available on all Exynos4 SoC series, so the misleading comment has been
> removed.
> 
> Signed-off-by: Marek Szyprowski 
> Acked-by: Tomasz Figa 
> ---
>  drivers/clk/samsung/clk-exynos4.c   |  1 +
>  include/dt-bindings/clock/exynos4.h | 10 +-
>  2 files changed, 6 insertions(+), 5 deletions(-)
> 

Applied for next.

Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[RFC PATCH 0/2] Fix rockchip IOMMU driver vs PM issues

[Tomasz Figa]

On Rockchip SoCs, the IOMMUs are located inside the same power domains as
IP blocks using them. This means that as soon as we runtime suspend such
IP block, causing the power domain to be powered off, the IOMMU would also
be powered off, losing all its state.

This means that whenever the power domain is being powered off, the IOMMU
driver needs to be able to deinitialize the hardware and whenever the domain
is being powered on, it needs to restore all the state, so the consumer
device is able to perform memory transactions.

The solution proposed here revives the idea of PM domain notifiers submitted
originally by Samsung's Sylwester Nawrocki and Marek Szyprowski in [1].
The main benefit of this idea that it is very simple, adding just 84 lines
of code, but effective and also useful for other purposes, what can be seen
in thread [2] and [3]. Moreover, it lets us avoid adding device specific
code (in this case specific to single IOMMU type) to power domain drivers,
which can be used for more devices than just IOMMU and which may vary
between platforms using the same IOMMU driver.

The IOMMU-specific part of the solution simply uses the pre-power-off and
post-power-on notifications to do the necessary setup without the need
of any action from inside drivers of devices behind the IOMMU, so all the
code specific to this particular IOMMU type stays outside other drivers
that should not rely on being run with a particular type of IOMMU (and
often even SoC).

Tested with a custom board based on RK3288 SoC.

[1] http://thread.gmane.org/gmane.linux.kernel.samsung-soc/36079/focus=346956
[2] 
http://thread.gmane.org/gmane.linux.power-management.general/51993/focus=39158
[3] http://marc.info/?l=linux-pm&m=136448756308203&w=2

Sylwester Nawrocki (1):
  pm: Add PM domain notifications

Tomasz Figa (1):
  iommu: rockchip: Handle system-wide and runtime PM

 Documentation/power/notifiers.txt |  14 +++
 drivers/base/power/domain.c   |  50 +
 drivers/iommu/rockchip-iommu.c| 208 +++---
 include/linux/pm_domain.h |  20 
 4 files changed, 256 insertions(+), 36 deletions(-)

-- 
2.2.0.rc0.207.ga3a616c

___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[RFC PATCH 1/2] pm: Add PM domain notifications

[Tomasz Figa]

From: Sylwester Nawrocki 

This patch adds notifiers to the runtime PM/genpd subsystem. It is now
possible to register a notifier, which will be called before and after
the generic power domain subsystem calls the power domain's power_on
and power_off callbacks.

Signed-off-by: Sylwester Nawrocki 
[tf...@chromium.org: rebased]
Signed-off-by: Tomasz Figa 
---
 Documentation/power/notifiers.txt | 14 +++
 drivers/base/power/domain.c   | 50 +++
 include/linux/pm_domain.h | 20 
 3 files changed, 84 insertions(+)

diff --git a/Documentation/power/notifiers.txt 
b/Documentation/power/notifiers.txt
index a81fa25..62303f6 100644
--- a/Documentation/power/notifiers.txt
+++ b/Documentation/power/notifiers.txt
@@ -53,3 +53,17 @@ NULL).  To register and/or unregister a suspend notifier use 
the functions
 register_pm_notifier() and unregister_pm_notifier(), respectively, defined in
 include/linux/suspend.h .  If you don't need to unregister the notifier, you 
can
 also use the pm_notifier() macro defined in include/linux/suspend.h .
+
+Power Domain notifiers
+--
+
+The power domain notifiers allow subsystems or drivers to register for power
+domain on/off notifications, should they need to perform any actions right
+before or right after the power domain on/off.  The device must be already
+added to a power domain before its subsystem or driver registers the notifier.
+Following events are supported:
+
+PM_GENPD_POWER_ON_PREPARE  The power domain is about to turn on.
+PM_GENPD_POST_POWER_ON The power domain has just turned on.
+PM_GENPD_POWER_OFF_PREPARE The power domain is about to turn off.
+PM_GENPD_POST_POWER_OFFThe power domain has just turned off.
diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c
index 6a103a3..0d6f84e 100644
--- a/drivers/base/power/domain.c
+++ b/drivers/base/power/domain.c
@@ -68,6 +68,45 @@ static struct generic_pm_domain *pm_genpd_lookup_name(const 
char *domain_name)
return genpd;
 }
 
+int pm_genpd_register_notifier(struct device *dev, struct notifier_block *nb)
+{
+   struct pm_domain_data *pdd;
+   int ret = -EINVAL;
+
+   spin_lock_irq(&dev->power.lock);
+   if (dev->power.subsys_data) {
+   pdd = dev->power.subsys_data->domain_data;
+   ret = blocking_notifier_chain_register(&pdd->notify_chain_head,
+  nb);
+   }
+   spin_unlock_irq(&dev->power.lock);
+   return ret;
+}
+EXPORT_SYMBOL_GPL(pm_genpd_register_notifier);
+
+void pm_genpd_unregister_notifier(struct device *dev, struct notifier_block 
*nb)
+{
+   struct pm_domain_data *pdd;
+
+   spin_lock_irq(&dev->power.lock);
+   if (dev->power.subsys_data) {
+   pdd = dev->power.subsys_data->domain_data;
+   blocking_notifier_chain_unregister(&pdd->notify_chain_head, nb);
+   }
+   spin_unlock_irq(&dev->power.lock);
+}
+EXPORT_SYMBOL_GPL(pm_genpd_unregister_notifier);
+
+static void pm_genpd_notifier_call(unsigned long event,
+  struct generic_pm_domain *genpd)
+{
+   struct pm_domain_data *pdd;
+
+   list_for_each_entry(pdd, &genpd->dev_list, list_node)
+   blocking_notifier_call_chain(&pdd->notify_chain_head,
+event, pdd->dev);
+}
+
 struct generic_pm_domain *dev_to_genpd(struct device *dev)
 {
if (IS_ERR_OR_NULL(dev->pm_domain))
@@ -161,12 +200,17 @@ static int genpd_power_on(struct generic_pm_domain *genpd)
if (!genpd->power_on)
return 0;
 
+   pm_genpd_notifier_call(PM_GENPD_POWER_ON_PREPARE, genpd);
+
time_start = ktime_get();
ret = genpd->power_on(genpd);
if (ret)
return ret;
 
elapsed_ns = ktime_to_ns(ktime_sub(ktime_get(), time_start));
+
+   pm_genpd_notifier_call(PM_GENPD_POST_POWER_ON, genpd);
+
if (elapsed_ns <= genpd->power_on_latency_ns)
return ret;
 
@@ -188,12 +232,17 @@ static int genpd_power_off(struct generic_pm_domain 
*genpd)
if (!genpd->power_off)
return 0;
 
+   pm_genpd_notifier_call(PM_GENPD_POWER_OFF_PREPARE, genpd);
+
time_start = ktime_get();
ret = genpd->power_off(genpd);
if (ret == -EBUSY)
return ret;
 
elapsed_ns = ktime_to_ns(ktime_sub(ktime_get(), time_start));
+
+   pm_genpd_notifier_call(PM_GENPD_POST_POWER_OFF, genpd);
+
if (elapsed_ns <= genpd->power_off_latency_ns)
return ret;
 
@@ -1466,6 +1515,7 @@ int __pm_genpd_add_device(struct generic_pm_domain 
*genpd, struct device *dev,
genpd->attach_dev(genpd, dev);
 
mutex_lock(&gpd_data->lock);
+  

Re: [RFC PATCH 1/2] pm: Add PM domain notifications

[Tomasz Figa]

Hi Sylwester,

On Thu, Dec 11, 2014 at 7:36 PM, Sylwester Nawrocki
 wrote:
>
> Hi Tomasz,
>
> On 11/12/14 09:26, Tomasz Figa wrote:
> > From: Sylwester Nawrocki 
> >
> > This patch adds notifiers to the runtime PM/genpd subsystem. It is now
> > possible to register a notifier, which will be called before and after
> > the generic power domain subsystem calls the power domain's power_on
> > and power_off callbacks.
> >
> > Signed-off-by: Sylwester Nawrocki 
> > [tf...@chromium.org: rebased]
> > Signed-off-by: Tomasz Figa 
>
> Not sure if you've noticed it, I posted an updated version of this patch
> recently [1]. The notifiers list is moved to struct generic_pm_domain
> there and it also allows to register a notifier for selected power domain
> by name.
[snip]
> [1] http://www.spinics.net/lists/linux-samsung-soc/msg38549.html

Ah, haven't noticed, sorry. The API using devices looks the same, so I
guess we can simply have patch 2/2 of this series applied on top of
your patch.

By the way, look-up by name (presumably hardcoded somewhere?) sounds a
bit strange to me. What was the reason for it to be added?

Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[RFC PATCH 2/2] iommu: rockchip: Handle system-wide and runtime PM

[Tomasz Figa]

This patch modifies the rockchip-iommu driver to consider state of
the power domain the IOMMU is located in. When the power domain
is powered off, the IOMMU cannot be accessed and register programming
must be deferred until the power domain becomes enabled.

The solution implemented in this patch uses power domain notifications
to perform necessary IOMMU initialization. Race with runtime PM core
is avoided by protecting code accessing the hardware, including startup
and shutdown functions, with a spinlock with a check for power state
inside.

Signed-off-by: Tomasz Figa 
---
 drivers/iommu/rockchip-iommu.c | 208 ++---
 1 file changed, 172 insertions(+), 36 deletions(-)

diff --git a/drivers/iommu/rockchip-iommu.c b/drivers/iommu/rockchip-iommu.c
index b2023af..9120655 100644
--- a/drivers/iommu/rockchip-iommu.c
+++ b/drivers/iommu/rockchip-iommu.c
@@ -20,6 +20,8 @@
 #include 
 #include 
 #include 
+#include 
+#include 
 #include 
 #include 
 
@@ -88,6 +90,9 @@ struct rk_iommu {
int irq;
struct list_head node; /* entry in rk_iommu_domain.iommus */
struct iommu_domain *domain; /* domain to which iommu is attached */
+   struct notifier_block genpd_nb;
+   spinlock_t hw_lock; /* lock for register access */
+   bool is_powered; /* power domain is on and register clock is enabled */
 };
 
 static inline void rk_table_flush(u32 *va, unsigned int count)
@@ -283,6 +288,9 @@ static void rk_iommu_zap_lines(struct rk_iommu *iommu, 
dma_addr_t iova,
   size_t size)
 {
dma_addr_t iova_end = iova + size;
+
+   assert_spin_locked(&iommu->hw_lock);
+
/*
 * TODO(djkurtz): Figure out when it is more efficient to shootdown the
 * entire iotlb rather than iterate over individual iovas.
@@ -293,11 +301,15 @@ static void rk_iommu_zap_lines(struct rk_iommu *iommu, 
dma_addr_t iova,
 
 static bool rk_iommu_is_stall_active(struct rk_iommu *iommu)
 {
+   assert_spin_locked(&iommu->hw_lock);
+
return rk_iommu_read(iommu, RK_MMU_STATUS) & RK_MMU_STATUS_STALL_ACTIVE;
 }
 
 static bool rk_iommu_is_paging_enabled(struct rk_iommu *iommu)
 {
+   assert_spin_locked(&iommu->hw_lock);
+
return rk_iommu_read(iommu, RK_MMU_STATUS) &
 RK_MMU_STATUS_PAGING_ENABLED;
 }
@@ -306,6 +318,8 @@ static int rk_iommu_enable_stall(struct rk_iommu *iommu)
 {
int ret;
 
+   assert_spin_locked(&iommu->hw_lock);
+
if (rk_iommu_is_stall_active(iommu))
return 0;
 
@@ -327,6 +341,8 @@ static int rk_iommu_disable_stall(struct rk_iommu *iommu)
 {
int ret;
 
+   assert_spin_locked(&iommu->hw_lock);
+
if (!rk_iommu_is_stall_active(iommu))
return 0;
 
@@ -344,6 +360,8 @@ static int rk_iommu_enable_paging(struct rk_iommu *iommu)
 {
int ret;
 
+   assert_spin_locked(&iommu->hw_lock);
+
if (rk_iommu_is_paging_enabled(iommu))
return 0;
 
@@ -361,6 +379,8 @@ static int rk_iommu_disable_paging(struct rk_iommu *iommu)
 {
int ret;
 
+   assert_spin_locked(&iommu->hw_lock);
+
if (!rk_iommu_is_paging_enabled(iommu))
return 0;
 
@@ -379,6 +399,8 @@ static int rk_iommu_force_reset(struct rk_iommu *iommu)
int ret;
u32 dte_addr;
 
+   assert_spin_locked(&iommu->hw_lock);
+
/*
 * Check if register DTE_ADDR is working by writing DTE_ADDR_DUMMY
 * and verifying that upper 5 nybbles are read back.
@@ -401,6 +423,50 @@ static int rk_iommu_force_reset(struct rk_iommu *iommu)
return ret;
 }
 
+static int rk_iommu_startup(struct rk_iommu *iommu)
+{
+   struct iommu_domain *domain = iommu->domain;
+   struct rk_iommu_domain *rk_domain;
+   phys_addr_t dte_addr;
+   int ret;
+
+   assert_spin_locked(&iommu->hw_lock);
+
+   ret = rk_iommu_enable_stall(iommu);
+   if (ret)
+   return ret;
+
+   ret = rk_iommu_force_reset(iommu);
+   if (ret)
+   return ret;
+
+   rk_domain = domain->priv;
+   dte_addr = virt_to_phys(rk_domain->dt);
+   rk_iommu_write(iommu, RK_MMU_DTE_ADDR, dte_addr);
+   rk_iommu_command(iommu, RK_MMU_CMD_ZAP_CACHE);
+   rk_iommu_write(iommu, RK_MMU_INT_MASK, RK_MMU_IRQ_MASK);
+
+   ret = rk_iommu_enable_paging(iommu);
+   if (ret)
+   return ret;
+
+   rk_iommu_disable_stall(iommu);
+
+   return ret;
+}
+
+static void rk_iommu_shutdown(struct rk_iommu *iommu)
+{
+   assert_spin_locked(&iommu->hw_lock);
+
+   /* Ignore error while disabling, just keep going */
+   rk_iommu_enable_stall(iommu);
+   rk_iommu_disable_paging(iommu);
+   rk_iommu_write(iommu, RK_MMU_INT_MASK, 0);
+   rk_iommu_write(iommu, RK_MMU_DTE_ADDR, 0);
+   rk_iommu_disable_stall(iommu);
+}
+
 static voi

Re: [RFC PATCH 2/2] iommu: rockchip: Handle system-wide and runtime PM

[Tomasz Figa]

Hi Ulf,

On Thu, Dec 11, 2014 at 8:58 PM, Ulf Hansson  wrote:
> On 11 December 2014 at 09:26, Tomasz Figa  wrote:
>> This patch modifies the rockchip-iommu driver to consider state of
>> the power domain the IOMMU is located in. When the power domain
>> is powered off, the IOMMU cannot be accessed and register programming
>> must be deferred until the power domain becomes enabled.

[snip]

>> @@ -988,11 +1107,28 @@ static int rk_iommu_probe(struct platform_device 
>> *pdev)
>> return -ENXIO;
>> }
>>
>> +   pm_runtime_no_callbacks(dev);
>> +   pm_runtime_enable(dev);
>> +
>> +   /* Synchronize state of the domain with driver data. */
>> +   pm_runtime_get_sync(dev);
>> +   iommu->is_powered = true;
>
> Doesn't the runtime PM status reflect the value of "is_powered", thus
> why do you need to have a copy of it? Could it perpahps be that you
> try to cope with the case when CONFIG_PM is unset?
>

It's worth noting that this driver fully relies on status of other
devices in the power domain the IOMMU is in and does not enforce the
status on its own. So in general, as far as my understanding of PM
runtime subsystem, the status of the IOMMU device will be always
suspended, because nobody will call pm_runtime_get() on it (except the
get and put pair in probe). So is_powered is here to track status of
the domain, not the device. Feel free to suggest a better way, though.

Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [RFC PATCH 2/2] iommu: rockchip: Handle system-wide and runtime PM

[Tomasz Figa]

Hi Rafael,

On Fri, Dec 12, 2014 at 5:48 AM, Rafael J. Wysocki  wrote:
> On Thursday, December 11, 2014 04:51:37 PM Ulf Hansson wrote:
>> On 11 December 2014 at 16:31, Kevin Hilman  wrote:
>> > [+ Laurent Pinchart]
>> >
>> > Tomasz Figa  writes:
>> >
>> >> On Thu, Dec 11, 2014 at 8:58 PM, Ulf Hansson  
>> >> wrote:
>> >
>> > [...]
>> >
>> >>>> @@ -988,11 +1107,28 @@ static int rk_iommu_probe(struct platform_device 
>> >>>> *pdev)
>> >>>> return -ENXIO;
>> >>>> }
>> >>>>
>> >>>> +   pm_runtime_no_callbacks(dev);
>> >>>> +   pm_runtime_enable(dev);
>> >>>> +
>> >>>> +   /* Synchronize state of the domain with driver data. */
>> >>>> +   pm_runtime_get_sync(dev);
>> >>>> +   iommu->is_powered = true;
>> >>>
>> >>> Doesn't the runtime PM status reflect the value of "is_powered", thus
>> >>> why do you need to have a copy of it? Could it perpahps be that you
>> >>> try to cope with the case when CONFIG_PM is unset?
>> >>>
>> >>
>> >> It's worth noting that this driver fully relies on status of other
>> >> devices in the power domain the IOMMU is in and does not enforce the
>> >> status on its own. So in general, as far as my understanding of PM
>> >> runtime subsystem, the status of the IOMMU device will be always
>> >> suspended, because nobody will call pm_runtime_get() on it (except the
>> >> get and put pair in probe). So is_powered is here to track status of
>> >> the domain, not the device. Feel free to suggest a better way, though.
>> >
>> > I still don't like these notifiers.  I think they add ways to bypass
>> > having proper runtime PM implemented for devices/subsystems.
>>
>> I do agree, but I haven't found another good solution to the problem.
>
> For the record, I'm not liking this mostly because it "fixes" a generic 
> problem
> in a way that's hidden in the genpd code and very indirect.

Well, that's true. This is indeed a generic problem of PM dependencies
between devices (other than those represented by parent-child
relation), which in fact doesn't have much to do with genpd, but
rather with those devices directly. It is just that genpd is the most
convenient location to solve this in current code and in a simple way.
In other words, I see this solution as a reasonable way to get the
problem solved quickly for now, so that we can start thinking about a
more elegant solution.

>
>> > From a high-level, the IOMMU is just another device inside the PM
>> > domain, so ideally it should be doing it's own _get() and _put() calls
>> > so the PM domain code would just do the right thing without the need for
>> > notifiers.
>>
>> As I understand it, the IOMMU (or for other similar cases) shouldn't
>> be doing any get() and put() at all because there are no IO API to
>> serve request from.
>>
>> In principle we could consider these kind devices as "parent" devices
>> to those other devices that needs them. Then runtime PM core would
>> take care of things for us, right!?
>>
>> Now, I am not so sure using the "parent" approach is actually viable,
>> since it will likely have other complications, but I haven't
>> thoroughly thought it though yet.
>
> That actually need not be a "parent".
>
> What's needed in this case is to do a pm_runtime_get_sync() on a device
> depended on every time a dependent device is runtime-resumed (and analogously
> for suspending).
>
> The core doesn't have a way to do that, but it looks like we'll need to add
> it anyway for various reasons (ACPI _DEP is one of them as I mentioned some
> time ago, but people dismissed it basically as not their problem).

Let me show you our exact use case.

We have a power domain, which contains an IOMMU and an IP block, which
can do bus transactions through that IOMMU. Of course the IP block is
not aware of the IOMMU, because this is just an integration detail and
on other platforms using the same IP block the IOMMU might not be
there. This implies that the driver for this IP block should not be
aware of the IOMMU either, which, on the buffer allocation and mapping
side, is achieved by DMA mapping subsystem. We would also want the
IOMMU to be fully transparent to that driver on PM side.

Now, for PM related details, they are located 

Re: [RFC PATCH 2/2] iommu: rockchip: Handle system-wide and runtime PM

[Tomasz Figa]

On Sat, Dec 13, 2014 at 5:04 AM, Kevin Hilman  wrote:
> Tomasz Figa  writes:
>
> [...]
>
>> We have a power domain, which contains an IOMMU and an IP block, which
>> can do bus transactions through that IOMMU. Of course the IP block is
>> not aware of the IOMMU, because this is just an integration detail and
>> on other platforms using the same IP block the IOMMU might not be
>> there. This implies that the driver for this IP block should not be
>> aware of the IOMMU either, which, on the buffer allocation and mapping
>> side, is achieved by DMA mapping subsystem. We would also want the
>> IOMMU to be fully transparent to that driver on PM side.
>
> An even more exciting problem exists when a CPU is in the same domain as
> other peripherals, those peripherals are all idle and the power domain
> is gated. :)
>

Definitely an exciting problem. :)

Although I think it's quite opposite to ours, because the kernel knows
when the CPU is online and can simply get() the CPU device from one of
the running CPUs when onlining it and put() when offlining. The same
can't be said about IOMMUs, which are essentially bus interfaces for
other IP blocks, not really standalone devices (more than being
standalone IPs).

>> Now, for PM related details, they are located in the same power
>> domain, which means that whenever the power domain is turned off, the
>> CPU can't access their registers and all the hardware state is gone.
>> To make the case more interesting, there is no point in programming
>> the IOMMU unless the device using it is powered on. Similarly, there
>> is no point in powering the domain on to just access the IOMMU. This
>> implies that the power domain should be fully controlled by the
>> stand-alone IP block, while the peripheral IOMMU shouldn't affect its
>> state and its driver only respond to operations performed by driver of
>> that stand-alone IP block.
>
> Which implies that the IOMMU driver should have it's own set of
> runtime_suspend/runtime_resume callbacks to properly save/restore state
> as well.

Yes, but someone needs to be calling them. Right now when genpd is
used, put()ting the last runtime active device in a domain will
trigger domain power off and calling .suspend() callbacks, but
.resume() callback of each device will be called only on first get()
of this device. This means that get()ting e.g. an video codec device
will not cause the IOMMU's .resume() callback to be called.

>
>> A solution like below could work for the above:
>>
>> - There is a per-device list of peripheral devices, which need to be
>> powered on for this device to work.
>> - Whenever the IOMMU subsystem/driver binds an IOMMU to a device, it
>> adds the IOMMU device to the list of peripheral devices of that device
>> (and similarly for removal).
>> - A runtime PM operation on a device will also perform the same
>> operation on all its peripheral devices.
>
> Yes, that is exactly what we need.
>
> I'd rather use the term "dependent" devices rather than peripheral
> devices, but otherwise it sounds like the right approach to me.

I will leave the terminology to you and other people. I'm not good at
inventing names. :)

Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [RFC PATCH 2/2] iommu: rockchip: Handle system-wide and runtime PM

[Tomasz Figa]

On Sat, Dec 13, 2014 at 5:47 AM, Laurent Pinchart
 wrote:
> Hello,
>
> On Friday 12 December 2014 13:15:51 Tomasz Figa wrote:
>> On Fri, Dec 12, 2014 at 5:48 AM, Rafael J. Wysocki wrote:
>> > On Thursday, December 11, 2014 04:51:37 PM Ulf Hansson wrote:
>> >> On 11 December 2014 at 16:31, Kevin Hilman  wrote:
>> >> > [+ Laurent Pinchart]
>> >> >
>> >> > Tomasz Figa  writes:
>> >> >> On Thu, Dec 11, 2014 at 8:58 PM, Ulf Hansson wrote:
>> >> > [...]
>> >> >
>> >> >>>> @@ -988,11 +1107,28 @@ static int rk_iommu_probe(struct
>> >> >>>> platform_device *pdev)>> >>>>
>> >> >>>> return -ENXIO;
>> >> >>>>
>> >> >>>> }
>> >> >>>>
>> >> >>>> +   pm_runtime_no_callbacks(dev);
>> >> >>>> +   pm_runtime_enable(dev);
>> >> >>>> +
>> >> >>>> +   /* Synchronize state of the domain with driver data. */
>> >> >>>> +   pm_runtime_get_sync(dev);
>> >> >>>> +   iommu->is_powered = true;
>> >> >>>
>> >> >>> Doesn't the runtime PM status reflect the value of "is_powered", thus
>> >> >>> why do you need to have a copy of it? Could it perpahps be that you
>> >> >>> try to cope with the case when CONFIG_PM is unset?
>> >> >>
>> >> >> It's worth noting that this driver fully relies on status of other
>> >> >> devices in the power domain the IOMMU is in and does not enforce the
>> >> >> status on its own. So in general, as far as my understanding of PM
>> >> >> runtime subsystem, the status of the IOMMU device will be always
>> >> >> suspended, because nobody will call pm_runtime_get() on it (except the
>> >> >> get and put pair in probe). So is_powered is here to track status of
>> >> >> the domain, not the device. Feel free to suggest a better way, though.
>> >> >
>> >> > I still don't like these notifiers.  I think they add ways to bypass
>> >> > having proper runtime PM implemented for devices/subsystems.
>> >>
>> >> I do agree, but I haven't found another good solution to the problem.
>> >
>> > For the record, I'm not liking this mostly because it "fixes" a generic
>> > problem in a way that's hidden in the genpd code and very indirect.
>>
>> Well, that's true. This is indeed a generic problem of PM dependencies
>> between devices (other than those represented by parent-child
>> relation), which in fact doesn't have much to do with genpd, but
>> rather with those devices directly. It is just that genpd is the most
>> convenient location to solve this in current code and in a simple way.
>> In other words, I see this solution as a reasonable way to get the
>> problem solved quickly for now, so that we can start thinking about a
>> more elegant solution.
>>
>> >> > From a high-level, the IOMMU is just another device inside the PM
>> >> > domain, so ideally it should be doing it's own _get() and _put() calls
>> >> > so the PM domain code would just do the right thing without the need
>> >> > for notifiers.
>> >>
>> >> As I understand it, the IOMMU (or for other similar cases) shouldn't
>> >> be doing any get() and put() at all because there are no IO API to
>> >> serve request from.
>
> Speaking purely from an IOMMU point of view that's not entirely tree. IOMMU
> drivers expose map and unmap operations, so they can track whether any memory
> is mapped. From a bus master point of view the IOMMU map and unmap operations
> are hidden by the DMA mapping API. The IOMMU can thus track the existence of
> mappings without any IOMMU awareness in the bus master driver.
>
> If no mapping exist the IOMMU shouldn't receive any translation request. An
> IOMMU driver could thus call pm_runtime_get_sync() in the map handler when
> creating the first mapping, and pm_runtime_put() in the unmap handler when
> tearing the last mapping down.
>
> One could argue that the IOMMU would end up being powered more often than
> strictly needed, as bus masters drivers, even when written properly, could
> keep mappings around at times they don't perform bus access. This is

Re: [RFC PATCH 2/2] iommu: rockchip: Handle system-wide and runtime PM

[Tomasz Figa]

On Tue, Dec 16, 2014 at 4:53 AM, Laurent Pinchart
 wrote:
> Hi Tomasz,
>
> On Monday 15 December 2014 11:39:01 Tomasz Figa wrote:
>> On Sat, Dec 13, 2014 at 5:47 AM, Laurent Pinchart wrote:
>> > On Friday 12 December 2014 13:15:51 Tomasz Figa wrote:
>> >> On Fri, Dec 12, 2014 at 5:48 AM, Rafael J. Wysocki wrote:
>> >>> On Thursday, December 11, 2014 04:51:37 PM Ulf Hansson wrote:
>> >>>> On 11 December 2014 at 16:31, Kevin Hilman  wrote:
>> >>>>> [+ Laurent Pinchart]
>> >>>>>
>> >>>>> Tomasz Figa  writes:
>> >>>>>> On Thu, Dec 11, 2014 at 8:58 PM, Ulf Hansson wrote:
>> >>>>> [...]
>> >>>>>
>> >>>>>>>> @@ -988,11 +1107,28 @@ static int rk_iommu_probe(struct
>> >>>>>>>> platform_device *pdev)
>> >>>>>>>> return -ENXIO;
>> >>>>>>>> }
>> >>>>>>>>
>> >>>>>>>> +   pm_runtime_no_callbacks(dev);
>> >>>>>>>> +   pm_runtime_enable(dev);
>> >>>>>>>> +
>> >>>>>>>> +   /* Synchronize state of the domain with driver data. */
>> >>>>>>>> +   pm_runtime_get_sync(dev);
>> >>>>>>>> +   iommu->is_powered = true;
>> >>>>>>>
>> >>>>>>> Doesn't the runtime PM status reflect the value of "is_powered",
>> >>>>>>> thus why do you need to have a copy of it? Could it perpahps be
>> >>>>>>> that you try to cope with the case when CONFIG_PM is unset?
>> >>>>>>
>> >>>>>> It's worth noting that this driver fully relies on status of other
>> >>>>>> devices in the power domain the IOMMU is in and does not enforce
>> >>>>>> the status on its own. So in general, as far as my understanding of
>> >>>>>> PM runtime subsystem, the status of the IOMMU device will be always
>> >>>>>> suspended, because nobody will call pm_runtime_get() on it (except
>> >>>>>> the get and put pair in probe). So is_powered is here to track
>> >>>>>> status of the domain, not the device. Feel free to suggest a better
>> >>>>>> way, though.
>> >>>>>
>> >>>>> I still don't like these notifiers.  I think they add ways to bypass
>> >>>>> having proper runtime PM implemented for devices/subsystems.
>> >>>>
>> >>>> I do agree, but I haven't found another good solution to the problem.
>> >>>
>> >>> For the record, I'm not liking this mostly because it "fixes" a generic
>> >>> problem in a way that's hidden in the genpd code and very indirect.
>> >>
>> >> Well, that's true. This is indeed a generic problem of PM dependencies
>> >> between devices (other than those represented by parent-child
>> >> relation), which in fact doesn't have much to do with genpd, but
>> >> rather with those devices directly. It is just that genpd is the most
>> >> convenient location to solve this in current code and in a simple way.
>> >> In other words, I see this solution as a reasonable way to get the
>> >> problem solved quickly for now, so that we can start thinking about a
>> >> more elegant solution.
>> >>
>> >> >> > From a high-level, the IOMMU is just another device inside the PM
>> >> >> > domain, so ideally it should be doing it's own _get() and _put()
>> >> >> > calls so the PM domain code would just do the right thing without
>> >> >> > the need for notifiers.
>> >> >>
>> >> >> As I understand it, the IOMMU (or for other similar cases) shouldn't
>> >> >> be doing any get() and put() at all because there are no IO API to
>> >> >> serve request from.
>> >
>> > Speaking purely from an IOMMU point of view that's not entirely tree.
>> > IOMMU drivers expose map and unmap operations, so they can track whether
>> > any memory is mapped. From a bus master point of view the IOMMU map and
>> > unmap operations are hidden by the DMA mapping API. The IOMMU can thus
>> >

[PATCH] CHROMIUM: iommu: rockchip: Make sure that page table state is coherent

[Tomasz Figa]

Even though the code uses the dt_lock spin lock to serialize mapping
operation from different threads, it does not protect from IOMMU
accesses that might be already taking place and thus altering state
of the IOTLB. This means that current mapping code which first zaps
the page table and only then updates it with new mapping which is
prone to mentioned race.

In addition, current code assumes that mappings are always > 4 MiB
(which translates to 1024 PTEs) and so they would always occupy
entire page tables. This is not true for mappings created by V4L2
Videobuf2 DMA contig allocator.

This patch changes the mapping code to always zap the page table
after it is updated, which avoids the aforementioned race and also
zap the last page of the mapping to make sure that stale data is
not cached from an already existing mapping.

Signed-off-by: Tomasz Figa 
Reviewed-by: Daniel Kurtz 
---
 drivers/iommu/rockchip-iommu.c | 23 +--
 1 file changed, 17 insertions(+), 6 deletions(-)

diff --git a/drivers/iommu/rockchip-iommu.c b/drivers/iommu/rockchip-iommu.c
index 6a8b1ec..b06fe76 100644
--- a/drivers/iommu/rockchip-iommu.c
+++ b/drivers/iommu/rockchip-iommu.c
@@ -544,6 +544,15 @@ static void rk_iommu_zap_iova(struct rk_iommu_domain 
*rk_domain,
spin_unlock_irqrestore(&rk_domain->iommus_lock, flags);
 }
 
+static void rk_iommu_zap_iova_first_last(struct rk_iommu_domain *rk_domain,
+dma_addr_t iova, size_t size)
+{
+   rk_iommu_zap_iova(rk_domain, iova, SPAGE_SIZE);
+   if (size > SPAGE_SIZE)
+   rk_iommu_zap_iova(rk_domain, iova + size - SPAGE_SIZE,
+   SPAGE_SIZE);
+}
+
 static u32 *rk_dte_get_page_table(struct rk_iommu_domain *rk_domain,
  dma_addr_t iova)
 {
@@ -568,12 +577,6 @@ static u32 *rk_dte_get_page_table(struct rk_iommu_domain 
*rk_domain,
rk_table_flush(page_table, NUM_PT_ENTRIES);
rk_table_flush(dte_addr, 1);
 
-   /*
-* Zap the first iova of newly allocated page table so iommu evicts
-* old cached value of new dte from the iotlb.
-*/
-   rk_iommu_zap_iova(rk_domain, iova, SPAGE_SIZE);
-
 done:
pt_phys = rk_dte_pt_address(dte);
return (u32 *)phys_to_virt(pt_phys);
@@ -623,6 +626,14 @@ static int rk_iommu_map_iova(struct rk_iommu_domain 
*rk_domain, u32 *pte_addr,
 
rk_table_flush(pte_addr, pte_count);
 
+   /*
+* Zap the first and last iova to evict from iotlb any previously
+* mapped cachelines holding stale values for its dte and pte.
+* We only zap the first and last iova, since only they could have
+* dte or pte shared with an existing mapping.
+*/
+   rk_iommu_zap_iova_first_last(rk_domain, iova, size);
+
return 0;
 unwind:
/* Unmap the range of iovas that we just mapped */
-- 
2.2.0.rc0.207.ga3a616c

___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH] CHROMIUM: iommu: rockchip: Make sure that page table state is coherent

[Tomasz Figa]

On Mon, Feb 9, 2015 at 8:19 PM, Tomasz Figa  wrote:
> Even though the code uses the dt_lock spin lock to serialize mapping
> operation from different threads, it does not protect from IOMMU
> accesses that might be already taking place and thus altering state
> of the IOTLB. This means that current mapping code which first zaps
> the page table and only then updates it with new mapping which is
> prone to mentioned race.

Oops, forgot to remove the "CHROMIUM" tag. Is that something that
could be fixed when applying (if the patch is otherwise okay) or
should I resend?

Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH 2/5] iommu/mediatek: Add mt8173 IOMMU driver

[Tomasz Figa]

Hi,

You can find part 2 of my comments inline.

On Fri, Mar 6, 2015 at 7:48 PM,   wrote:

[snip]

> +static irqreturn_t mtk_iommu_isr(int irq, void *dev_id)
> +{
> +   struct iommu_domain *domain = dev_id;
> +   struct mtk_iommu_domain *mtkdomain = domain->priv;
> +   struct mtk_iommu_info *piommu = mtkdomain->piommuinfo;
> +
> +   if (irq == piommu->irq)
> +   report_iommu_fault(domain, piommu->dev, 0, 0);

In addition to my previous comment about how this gets called from
this handler, you need to keep in mind that the function called by
report_iommu_fault() might actually be a different function than
mtk_iommu_fault_handler(), because upper layers can provide their own
handlers. This means that you need to perform any operations on
hardware from this handler and only use the iommu fault handler as a
way to tell an upper layer about the fault (including notifying the
user through kernel log if there is no special handler installed and
the generic fallback is used).

> +   else
> +   dev_err(piommu->dev, "irq number:%d\n", irq);
> +
> +   return IRQ_HANDLED;
> +}

[snip]

> +static int mtk_iommu_fault_handler(struct iommu_domain *imudomain,
> +  struct device *dev, unsigned long iova,
> +  int m4uindex, void *pimu)
> +{
> +   void __iomem *m4u_base;
> +   u32 int_state, regval;
> +   int m4u_slave_id = 0;
> +   unsigned int layer, write, m4u_port;
> +   unsigned int fault_mva, fault_pa;
> +   struct mtk_iommu_info *piommu = pimu;
> +   struct mtk_iommu_domain *mtkdomain = imudomain->priv;
> +
> +   m4u_base = piommu->m4u_base;
> +   int_state = readl(m4u_base + REG_MMU_MAIN_FAULT_ST);
> +
> +   /* read error info from registers */
> +   fault_mva = readl(m4u_base + REG_MMU_FAULT_VA(m4u_slave_id));
> +   layer = !!(fault_mva & F_MMU_FAULT_VA_LAYER_BIT);
> +   write = !!(fault_mva & F_MMU_FAULT_VA_WRITE_BIT);
> +   fault_mva &= F_MMU_FAULT_VA_MSK;
> +   fault_pa = readl(m4u_base + REG_MMU_INVLD_PA(m4u_slave_id));
> +   regval = readl(m4u_base + REG_MMU_INT_ID(m4u_slave_id));
> +   regval &= F_MMU0_INT_ID_TF_MSK;
> +   m4u_port = mtk_iommu_get_port_by_tfid(piommu, regval);
> +
> +   if (int_state & F_INT_TRANSLATION_FAULT(m4u_slave_id)) {
> +   struct m4u_pte_info_t pte;
> +   unsigned long flags;
> +
> +   spin_lock_irqsave(&mtkdomain->pgtlock, flags);
> +   m4u_get_pte_info(mtkdomain, fault_mva, &pte);
> +   spin_unlock_irqrestore(&mtkdomain->pgtlock, flags);
> +
> +   if (pte.size == MMU_SMALL_PAGE_SIZE ||
> +   pte.size == MMU_LARGE_PAGE_SIZE) {
> +   dev_err_ratelimited(
> +   dev,
> +   "fault:port=%s iova=0x%x pa=0x%x layer=%d %s;"
> +   
> "pgd(0x%x)->pte(0x%x)->pa(%pad)sz(0x%x)Valid(%d)\n",
> +   mtk_iommu_get_port_name(piommu, m4u_port),
> +   fault_mva, fault_pa, layer,
> +   write ? "write" : "read",
> +   imu_pgd_val(*pte.pgd), imu_pte_val(*pte.pte),
> +   &pte.pa, pte.size, pte.valid);
> +   } else {
> +   dev_err_ratelimited(
> +   dev,
> +   "fault:port=%s iova=0x%x pa=0x%x layer=%d %s;"
> +   "pgd(0x%x)->pa(%pad)sz(0x%x)Valid(%d)\n",
> +   mtk_iommu_get_port_name(piommu, m4u_port),
> +   fault_mva, fault_pa, layer,
> +   write ? "write" : "read",
> +   imu_pgd_val(*pte.pgd),
> +   &pte.pa, pte.size, pte.valid);
> +   }
> +   }
> +
> +   if (int_state & F_INT_MAIN_MULTI_HIT_FAULT(m4u_slave_id))
> +   dev_err_ratelimited(dev, "multi-hit!port=%s iova=0x%x\n",
> +   mtk_iommu_get_port_name(piommu, m4u_port),
> +   fault_mva);
> +
> +   if (int_state & F_INT_INVALID_PA_FAULT(m4u_slave_id)) {
> +   if (!(int_state & F_INT_TRANSLATION_FAULT(m4u_slave_id)))
> +   dev_err_ratelimited(dev, "invalid pa!port=%s 
> iova=0x%x\n",
> +   mtk_iommu_get_port_name(piommu,
> +   m4u_port),
> +   fault_mva);
> +   }
> +   if (int_state & F_INT_ENTRY_REPLACEMENT_FAULT(m4u_slave_id))
> +   dev_err_ratelimited(dev, "replace-fault!port=%s iova=0x%x\n",
> +   mtk_iommu_get_port_name(piommu, m4u_port),
> + 

Re: [PATCH 2/5] iommu/mediatek: Add mt8173 IOMMU driver

[Tomasz Figa]

On Mon, Mar 9, 2015 at 11:46 PM, Yingjoe Chen  wrote:
> On Mon, 2015-03-09 at 20:11 +0900, Tomasz Figa wrote:
> <...>
>> > +/*
>> > + * pimudev is a global var for dma_alloc_coherent.
>> > + * It is not accepatable, we will delete it if "domain_alloc" is enabled
>> > + */
>> > +static struct device *pimudev;
>>
>> This is indeed not acceptable. Could you replace dma_alloc_coherent()
>> with something that doesn't require device pointer, e.g.
>> alloc_pages()? (Although that would require you to handle cache
>> maintenance in the driver, due to cached memory allocated.) I need to
>> think about a better solution for this.
>
> Hi,
>
> For 2nd level page table, we use cached memory now. Currently we are
> using __dma_flush_range to flush the cache, which is also unacceptable.
>
> For proper cache management, we'll need to use dma_map_single or
> dma_sync_*, which still need a deivce*.

Looking at how already mainlined drivers do this, they either use
dmac_flush_range()
(https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/drivers/iommu/msm_iommu.c?id=refs/tags/v4.0-rc3#n80)
or directly __cpuc_flush_dcache_area() and outer_flush_range()
(https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/drivers/iommu/rockchip-iommu.c?id=refs/tags/v4.0-rc3#n93).

Joerg, what's your opinion on this?

Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH 2/5] iommu/mediatek: Add mt8173 IOMMU driver

[Tomasz Figa]

On Tue, Mar 10, 2015 at 12:41 PM, Yingjoe Chen
 wrote:
> On Tue, 2015-03-10 at 02:00 +0900, Tomasz Figa wrote:
>> On Mon, Mar 9, 2015 at 11:46 PM, Yingjoe Chen  
>> wrote:
>> > On Mon, 2015-03-09 at 20:11 +0900, Tomasz Figa wrote:
>> > <...>
>> >> > +/*
>> >> > + * pimudev is a global var for dma_alloc_coherent.
>> >> > + * It is not accepatable, we will delete it if "domain_alloc" is 
>> >> > enabled
>> >> > + */
>> >> > +static struct device *pimudev;
>> >>
>> >> This is indeed not acceptable. Could you replace dma_alloc_coherent()
>> >> with something that doesn't require device pointer, e.g.
>> >> alloc_pages()? (Although that would require you to handle cache
>> >> maintenance in the driver, due to cached memory allocated.) I need to
>> >> think about a better solution for this.
>> >
>> > Hi,
>> >
>> > For 2nd level page table, we use cached memory now. Currently we are
>> > using __dma_flush_range to flush the cache, which is also unacceptable.
>> >
>> > For proper cache management, we'll need to use dma_map_single or
>> > dma_sync_*, which still need a deivce*.
>>
>> Looking at how already mainlined drivers do this, they either use
>> dmac_flush_range()
>> (https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/drivers/iommu/msm_iommu.c?id=refs/tags/v4.0-rc3#n80)
>> or directly __cpuc_flush_dcache_area() and outer_flush_range()
>> (https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/drivers/iommu/rockchip-iommu.c?id=refs/tags/v4.0-rc3#n93).
>
> Hi,
>
> These only exist in arch/arm, not arm64. I think we should avoid using
> API start with __ in drivers. This driver might be used in both
> arm/arm64, I think the only option for us is DMA APIs.
>
> Actually, I'm thinking that we should change to use coherent memory for
> 2nd level page table as well and totally skip the cache flush. It seems
> dma_pool_create is suitable to replace kmem_cache we are using right
> now. However it still need a device*, which we have to fix anyway.

That sounds like a reasonable option, because this is what we have DMA
mapping API for.

Do you expect to have more than one M4U block inside a SoC? Maybe this
static variable actually isn't that bad, with a comment added
explaining that there is always only one such block and that a rework
will be needed if future SoCs will have more.

Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH 2/5] iommu/mediatek: Add mt8173 IOMMU driver

[Tomasz Figa]

Hi,

Please find next part of my comments inline.

On Fri, Mar 6, 2015 at 7:48 PM,   wrote:

[snip]

> +/*
> + * pimudev is a global var for dma_alloc_coherent.
> + * It is not accepatable, we will delete it if "domain_alloc" is enabled

It looks like we indeed need to use dma_alloc_coherent() and we don't
have a good way to pass the device pointer to domain_init callback.

If you don't expect SoCs in the nearest future to have multiple M4U
blocks, then I guess this global variable could stay, after changing
the comment into an explanation why it's correct. Also it should be
moved to the top of the file, below #include directives, as this is
where usually global variables are located.

> + */
> +static struct device *pimudev;
> +
> +static int mtk_iommu_domain_init(struct iommu_domain *domain)
> +{
> +   struct mtk_iommu_domain *priv;
> +
> +   priv = kzalloc(sizeof(*priv), GFP_KERNEL);
> +   if (!priv)
> +   return -ENOMEM;
> +
> +   priv->pgd = dma_alloc_coherent(pimudev, M4U_PGD_SIZE, &priv->pgd_pa,
> +  GFP_KERNEL);
> +   if (!priv->pgd) {
> +   pr_err("dma_alloc_coherent pagetable fail\n");
> +   goto err_pgtable;

ret = -ENOMEM;
goto err_free_priv;

> +   }
> +
> +   if (!IS_ALIGNED(priv->pgd_pa, M4U_PGD_SIZE)) {
> +   pr_err("pagetable not aligned pa 0x%pad-0x%p align 0x%x\n",
> +  &priv->pgd_pa, priv->pgd, M4U_PGD_SIZE);
> +   goto err_pgtable;

ret = -ENOMEM;
goto err_dmafree;

> +   }
> +
> +   memset(priv->pgd, 0, M4U_PGD_SIZE);
> +
> +   spin_lock_init(&priv->pgtlock);
> +   spin_lock_init(&priv->portlock);
> +   domain->priv = priv;
> +
> +   domain->geometry.aperture_start = 0;
> +   domain->geometry.aperture_end   = (unsigned int)~0;

Please replace the cast with ~0U and fix multiple spaces between =.

> +   domain->geometry.force_aperture = true;
> +
> +   return 0;
> +
> +err_pgtable:

err_dma_free:

> +   if (priv->pgd)

Remove this check.

> +   dma_free_coherent(pimudev, M4U_PGD_SIZE, priv->pgd,
> + priv->pgd_pa);

err_free_priv:

> +   kfree(priv);
> +   return -ENOMEM;

return ret;

> +}
> +
> +static void mtk_iommu_domain_destroy(struct iommu_domain *domain)
> +{
> +   struct mtk_iommu_domain *priv = domain->priv;
> +
> +   dma_free_coherent(priv->piommuinfo->dev, M4U_PGD_SIZE,
> + priv->pgd, priv->pgd_pa);
> +   kfree(domain->priv);
> +   domain->priv = NULL;
> +}
> +
> +static int mtk_iommu_attach_device(struct iommu_domain *domain,
> +  struct device *dev)
> +{
> +   unsigned long flags;
> +   struct mtk_iommu_domain *priv = domain->priv;
> +   struct mtk_iommu_info *piommu = priv->piommuinfo;
> +   struct of_phandle_args out_args = {0};
> +   struct device *imudev;
> +   unsigned int i = 0;
> +
> +   if (!piommu)

Could you explain when this can happen?

> +   goto imudev;

return 0;

> +   else

No else needed.

> +   imudev = piommu->dev;
> +
> +   spin_lock_irqsave(&priv->portlock, flags);

What is protected by this spinlock?

> +
> +   while (!of_parse_phandle_with_args(dev->of_node, "iommus",
> +  "#iommu-cells", i, &out_args)) {
> +   if (1 == out_args.args_count) {

Can we be sure that this is actually referring to our IOMMU?

Maybe this should be rewritten to

if (out_args.np != imudev->of_node)
continue;
if (out_args.args_count != 1) {
dev_err(imudev, "invalid #iommu-cells property for IOMMU %s\n",

}

> +   unsigned int portid = out_args.args[0];
> +
> +   dev_dbg(dev, "iommu add port:%d\n", portid);

imudev should be used here instead of dev.

> +
> +   mtk_iommu_config_port(piommu, portid);
> +
> +   if (i == 0)
> +   dev->archdata.dma_ops =
> +   piommu->dev->archdata.dma_ops;

Shouldn't this be set automatically by IOMMU or DMA mapping core?

> +   }
> +   i++;
> +   }
> +
> +   spin_unlock_irqrestore(&priv->portlock, flags);
> +
> +imudev:
> +   return 0;
> +}
> +
> +static void mtk_iommu_detach_device(struct iommu_domain *domain,
> +   struct device *dev)
> +{

No hardware (de)configuration or clean-up necessary?

> +}
> +
> +static int mtk_iommu_map(struct iommu_domain *domain, unsigned long iova,
> +phys_addr_t paddr, size_t size, int prot)
> +{
> +   struct mtk_iommu_domain *priv = domain->priv;
> +   unsigned long flags;
> +   int ret;
> +
> +   spin_lock_irqsave(&priv->pgtlock, flags);
> +   ret = m4u_map(priv, (unsigned int)iova, paddr, size, prot);
> +   mtk_iommu_invali

Re: [PATCH] CHROMIUM: iommu: rockchip: Make sure that page table state is coherent

[Tomasz Figa]

Sorry, I had to dig my way out through my backlog.

On Tue, Mar 3, 2015 at 10:36 PM, Joerg Roedel  wrote:
> On Mon, Feb 09, 2015 at 08:19:21PM +0900, Tomasz Figa wrote:
>> Even though the code uses the dt_lock spin lock to serialize mapping
>> operation from different threads, it does not protect from IOMMU
>> accesses that might be already taking place and thus altering state
>> of the IOTLB. This means that current mapping code which first zaps
>> the page table and only then updates it with new mapping which is
>> prone to mentioned race.
>
> Could you elabortate a bit on the race and why it is sufficient to zap
> only the first and the last iova? From the description and the comments
> in the patch this is not clear to me.

Let's start with why it's sufficient to zap only first and last iova.

While unmapping, the driver zaps all iovas belonging to the mapping,
so the page tables not used by any mapping won't be cached. Now when
the driver creates a mapping it might end up occupying several page
tables. However, since the mapping area is virtually contiguous, only
the first and last page table can be shared with different mappings.
This means that only first and last iovas can be already cached. In
fact, we could detect if first and last page tables are shared and do
not zap at all, but this wouldn't really optimize too much. Why
invalidating one iova is enough to invalidate the whole page table is
unclear to me as well, but it seems to be the correct way on this
hardware.

As for the race, it's also kind of explained by the above. The already
running hardware can trigger page table look-ups in the IOMMU and so
caching of the page table between our zapping and updating its
contents. With this patch zapping is performed after updating the page
table so the race is gone.

Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH 2/5] iommu/mediatek: Add mt8173 IOMMU driver

[Tomasz Figa]

On Tue, Apr 14, 2015 at 3:31 PM, Yong Wu  wrote:
>> >>
>> >> > +
>> >> > +   piommu->protect_va = devm_kmalloc(piommu->dev, 
>> >> > MTK_PROTECT_PA_ALIGN*2,
>> >>
>> >> style: Operators like * should have space on both sides.
>> >>
>> >> > + GFP_KERNEL);
>> >>
>> >> Shouldn't dma_alloc_coherent() be used for this?
>> >  We don't care the data in it. I think they are the same. Could you
>> > help tell me why dma_alloc_coherent may be better.
>>
>> Can you guarantee that at the time you allocate the memory using
>> devm_kmalloc() the memory is not dirty (i.e. some write back data are
>> stored in CPU cache) and is not going to be written back in some time,
>> overwriting data put there by IOMMU hardware?
>>
> As I noted in the function "mtk_iommu_hw_init":
>
>/* protect memory,HW will write here while translation fault */
>protectpa = __virt_to_phys(piommu->protect_va);
>
>  We don’t care the content of this buffer, It is ok even though its
> data is dirty.
> It seem to be a the protect memory. While a translation fault
> happened, The iommu HW will overwrite here instead of writing to the
> fault physical address which may be 0 or some random address.
>

Do you mean that it's just a dummy page for hardware behind the IOMMU
to access when the mapping is not available? How would that work with
potential on demand paging when the hardware needs to be blocked until
the mapping is created?

Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH 2/5] iommu/mediatek: Add mt8173 IOMMU driver

[Tomasz Figa]

On Wed, Apr 15, 2015 at 4:06 PM, Yong Wu  wrote:
> On Wed, 2015-04-15 at 11:20 +0900, Tomasz Figa wrote:
>> On Tue, Apr 14, 2015 at 3:31 PM, Yong Wu  wrote:
>> >> >>
>> >> >> > +
>> >> >> > +   piommu->protect_va = devm_kmalloc(piommu->dev, 
>> >> >> > MTK_PROTECT_PA_ALIGN*2,
>> >> >>
>> >> >> style: Operators like * should have space on both sides.
>> >> >>
>> >> >> > + GFP_KERNEL);
>> >> >>
>> >> >> Shouldn't dma_alloc_coherent() be used for this?
>> >> >  We don't care the data in it. I think they are the same. Could you
>> >> > help tell me why dma_alloc_coherent may be better.
>> >>
>> >> Can you guarantee that at the time you allocate the memory using
>> >> devm_kmalloc() the memory is not dirty (i.e. some write back data are
>> >> stored in CPU cache) and is not going to be written back in some time,
>> >> overwriting data put there by IOMMU hardware?
>> >>
>> > As I noted in the function "mtk_iommu_hw_init":
>> >
>> >/* protect memory,HW will write here while translation fault */
>> >protectpa = __virt_to_phys(piommu->protect_va);
>> >
>> >  We don’t care the content of this buffer, It is ok even though its
>> > data is dirty.
>> > It seem to be a the protect memory. While a translation fault
>> > happened, The iommu HW will overwrite here instead of writing to the
>> > fault physical address which may be 0 or some random address.
>> >
>>
>> Do you mean that it's just a dummy page for hardware behind the IOMMU
>> to access when the mapping is not available? How would that work with
>> potential on demand paging when the hardware needs to be blocked until
>> the mapping is created?
>>
>> Best regards,
>> Tomasz
>  1. YES
>  2. Sorry. Our iommu HW can not support this right now. The HW can not
> be blocked until the mapping is created.

OK, that explains it. Well, then I guess this is necessary and
contents of that memory don't matter that much. (Although, this might
be a minor security issue, because the faulting hardware would get
access to some data previously stored by kernel code. Not sure how
much of a threat would that be, though.)

> If the page is not ready, we can not get the physical address, then
> How to fill the pagetable for that memory. I think the dma&iommu may
> guaranty it?

If your hardware can't block until the mapping is created then what
you do currently seems to be the only option. (+/- the missing cache
maintenance at initialization)

Best regards,
Tomasz
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[PATCH v2] CHROMIUM: iommu: rockchip: Make sure that page table state is coherent

[Tomasz Figa]

To flush created mappings, current mapping code relies on the fact that
during unmap the driver zaps every IOVA being unmapped and that it is
enough to zap a single IOVA of page table to remove the entire page
table from IOMMU cache. Based on these assumptions the driver was made to
simply zap the first IOVA of the mapping being created. This is enough
to invalidate first page table, which could be shared with another
mapping (and thus could be already present in IOMMU cache), but
unfortunately it does not do anything about the last page table that
could be shared with other mappings as well.

Moreover, the flushing is performed before page table contents are
actually modified, so there is a race between the CPU updating the page
tables and hardware that could be possibly running at the same time and
triggering IOMMU look-ups, which could bring back the page tables back
to the cache.

To fix both issues, this patch makes the mapping code zap first and last
(if they are different) IOVAs of new mapping after the page table is
updated.

Signed-off-by: Tomasz Figa 
Reviewed-by: Daniel Kurtz 
Tested-by: Heiko Stuebner 
---
 drivers/iommu/rockchip-iommu.c | 23 +--
 1 file changed, 17 insertions(+), 6 deletions(-)

diff --git a/drivers/iommu/rockchip-iommu.c b/drivers/iommu/rockchip-iommu.c
index 4015560..31004c0 100644
--- a/drivers/iommu/rockchip-iommu.c
+++ b/drivers/iommu/rockchip-iommu.c
@@ -551,6 +551,15 @@ static void rk_iommu_zap_iova(struct rk_iommu_domain 
*rk_domain,
spin_unlock_irqrestore(&rk_domain->iommus_lock, flags);
 }
 
+static void rk_iommu_zap_iova_first_last(struct rk_iommu_domain *rk_domain,
+dma_addr_t iova, size_t size)
+{
+   rk_iommu_zap_iova(rk_domain, iova, SPAGE_SIZE);
+   if (size > SPAGE_SIZE)
+   rk_iommu_zap_iova(rk_domain, iova + size - SPAGE_SIZE,
+   SPAGE_SIZE);
+}
+
 static u32 *rk_dte_get_page_table(struct rk_iommu_domain *rk_domain,
  dma_addr_t iova)
 {
@@ -575,12 +584,6 @@ static u32 *rk_dte_get_page_table(struct rk_iommu_domain 
*rk_domain,
rk_table_flush(page_table, NUM_PT_ENTRIES);
rk_table_flush(dte_addr, 1);
 
-   /*
-* Zap the first iova of newly allocated page table so iommu evicts
-* old cached value of new dte from the iotlb.
-*/
-   rk_iommu_zap_iova(rk_domain, iova, SPAGE_SIZE);
-
 done:
pt_phys = rk_dte_pt_address(dte);
return (u32 *)phys_to_virt(pt_phys);
@@ -630,6 +633,14 @@ static int rk_iommu_map_iova(struct rk_iommu_domain 
*rk_domain, u32 *pte_addr,
 
rk_table_flush(pte_addr, pte_count);
 
+   /*
+* Zap the first and last iova to evict from iotlb any previously
+* mapped cachelines holding stale values for its dte and pte.
+* We only zap the first and last iova, since only they could have
+* dte or pte shared with an existing mapping.
+*/
+   rk_iommu_zap_iova_first_last(rk_domain, iova, size);
+
return 0;
 unwind:
/* Unmap the range of iovas that we just mapped */
-- 
2.2.0.rc0.207.ga3a616c

___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [V3, 2/2] media: i2c: Add Omnivision OV02A10 camera sensor driver

[Tomasz Figa]

On Wed, Aug 21, 2019 at 8:05 PM Sakari Ailus
 wrote:
>
> Hi Tomasz,
>
> On Wed, Aug 21, 2019 at 07:30:38PM +0900, Tomasz Figa wrote:
[snip]
> > Is it really correct to enable the clock before the regulators?
> >
> > According to the datasheet, it should be:
> >  - PD pin HIGH,
> >  - nRST pin LOW,
> >  - DVDDIO and AVDD28 power up and stabilize,
> >  - clock enabled,
> >  - min 5 ms delay,
> >  - PD pin LOW,
> >  - min 4 ms delay,
> >  - nRST pin HIGH,
> >  - min 5 ms delay,
> >  - I2C interface ready.
> >
> > > +
> > > +   /* Note: set 0 is high, set 1 is low */
> >
> > Why is that? If there is some inverter on the way that should be handled
> > outside of this driver. (GPIO DT bindings have flags for this purpose.
> >
> > If the pins are nRESET and nPOWERDOWN in the hardware datasheet, we should
> > call them like this in the driver too (+/- the lowercase and underscore
> > convention).
> >
> > According to the datasheet, the reset pin is called RST and inverted, so we 
> > should
> > call it n_rst, but the powerdown signal, called PD, is not inverted, so pd
> > would be the right name.
>
> For what it's worth sensors generally have xshutdown (or reset) pin that is
> active high. Looking at the code, it is not the case here. It's a bit odd
> since the usual arrangement saves power when the camera is not in use; it's
> not a lot but still. Oh well.
>

I guess we could drive powerdown low after disabling the regulators
and clocks, but that wouldn't work for the cases where the regulators
are actually shared with something else, especially if that is not
related to the same camera module.

> ...
>
> > > +static struct i2c_driver ov02a10_i2c_driver = {
> > > +   .driver = {
> > > +   .name = "ov02a10",
> > > +   .pm = &ov02a10_pm_ops,
> > > +   .of_match_table = ov02a10_of_match,
> >
> > Please use of_match_ptr() wrapper.
>
> Not really needed; the driver does expect regulators, GPIOs etc., but by
> leaving out of_match_ptr(), the driver will also probe on ACPI based
> systems.

Good point, I always keep forgetting about the ability to probe OF
drivers from ACPI. Then we also need to remove the #if
IS_ENABLED(CONFIG_OF) from ov02a10_of_match.

Best regards,
Tomasz

Re: [PATCH 2/2] media: i2c: dw9768: Add DW9768 VCM driver

[Tomasz Figa]

Hi Dongchun,

On Tue, Sep 3, 2019 at 12:02 AM Dongchun Zhu  wrote:
>
> Hi Tomasz,
>
> On Fri, 2019-08-23 at 17:17 +0900, Tomasz Figa wrote:
> > Hi Dongchun,
> >
> > On Mon, Jul 08, 2019 at 06:06:41PM +0800, dongchun@mediatek.com wrote:
> > > From: Dongchun Zhu 
> > >
> > > This patch adds a V4L2 sub-device driver for DW9768 lens voice coil,
> > > and provides control to set the desired focus.
> > >
> > > The DW9807 is a 10 bit DAC from Dongwoon, designed for linear
> > > control of voice coil motor.
> > >
> > > Signed-off-by: Dongchun Zhu 
> > > ---
> > >  MAINTAINERS|   1 +
> > >  drivers/media/i2c/Kconfig  |  10 +
> > >  drivers/media/i2c/Makefile |   1 +
> > >  drivers/media/i2c/dw9768.c | 458 
> > > +
> > >  4 files changed, 470 insertions(+)
> > >  create mode 100644 drivers/media/i2c/dw9768.c
> > >
> >
> > Thanks for the patch! Please see my comments inline.
> >
> > > diff --git a/MAINTAINERS b/MAINTAINERS
> > > index 8f6ac93..17152d7 100644
> > > --- a/MAINTAINERS
> > > +++ b/MAINTAINERS
> > > @@ -4877,6 +4877,7 @@ M:Dongchun Zhu 
> > >  L: linux-me...@vger.kernel.org
> > >  T: git git://linuxtv.org/media_tree.git
> > >  S: Maintained
> > > +F: drivers/media/i2c/dw9768.c
> > >  F: Documentation/devicetree/bindings/media/i2c/dongwoon,dw9768.txt
> > >
> > >  DONGWOON DW9807 LENS VOICE COIL DRIVER
> > > diff --git a/drivers/media/i2c/Kconfig b/drivers/media/i2c/Kconfig
> > > index 7793358..8ff6c95 100644
> > > --- a/drivers/media/i2c/Kconfig
> > > +++ b/drivers/media/i2c/Kconfig
> > > @@ -1014,6 +1014,16 @@ config VIDEO_DW9714
> > >   capability. This is designed for linear control of
> > >   voice coil motors, controlled via I2C serial interface.
> > >
> > > +config VIDEO_DW9768
> > > +   tristate "DW9768 lens voice coil support"
> > > +   depends on I2C && VIDEO_V4L2 && MEDIA_CONTROLLER
> > > +   depends on VIDEO_V4L2_SUBDEV_API
> > > +   help
> > > + This is a driver for the DW9768 camera lens voice coil.
> > > + DW9768 is a 10 bit DAC with 100mA output current sink
> > > + capability. This is designed for linear control of
> > > + voice coil motors, controlled via I2C serial interface.
> > > +
> > >  config VIDEO_DW9807_VCM
> > > tristate "DW9807 lens voice coil support"
> > > depends on I2C && VIDEO_V4L2 && MEDIA_CONTROLLER
> > > diff --git a/drivers/media/i2c/Makefile b/drivers/media/i2c/Makefile
> > > index d8ad9da..944fbf6 100644
> > > --- a/drivers/media/i2c/Makefile
> > > +++ b/drivers/media/i2c/Makefile
> > > @@ -24,6 +24,7 @@ obj-$(CONFIG_VIDEO_SAA6752HS) += saa6752hs.o
> > >  obj-$(CONFIG_VIDEO_AD5820)  += ad5820.o
> > >  obj-$(CONFIG_VIDEO_AK7375)  += ak7375.o
> > >  obj-$(CONFIG_VIDEO_DW9714)  += dw9714.o
> > > +obj-$(CONFIG_VIDEO_DW9768)  += dw9768.o
> > >  obj-$(CONFIG_VIDEO_DW9807_VCM)  += dw9807-vcm.o
> > >  obj-$(CONFIG_VIDEO_ADV7170) += adv7170.o
> > >  obj-$(CONFIG_VIDEO_ADV7175) += adv7175.o
> > > diff --git a/drivers/media/i2c/dw9768.c b/drivers/media/i2c/dw9768.c
> > > new file mode 100644
> > > index 000..f5b5591
> > > --- /dev/null
> > > +++ b/drivers/media/i2c/dw9768.c
> > > @@ -0,0 +1,458 @@
> > > +// SPDX-License-Identifier: GPL-2.0
> > > +/*
> > > + * Copyright (c) 2018 MediaTek Inc.
> > > + */
> > > +
> > > +#include 
> > > +#include 
> > > +#include 
> > > +#include 
> > > +#include 
> > > +#include 
> > > +#include 
> > > +#include 
> > > +
> > > +#define DW9768_VOLTAGE_ANALOG  280
> >
> > This is a platform detail and should be defined in the platform data, for
> > example DTS on platforms using DT.
> >
>
> Thanks for your reminder.
> This would be fixed in next release.
>
> > > +#define DW9768_NAME"dw9768"
> >
> > The chip we seem to be using this driver for is called gt9769. Shouldn't we
> > call the driver the same?
> >
>
> It is also called DW9768 from camera module specification, which was
> initially confirmed with vendor.
>

Okay, thanks for clarifying.

Best regards,
Tomasz

Re: [V2, 2/2] media: i2c: Add DW9768 VCM driver

[Tomasz Figa]

Hi Dongchun,

On Thu, Sep 5, 2019 at 4:22 PM  wrote:
>
> From: Dongchun Zhu 
>
> This patch adds a V4L2 sub-device driver for DW9768 lens voice coil,
> and provides control to set the desired focus.
>
> The DW9768 is a 10 bit DAC with 100mA output current sink capability
> from Dongwoon, designed for linear control of voice coil motor,
> and controlled via I2C serial interface.
>
> Signed-off-by: Dongchun Zhu 
> ---
>  MAINTAINERS|   1 +
>  drivers/media/i2c/Kconfig  |  10 ++
>  drivers/media/i2c/Makefile |   1 +
>  drivers/media/i2c/dw9768.c | 349 
> +
>  4 files changed, 361 insertions(+)
>  create mode 100644 drivers/media/i2c/dw9768.c
>

Thanks for v2! Please see my comments inline.

> diff --git a/MAINTAINERS b/MAINTAINERS
> index 192a671..c5c9a0e 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -4976,6 +4976,7 @@ M:Dongchun Zhu 
>  L: linux-me...@vger.kernel.org
>  T: git git://linuxtv.org/media_tree.git
>  S: Maintained
> +F: drivers/media/i2c/dw9768.c
>  F: Documentation/devicetree/bindings/media/i2c/dongwoon,dw9768.txt
>
>  DONGWOON DW9807 LENS VOICE COIL DRIVER
> diff --git a/drivers/media/i2c/Kconfig b/drivers/media/i2c/Kconfig
> index 79ce9ec..dfb665c 100644
> --- a/drivers/media/i2c/Kconfig
> +++ b/drivers/media/i2c/Kconfig
> @@ -1016,6 +1016,16 @@ config VIDEO_DW9714
>   capability. This is designed for linear control of
>   voice coil motors, controlled via I2C serial interface.
>
> +config VIDEO_DW9768
> +   tristate "DW9768 lens voice coil support"
> +   depends on I2C && VIDEO_V4L2 && MEDIA_CONTROLLER
> +   depends on VIDEO_V4L2_SUBDEV_API
> +   help
> + This is a driver for the DW9768 camera lens voice coil.
> + DW9768 is a 10 bit DAC with 100mA output current sink
> + capability. This is designed for linear control of
> + voice coil motors, controlled via I2C serial interface.
> +
>  config VIDEO_DW9807_VCM
> tristate "DW9807 lens voice coil support"
> depends on I2C && VIDEO_V4L2 && MEDIA_CONTROLLER
> diff --git a/drivers/media/i2c/Makefile b/drivers/media/i2c/Makefile
> index fd4ea86..2561239 100644
> --- a/drivers/media/i2c/Makefile
> +++ b/drivers/media/i2c/Makefile
> @@ -24,6 +24,7 @@ obj-$(CONFIG_VIDEO_SAA6752HS) += saa6752hs.o
>  obj-$(CONFIG_VIDEO_AD5820)  += ad5820.o
>  obj-$(CONFIG_VIDEO_AK7375)  += ak7375.o
>  obj-$(CONFIG_VIDEO_DW9714)  += dw9714.o
> +obj-$(CONFIG_VIDEO_DW9768)  += dw9768.o
>  obj-$(CONFIG_VIDEO_DW9807_VCM)  += dw9807-vcm.o
>  obj-$(CONFIG_VIDEO_ADV7170) += adv7170.o
>  obj-$(CONFIG_VIDEO_ADV7175) += adv7175.o
> diff --git a/drivers/media/i2c/dw9768.c b/drivers/media/i2c/dw9768.c
> new file mode 100644
> index 000..66d1712
> --- /dev/null
> +++ b/drivers/media/i2c/dw9768.c
> @@ -0,0 +1,349 @@
> +// SPDX-License-Identifier: GPL-2.0
> +// Copyright (c) 2019 MediaTek Inc.
> +
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +
> +#define DW9768_NAME"dw9768"
> +#define DW9768_MAX_FOCUS_POS   1023
> +/*
> + * This sets the minimum granularity for the focus positions.
> + * A value of 1 gives maximum accuracy for a desired focus position
> + */
> +#define DW9768_FOCUS_STEPS 1
> +/*
> + * DW9768 separates two registers to control the VCM position.
> + * One for MSB value, another is LSB value.
> + */
> +#define DW9768_REG_MASK_MSB0x03
> +#define DW9768_REG_MASK_LSB0xff
> +#define DW9768_SET_POSITION_ADDR0x03
> +
> +#define DW9768_CMD_DELAY   0xff
> +#define DW9768_CTRL_DELAY_US   5000
> +
> +#define DW9768_DAC_SHIFT   8
> +
> +/* dw9768 device structure */
> +struct dw9768 {
> +   struct v4l2_ctrl_handler ctrls;
> +   struct v4l2_subdev sd;
> +   struct regulator *vin;
> +   struct regulator *vdd;
> +};
> +
> +static inline struct dw9768 *to_dw9768_vcm(struct v4l2_ctrl *ctrl)
> +{
> +   return container_of(ctrl->handler, struct dw9768, ctrls);
> +}
> +
> +static inline struct dw9768 *sd_to_dw9768_vcm(struct v4l2_subdev *subdev)
> +{
> +   return container_of(subdev, struct dw9768, sd);
> +}
> +
> +struct regval_list {
> +   unsigned char reg_num;
> +   unsigned char value;
> +};
> +
> +static struct regval_list dw9768_init_regs[] = {
> +   {0x02, 0x02},
> +   {DW9768_CMD_DELAY, DW9768_CMD_DELAY},
> +   {0x06, 0x41},
> +   {0x07, 0x39},
> +   {DW9768_CMD_DELAY, DW9768_CMD_DELAY},
> +};
> +
> +static struct regval_list dw9768_release_regs[] = {
> +   {0x02, 0x00},
> +   {DW9768_CMD_DELAY, DW9768_CMD_DELAY},
> +   {0x01, 0x00},
> +   {DW9768_CMD_DELAY, DW9768_CMD_DELAY},
> +};
> +
> +static int dw9768_write_smbus(struct dw9768 *dw9768, unsigned char reg,
> + unsigned char va

Re: [V3, 2/2] media: i2c: Add Omnivision OV02A10 camera sensor driver

[Tomasz Figa]

On Thu, Sep 5, 2019 at 7:45 PM Sakari Ailus
 wrote:
>
> Hi Dongchun,
>
> On Thu, Sep 05, 2019 at 05:41:05PM +0800, Dongchun Zhu wrote:
>
> ...
>
> > > > + ret = regulator_bulk_enable(OV02A10_NUM_SUPPLIES, ov02a10->supplies);
> > > > + if (ret < 0) {
> > > > + dev_err(dev, "Failed to enable regulators\n");
> > > > + goto disable_clk;
> > > > + }
> > > > + msleep_range(7);
> > >
> > > This has some potential of clashing with more generic functions in the
> > > future. Please use usleep_range directly, or msleep.
> > >
> >
> > Did you mean using usleep_range(7*1000, 8*1000), as used in patch v1?
> > https://patchwork.kernel.org/patch/10957225/
>
> Yes, please.

Why not just msleep()?