Thank you for your help! That bit resolved all of the RDMA/WDMA coherency issues on the CSB side...except:
We expose a 1MB region of memory from CSB via a BAR (BAR1, if it matters) to the Host. This region is also not behaving correctly with respect to coherency on SOME hosts; again, disabling our cache makes it work correctly on all hosts. We have set PEX_DEVICE_CONTROL in PCI-E Config Space (0x54) to 0x2010 (sorry about the endianness below). We thought that CLEARING the no-snoop bit here would indicate that snooping was required for this region...is this a similar issue? - Ben On Fri, Mar 6, 2009 at 10:12 AM, Ben Menchaca <ben.mench...@gmail.com>wrote: > Testing now...it looks like it (almost) works, though! Why does setting > no-snoop cause snooping to work? More on the effect on setting that bit in > a few minutes...need more testing. > > ACR is 0x00030300. > > - Ben > > > On Fri, Mar 6, 2009 at 12:30 AM, Liu Dave-R63238 <dave...@freescale.com>wrote: > >> Did you enable the descriptor bit 3 to have a try? >> >> ------------------------------ >> *From:* Ben Menchaca [mailto:ben.mench...@gmail.com] >> *Sent:* Friday, March 06, 2009 2:10 PM >> >> *To:* Liu Dave-R63238 >> *Cc:* linuxppc-dev@ozlabs.org >> *Subject:* Re: 83xx: Marking or Allocating Pages as Cache-Inhibited >> >> I can look at ACR morning...although I can say with a fair amount of >> certainty that I have not changed it from the POR value. >> >> I will try enabling No Snoop for CSB in the descriptor (bit 3, >> yes?)...this seems a bit counterintuitive to me. >> >> What is the hope regarding these two? Some combination I am not seeing? >> >> >> On Thu, Mar 5, 2009 at 11:40 PM, Liu Dave-R63238 >> <dave...@freescale.com>wrote: >> >>> what is the value of ACR register? >>> >>> ------------------------------ >>> *From:* Ben Menchaca [mailto:ben.mench...@gmail.com] >>> *Sent:* Friday, March 06, 2009 1:38 PM >>> *To:* Liu Dave-R63238 >>> *Cc:* linuxppc-dev@ozlabs.org >>> *Subject:* Re: 83xx: Marking or Allocating Pages as Cache-Inhibited >>> >>> 1. BAT2 in linux is set to WIMG=0010, and covers all 64M >>> 2. PEX_DEVICE_CONTROL in PCI-E Config Space (0x54): 0x1020 >>> 3. PEX_xDMA_CTRL is set to 0x00000401 at the initiation of the DMA. >>> 4. OWAR0 is set to 0xFFFFF005, so NSNP is 0. >>> 5. The DMA descriptor (randomly chosen when I hit a trigger...just >>> ignore the size...) contains 0002AFF3 at offset 0, so nosnoops are cleared. >>> >>> >>> Core is 400MHz, and CSB is 133MHz. >>> >>> - Ben >>> >>> On Thu, Mar 5, 2009 at 11:27 PM, Liu Dave-R63238 >>> <dave...@freescale.com>wrote: >>> >>>> and what settings is DMA description bit 3? >>>> >>>> > -----Original Message----- >>>> > From: linuxppc-dev-bounces+daveliu=freescale....@ozlabs.org >>>> > [mailto:linuxppc-dev-bounces+daveliu <linuxppc-dev-bounces%2Bdaveliu> >>>> =freescale....@ozlabs.org] >>>> > On Behalf Of Liu Dave-R63238 >>>> > Sent: Friday, March 06, 2009 1:22 PM >>>> > To: Ben Menchaca; linuxppc-dev@ozlabs.org >>>> > Subject: RE: 83xx: Marking or Allocating Pages as Cache-Inhibited >>>> > >>>> > Did you enable the snoop bit at PEX_WDMA_CTRL[SNOOP] and >>>> > PEX_RDMA_CTRL[SNOOP]? >>>> > >>>> > What is the freq settings? CORE/CSB bus. >>>> > >>>> > Thanks, Dave >>>> > >>>> > ________________________________ >>>> > >>>> > From: linuxppc-dev-bounces+daveliu=freescale....@ozlabs.org >>>> > [mailto:linuxppc-dev-bounces+daveliu <linuxppc-dev-bounces%2Bdaveliu> >>>> =freescale....@ozlabs.org] >>>> > On Behalf Of Ben Menchaca >>>> > Sent: Friday, March 06, 2009 12:33 PM >>>> > To: linuxppc-dev@ozlabs.org >>>> > Subject: 83xx: Marking or Allocating Pages as Cache-Inhibited >>>> > >>>> > >>>> > I am working on a Freescale 8314e design, and the >>>> > embedded device is configured as a PCI-e endpoint running a >>>> > 2.6.27-5 kernel. For context, we have written a kernel >>>> > module which, among other things, uses the RDMA/WDMA engine >>>> > in the PCI-e IP block. On the host side, these DMAs are >>>> > coherent. However, on the embedded side, things are quite a >>>> > bit less rosy; we must manually flush/invalidate cache lines >>>> > for WDMA/RDMAs to occur successfully. After speaking with >>>> > (several) FAEs at Freescale, we believe there is a >>>> > configuration issue that is the cause, but we have yet to >>>> > have anyone successfully point to it. >>>> > >>>> > Disabling the data cache altogether resolves the issue >>>> > entirely, but of course, also completely tanks performance. >>>> > As a temporary workaround, I would like to simply mark the >>>> > pages (obtained currently via dma_alloc_coherent) involved as >>>> > cache-inhibited. I have attempted to do this via some >>>> > snippets remaining in fec.c (va_to_pte, uncache_pte to set >>>> > _PAGE_NO_CACHE, flush_tlb_page, then unmap_pte), but this is >>>> > almost certainly braindead; va_to_pte is not a part of the >>>> > 83xx source, as far as I can tell; 8xx only. >>>> > >>>> > A quick pointer in the correct direction for marking >>>> > pages as cache-inhibited on a 2.6.27-5 kernel would be >>>> > appreciated, or if my approach to a workaround is flawed, a >>>> > pointer to the correct way would be great. >>>> > >>>> > Ben Menchaca >>>> > >>>> > >>>> > _______________________________________________ >>>> > Linuxppc-dev mailing list >>>> > Linuxppc-dev@ozlabs.org >>>> > https://ozlabs.org/mailman/listinfo/linuxppc-dev >>>> > >>>> > >>>> >>> >>> >> >
_______________________________________________ Linuxppc-dev mailing list Linuxppc-dev@ozlabs.org https://ozlabs.org/mailman/listinfo/linuxppc-dev
