Hi Eric,
> -----Original Message----- > From: dev <dev-boun...@dpdk.org> On Behalf Of Wangyu (Eric) > Sent: Monday, November 11, 2019 5:38 PM > To: Burakov, Anatoly <anatoly.bura...@intel.com>; David Marchand > <david.march...@redhat.com> > Cc: dev@dpdk.org; ferruh.yi...@intel.com; Linuxarm > <linux...@huawei.com>; humin (Q) <humi...@huawei.com>; Liyuan > (Larry) <larr...@huawei.com>; dengxiaofeng <dengxiaof...@huawei.com> > Subject: [dpdk-dev] 答复: 答复: [PATCH v2] bus/pci: resolve multiple NICs > address conflicts > > > Sorry, I didn't explain it clearly, and I will explain this problem step by > step. > > Precondition, we have a 64K page size system and two 82599 NICs. The > memory required for each NIC is as follows: > Map0 : size = 0x0000000000400000 > Map1 : size = 0x0000000000004000 > > 1. Primary process start, process mmap() first NIC map0, mmap()'s input > address is 0x8202000000, and output address is 0x8202000000, size is > 0x0000000000400000, next_addr is 0x8202400000, mmap() executed > correctly. > > 2. Primary mmap() first NIC map1, mmap()'s input address is 0x8202400000, > and output address is 0x8202400000, size is 0x0000000000004000, > next_addr is 0x8202404000, now mmap() applied from 0x8202400000 to > 0x8202410000 actually(because page size is 64K), and next_addr is > 0x8202404000, but mmap() executed correctly. So the problem begins to happen here, next_addr should be equal to: 0x8202400000 + 64K(other than 16K) = 0x8202410000, taking account of the real mapping size(page size). > > 3. Primary mmap() second NIC map0, mmap()'s input address is > 0x8202404000, but it's conflict, so output address is 0xffffbcdc0000(system > assigned), size is 0x0000000000400000, next_addr is 0xffffbd1c0000, now > the address is abnormal, and mmap() executed correctly. If the step 2) is correct with the next_addr, this step should go on correctly with addresses. > > 4. Primary mmap() second NIC map1, mmap()'s input address is > 0xffffbd1c0000, and output address is 0xffffbcdb0000 (system assigned), > size is 0x0000000000004000, now the address is abnormal, and mmap() > executed correctly. > > 5. Secondary process start, process mmap() first NIC map0, it's normal. > > 6. Secondary process mmap() first NIC map1, it's normal. > > 7. Secondary process mmap() second NIC map0, mmap()'s input address is > 0xffffbcdc0000, but it's conflict on secondary process, so we get another > address, but secondary will check if the input address is equal with output > address, it's not equal, so secondary will exit with " Cannot mmap device > resource file %s to address: %p ". > > > Now I use (next_addr = RTE_PTR_ALIGN(current.addr + current.len, > pagesize)) to solve the problem, and it worked. If it is right, I will submit > a > patch later. > > By the way, I made a mistake, the problem won't happen on VFIO, because > VFIO don't apply for 16K memory, only apply for 4M size(map0).But I think > VFIO also needs to be modified. > > > -----邮件原件----- > 发件人: Burakov, Anatoly [mailto:anatoly.bura...@intel.com] > 发送时间: 2019年11月7日 20:25 > 收件人: Wangyu (Eric) <seven.wan...@huawei.com>; David Marchand > <david.march...@redhat.com> > 抄送: dev@dpdk.org; ferruh.yi...@intel.com; Linuxarm > <linux...@huawei.com>; humin (Q) <humi...@huawei.com>; Liyuan > (Larry) <larr...@huawei.com>; dengxiaofeng <dengxiaof...@huawei.com> > 主题: Re: 答复: [dpdk-dev] [PATCH v2] bus/pci: resolve multiple NICs > address conflicts > > On 07-Nov-19 5:44 AM, Wangyu (Eric) wrote: > > Hi, Anatoly > > > > Thank you for advices. This problem will happen in both VFIO and UIO, I > will modify both according to your advices and test them. > > > > I did some tests with mmap() on my system, when I provided address not > page-aligned, mmap() could return page-aligned address too, but the code > will return fault because mmap() return address was not equal with address > I provided(problem occurs in pci_uio_map_secondary()). > > > > I still don't understand how do you get addresses aligned on a 16K boundary > with 64K page size. > > The mapping process is as follows: > > 0) start with max_va_end, or with previous addres + previous len > 1) reserve virtual area with mmap() (accepts any return address) > 2) map the BAR with MAP_FIXED (checks return address, but should work > because we already have that area reserved) > > The error you're referring to would've happened at step 2 (MAP_FIXED with > unaligned addresses will cause the mmap() to fail), but at that point we > already have a valid virtual area for the bar. If you get a 16K-aligned page > address for the BAR, you get it on step 1, not step 2. > > So, if, by your own admission, your mmap() implementation does return a > 64K-aligned address... What exactly is the issue then? How does your BAR > end up with an invalid address? > > -- > Thanks, > Anatoly
