On 11-Nov-19 9:37 AM, Wangyu (Eric) wrote:
Sorry, I didn't explain it clearly, and I will explain this problem step by step. Precondition, we have a 64K page size system and two 82599 NICs. The memory required for each NIC is as follows: Map0 : size = 0x0000000000400000 Map1 : size = 0x0000000000004000 1. Primary process start, process mmap() first NIC map0, mmap()'s input address is 0x8202000000, and output address is 0x8202000000, size is 0x0000000000400000, next_addr is 0x8202400000, mmap() executed correctly. 2. Primary mmap() first NIC map1, mmap()'s input address is 0x8202400000, and output address is 0x8202400000, size is 0x0000000000004000, next_addr is 0x8202404000, now mmap() applied from 0x8202400000 to 0x8202410000 actually(because page size is 64K), and next_addr is 0x8202404000, but mmap() executed correctly. 3. Primary mmap() second NIC map0, mmap()'s input address is 0x8202404000, but it's conflict, so output address is 0xffffbcdc0000(system assigned), size is 0x0000000000400000, next_addr is 0xffffbd1c0000, now the address is abnormal, and mmap() executed correctly. 4. Primary mmap() second NIC map1, mmap()'s input address is 0xffffbd1c0000, and output address is 0xffffbcdb0000 (system assigned), size is 0x0000000000004000, now the address is abnormal, and mmap() executed correctly. 5. Secondary process start, process mmap() first NIC map0, it's normal. 6. Secondary process mmap() first NIC map1, it's normal. 7. Secondary process mmap() second NIC map0, mmap()'s input address is 0xffffbcdc0000, but it's conflict on secondary process, so we get another address, but secondary will check if the input address is equal with output address, it's not equal, so secondary will exit with " Cannot mmap device resource file %s to address: %p ". Now I use (next_addr = RTE_PTR_ALIGN(current.addr + current.len, pagesize)) to solve the problem, and it worked. If it is right, I will submit a patch later. By the way, I made a mistake, the problem won't happen on VFIO, because VFIO don't apply for 16K memory, only apply for 4M size(map0).But I think VFIO also needs to be modified.
OK, that makes more sense. In that case, aligning next address on page boundary is the right approach.
-- Thanks, Anatoly
