Joerg, I am seeing IO_PAGE_FAULTs on AMD system running releases prior to 3.7. I focused my debug and testing on 3.4. I am hoping to find a solution for this problem in 3.4. I don't see any IO_PAGE_FAULTs with 3.7 and later releases on this system.
On this system BIOS specifies Unity mapped (direct mapped) exclusion ranges in IVMDs for several devices. These regions are in use during BIOS hand-off to kernel and continue to be used during kernel boot and run-time. Access to these ranges continues to work with no errors until AMD IOMMU driver disables and re-enables IOMMU in enable_iommus(). These faults don't persist and appear between the enable_iommus() call and before amd_iommu_init() gets done printing "AMD-Vi: Lazy IO/TLB flushing enabled" message. Read requests from device 02:00.2 and write request from device 03:00.0 to these unity mapped regions fail. The reason appears to be because domain id is 0. Domain gets assigned in amd_iommu_init_dma_ops() and unity maps are handled. I don't see enable_iommus() doing anything to these unity mapped exclusion ranges. So I am assuming that is not the issue, however, could domain ids get flushed? More like, why do these faults show up in this window? These are direct mapped, so there is no need for any translations. Please see below for IVMD dump and IO_PAGE_FAULT analysis. Dump of these ranges from dmesg: [ 5.322280] AMD-Vi: IVMD_TYPE_ALL devid_start: 00:00.0 devid_end: 04:00.3 range_start: 00000000000f0000 range_end: 0000000000100000 flags: 7 [ 5.322367] AMD-Vi: IVMD_TYPE_ALL devid_start: 00:00.0 devid_end: 04:00.3 range_start: 00000000bff70000 range_end: 00000000bfff0000 flags: 7 [ 5.322454] AMD-Vi: IVMD_TYPE_ALL devid_start: 00:00.0 devid_end: 04:00.3 range_start: 00000000000e8000 range_end: 00000000000e9000 flags: 7 [ 5.322540] AMD-Vi: IVMD_TYPE_ALL devid_start: 00:00.0 devid_end: 04:00.3 range_start: 00000000bdffe000 range_end: 00000000be000000 flags: 7 [ 5.322627] AMD-Vi: IVMD_TYPE_ALL devid_start: 00:00.0 devid_end: 04:00.3 range_start: 00000000bdff9000 range_end: 00000000bdffd000 flags: 7 [ 5.322714] AMD-Vi: IVMD_TYPE_ALL devid_start: 00:00.0 devid_end: 04:00.3 range_start: 00000000bdfe9000 range_end: 00000000bdff9000 flags: 7 Now to IO_PAGE_FAULT analysis: My observations in "" [ 15.281594] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.2 domain=0x0000 address=0x00000000bdffe000 flags=0x0050] [ 15.281861] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.2 domain=0x0000 address=0x00000000bdff9080 flags=0x0050] [ 15.281990] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.2 domain=0x0000 address=0x00000000bdff9100 flags=0x0050] Domain ID is zero - "PASID not valid" flags=0x0050 - "Bits PE and PR are set in the Event." TR: translation TR=0 "TR is 0 that means it is a transaction request" RZ: reserved bit RZ=0 "Since PR is set RZ is meaningful, I/O page fault is due to an invalid level encoding" PE: permission indicator PE=1 "Device doesn't have permission for this transaction" RW: read-write RW=0 "RW is meaningful since PR=1, TR=0, and I=0. It is a Read transaction" PR: Present PR=1 "PR = 1 means transaction is to a page marked present" I: interrupt I=0 "transaction is a memory request" US: user-supervisor US=0 "Supervisor privileges were asserted." NX: no execute NX=0 "0 upstream transaction lacks a PASID TLP prefix. Domain ID is zero." GN: guest/nested GN=0 "Transaction used a nested address (GPA)." [ 15.281733] AMD-Vi: Event logged [IO_PAGE_FAULT device=03:00.0 domain=0x0000 address=0x00000000bdff9160 flags=0x0070] Domain ID is zero - "PASID is not valid" flags=0x0070 - "Bits PE, RW, and PR are set in the Event." TR: translation TR=0 "TR is 0 that means it is a transaction request" RZ: reserved bit RZ=0 "Since PR is set RZ is meaningful, I/O page fault is due to an invalid level encoding" PE: permission indicator PE=1 "Device doesn't have permission for this transaction" RW: read-write RW=1 "RW is meaningful since PR=1, TR=0, and I=0. It is a Write transaction" PR: Present PR=1 "PR = 1 means transaction is to a page marked present" I: interrupt I=0 "transaction is a memory request" US: user-supervisor US=0 "Supervisor privileges were asserted." NX: no execute NX=0 "0 upstream transaction lacks a PASID TLP prefix. Domain ID is zero." GN: guest/nested GN=0 "Transaction used a nested address (GPA)." Thanks, -- Shuah -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
