Am Dienstag, dem 25.03.2025 um 13:23 +0100 schrieb Christian König:
> Am 25.03.25 um 11:14 schrieb Bert Karwatzki:
> > My /proc/iomem contans two memory areas of 8G size which are
> > belonging to PCI 0000:03:00.0, one of the is the BAR reported by dmesg
> > [ 0.312692] [ T1] pci 0000:03:00.0: BAR 0 [mem 0xfc00000000-0xfdffffffff
> > 64bit pref]
>
> > the other one is "afe00000000-affffffffff : 0000:03:00.0" (in the case
> > without nokaslr) which shifts
> > to "3ffe00000000-3fffffffffff : 0000:03:00.0" in the case with nokaslr.
>
> You need to figure out where that stuff is coming from.
>
> See below for another extremely odd thing.
>
> >
> > Here's /proc/iomem in the case without nokaslr:
> > 00000000-00000fff : Reserved
> > 00001000-0009ffff : System RAM
> > 000a0000-000fffff : Reserved
> > 000a0000-000dffff : PCI Bus 0000:00
> > 000f0000-000fffff : System ROM
> > 00100000-09bfefff : System RAM
> > 09bff000-0a000fff : Reserved
> > 0a001000-0a1fffff : System RAM
> > 0a200000-0a20efff : ACPI Non-volatile Storage
> > 0a20f000-e62edfff : System RAM
> > e62ee000-e63e1fff : Reserved
> > e63e2000-e87cafff : System RAM
> > e87cb000-e87cbfff : Reserved
> > e87cc000-e9e1ffff : System RAM
> > e9e20000-eb33efff : Reserved
> > eb31e000-eb321fff : MSFT0101:00
> > eb322000-eb325fff : MSFT0101:00
> > eb33f000-eb39efff : ACPI Tables
> > eb39f000-eb556fff : ACPI Non-volatile Storage
> > eb557000-ed1fefff : Reserved
> > ed1ff000-edffffff : System RAM
> > ee000000-efffffff : Reserved
> > f0000000-fcffffff : PCI Bus 0000:00
> > f0000000-f7ffffff : PCI ECAM 0000 [bus 00-7f]
> > f0000000-f7ffffff : pnp 00:00
> > fc500000-fc9fffff : PCI Bus 0000:08
> > fc500000-fc5fffff : 0000:08:00.7
> > fc500000-fc5fffff : pcie_mp2_amd
> > fc600000-fc6fffff : 0000:08:00.4
> > fc600000-fc6fffff : xhci-hcd
> > fc700000-fc7fffff : 0000:08:00.3
> > fc700000-fc7fffff : xhci-hcd
> > fc800000-fc8fffff : 0000:08:00.2
> > fc800000-fc8fffff : ccp
> > fc900000-fc97ffff : 0000:08:00.0
> > fc980000-fc9bffff : 0000:08:00.5
> > fc980000-fc9bffff : AMD ACP3x audio
> > fc980000-fc990200 : acp_pdm_iomem
> > fc9c0000-fc9c7fff : 0000:08:00.6
> > fc9c0000-fc9c7fff : ICH HD audio
> > fc9c8000-fc9cbfff : 0000:08:00.1
> > fc9c8000-fc9cbfff : ICH HD audio
> > fc9cc000-fc9cdfff : 0000:08:00.7
> > fc9ce000-fc9cffff : 0000:08:00.2
> > fc9ce000-fc9cffff : ccp
> > fca00000-fccfffff : PCI Bus 0000:01
> > fca00000-fcbfffff : PCI Bus 0000:02
> > fca00000-fcbfffff : PCI Bus 0000:03
> > fca00000-fcafffff : 0000:03:00.0
> > fcb00000-fcb1ffff : 0000:03:00.0
> > fcb20000-fcb23fff : 0000:03:00.1
> > fcb20000-fcb23fff : ICH HD audio
> > fcc00000-fcc03fff : 0000:01:00.0
> > fcd00000-fcdfffff : PCI Bus 0000:07
> > fcd00000-fcd03fff : 0000:07:00.0
> > fcd00000-fcd03fff : nvme
> > fce00000-fcefffff : PCI Bus 0000:06
> > fce00000-fce03fff : 0000:06:00.0
> > fce00000-fce03fff : nvme
> > fcf00000-fcffffff : PCI Bus 0000:05
> > fcf00000-fcf03fff : 0000:05:00.0
> > fcf04000-fcf04fff : 0000:05:00.0
> > fcf04000-fcf04fff : r8169
> > fd300000-fd37ffff : amd_iommu
> > fec00000-fec003ff : IOAPIC 0
> > fec01000-fec013ff : IOAPIC 1
> > fec10000-fec10fff : Reserved
> > fec10000-fec10fff : pnp 00:04
> > fed00000-fed00fff : Reserved
> > fed00000-fed003ff : HPET 0
> > fed00000-fed003ff : PNP0103:00
> > fed40000-fed44fff : Reserved
> > fed80000-fed8ffff : Reserved
> > fed81200-fed812ff : AMDI0030:00
> > fed81500-fed818ff : AMDI0030:00
> > fed81500-fed818ff : AMDI0030:00 AMDI0030:00
> > fedc0000-fedc0fff : pnp 00:04
> > fedc4000-fedc9fff : Reserved
> > fedc5000-fedc5fff : AMDI0010:03
> > fedc5000-fedc5fff : AMDI0010:03 AMDI0010:03
> > fedcc000-fedcefff : Reserved
> > fedd5000-fedd5fff : Reserved
> > fee00000-fee00fff : pnp 00:04
> > ff000000-ffffffff : pnp 00:04
> > 100000000-fee2fffff : System RAM
> > 825600000-8261fa1b1 : Kernel code
> > 826200000-82663dfff : Kernel rodata
> > 826800000-82692ef3f : Kernel data
> > 826eaf000-826ffffff : Kernel bss
>
> > fee300000-100fffffff : Reserved
>
> First it says that this range is reserved.
>
> > 1010000000-ffffffffff : PCI Bus 0000:00
>
> And this range here used for the PCI Bus.
>
> > fc00000000-fe0fffffff : PCI Bus 0000:01
> > fc00000000-fe0fffffff : PCI Bus 0000:02
> > fc00000000-fe0fffffff : PCI Bus 0000:03
> > fc00000000-fdffffffff : 0000:03:00.0 This is the usual BAR reported
> > by dmesg (and lspci -vv)
> > fe00000000-fe0fffffff : 0000:03:00.0
>
> And then it says that the PCIe devices are inside the reserved range.
>
> > fe20000000-fe301fffff : PCI Bus 0000:08
> > fe20000000-fe2fffffff : 0000:08:00.0
> > fe30000000-fe301fffff : 0000:08:00.0
> > fe30300000-fe304fffff : PCI Bus 0000:04
> > fe30300000-fe303fffff : 0000:04:00.0
> > fe30300000-fe303fffff : 0000:04:00.0
> > fe30400000-fe30403fff : 0000:04:00.0
> > fe30404000-fe30404fff : 0000:04:00.0
>
> > afe00000000-affffffffff : 0000:03:00.0 This is the memory which shifts with
> > nokaslr
>
> To be honest that looks like a broken ACPI table to me, but it doesn't
> explain why this range here shifts when activating/deactivating nokaslr.
>
> Do we have some option to enable printing when ranges are added to the iomem
> reservation? If yes than we should probably do that and take a look at the
> dmesg again.
>
> Regards,
> Christian.
>
> >
> > Bert Karwatzki
As Balbir Singh found out this memory comes from amdkfd
(kgd2kfd_init_zone_device()) with CONFIG_HSA_AMD_SVM=y. The memory gets placed
by devm_request_free_mem_region() which places the memory at the end of the
physical address space (DIRECT_MAP_PHYSMEM_END). DIRECT_MAP_PHYSMEM_END changes
when using nokaslr and so the memory shifts.
One can work around this by removing the GFR_DESCENDING flag from
devm_request_free_mem_region() so the memory gets placed right after the other
resources:
tail -n 15 /proc/iomem
1010000000-ffffffffff : PCI Bus 0000:00
fc00000000-fe0fffffff : PCI Bus 0000:01
fc00000000-fe0fffffff : PCI Bus 0000:02
fc00000000-fe0fffffff : PCI Bus 0000:03
fc00000000-fdffffffff : 0000:03:00.0
fe00000000-fe0fffffff : 0000:03:00.0
fe20000000-fe301fffff : PCI Bus 0000:08
fe20000000-fe2fffffff : 0000:08:00.0
fe30000000-fe301fffff : 0000:08:00.0
fe30300000-fe304fffff : PCI Bus 0000:04
fe30300000-fe303fffff : 0000:04:00.0
fe30300000-fe303fffff : 0000:04:00.0
fe30400000-fe30403fff : 0000:04:00.0
fe30404000-fe30404fff : 0000:04:00.0
10000000000-101ffffffff : 0000:03:00.0
Bert Karwatzki
