Lots of XEN init functions called in non-XEN environment
Dear Linux folks, Booting Debian’s Linux kernel with `initcall_debug` on a laptop with no XEN being used, I see a lot of Xen related init functions to be called. ``` $ sudo dmesg | grep -e balloon -e xen [0.066207] calling xen_cons_init+0x0/0x50 @ 0 [0.066210] initcall xen_cons_init+0x0/0x50 returned 0 after 0 usecs [0.096491] calling xen_pvh_gnttab_setup+0x0/0x34 @ 1 [0.096491] initcall xen_pvh_gnttab_setup+0x0/0x34 returned -19 after 0 usecs [0.100353] calling xenbus_init+0x0/0x322 @ 1 [0.100353] initcall xenbus_init+0x0/0x322 returned -19 after 0 usecs [0.100353] calling register_xen_pci_notifier+0x0/0x2d @ 1 [0.100353] initcall register_xen_pci_notifier+0x0/0x2d returned 0 after 0 usecs [0.100353] calling xen_pcpu_init+0x0/0xb5 @ 1 [0.100353] initcall xen_pcpu_init+0x0/0xb5 returned -19 after 0 usecs [0.169465] calling balloon_init+0x0/0x1e0 @ 1 [0.169467] initcall balloon_init+0x0/0x1e0 returned -19 after 0 usecs [0.169470] calling xen_setup_shutdown_event+0x0/0x30 @ 1 [0.169473] initcall xen_setup_shutdown_event+0x0/0x30 returned -19 after 0 usecs [0.169476] calling xenbus_probe_backend_init+0x0/0x6b @ 1 [0.169482] initcall xenbus_probe_backend_init+0x0/0x6b returned 0 after 0 usecs [0.169485] calling xenbus_probe_frontend_init+0x0/0x4f @ 1 [0.169489] initcall xenbus_probe_frontend_init+0x0/0x4f returned 0 after 0 usecs [0.169491] calling xen_acpi_pad_init+0x0/0x3c @ 1 [0.169493] initcall xen_acpi_pad_init+0x0/0x3c returned -19 after 0 usecs [0.257640] calling xenfb_init+0x0/0x3b @ 1 [0.257642] initcall xenfb_init+0x0/0x3b returned -19 after 0 usecs [0.259498] calling xenbus_probe_initcall+0x0/0x6f @ 1 [0.259599] initcall xenbus_probe_initcall+0x0/0x6f returned 0 after 98 usecs [0.259615] calling xenbus_init+0x0/0x3b @ 1 [0.259617] initcall xenbus_init+0x0/0x3b returned -19 after 0 usecs [0.259620] calling xenbus_backend_init+0x0/0x44 @ 1 [0.259622] initcall xenbus_backend_init+0x0/0x44 returned -19 after 0 usecs [0.259666] calling xen_late_init_mcelog+0x0/0x5e @ 1 [0.259668] initcall xen_late_init_mcelog+0x0/0x5e returned -19 after 0 usecs [0.259811] calling xen_hvc_init+0x0/0x1de @ 1 [0.259813] initcall xen_hvc_init+0x0/0x1de returned -19 after 0 usecs [0.263794] calling xenkbd_init+0x0/0x3b @ 1 [0.263796] initcall xenkbd_init+0x0/0x3b returned -19 after 0 usecs [0.285181] calling balloon_wait_finish+0x0/0xda @ 1 [0.285183] initcall balloon_wait_finish+0x0/0xda returned -19 after 0 usecs ``` All these drivers(?) are enabled in Debian’s Linux configuration to also support XEN setups, but I wonder, if the system can’t detect once if it’s running in a XEN environment, and if it’s not then to skip all the XEN related init functions. Kind regards, Paul
Re: [patch 00/37] cpu/hotplug, x86: Reworked parallel CPU bringup
Dear Thomas, Am 15.04.23 um 01:44 schrieb Thomas Gleixner: This is a complete rework of the parallel bringup patch series (V17) https://lore.kernel.org/lkml/20230328195758.1049469-1-usama.a...@bytedance.com to address the issues which were discovered in review: […] Thank you very much for your rework. I tested this on the ASUS F2A85-M PRO, and get a delay of ten seconds. ``` […] [0.258193] smpboot: CPU0: AMD A6-6400K APU with Radeon(tm) HD Graphics (family: 0x15, model: 0x13, stepping: 0x1) […] [0.259329] smp: Bringing up secondary CPUs ... [0.259527] x86: Booting SMP configuration: [0.259528] node #0, CPUs: #1 [0.261007] After schedule_preempt_disabled [ 10.260990] CPU1 failed to report alive state [ 10.261070] smp: Brought up 1 node, 1 CPU [ 10.261073] smpboot: Max logical packages: 2 [ 10.261074] smpboot: Total of 1 processors activated (7800.54 BogoMIPS) [ 10.261601] devtmpfs: initialized [ 10.261697] x86/mm: Memory block size: 128MB ``` This delay has been there with v6.3-rc6-46-gde4664485abbc and some custom (printk) patches on top and merging dwmw2/parallel-6.2-rc3-v16 into it. I only tested this. I think dwmw2/parallel-6.2-v17 failed to build for me, when trying to merge it into Linus’ master version at that time. I didn’t come around to report it, and you posted your rework, so I am replying here. I am going to try your branch directly in the next days, but just wanted to report back already. Kind regards, Paul[0.00] Linux version 6.3.0-rc6-00311-gde8224969f66 (root@bf16f3646a84) (gcc (Debian 11.2.0-12) 11.2.0, GNU ld (GNU Binutils for Debian) 2.40) #446 SMP PREEMPT_DYNAMIC Sat Apr 15 14:12:29 UTC 2023 [0.00] Command line: BOOT_IMAGE=/boot/vmlinuz-6.3.0-rc6-00311-gde8224969f66 root=/dev/sda3 rw quiet noisapnp cryptomgr.notests ipv6.disable_ipv6=1 selinux=0 [0.00] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [0.00] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [0.00] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [0.00] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [0.00] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [0.00] signal: max sigframe size: 1776 [0.00] BIOS-provided physical RAM map: [0.00] BIOS-e820: [mem 0x-0x0009fbff] usable [0.00] BIOS-e820: [mem 0x0009fc00-0x0009] reserved [0.00] BIOS-e820: [mem 0x000f-0x000f] reserved [0.00] BIOS-e820: [mem 0x0010-0x5fe4cfff] usable [0.00] BIOS-e820: [mem 0x5fe4d000-0x7fff] reserved [0.00] BIOS-e820: [mem 0xf800-0xfbff] reserved [0.00] BIOS-e820: [mem 0xfec1-0xfec10fff] reserved [0.00] BIOS-e820: [mem 0x0001-0x00017eff] usable [0.00] NX (Execute Disable) protection: active [0.00] SMBIOS 3.0.0 present. [0.00] DMI: ASUS F2A85-M_PRO/F2A85-M_PRO, BIOS 4.18-9-g9917d2d915 04/17/2023 [0.00] tsc: Fast TSC calibration using PIT [0.00] tsc: Initial usec timer 6035615 [0.00] tsc: Detected 3900.273 MHz processor [0.000756] e820: update [mem 0x-0x0fff] usable ==> reserved [0.000759] e820: remove [mem 0x000a-0x000f] usable [0.000763] last_pfn = 0x17f000 max_arch_pfn = 0x4 [0.000768] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [0.000942] last_pfn = 0x5fe4d max_arch_pfn = 0x4 [0.004000] Using GB pages for direct mapping [0.004000] ACPI: Early table checksum verification disabled [0.004000] ACPI: RSDP 0x000F6830 24 (v02 COREv4) [0.004000] ACPI: XSDT 0x5FE5A0E0 74 (v01 COREv4 COREBOOT CORE 20200925) [0.004000] ACPI: FACP 0x5FE5BBC0 000114 (v06 COREv4 COREBOOT CORE 20200925) [0.004000] ACPI: DSDT 0x5FE5A280 00193A (v02 COREv4 COREBOOT 00010001 INTL 20200925) [0.004000] ACPI: FACS 0x5FE5A240 40 [0.004000] ACPI: FACS 0x5FE5A240 40 [0.004000] ACPI: SSDT 0x5FE5BCE0 8A (v02 COREv4 COREBOOT 002A CORE 20200925) [0.004000] ACPI: MCFG 0x5FE5BD70 3C (v01 COREv4 COREBOOT CORE 20200925) [0.004000] ACPI: APIC 0x5FE5BDB0 62 (v03 COREv4 COREBOOT CORE 20200925) [0.004000] ACPI: HPET 0x5FE5BE20 38 (v01 COREv4 COREBOOT CORE 20200925) [0.004000] ACPI: HEST 0x5FE5BE60 0001D0 (v01 COREv4 COREBOOT CORE 20200925) [0.004000] ACPI: IVRS 0x5FE5C030 70 (v02 AMDAMDIOMMU 0001 AMD ) [0.004000] ACPI: SSDT 0x5FE5C0A0 00051F (v02 AMDALIB 0001 MSFT 0400) [0.004000] ACPI: SSDT 0x5FE5C5C0 0006
Re: [patch 00/37] cpu/hotplug, x86: Reworked parallel CPU bringup
[Correct David’s address] Am 17.04.23 um 13:19 schrieb Paul Menzel: Dear Thomas, Am 15.04.23 um 01:44 schrieb Thomas Gleixner: This is a complete rework of the parallel bringup patch series (V17) https://lore.kernel.org/lkml/20230328195758.1049469-1-usama.a...@bytedance.com to address the issues which were discovered in review: […] Thank you very much for your rework. I tested this on the ASUS F2A85-M PRO, and get a delay of ten seconds. ``` […] [ 0.258193] smpboot: CPU0: AMD A6-6400K APU with Radeon(tm) HD Graphics (family: 0x15, model: 0x13, stepping: 0x1) […] [ 0.259329] smp: Bringing up secondary CPUs ... [ 0.259527] x86: Booting SMP configuration: [ 0.259528] node #0, CPUs: #1 [ 0.261007] After schedule_preempt_disabled [ 10.260990] CPU1 failed to report alive state [ 10.261070] smp: Brought up 1 node, 1 CPU [ 10.261073] smpboot: Max logical packages: 2 [ 10.261074] smpboot: Total of 1 processors activated (7800.54 BogoMIPS) [ 10.261601] devtmpfs: initialized [ 10.261697] x86/mm: Memory block size: 128MB ``` This delay has been there with v6.3-rc6-46-gde4664485abbc and some custom (printk) patches on top and merging dwmw2/parallel-6.2-rc3-v16 into it. I only tested this. I think dwmw2/parallel-6.2-v17 failed to build for me, when trying to merge it into Linus’ master version at that time. I didn’t come around to report it, and you posted your rework, so I am replying here. I am going to try your branch directly in the next days, but just wanted to report back already. Kind regards, Paul
Re: [patch 00/37] cpu/hotplug, x86: Reworked parallel CPU bringup
Dear Thomas, Am 17.04.23 um 16:48 schrieb Thomas Gleixner: On Mon, Apr 17 2023 at 13:19, Paul Menzel wrote: Am 15.04.23 um 01:44 schrieb Thomas Gleixner: [0.258193] smpboot: CPU0: AMD A6-6400K APU with Radeon(tm) HD Graphics (family: 0x15, model: 0x13, stepping: 0x1) […] [0.259329] smp: Bringing up secondary CPUs ... [0.259527] x86: Booting SMP configuration: [0.259528] node #0, CPUs: #1 [0.261007] After schedule_preempt_disabled [ 10.260990] CPU1 failed to report alive state Weird. CPU1 fails to come up and report that it has reached the synchronization point. Does it work when you add cpuhp.parallel=off on the kernel command line? Yes, the ten seconds delay is gone with `cpuhp.parallel=off`. There was a patch set in the past, that worked on that device. I think up to v4 it did *not* work at all and hung [1]. I need some days to collect the results again. Kind regards, Paul [1]: https://lore.kernel.org/lkml/ab28d2ce-4a9c-387d-9eda-558045a0c...@molgen.mpg.de/[0.00] Linux version 6.3.0-rc6-00311-gde8224969f66 (root@bf16f3646a84) (gcc (Debian 11.2.0-12) 11.2.0, GNU ld (GNU Binutils for Debian) 2.40) #446 SMP PREEMPT_DYNAMIC Sat Apr 15 14:12:29 UTC 2023 [0.00] Command line: BOOT_IMAGE=/boot/vmlinuz-6.3.0-rc6-00311-gde8224969f66 root=/dev/sda3 rw quiet noisapnp cryptomgr.notests ipv6.disable_ipv6=1 selinux=0 cpuhp.parallel=off [0.00] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [0.00] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [0.00] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [0.00] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [0.00] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [0.00] signal: max sigframe size: 1776 [0.00] BIOS-provided physical RAM map: [0.00] BIOS-e820: [mem 0x-0x0009fbff] usable [0.00] BIOS-e820: [mem 0x0009fc00-0x0009] reserved [0.00] BIOS-e820: [mem 0x000f-0x000f] reserved [0.00] BIOS-e820: [mem 0x0010-0x5fe4cfff] usable [0.00] BIOS-e820: [mem 0x5fe4d000-0x7fff] reserved [0.00] BIOS-e820: [mem 0xf800-0xfbff] reserved [0.00] BIOS-e820: [mem 0xfec1-0xfec10fff] reserved [0.00] BIOS-e820: [mem 0x0001-0x00017eff] usable [0.00] NX (Execute Disable) protection: active [0.00] SMBIOS 3.0.0 present. [0.00] DMI: ASUS F2A85-M_PRO/F2A85-M_PRO, BIOS 4.18-9-gb640ed51b2 04/17/2023 [0.00] tsc: Fast TSC calibration using PIT [0.00] tsc: Initial usec timer 9249065 [0.00] tsc: Detected 3899.954 MHz processor [0.000755] e820: update [mem 0x-0x0fff] usable ==> reserved [0.000759] e820: remove [mem 0x000a-0x000f] usable [0.000763] last_pfn = 0x17f000 max_arch_pfn = 0x4 [0.000768] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [0.000938] last_pfn = 0x5fe4d max_arch_pfn = 0x4 [0.004000] Using GB pages for direct mapping [0.004000] ACPI: Early table checksum verification disabled [0.004000] ACPI: RSDP 0x000F6830 24 (v02 COREv4) [0.004000] ACPI: XSDT 0x5FE5A0E0 74 (v01 COREv4 COREBOOT CORE 20200925) [0.004000] ACPI: FACP 0x5FE5BBC0 000114 (v06 COREv4 COREBOOT CORE 20200925) [0.004000] ACPI: DSDT 0x5FE5A280 00193A (v02 COREv4 COREBOOT 00010001 INTL 20200925) [0.004000] ACPI: FACS 0x5FE5A240 40 [0.004000] ACPI: FACS 0x5FE5A240 40 [0.004000] ACPI: SSDT 0x5FE5BCE0 8A (v02 COREv4 COREBOOT 002A CORE 20200925) [0.004000] ACPI: MCFG 0x5FE5BD70 3C (v01 COREv4 COREBOOT CORE 20200925) [0.004000] ACPI: APIC 0x5FE5BDB0 62 (v03 COREv4 COREBOOT CORE 20200925) [0.004000] ACPI: HPET 0x5FE5BE20 38 (v01 COREv4 COREBOOT CORE 20200925) [0.004000] ACPI: HEST 0x5FE5BE60 0001D0 (v01 COREv4 COREBOOT CORE 20200925) [0.004000] ACPI: IVRS 0x5FE5C030 70 (v02 AMDAMDIOMMU 0001 AMD ) [0.004000] ACPI: SSDT 0x5FE5C0A0 00051F (v02 AMDALIB 0001 MSFT 0400) [0.004000] ACPI: SSDT 0x5FE5C5C0 0006B2 (v01 AMDPOWERNOW 0001 AMD 0001) [0.004000] ACPI: VFCT 0x5FE5CC80 00F269 (v01 COREv4 COREBOOT CORE 20200925) [0.004000] ACPI: Reserving FACP table memory at [mem 0x5fe5bbc0-0x5fe5bcd3] [0.004000] ACPI: Reserving DSDT table memory at [mem 0x5fe5a280-0x5fe5bbb9] [0.004000] ACPI: Reserving FACS table memory at [mem 0x5fe5a240-0x5fe5a27f] [0.004000] ACPI: Reserving FACS
Re: [SPECIFICATION RFC] The firmware and bootloader log specification
Dear Wim, dear Daniel, First, thank you for including all parties in the discussion. Am 04.12.20 um 13:52 schrieb Wim Vervoorn: I agree with you. Using an existing standard is better than inventing a new one in this case. I think using the coreboot logging is a good idea as there is indeed a lot of support already available and it is lightweight and simple. In my opinion coreboot’s format is lacking, that it does not record the timestamp, and the log level is not stored as metadata, but (in coreboot) only used to decide if to print the message or not. I agree with you, that an existing standard should be used, and in my opinion it’s Linux message format. That is most widely supported, and existing tools could then also work with pre-Linux messages. Sean Hudson from Mentor Graphics presented that idea at Embedded Linux Conference Europe 2016 [1]. No idea, if anything came out of that effort. (Unfortunately, I couldn’t find an email. Does somebody have contacts at Mentor to find out, how to reach him?) Kind regards, Paul [1]: http://events17.linuxfoundation.org/sites/events/files/slides/2016-10-12%20-%20ELCE%20-%20Shared%20Logging%20-%20Part%20Deux.pdf
Re: [patch 00/37] cpu/hotplug, x86: Reworked parallel CPU bringup
Dear Thomas, Am 18.04.23 um 10:40 schrieb Thomas Gleixner: On Tue, Apr 18 2023 at 08:58, Thomas Gleixner wrote: On Mon, Apr 17 2023 at 19:40, Paul Menzel wrote: Am 17.04.23 um 16:48 schrieb Thomas Gleixner: On Mon, Apr 17 2023 at 13:19, Paul Menzel wrote: Am 15.04.23 um 01:44 schrieb Thomas Gleixner: [0.258193] smpboot: CPU0: AMD A6-6400K APU with Radeon(tm) HD Graphics (family: 0x15, model: 0x13, stepping: 0x1) […] [0.259329] smp: Bringing up secondary CPUs ... [0.259527] x86: Booting SMP configuration: [0.259528] node #0, CPUs: #1 [0.261007] After schedule_preempt_disabled [ 10.260990] CPU1 failed to report alive state Weird. CPU1 fails to come up and report that it has reached the synchronization point. Does it work when you add cpuhp.parallel=off on the kernel command line? Yes, the ten seconds delay is gone with `cpuhp.parallel=off`. There was a patch set in the past, that worked on that device. I think up to v4 it did *not* work at all and hung [1]. I need some days to collect the results again. Can you please apply the patch below on top of the pile remove the command line option again? Bah. That patch does not make any sense at all. Not enough coffee. Can you please provide the output of cpuid? Of course. Here the top, and the whole output is attached. ``` CPU 0: vendor_id = "AuthenticAMD" version information (1/eax): processor type = primary processor (0) family = 0xf (15) model = 0x3 (3) stepping id = 0x1 (1) extended family = 0x6 (6) extended model = 0x1 (1) (family synth) = 0x15 (21) (model synth) = 0x13 (19) (simple synth) = AMD (unknown type) (Richland RL-A1) [Piledriver], 32nm […] ``` Kind regards, PaulCPU 0: vendor_id = "AuthenticAMD" version information (1/eax): processor type = primary processor (0) family = 0xf (15) model = 0x3 (3) stepping id = 0x1 (1) extended family = 0x6 (6) extended model = 0x1 (1) (family synth) = 0x15 (21) (model synth) = 0x13 (19) (simple synth) = AMD (unknown type) (Richland RL-A1) [Piledriver], 32nm miscellaneous (1/ebx): process local APIC physical ID = 0x0 (0) maximum IDs for CPUs in pkg= 0x2 (2) CLFLUSH line size = 0x8 (8) brand index= 0x0 (0) brand id = 0x00 (0): unknown feature information (1/edx): x87 FPU on chip= true VME: virtual-8086 mode enhancement = true DE: debugging extensions = true PSE: page size extensions = true TSC: time stamp counter= true RDMSR and WRMSR support= true PAE: physical address extensions = true MCE: machine check exception = true CMPXCHG8B inst.= true APIC on chip = true SYSENTER and SYSEXIT = true MTRR: memory type range registers = true PTE global bit = true MCA: machine check architecture= true CMOV: conditional move/compare instr = true PAT: page attribute table = true PSE-36: page size extension= true PSN: processor serial number = false CLFLUSH instruction= true DS: debug store= false ACPI: thermal monitor and clock ctrl = false MMX Technology = true FXSAVE/FXRSTOR = true SSE extensions = true SSE2 extensions= true SS: self snoop = false hyper-threading / multi-core supported = true TM: therm. monitor = false IA64 = false PBE: pending break event = false feature information (1/ecx): PNI/SSE3: Prescott New Instructions = true PCLMULDQ instruction= true DTES64: 64-bit debug store = false MONITOR/MWAIT = true CPL-qualified debug store = false VMX: virtual machine extensions = false SMX: safer mode extensions = false Enhanced Intel SpeedStep Technology = false TM2: thermal monitor 2 = false SSSE3 extensions= true context ID: adaptive or shared L1 data = false SDBG: IA32_DEBUG_INTERFACE = false FMA instruction = true CMPXCHG16B instruction = true xTPR disable= false PDCM: perfmon and debug = false PCID: process
Re: [patch 00/37] cpu/hotplug, x86: Reworked parallel CPU bringup
Dear Thomas, Am 19.04.23 um 14:38 schrieb Thomas Gleixner: On Wed, Apr 19 2023 at 11:38, Thomas Gleixner wrote: On Tue, Apr 18 2023 at 22:10, Paul Menzel wrote: Am 18.04.23 um 10:40 schrieb Thomas Gleixner: Can you please provide the output of cpuid? Of course. Here the top, and the whole output is attached. Thanks for the data. Can you please apply the debug patch below and provide the dmesg output? Just the line which is added by the patch is enough. You can boot with cpuhp.parallel=off so you don't have wait for 10 seconds. Borislav found some a machine which also refuses to boot. It turns of the debug patch was spot on: [0.462724] node #0, CPUs: #1 [0.462731] smpboot: Kicking AP alive: 17 [0.465723] #2 [0.465732] smpboot: Kicking AP alive: 18 [0.467641] #3 [0.467641] smpboot: Kicking AP alive: 19 So the kernel gets APICID 17, 18, 19 from ACPI but CPUID leaf 0x1 ebx[31:24], which is the initial APICID has: CPU10x01 CPU20x02 CPU30x03 Which means the APICID to Linux CPU number lookup based on CPUID 0x01 fails for all of them and stops them dead in the low level startup code. I am attaching the logs for completeness. Linux is build from your branch with the debug print on top. The firmware, coreboot based, is built from [1], but it also happened non-parallel MP init. The code has better debug prints (attached) though as far as I can see. As Borislav is able to reproduce this too with some non-coreboot firmware, I assume it’s unrelated to coreboot. ``` [0.259247] smp: Bringing up secondary CPUs ... [0.259446] x86: Booting SMP configuration: [0.259448] node #0, CPUs: #1 [0.259453] smpboot: Kicking AP alive: 17 [ 10.260918] CPU1 failed to report alive state [ 10.260998] smp: Brought up 1 node, 1 CPU [ 10.261000] smpboot: Max logical packages: 2 [ 10.261001] smpboot: Total of 1 processors activated (7801.09 BogoMIPS) ``` IOW, the BIOS assignes random numbers to the AP APICs for whatever raisins, which leaves the parallel startup low level code up a creek without a paddle, except for actually reading the APICID back from the APIC. *SHUDDER* I'm leaning towards disabling the CPUID lead 0x01 based discovery and be done with it. Kind regards, Paul [1]: https://review.coreboot.org/68169[0.00] Linux version 6.3.0-rc3-00045-g64de4df9c80b (root@bf16f3646a84) (gcc (Debian 11.2.0-12) 11.2.0, GNU ld (GNU Binutils for Debian) 2.40) #449 SMP PREEMPT_DYNAMIC Wed Apr 19 16:13:54 UTC 2023 [0.00] Command line: BOOT_IMAGE=/boot/vmlinuz-6.3.0-rc3-00045-g64de4df9c80b root=/dev/sda3 rw quiet noisapnp cryptomgr.notests ipv6.disable_ipv6=1 selinux=0 [0.00] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [0.00] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [0.00] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [0.00] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [0.00] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [0.00] signal: max sigframe size: 1776 [0.00] BIOS-provided physical RAM map: [0.00] BIOS-e820: [mem 0x-0x0009fbff] usable [0.00] BIOS-e820: [mem 0x0009fc00-0x0009] reserved [0.00] BIOS-e820: [mem 0x000f-0x000f] reserved [0.00] BIOS-e820: [mem 0x0010-0x5fe3cfff] usable [0.00] BIOS-e820: [mem 0x5fe3d000-0x7fff] reserved [0.00] BIOS-e820: [mem 0xf800-0xfbff] reserved [0.00] BIOS-e820: [mem 0xfec1-0xfec10fff] reserved [0.00] BIOS-e820: [mem 0x0001-0x00017eff] usable [0.00] NX (Execute Disable) protection: active [0.00] SMBIOS 3.0.0 present. [0.00] DMI: ASUS F2A85-M_PRO/F2A85-M_PRO, BIOS 4.18-15-gc782ef4345 04/19/2023 [0.00] tsc: Fast TSC calibration using PIT [0.00] tsc: Detected 3900.549 MHz processor [0.000756] e820: update [mem 0x-0x0fff] usable ==> reserved [0.000759] e820: remove [mem 0x000a-0x000f] usable [0.000763] last_pfn = 0x17f000 max_arch_pfn = 0x4 [0.000768] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [0.000940] last_pfn = 0x5fe3d max_arch_pfn = 0x4 [0.004000] Using GB pages for direct mapping [0.004000] ACPI: Early table checksum verification disabled [0.004000] ACPI: RSDP 0x000F6830 24 (v02 COREv4) [0.004000] ACPI: XSDT 0x5FE4A0E0 74 (v01 COREv4 COREBOOT CORE 20200925) [0.004000] ACPI: FACP 0x5FE4BBC0 000114 (v06 COREv4 COREBOOT CORE 20200925) [0.004000] ACPI: DSDT 0x5FE4A280 00193A (v02 COREv4 COREBOOT 00010001 INTL 2
Re: [patch 00/37] cpu/hotplug, x86: Reworked parallel CPU bringup
Dear Thomas,
Am 20.04.23 um 17:57 schrieb Thomas Gleixner:
On Thu, Apr 20 2023 at 07:51, Sean Christopherson wrote:
On Thu, Apr 20, 2023, Thomas Gleixner wrote:
On Thu, Apr 20 2023 at 10:23, Andrew Cooper wrote:
On 20/04/2023 9:32 am, Thomas Gleixner wrote:
On Wed, Apr 19, 2023, Andrew Cooper wrote:
This was changed in x2APIC, which made the x2APIC_ID immutable.
I'm pondering to simply deny parallel mode if x2APIC is not there.
I'm not sure if that will help much.
Spoilsport.
LOL, well let me pile on then. x2APIC IDs aren't immutable on AMD hardware.
The
ID is read-only when the CPU is in x2APIC mode, but any changes made to the ID
while the CPU is in xAPIC mode survive the transition to x2APIC. From the APM:
A value previously written by software to the 8-bit APIC_ID register (MMIO
offset
30h) is converted by hardware into the appropriate format and reflected into
the
32-bit x2APIC_ID register (MSR 802h).
FWIW, my observations from testing on bare metal are that the xAPIC ID is
effectively
read-only (writes are dropped) on Intel CPUs as far back as Haswell, while the
above
behavior described in the APM holds true on at least Rome and Milan.
My guess is that Intel's uArch specific behavior of the xAPIC ID being read-only
was introduced when x2APIC came along, but I didn't test farther back than
Haswell.
I'm not so worried about modern hardware. The horrorshow is the old muck
as demonstrated and of course there is virt :)
Something like the completely untested below should just work whatever
APIC ID the BIOS decided to dice.
That might just work on SEV too without that GHCB muck, but what do I
know.
Thanks,
tglx
---
--- a/arch/x86/include/asm/apicdef.h
+++ b/arch/x86/include/asm/apicdef.h
@@ -138,7 +138,8 @@
#define APIC_EILVT_MASKED (1 << 16)
#define APIC_BASE (fix_to_virt(FIX_APIC_BASE))
-#define APIC_BASE_MSR 0x800
+#define APIC_BASE_MSR 0x800
+#define APIC_X2APIC_ID_MSR 0x802
#define XAPIC_ENABLE (1UL << 11)
#define X2APIC_ENABLE (1UL << 10)
@@ -162,6 +163,7 @@
#define APIC_CPUID(apicid)((apicid) & XAPIC_DEST_CPUS_MASK)
#define NUM_APIC_CLUSTERS ((BAD_APICID + 1) >> XAPIC_DEST_CPUS_SHIFT)
+#ifndef __ASSEMBLY__
/*
* the local APIC register structure, memory mapped. Not terribly well
* tested, but we might eventually use this one in the future - the
@@ -435,4 +437,5 @@ enum apic_delivery_modes {
APIC_DELIVERY_MODE_EXTINT = 7,
};
+#endif /* !__ASSEMBLY__ */
#endif /* _ASM_X86_APICDEF_H */
--- a/arch/x86/include/asm/smp.h
+++ b/arch/x86/include/asm/smp.h
@@ -195,14 +195,13 @@ extern void nmi_selftest(void);
#endif
extern unsigned int smpboot_control;
+extern unsigned long apic_mmio_base;
#endif /* !__ASSEMBLY__ */
/* Control bits for startup_64 */
-#define STARTUP_APICID_CPUID_1F 0x8000
-#define STARTUP_APICID_CPUID_0B 0x4000
-#define STARTUP_APICID_CPUID_01 0x2000
-#define STARTUP_APICID_SEV_ES 0x1000
+#define STARTUP_READ_APICID0x8000
+#define STARTUP_APICID_SEV_ES 0x4000
/* Top 8 bits are reserved for control */
#define STARTUP_PARALLEL_MASK 0xFF00
--- a/arch/x86/kernel/apic/apic.c
+++ b/arch/x86/kernel/apic/apic.c
@@ -101,6 +101,8 @@ static int apic_extnmi __ro_after_init =
*/
static bool virt_ext_dest_id __ro_after_init;
+unsigned long apic_mmio_base __ro_after_init;
+
/*
* Map cpu index to physical APIC ID
*/
@@ -2164,6 +2166,7 @@ void __init register_lapic_address(unsig
if (!x2apic_mode) {
set_fixmap_nocache(FIX_APIC_BASE, address);
+ apic_mmio_base = APIC_BASE;
apic_printk(APIC_VERBOSE, "mapped APIC to %16lx (%16lx)\n",
APIC_BASE, address);
}
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -24,8 +24,10 @@
#include "../entry/calling.h"
#include
#include
+#include
#include
#include
+
#include
/*
@@ -237,37 +239,24 @@ SYM_INNER_LABEL(secondary_startup_64_no_
#ifdef CONFIG_SMP
/*
-* For parallel boot, the APIC ID is retrieved from CPUID, and then
-* used to look up the CPU number. For booting a single CPU, the
-* CPU number is encoded in smpboot_control.
+* For parallel boot, the APIC ID is either retrieved the APIC or
+* from CPUID, and then used to look up the CPU number.
+* For booting a single CPU, the CPU number is encoded in
+* smpboot_control.
*
-* Bit 31 STARTUP_APICID_CPUID_1F flag (use CPUID 0x1f)
-* Bit 30 STARTUP_APICID_CPUID_0B flag (use CPUID 0x0b)
-* Bit 29 STARTUP_APICID_CPUID_01 flag (use CPUID 0x01)
-* Bit 28 STARTUP_APICID_SEV_ES flag (CPUID 0x0b via GHCB MSR)
+* Bit 31 STARTUP_APICID_READ (Read APICID from APIC)
+* Bit 30 STARTUP_APICID_SEV_ES flag (CPUID 0x0b via GHCB
Re: [patch 00/37] cpu/hotplug, x86: Reworked parallel CPU bringup
Dear Thomas, Am 20.04.23 um 21:10 schrieb Thomas Gleixner: On Thu, Apr 20 2023 at 18:47, Paul Menzel wrote: Am 20.04.23 um 17:57 schrieb Thomas Gleixner: I quickly applied it on top of your branch, but I am getting: As I said it was untested. I was traveling and did not have access to a machine to even build it completely. Fixed up and tested version below. Sorry, if it sounded like a complaint. I just wanted to give a quick feedback. […] I tested your new version even on Friday, and it worked fine – no ten seconds delay. Please find the messages attached. Thank you all for your great work. Kind regards, Paul PS: I am going to try to test your updated branch at the end of the week.[0.00] Linux version 6.3.0-rc3-00046-g8ba643d7e1c7 (root@bf16f3646a84) (gcc (Debian 11.2.0-12) 11.2.0, GNU ld (GNU Binutils for Debian) 2.40) #452 SMP PREEMPT_DYNAMIC Thu Apr 20 20:15:01 UTC 2023 [0.00] Command line: BOOT_IMAGE=/boot/vmlinuz-6.3.0-rc3-00046-g8ba643d7e1c7 root=/dev/sda3 rw quiet noisapnp cryptomgr.notests ipv6.disable_ipv6=1 selinux=0 [0.00] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [0.00] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [0.00] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [0.00] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [0.00] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [0.00] signal: max sigframe size: 1776 [0.00] BIOS-provided physical RAM map: [0.00] BIOS-e820: [mem 0x-0x0009fbff] usable [0.00] BIOS-e820: [mem 0x0009fc00-0x0009] reserved [0.00] BIOS-e820: [mem 0x000f-0x000f] reserved [0.00] BIOS-e820: [mem 0x0010-0x5fe4cfff] usable [0.00] BIOS-e820: [mem 0x5fe4d000-0x7fff] reserved [0.00] BIOS-e820: [mem 0xf800-0xfbff] reserved [0.00] BIOS-e820: [mem 0xfec1-0xfec10fff] reserved [0.00] BIOS-e820: [mem 0x0001-0x00017eff] usable [0.00] NX (Execute Disable) protection: active [0.00] SMBIOS 3.0.0 present. [0.00] DMI: ASUS F2A85-M_PRO/F2A85-M_PRO, BIOS 4.18-9-gb640ed51b2 04/17/2023 [0.00] tsc: Fast TSC calibration using PIT [0.00] tsc: Detected 3900.440 MHz processor [0.000756] e820: update [mem 0x-0x0fff] usable ==> reserved [0.000759] e820: remove [mem 0x000a-0x000f] usable [0.000763] last_pfn = 0x17f000 max_arch_pfn = 0x4 [0.000768] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [0.000938] last_pfn = 0x5fe4d max_arch_pfn = 0x4 [0.004000] Using GB pages for direct mapping [0.004000] ACPI: Early table checksum verification disabled [0.004000] ACPI: RSDP 0x000F6830 24 (v02 COREv4) [0.004000] ACPI: XSDT 0x5FE5A0E0 74 (v01 COREv4 COREBOOT CORE 20200925) [0.004000] ACPI: FACP 0x5FE5BBC0 000114 (v06 COREv4 COREBOOT CORE 20200925) [0.004000] ACPI: DSDT 0x5FE5A280 00193A (v02 COREv4 COREBOOT 00010001 INTL 20200925) [0.004000] ACPI: FACS 0x5FE5A240 40 [0.004000] ACPI: FACS 0x5FE5A240 40 [0.004000] ACPI: SSDT 0x5FE5BCE0 8A (v02 COREv4 COREBOOT 002A CORE 20200925) [0.004000] ACPI: MCFG 0x5FE5BD70 3C (v01 COREv4 COREBOOT CORE 20200925) [0.004000] ACPI: APIC 0x5FE5BDB0 62 (v03 COREv4 COREBOOT CORE 20200925) [0.004000] ACPI: HPET 0x5FE5BE20 38 (v01 COREv4 COREBOOT CORE 20200925) [0.004000] ACPI: HEST 0x5FE5BE60 0001D0 (v01 COREv4 COREBOOT CORE 20200925) [0.004000] ACPI: IVRS 0x5FE5C030 70 (v02 AMDAMDIOMMU 0001 AMD ) [0.004000] ACPI: SSDT 0x5FE5C0A0 00051F (v02 AMDALIB 0001 MSFT 0400) [0.004000] ACPI: SSDT 0x5FE5C5C0 0006B2 (v01 AMDPOWERNOW 0001 AMD 0001) [0.004000] ACPI: VFCT 0x5FE5CC80 00F269 (v01 COREv4 COREBOOT CORE 20200925) [0.004000] ACPI: Reserving FACP table memory at [mem 0x5fe5bbc0-0x5fe5bcd3] [0.004000] ACPI: Reserving DSDT table memory at [mem 0x5fe5a280-0x5fe5bbb9] [0.004000] ACPI: Reserving FACS table memory at [mem 0x5fe5a240-0x5fe5a27f] [0.004000] ACPI: Reserving FACS table memory at [mem 0x5fe5a240-0x5fe5a27f] [0.004000] ACPI: Reserving SSDT table memory at [mem 0x5fe5bce0-0x5fe5bd69] [0.004000] ACPI: Reserving MCFG table memory at [mem 0x5fe5bd70-0x5fe5bdab] [0.004000] ACPI: Reserving APIC table memory at [mem 0x5fe5bdb0-0x5fe5be11] [0.004000] ACPI: Reserving HPET table memory at [mem 0x5fe5be20-0x5fe5be57] [0.004000] ACPI: Res
