On Mon, 2 Sep 2019 16:27:18 +1000 David Gibson <da...@gibson.dropbear.id.au> wrote:
> On Fri, Aug 30, 2019 at 07:45:43PM +0200, Greg Kurz wrote: > > On Fri, 30 Aug 2019 17:34:13 +0100 > > Daniel P. Berrangé <berra...@redhat.com> wrote: > > > > > On Fri, Aug 30, 2019 at 06:13:45PM +0200, Laurent Vivier wrote: > > > > When we hotplug a CPU on memory-less/cpu-less node, the linux kernel > > > > crashes. > > > > > > > > This happens because linux kernel needs to know the NUMA topology at > > > > start to be able to initialize the distance lookup table. > > > > > > > > On pseries, the topology is provided by the firmware via the existing > > > > CPUs and memory information. Thus a node without memory and CPU cannot > > > > be > > > > discovered by the kernel. > > > > > > > > To avoid the kernel crash, do not allow to start pseries with empty > > > > nodes. > > > > > > This describes one possible guest OS. Is there any reasonable chance > > > that a non-Linux guest might be able to handle this situation correctly, > > > or do you expect any guest to have the same restriction ? > > That's... a more complicated question than you'd think. > > The problem here is it's not really obvious in PAPR how topology > information for nodes without memory should be described in the device > tree (which is the only way we given that information to the guest). > The reported issue is to have a node without memory AND without cpu. > It's possible there's some way to encode this information that would > make AIX happy and we just need to fix Linux to cope with that, but > it's not really clear what it would be. > > > I can try to grab an AIX image and give a try, but anyway this looks like > > a very big hammer to me... :-\ > > I'm not really sure why everyone seems to think losing zero-memory > node capability is such a big deal. It's never worked in practice on > POWER and we can always put it back if we figure out a sensible way to > do it. > It isn't really about losing the memory-less/cpu-less node capability, but more about finding the appropriate fix. The changelog doesn't give much clues on what's happening exactly: QEMU command line ? linux call stack ? For example, I could hit a crash with the following command line: -smp 1,maxcpus=2 \ -object memory-backend-ram,size=512M,id=node0 \ -numa node,nodeid=0,memdev=node0 \ -numa node,nodeid=1 (qemu) info numa 2 nodes node 0 cpus: 0 node 0 size: 512 MB node 0 plugged: 0 MB node 1 cpus: node 1 size: 0 MB node 1 plugged: 0 MB (qemu) device_add host-spapr-cpu-core,core-id=1 [ 24.507552] Built 1 zonelists, mobility grouping on. Total pages: 7656 [ 24.507592] Policy zone: Normal [ 24.553481] WARNING: workqueue cpumask: online intersect > possible intersect [ 24.608814] BUG: Unable to handle kernel data access at 0x14e13da04c5bc37e [ 24.608875] Faulting instruction address: 0xc000000000175650 [ 24.608931] Oops: Kernel access of bad area, sig: 11 [#1] [ 24.608976] LE PAGE_SIZE=64K MMU=Radix MMU=Hash SMP NR_CPUS=1024 NUMA pSeries [ 24.609042] Modules linked in: virtio_net vmx_crypto net_failover failover crct10dif_vpmsum ip_tables xfs libcrc32c crc32c_vpmsum virtio_blk kvm rpadlpar_io rpaphp 9p fscache 9pnet_virtio 9pnet [ 24.609222] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.1.17-300.fc30.ppc64le #1 [ 24.609286] NIP: c000000000175650 LR: c000000000175310 CTR: 0000000000000000 [ 24.609351] REGS: c00000001e597210 TRAP: 0380 Not tainted (5.1.17-300.fc30.ppc64le) [ 24.609414] MSR: 8000000000009033 <SF,EE,ME,IR,DR,RI,LE> CR: 44444248 XER: 00000000 [ 24.609482] CFAR: c000000000175528 IRQMASK: 0 [ 24.609482] GPR00: c000000000175310 c00000001e5974a0 c0000000015fc400 0000000000000002 [ 24.609482] GPR04: 0000000000000001 0000000000000001 0000000000000001 0000000000000400 [ 24.609482] GPR08: 14e13da04c5bc37e 0000000000000000 0000000000000000 0000000000000000 [ 24.609482] GPR12: 0000000024022248 c00000000fffee00 0000000000000007 c00000001e0e8fb0 [ 24.609482] GPR16: c00000000162dc70 0000000000000008 c00000001e5976d8 0000000020000000 [ 24.609482] GPR20: 0000000100000003 0000000000000001 0000000000000000 14e13da04c5bc35e [ 24.609482] GPR24: c000000001630164 0000000000000010 14e13da04c5bc37e 0000000000000000 [ 24.609482] GPR28: 0000000000000002 c0000000142a0e00 c00000001ff25d80 c00000001e5975a8 [ 24.610052] NIP [c000000000175650] find_busiest_group+0x510/0xe10 [ 24.610107] LR [c000000000175310] find_busiest_group+0x1d0/0xe10 [ 24.610169] Call Trace: [ 24.610203] [c00000001e5974a0] [c000000000175310] find_busiest_group+0x1d0/0xe10 (unreliable) [ 24.610304] [c00000001e597680] [c000000000176110] load_balance+0x1c0/0xe80 [ 24.610377] [c00000001e5977d0] [c000000000176ff8] rebalance_domains+0x228/0x380 [ 24.610467] [c00000001e597880] [c000000000c7c170] __do_softirq+0x170/0x404 [ 24.610542] [c00000001e597980] [c000000000124368] irq_exit+0xd8/0x110 [ 24.610617] [c00000001e5979a0] [c000000000028778] timer_interrupt+0x128/0x2e0 [ 24.610706] [c00000001e597a00] [c000000000009314] decrementer_common+0x154/0x160 [ 24.610799] --- interrupt: 901 at plpar_hcall_norets+0x1c/0x28 [ 24.610799] LR = check_and_cede_processor+0x48/0x60 [ 24.610915] [c00000001e597d00] [c00000001e597d60] 0xc00000001e597d60 (unreliable) [ 24.611004] [c00000001e597d60] [c0000000009e22a8] shared_cede_loop+0x68/0x180 [ 24.611096] [c00000001e597da0] [c0000000009dec64] cpuidle_enter_state+0xa4/0x660 [ 24.611191] [c00000001e597e30] [c0000000001647a0] call_cpuidle+0x50/0xa0 [ 24.611270] [c00000001e597e50] [c000000000164d6c] do_idle+0x2cc/0x3b0 [ 24.611350] [c00000001e597ec0] [c00000000016508c] cpu_startup_entry+0x3c/0x50 [ 24.611445] [c00000001e597ef0] [c000000000051dd0] start_secondary+0x630/0x660 [ 24.611539] [c00000001e597f90] [c00000000000b25c] start_secondary_prolog+0x10/0x14 [ 24.611632] Instruction dump: [ 24.611680] 7c374800 41820234 e8920016 3b570020 8152002c 7c893670 7d290194 548506be [ 24.611775] 788606a0 7d2907b4 79291f24 7d1a4a14 <7cfa482a> 7ce72c36 78e707e0 2d270000 [ 24.611871] ---[ end trace 0e5e3ed14d31f59d ]--- [ 24.617852] [ 25.617885] Kernel panic - not syncing: Aiee, killing interrupt handler! (qemu) info numa 2 nodes node 0 cpus: 0 node 0 size: 512 MB node 0 plugged: 0 MB node 1 cpus: 1 node 1 size: 0 MB node 1 plugged: 0 MB but the crash doesn't occur with: -smp 1,maxcpus=2 \ -object memory-backend-ram,size=512M,id=node0 \ -numa node,nodeid=0,memdev=node0 \ -numa node,nodeid=1 \ -device spapr-pci-host-bridge,index=1,id=phb1,numa_node=1 (qemu) info numa 2 nodes node 0 cpus: 0 node 0 size: 512 MB node 0 plugged: 0 MB node 1 cpus: node 1 size: 0 MB node 1 plugged: 0 MB (qemu) device_add host-spapr-cpu-core,core-id=1 [ 154.637304] Policy zone: Normal [ 154.665463] WARNING: workqueue cpumask: online intersect > possible intersect (qemu) info numa 2 nodes node 0 cpus: 0 node 0 size: 512 MB node 0 plugged: 0 MB node 1 cpus: 1 node 1 size: 0 MB node 1 plugged: 0 MB nor with: -smp 1,maxcpus=2 \ -object memory-backend-ram,size=512M,id=node0 \ -numa node,nodeid=0,memdev=node0,cpus=0 \ -numa node,nodeid=1 qemu-system-ppc64: warning: CPU(s) not present in any NUMA nodes: CPU 1 [core-id: 1] qemu-system-ppc64: warning: All CPU(s) up to maxcpus should be described in NUMA config, ability to start up with partial NUMA mappings is obsoleted and will be removed in future (qemu) device_add host-spapr-cpu-core,core-id=1 (qemu) info numa 2 nodes node 0 cpus: 0 1 node 0 size: 512 MB node 0 plugged: 0 MB node 1 cpus: node 1 size: 0 MB node 1 plugged: 0 MB so I don't know why linux crashes, but it isn't exactly because of having a cpu-less/memory-less node and this patch catches the non-crashing cases anyway.
pgpsccACsbx11.pgp
Description: OpenPGP digital signature
