This is also been seen on j/gcp-fips with version 5.15.0-1048.56+fips1 during development cycle d2024.01.02 on the n2d-standard-4.sev_snp instance. If necessary, another bug could be filed targeting GCP specifically.
-- You received this bug notification because you are a member of Canonical Platform QA Team, which is subscribed to ubuntu-kernel-tests. https://bugs.launchpad.net/bugs/2034057 Title: function_graph tracer in ftrace related tests triggers kernel bug (arch/x86/xen/spinlock.c:62) and kernel panic on AWS cloud c3.xlarge Status in ubuntu-kernel-tests: New Status in linux-aws package in Ubuntu: New Status in linux-aws source package in Bionic: New Status in linux-aws source package in Focal: New Status in linux-aws source package in Jammy: New Status in linux-aws source package in Lunar: Won't Fix Bug description: Test: * ftrace:test.d--00basic--basic2.tc from ubuntu_kselftests_ftrace * test_enable_all_tracers() and test_enable_all_tracers() from ubuntu_ftrace_smoke_test Will crash AWS instance c3.xlarge when testing the "function_graph" tracer. We have a similar issue filed against Azure (bug 1882669). Filing a new bug report because on AWS this is affecting 5.4 ~ 6.2 AWS kernel. However on Azure this is not affecting newer kernels. Take B-aws-5.4-1108 for example, with the ubuntu_kselftests_ftrace test the system will crash with the following error message and reboot (similar to bug 2029917): [ 211.675624] kernel BUG at /build/linux-aws-5.4-VHOhmt/linux-aws-5.4-5.4.0/arch/x86/xen/spinlock.c:62! [ 211.678258] invalid opcode: 0000 [#1] SMP PTI [ 211.679596] CPU: 1 PID: 14 Comm: cpuhp/1 Not tainted 5.4.0-1108-aws #116~18.04.1-Ubuntu [ 211.681825] Hardware name: Xen HVM domU, BIOS 4.11.amazon 08/24/2006 [ 211.683728] RIP: 0010:dummy_handler+0x4/0x10 [ 211.685042] Code: 8b 75 e4 74 d6 44 89 e7 e8 f9 88 61 00 eb d6 44 89 e7 e8 6f ab 61 00 eb cc 0f 1f 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 <0f> 0b 66 2e 0f 1f 84 00 00 00 00 00 80 3d 59 d0 9f 01 00 75 02 f3 [ 211.690314] RSP: 0000:ffffaecd4000ee38 EFLAGS: 00010046 [ 211.691934] RAX: ffffffffb462e3e0 RBX: 000000000000003b RCX: 0000000000000000 [ 211.694036] RDX: 0000000000400e00 RSI: 0000000000000000 RDI: 000000000000003b [ 211.696159] RBP: ffffaecd4000ee38 R08: ffff8aefa6c036c0 R09: ffff8aefa6c038c0 [ 211.698232] R10: 0000000000000000 R11: ffffffffb6064da8 R12: 0000000000000000 [ 211.700365] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8aefa34b8700 [ 211.702482] FS: 0000000000000000(0000) GS:ffff8aefa8040000(0000) knlGS:0000000000000000 [ 211.704894] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 211.706598] CR2: 0000000000000000 CR3: 00000001b400a001 CR4: 00000000001606e0 [ 211.708731] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 211.710835] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 211.713448] Call Trace: [ 211.714796] <IRQ> [ 211.715996] __handle_irq_event_percpu+0x44/0x1a0 [ 211.717907] handle_irq_event_percpu+0x32/0x80 [ 211.719759] handle_percpu_irq+0x3d/0x60 [ 211.721474] generic_handle_irq+0x28/0x40 [ 211.723184] handle_irq_for_port+0x8f/0xe0 [ 211.724988] evtchn_2l_handle_events+0x157/0x270 [ 211.726913] __xen_evtchn_do_upcall+0x76/0xe0 [ 211.728749] xen_evtchn_do_upcall+0x2b/0x40 [ 211.730520] xen_hvm_callback_vector+0xf/0x20 [ 211.732271] </IRQ> [ 211.733425] RIP: 0010:_raw_spin_unlock_irqrestore+0x15/0x20 [ 211.735470] Code: e8 70 3d 64 ff 4c 29 e0 4c 39 f0 76 cf 80 0b 08 eb 8a 90 90 90 0f 1f 44 00 00 55 48 89 e5 e8 a6 ad 66 ff 66 90 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 c6 07 [ 211.741920] RSP: 0000:ffffaecd400fbcf8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff0c [ 211.744980] RAX: 0000000000000001 RBX: ffff8aefa34b8700 RCX: 000000000002cc00 [ 211.747420] RDX: 0000000000000001 RSI: 0000000000000246 RDI: 0000000000000246 [ 211.749882] RBP: ffffaecd400fbcf8 R08: ffff8aefa6c036c0 R09: ffff8aefa6c038c0 [ 211.752340] R10: 0000000000000000 R11: 0000000000000246 R12: 000000000000003b [ 211.754785] R13: 0000000000000000 R14: ffff8aef95bdfa00 R15: ffff8aef95bdfaa4 [ 211.757249] __setup_irq+0x456/0x760 [ 211.758779] ? kmem_cache_alloc_trace+0x170/0x230 [ 211.760639] request_threaded_irq+0xfb/0x160 [ 211.762358] bind_ipi_to_irqhandler+0xba/0x1c0 [ 211.764124] ? xen_qlock_wait+0x90/0x90 [ 211.765734] ? snr_uncore_mmio_init+0x20/0x20 [ 211.767496] xen_init_lock_cpu+0x78/0xd0 [ 211.769135] ? snr_uncore_mmio_init+0x20/0x20 [ 211.770864] xen_cpu_up_online+0xe/0x20 [ 211.772500] cpuhp_invoke_callback+0x8a/0x580 [ 211.774233] cpuhp_thread_fun+0xb8/0x120 [ 211.775866] smpboot_thread_fn+0xfc/0x170 [ 211.777524] kthread+0x121/0x140 [ 211.778968] ? sort_range+0x30/0x30 [ 211.780508] ? kthread_park+0x90/0x90 [ 211.782073] ret_from_fork+0x35/0x40 [ 211.783622] Modules linked in: nls_iso8859_1 binfmt_misc serio_raw sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd glue_helper ixgbevf [ 211.796497] ---[ end trace bb7d4e9bb7f852cb ]--- [ 211.798303] RIP: 0010:dummy_handler+0x4/0x10 [ 211.800022] Code: 8b 75 e4 74 d6 44 89 e7 e8 f9 88 61 00 eb d6 44 89 e7 e8 6f ab 61 00 eb cc 0f 1f 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 <0f> 0b 66 2e 0f 1f 84 00 00 00 00 00 80 3d 59 d0 9f 01 00 75 02 f3 [ 211.806379] RSP: 0000:ffffaecd4000ee38 EFLAGS: 00010046 [ 211.808356] RAX: ffffffffb462e3e0 RBX: 000000000000003b RCX: 0000000000000000 [ 211.810792] RDX: 0000000000400e00 RSI: 0000000000000000 RDI: 000000000000003b [ 211.813263] RBP: ffffaecd4000ee38 R08: ffff8aefa6c036c0 R09: ffff8aefa6c038c0 [ 211.815713] R10: 0000000000000000 R11: ffffffffb6064da8 R12: 0000000000000000 [ 211.818162] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8aefa34b8700 [ 211.820637] FS: 0000000000000000(0000) GS:ffff8aefa8040000(0000) knlGS:0000000000000000 [ 211.823821] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 211.825904] CR2: 0000000000000000 CR3: 00000001b400a001 CR4: 00000000001606e0 [ 211.828357] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 211.830829] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 211.833284] Kernel panic - not syncing: Fatal exception in interrupt [ 211.835575] Kernel Offset: 0x33600000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) Test output: + echo 1 + . /home/ubuntu/autotest/client/tmp/ubuntu_kselftests_ftrace/src/linux/tools/testing/selftests/ftrace/test.d/00basic/basic2.tc + test -f available_tracers + cat available_tracers + echo hwlat + echo blk + echo mmiotrace + echo function_graph (Test interrupted here) To verify this, you can hack linux/tools/testing/selftests/ftrace/test.d/00basic/basic2.tc and add "read -p" to check what tracer we're testing: #!/bin/sh # SPDX-License-Identifier: GPL-2.0 # description: Basic test for tracers # flags: instance test -f available_tracers for t in `cat available_tracers`; do read -p "testing $t" foo echo $t > current_tracer done echo nop > current_tracer And run this manually with sudo ./ftracetest -vvv test.d/00basic/basic2.tc, you will see the system crashes right away with "function_graph". For ubuntu_ftrace_smoke_test, the "function_graph" tracer will be tested with test_function_graph_tracer(). And since "function_graph" tracer is in /sys/kernel/debug/tracing/available_tracers, it will be tested with test_enable_all_tracers() as well. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/2034057/+subscriptions -- Mailing list: https://launchpad.net/~canonical-ubuntu-qa Post to : canonical-ubuntu-qa@lists.launchpad.net Unsubscribe : https://launchpad.net/~canonical-ubuntu-qa More help : https://help.launchpad.net/ListHelp
