Package: src:linux Version: 3.16.7-ckt11-1+deb8u5 Severity: important Tags: upstream
Dear Maintainer, We have identified a failure in this kernel version to boot with low probability on Google Compute Engine, and have traced the issue back to a bug in upstream that has since been fixed. Please integrate the following upstream commit. https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=dd9d3843755da95f63dd3a376f62b3e45c011210 sched: Fix cpu_active_mask/cpu_online_mask race There is a race condition in SMP bootup code, which may result in WARNING: CPU: 0 PID: 1 at kernel/workqueue.c:4418 workqueue_cpu_up_callback() or kernel BUG at kernel/smpboot.c:135! It can be triggered with a bit of luck in Linux guests running on busy hosts. CPU0 CPUn ==== ==== _cpu_up() __cpu_up() start_secondary() set_cpu_online() cpumask_set_cpu(cpu, to_cpumask(cpu_online_bits)); cpu_notify(CPU_ONLINE) <do stuff, see below> cpumask_set_cpu(cpu, to_cpumask(cpu_active_bits)); During the various CPU_ONLINE callbacks CPUn is online but not active. Several things can go wrong at that point, depending on the scheduling of tasks on CPU0. Variant 1: cpu_notify(CPU_ONLINE) workqueue_cpu_up_callback() rebind_workers() set_cpus_allowed_ptr() This call fails because it requires an active CPU; rebind_workers() ends with a warning: WARNING: CPU: 0 PID: 1 at kernel/workqueue.c:4418 workqueue_cpu_up_callback() Variant 2: cpu_notify(CPU_ONLINE) smpboot_thread_call() smpboot_unpark_threads() .. __kthread_unpark() __kthread_bind() wake_up_state() .. select_task_rq() select_fallback_rq() The ->wake_cpu of the unparked thread is not allowed, making a call to select_fallback_rq() necessary. Then, select_fallback_rq() cannot find an allowed, active CPU and promptly resets the allowed CPUs, so that the task in question ends up on CPU0. When those unparked tasks are eventually executed, they run immediately into a BUG: kernel BUG at kernel/smpboot.c:135! Just changing the order in which the online/active bits are set (and adding some memory barriers), would solve the two issues above. However, it would change the order of operations back to the one before commit 6acbfb96976f ("sched: Fix hotplug vs. set_cpus_allowed_ptr()"), thus, reintroducing that particular problem. Going further back into history, we have at least the following commits touching this topic: - commit 2baab4e90495 ("sched: Fix select_fallback_rq() vs cpu_active/cpu_online") - commit 5fbd036b552f ("sched: Cleanup cpu_active madness") Together, these give us the following non-working solutions: - secondary CPU sets active before online, because active is assumed to be a subset of online; - secondary CPU sets online before active, because the primary CPU assumes that an online CPU is also active; - secondary CPU sets online and waits for primary CPU to set active, because it might deadlock. Commit 875ebe940d77 ("powerpc/smp: Wait until secondaries are active & online") introduces an arch-specific solution to this arch-independent problem. Now, go for a more general solution without explicit waiting and simply set active twice: once on the secondary CPU after online was set and once on the primary CPU after online was seen. set_cpus_allowed_ptr()") -- Package-specific info: ** Version: Linux version 3.16.0-4-amd64 (debian-kernel@lists.debian.org) (gcc version 4.8.4 (Debian 4.8.4-1) ) #1 SMP Debian 3.16.7-ckt11-1+deb8u5 (2015-10-09) ** Command line: BOOT_IMAGE=/boot/vmlinuz-3.16.0-4-amd64 root=UUID=f5d73494-1cf2-4811-8e2d-67884d4bd6e7 ro console=ttyS0,38400n8 elevator=noop console=hvc0 ** Not tainted ** Kernel log: [ 0.562395] sd 0:0:1:0: [sda] 20971520 512-byte logical blocks: (10.7 GB/10.0 GiB) [ 0.563436] sd 0:0:1:0: [sda] 4096-byte physical blocks [ 0.564355] sd 0:0:1:0: [sda] Write Protect is off [ 0.565007] sd 0:0:1:0: [sda] Mode Sense: 1f 00 00 08 [ 0.565054] sd 0:0:1:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 0.567526] sda: sda1 [ 0.568469] sd 0:0:1:0: [sda] Attached SCSI disk [ 0.569459] sd 0:0:1:0: Attached scsi generic sg0 type 0 [ 0.680208] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0 [ 0.699765] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null) [ 0.797732] systemd[1]: systemd 215 running in system mode. (+PAM +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ -SECCOMP -APPARMOR) [ 0.799627] systemd[1]: Detected virtualization 'kvm'. [ 0.800379] systemd[1]: Detected architecture 'x86-64'. [ 0.851257] systemd[1]: Inserted module 'autofs4' [ 0.852140] systemd[1]: No hostname configured. [ 0.852805] systemd[1]: Set hostname to <localhost>. [ 0.970228] systemd[1]: Cannot add dependency job for unit dbus.socket, ignoring: Unit dbus.socket failed to load: No such file or directory. [ 0.972036] systemd[1]: Cannot add dependency job for unit display-manager.service, ignoring: Unit display-manager.service failed to load: No such file or directory. [ 0.974247] systemd[1]: Expecting device dev-ttyS0.device... [ 0.980127] systemd[1]: Starting Forward Password Requests to Wall Directory Watch. [ 0.981489] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ 0.982613] systemd[1]: Starting Remote File Systems (Pre). [ 0.992114] systemd[1]: Reached target Remote File Systems (Pre). [ 0.993050] systemd[1]: Starting Encrypted Volumes. [ 1.000145] systemd[1]: Reached target Encrypted Volumes. [ 1.001026] systemd[1]: Starting Dispatch Password Requests to Console Directory Watch. [ 1.002331] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ 1.003665] systemd[1]: Starting Arbitrary Executable File Formats File System Automount Point. [ 1.012116] systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point. [ 1.013825] systemd[1]: Starting Swap. [ 1.020134] systemd[1]: Reached target Swap. [ 1.020845] systemd[1]: Starting Root Slice. [ 1.028123] systemd[1]: Created slice Root Slice. [ 1.028915] systemd[1]: Starting User and Session Slice. [ 1.036148] systemd[1]: Created slice User and Session Slice. [ 1.037231] systemd[1]: Starting /dev/initctl Compatibility Named Pipe. [ 1.044192] systemd[1]: Listening on /dev/initctl Compatibility Named Pipe. [ 1.045272] systemd[1]: Starting Delayed Shutdown Socket. [ 1.052106] systemd[1]: Listening on Delayed Shutdown Socket. [ 1.053042] systemd[1]: Starting Journal Socket (/dev/log). [ 1.060127] systemd[1]: Listening on Journal Socket (/dev/log). [ 1.060981] systemd[1]: Starting udev Kernel Socket. [ 1.068187] systemd[1]: Listening on udev Kernel Socket. [ 1.069127] systemd[1]: Starting udev Control Socket. [ 1.076125] systemd[1]: Listening on udev Control Socket. [ 1.077170] systemd[1]: Starting Journal Socket. [ 1.084126] systemd[1]: Listening on Journal Socket. [ 1.084928] systemd[1]: Starting System Slice. [ 1.092156] systemd[1]: Created slice System Slice. [ 1.093251] systemd[1]: Started File System Check on Root Device. [ 1.094295] systemd[1]: Starting system-getty.slice. [ 1.104163] systemd[1]: Created slice system-getty.slice. [ 1.105061] systemd[1]: Starting system-serial\x2dgetty.slice. [ 1.112120] systemd[1]: Created slice system-serial\x2dgetty.slice. [ 1.113196] systemd[1]: Starting Increase datagram queue length... [ 1.121761] systemd[1]: Started Set Up Additional Binary Formats. [ 1.123056] systemd[1]: Mounting Debug File System... [ 1.132379] systemd[1]: Starting udev Coldplug all Devices... [ 1.140365] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 1.148339] systemd[1]: Mounting POSIX Message Queue File System... [ 1.154357] systemd[1]: Starting Load Kernel Modules... [ 1.164514] systemd[1]: Mounting Huge Pages File System... [ 1.172345] systemd[1]: Starting Slices. [ 1.180137] systemd[1]: Reached target Slices. [ 1.180781] systemd[1]: Starting Remount Root and Kernel File Systems... [ 1.192873] EXT4-fs (sda1): re-mounted. Opts: (null) [ 1.196122] systemd[1]: Mounted Huge Pages File System. [ 1.204210] systemd[1]: Mounted POSIX Message Queue File System. [ 1.212153] systemd[1]: Mounted Debug File System. [ 1.220194] systemd[1]: Started Increase datagram queue length. [ 1.228268] systemd[1]: Started Create list of required static device nodes for the current kernel. [ 1.236178] systemd[1]: Started Load Kernel Modules. [ 1.244109] systemd[1]: Started Remount Root and Kernel File Systems. [ 1.252125] systemd[1]: Started udev Coldplug all Devices. [ 1.260988] systemd[1]: Starting Various fixups to make systemd work better on Debian... [ 1.268466] systemd[1]: Starting Load/Save Random Seed... [ 1.276347] systemd[1]: Starting Apply Kernel Variables... [ 1.284367] systemd[1]: Mounted FUSE Control File System. [ 1.285266] systemd[1]: Mounted Configuration File System. [ 1.286066] systemd[1]: Starting Create Static Device Nodes in /dev... [ 1.292330] systemd[1]: Starting Syslog Socket. [ 1.300152] systemd[1]: Listening on Syslog Socket. [ 1.301174] systemd[1]: Starting Journal Service... [ 1.316160] systemd[1]: Started Journal Service. [ 1.365177] systemd-udevd[175]: starting version 215 [ 1.368087] tsc: Refined TSC clocksource calibration: 2300.003 MHz [ 1.431978] systemd-journald[173]: Received request to flush runtime journal from PID 1 [ 1.443420] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input2 [ 1.444571] ACPI: Power Button [PWRF] [ 1.445370] input: Sleep Button as /devices/LNXSYSTM:00/LNXSLPBN:00/input/input3 [ 1.447470] ACPI: Sleep Button [SLPF] [ 1.452538] piix4_smbus 0000:00:01.3: SMBus base address uninitialized - upgrade BIOS or use force_addr=0xaddr [ 1.490217] AVX2 version of gcm_enc/dec engaged. [ 1.503138] ppdev: user-space parallel port driver [ 1.505023] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 1.509085] alg: No test for crc32 (crc32-pclmul) [ 1.511110] intel_rapl: no valid rapl domains found in package 0 [ 1.664655] psmouse serio1: alps: Unknown ALPS touchpad: E7=10 00 64, EC=10 00 64 [ 2.098149] input: ImPS/2 Generic Wheel Mouse as /devices/platform/i8042/serio1/input/input4 [ 2.324606] random: nonblocking pool is initialized ** Model information sys_vendor: Google product_name: Google product_version: chassis_vendor: Google chassis_version: bios_vendor: Google bios_version: Google ** Loaded modules: crc32_pclmul ghash_clmulni_intel ppdev aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd evdev processor thermal_sys parport_pc psmouse parport i2c_piix4 serio_raw i2c_core button autofs4 ext4 crc16 mbcache jbd2 sg sd_mod crc_t10dif crct10dif_generic virtio_net virtio_scsi scsi_mod crct10dif_pclmul crct10dif_common crc32c_intel virtio_pci virtio_ring virtio ** PCI devices: not available ** USB devices: not available -- System Information: Debian Release: 8.2 APT prefers stable-updates APT policy: (500, 'stable-updates'), (500, 'stable') Architecture: amd64 (x86_64) Kernel: Linux 3.16.0-4-amd64 (SMP w/1 CPU core) Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) Versions of packages linux-image-3.16.0-4-amd64 depends on: ii debconf [debconf-2.0] 1.5.56 ii initramfs-tools [linux-initramfs-tool] 0.120 ii kmod 18-3 ii linux-base 3.5 Versions of packages linux-image-3.16.0-4-amd64 recommends: pn firmware-linux-free <none> pn irqbalance <none> Versions of packages linux-image-3.16.0-4-amd64 suggests: pn debian-kernel-handbook <none> ii grub-pc 2.02~beta2-22 pn linux-doc-3.16 <none> Versions of packages linux-image-3.16.0-4-amd64 is related to: pn firmware-atheros <none> pn firmware-bnx2 <none> pn firmware-bnx2x <none> pn firmware-brcm80211 <none> pn firmware-intelwimax <none> pn firmware-ipw2x00 <none> pn firmware-ivtv <none> pn firmware-iwlwifi <none> pn firmware-libertas <none> pn firmware-linux <none> pn firmware-linux-nonfree <none> pn firmware-myricom <none> pn firmware-netxen <none> pn firmware-qlogic <none> pn firmware-ralink <none> pn firmware-realtek <none> pn xen-hypervisor <none> -- debconf information: linux-image-3.16.0-4-amd64/postinst/depmod-error-initrd-3.16.0-4-amd64: false linux-image-3.16.0-4-amd64/postinst/mips-initrd-3.16.0-4-amd64: linux-image-3.16.0-4-amd64/prerm/removing-running-kernel-3.16.0-4-amd64: true
