Test-case snippet on original kernel (3.13.0-170.220) [ 330.980173] cpu 5/pid 2126/task ONE :: Stage 4. T1 wake up T2... it may BUG! <...> [ 331.975122] cpu 6/pid 2125/task TWO :: Stage 4. T2 sleep for 1s... BUG afterward? [ 332.980237] BUG: unable to handle kernel paging request at ffffeaffffffffc0 [ 332.984081] IP: [<ffffffff811b3e81>] wait_migrate_huge_page+0x51/0x70 [ 332.984081] PGD 0 [ 332.984081] Oops: 0000 [#1] SMP [ 332.984081] Modules linked in: kmod(OX) <...> [ 332.984081] CPU: 6 PID: 2125 Comm: test Tainted: G OX 3.13.0-170-generic #220-Ubuntu <...> [ 332.984081] RIP: 0010:[<ffffffff811b3e81>] [<ffffffff811b3e81>] wait_migrate_huge_page+0x51/0x70 <...> [ 332.984081] Call Trace: [ 332.984081] [<ffffffff811b6f3c>] ? do_huge_pmd_numa_page+0x32c/0x390 [ 332.984081] [<ffffffff811839e2>] handle_mm_fault+0x8f2/0xfb0 [ 332.984081] [<ffffffff81748963>] __do_page_fault+0x183/0x570 <...> [ 332.984081] RIP [<ffffffff811b3e81>] wait_migrate_huge_page+0x51/0x70 [ 332.984081] RSP <ffff8800376c5d00> [ 332.984081] CR2: ffffeaffffffffc0 [ 332.984081] ---[ end trace dcd414863870ead7 ]---
-- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1813018 Title: Kernel Oops - unable to handle kernel paging request; RIP is at wait_migrate_huge_page+0x51/0x70 Status in linux package in Ubuntu: Invalid Status in linux source package in Trusty: In Progress Bug description: [Impact] * Users on NUMA systems (mostly servers) with NUMA balancing enabled (which is by default) might hit a crash/BUG() on a race condition if two simultaneous page faults of the same transparent hugepage go into the path for migration to another NUMA node. * The symptom is BUG() for 0xffffeaffffffffc0, which happens if the PMD is set to zero/NULL. BUG: unable to handle kernel paging request at ffffeaffffffffc0 IP: [<ffffffff811b3d31>] wait_migrate_huge_page+0x51/0x70 * NUMA balancing periodically unmaps pages so to force page faults to occur, and later find out using page faults where the NUMA memory access is coming from - if it's often from other NUMA node, it attempts to migrate the page contents to the other NUMA node (for more local access.) * The race condition is related to these 3 functions in the pagefault handling of transparent hugepages: do_huge_pmd_numa_page() -> wait_migrate_huge_page() do_huge_pmd_numa_page() -> migrate_misplaced_transhuge_page() The first task to hit the pagefault / migration path calls migrate_misplaced_transhuge_page(), which does: - ptl = pmd_lock(mm, pmd) // calls spin_lock(ptl) - pmdp_clear_flush(..., pmd); // set PMD to zero - set_pmd_at(..., pmd, ...); // set PMD to non-zero - spin_unlock(ptl); The second task to hit that path finds that the page is already being migrated (page is locked) and waits for that finish (ie, until page is unlocked), doing: - spin_unlock(ptl) - page = pmd_page(*pmd) - wait_on_page_locked(page) *BUT* it reads the PMD value *after* unlocking stuff. So, if the tasks/CPUs manage to run in the sequence below the PMD can be set to zero/NULL by first task and read by second task before it's set to non-NULL. Thus the second task miscalculates the page pointer, from PMD and hit BUG for address 0xffffeaffffffffc0. https://pastebin.ubuntu.com/p/pjMNxkJfJy/ * The fix just moves pmd_page() before spin_unlock(), and now the change perfomed in the other function (done within the spin_lock()/spin_unlock() region) can no longer run concurrently with this PMD read. * So, when the other function releases the spin lock, the PMD has already been set to non-NULL/valid PMD, and wait_on_page_locked() receives a valid address. * Fix commit 5d833062139d ("mm: numa: do not dereference pmd outside of the lock during NUMA hinting fault") [1] * Applied in v4.0 upstream; only Trusty/3.13 needs it. $ git describe --contains 5d833062139d v4.0-rc1~98^2~103 [Test Case] * A synthetic reproducer is available for this bug. It consists of two parts: 1) userspace program to allocate one transparent huge page and spawn two threads that write to it periodically - based on the period used by numa balancing to unmap pages - so it trigger page faults simultaneously on the two threads. 2) kernel module that inserts kprobes on several points in the involved functions, in order to synchronize the two threads's timing/progress in the way that reproduces the problem. Then the user forces the NUMA migration to occur by changing the affinity of userspace program to another NUMA node, and the kernel module/kprobes force the problem to occur. * Steps done with kprobes: 1) Task 1 and Task 2 join at do_huge_pmd_numa_page(). 2) Task 1 moves on to migrate_misplaced_transhuge_page(), signals Task 2 to move on, and waits for Task 2 signal. 3) Task 2 moves on to point in 'if pmd_trans_migrating()' just after spin_unlock() [before pmd_page() originally], signals Task 1 to move on, and waits for Task 1 signal. 4) Task 1 moves on until it sets the PMD to zero/NULL, signals Task 2 to move on, and waits for Task 2 signal. 5) Task 2 moves on to read the PMD and calculate its page, and attempt to wait on it to be unlocked. 5.1) BUG() on the original kernel / without the patch. 5.2) Go-on on the modified kernel / with the patch. [Regression Potential] * Low. The fix is both really targeted at this particular problem/race condition (see last paragraph in commit msg) and is sufficiently contained to fix/change _just_ that (despite the long commit message for intro to a series.) * The only change is actually move pmd_page() before spin_unlock() (rest is removing a wrapper function), and wait_on_page_locked() remains after spin_unlock(). * The pmd_page() code path does not acquire any locks, so not susceptible to a dead-lock or interplay with another lock. [Original Description] [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5d833062139d Kernel oops occurs randomly every now and then, seemingly when running memory-intensive processes (so far, it happened to me when using bowtie2 or STAR). Running Ubuntu 14.04 LTS on AWS EC2 instances (m4.* and c4.* family classes). After the error occurs, the server stays accessible through SSH, but the commands w, htop, ps (and maybe others) seem to hang, while commands like ls, cd, top and others keep working. Whatever process was running and (probably) caused the crash seems to go into a sleeping mode. Rebooting (sudo reboot) makes the instance refuse all connections (more than an hour after initiating the reboot). Stopping the (AWS EC2) instance and starting again makes the instance function normally again. Restarting the task that was running when the instance crashed on the newly (re)started instance usually works with no more problems. --- AlsaDevices: total 0 crw-rw---- 1 root audio 116, 1 Jan 23 12:49 seq crw-rw---- 1 root audio 116, 33 Jan 23 12:49 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.29 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory DistroRelease: Ubuntu 14.04 Ec2AMI: ami-4473183b Ec2AMIManifest: (unknown) Ec2AvailabilityZone: us-east-1c Ec2InstanceType: m4.16xlarge Ec2Kernel: unavailable Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 MachineType: Xen HVM domU Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=xterm PATH=(custom, no user) XDG_RUNTIME_DIR=<set> LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-164-generic root=UUID=d4f2aafc-946a-4514-930d-4c45e676f198 ro console=tty1 console=ttyS0 ProcVersionSignature: Ubuntu 3.13.0-164.214-generic 3.13.11-ckt39 RelatedPackageVersions: linux-restricted-modules-3.13.0-164-generic N/A linux-backports-modules-3.13.0-164-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-164-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: sudo WifiSyslog: _MarkForUpload: True dmi.bios.date: 08/24/2006 dmi.bios.vendor: Xen dmi.bios.version: 4.2.amazon dmi.chassis.type: 1 dmi.chassis.vendor: Xen dmi.modalias: dmi:bvnXen:bvr4.2.amazon:bd08/24/2006:svnXen:pnHVMdomU:pvr4.2.amazon:cvnXen:ct1:cvr: dmi.product.name: HVM domU dmi.product.version: 4.2.amazon dmi.sys.vendor: Xen To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1813018/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
