On Thu, Jan 10, 2013 at 5:39 PM, Liu, Chuansheng <chuansheng....@intel.com> wrote: > > >> -----Original Message----- >> From: Colin Cross [mailto:ccr...@android.com] >> Sent: Thursday, January 10, 2013 9:58 AM >> To: linux-kernel@vger.kernel.org >> Cc: Andrew Morton; Don Zickus; Ingo Molnar; Thomas Gleixner; Liu, >> Chuansheng; linux-arm-ker...@lists.infradead.org; Colin Cross >> Subject: [PATCH] hardlockup: detect hard lockups without NMIs using >> secondary cpus >> >> Emulate NMIs on systems where they are not available by using timer >> interrupts on other cpus. Each cpu will use its softlockup hrtimer >> to check that the next cpu is processing hrtimer interrupts by >> verifying that a counter is increasing. >> >> This patch is useful on systems where the hardlockup detector is not >> available due to a lack of NMIs, for example most ARM SoCs. >> Without this patch any cpu stuck with interrupts disabled can >> cause a hardware watchdog reset with no debugging information, >> but with this patch the kernel can detect the lockup and panic, >> which can result in useful debugging info. >> >> Signed-off-by: Colin Cross <ccr...@android.com> >> +static void watchdog_check_hardlockup_other_cpu(void) >> +{ >> + int cpu; >> + cpumask_t cpus = watchdog_cpus; >> + >> + /* >> + * Test for hardlockups every 3 samples. The sample period is >> + * watchdog_thresh * 2 / 5, so 3 samples gets us back to slightly over >> + * watchdog_thresh (over by 20%). >> + */ >> + if (__this_cpu_read(hrtimer_interrupts) % 3 != 0) >> + return; >> + >> + /* check for a hardlockup on the next cpu */ >> + cpu = cpumask_next(smp_processor_id(), &cpus); >> + if (cpu >= nr_cpu_ids) >> + cpu = cpumask_first(&cpus); >> + if (cpu == smp_processor_id()) >> + return; >> + >> + smp_rmb(); >> + >> + if (per_cpu(watchdog_nmi_touch, cpu) == true) { >> + per_cpu(watchdog_nmi_touch, cpu) = false; >> + return; >> + } >> + >> + if (is_hardlockup_other_cpu(cpu)) { >> + /* only warn once */ > One possible case for new hotplug CPU that false hardlockup case. > 1/ Assume CPU1, CPU2 are online, CPU3 is being hotplug: > CPU3: CPU2: > watchdog_nmi_enable() > per_cpu(watchdog_nmi_touch, cpu) = true; > cpumask_set_cpu(cpu, &watchdog_cpus); > > watchdog_check_hardlockup_other_cpu() > > per_cpu(watchdog_nmi_touch, cpu) = false; == > Here cpu is CPU3 > > 2/ Before CPU3's first hrtimer interrupt coming, CPU2 is been offlined. > Then CPU1's next CPU is CPU3. But we can not use CPU3's watchdog_nmi_touch > to defense > false CPU3 hardlock more. When CPU1's hrtimer interrupt coming, it is > possible report CPU3 > false hard lockup. > > Is it the case?
Yes, this is the same race condition I pointed out in reply to Don Zickus earlier in the thread. I think the easiest solution is to set per_cpu(watchdog_nmi_touch, next_cpu) = true during cpu offlining. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
