On Thu, Nov 13, 2014 at 2:23 PM, Andy Lutomirski <l...@amacapital.net> wrote:
> On Thu, Nov 13, 2014 at 10:43 AM, Luck, Tony <tony.l...@intel.com> wrote:
>>> printk seems to work just fine in do_machine_check. Any chance you
>>> can instrument, for each cpu, all entries to do_machine_check, all
>>> calls to do_machine_check, all returns, and everything that tries to
>>> do memory_failure?
>>
>> I first added a printk() just for the cpu that calls do_machine_check()
>>
>> printk("MCE: regs = %p\n", regs);
>>
>> to see if something went wonky when jumping to the kernel stack.
>> But that printed the same value every time (same process is consuming
>> and recovering from errors). Maybe this took longer to hit the problem
>> case - I ran to 1500ish errors instead of just 400 in the previous two tests.
>> But not sure if that is a significant change.
>>
>> Then I added printk() for every entry/return on all cpus. This just locked
>> up on the third injection. Serial console looked to have stopped printing
>> after the first - so I put in bigger delays into my test program between
>> injection
>> and consumption, and before looping around for the next cycle to give
>> time for all the messages (4-socket HSW-EX ... there are a lot of cpus
>> printing messages). But now it is taking a lot longer to get through
>> injection/consumption iterations. At 226 now and counting.
>>
>>> Also, shouldn't there be a local_irq_enable before memory_failure and
>>> a local_irq_disable after it? It wouldn't surprise me if you've
>>> deadlocked somewhere. Lockdep could also have something interesting
>>> to say.
>> Added enable/disable.
>>
>>> should still be deliverable. Is it possible that we really need an
>>> IRET to unmask NMIs? This seems unlikely.)
>>
>> If that were the problem, wouldn't we fail on iteration 2, instead of
>> 400+ ?
>>
>> -Tony
>
> There could be a timer interrupt or something. But I agree, it seems
> implausible.
>
> Are you sure that this works in an unmodified kernel? The timeout
> code seems highly questionable to me. For example, there's this:
>
> if ((c->x86 > 6 || (c->x86 == 6 && c->x86_model >= 0xe)) &&
> cfg->monarch_timeout < 0)
> cfg->monarch_timeout = USEC_PER_SEC;
>
> which presumably determines monarch_timeout on your system and sets it
> to 1000000. But then there's this:
>
> #define SPINUNIT 100 /* 100ns */
>
> which smells like unit error to me. On top of that, it seems likely
> to me that the cpu could execute a loop iteration in much less than
> 100ns, since the only thing that should be anything other than an L1
> hit or a correctly predicted branch is the rmb(), which is lfence,
> which is probably just a few ns. So you have 10k iterations at, say,
> 10ns each, allowing about 100µs to synchronize, and if an SMI hits at
> an inopportune time, boom.
This theory is consistent with the very quick failure with an extra
printk on entry.
--Andy
>
> Also, rmb, seriously? I would understand smp_rmb() or cpu_relax() or
> even barrier(), but rmb() seems completely bogus if harmless.
>
> --Andy
>
> --
> Andy Lutomirski
> AMA Capital Management, LLC
--
Andy Lutomirski
AMA Capital Management, LLC
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
