On Wed, 21 Aug 2013 10:08:31 +0200 Jan Kara <j...@suse.cz> wrote: > A CPU can be caught in console_unlock() for a long time (tens of seconds > are reported by our customers) when other CPUs are using printk heavily > and serial console makes printing slow. Despite serial console drivers > are calling touch_nmi_watchdog() this triggers softlockup warnings > because interrupts are disabled for the whole time console_unlock() runs > (e.g. vprintk() calls console_unlock() with interrupts disabled). Thus > IPIs cannot be processed and other CPUs get stuck spinning in calls like > smp_call_function_many(). Also RCU eventually starts reporting lockups. > > In my artifical testing I can also easily trigger a situation when disk > disappears from the system apparently because commands to / from it > could not be delivered for long enough. This is why just silencing > watchdogs isn't a reliable solution to the problem and we simply have to > avoid spending too long in console_unlock() with interrupts disabled. > > The solution this patch works toward is to postpone printing to a later > moment / different CPU when we already printed over 1000 characters in > current console_unlock() invocation. This is a crude heuristic but > measuring time we spent printing doesn't seem to be really viable - we > cannot rely on high resolution time being available and with interrupts > disabled jiffies are not updated. The value 1000 was chosen so that > things are still bearable with 9600 baud serial console and OTOH it > shouldn't cause offloading of printing in common cases. >
BTW, you can also test by running: # cd /sys/kernel/debug/tracing # echo function > current_tracer # sleep 1 # echo z > /proc/sysrq-trigger > Signed-off-by: Jan Kara <j...@suse.cz> Reviewed-by: Steven Rostedt <rost...@goodmis.org> -- Steve -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
