On (10/18/16 19:07), Peter Zijlstra wrote:
> > This patch set extends a lock-less NMI per-cpu buffers idea to
> > handle recursive printk() calls. The basic mechanism is pretty much the
> > same -- at the beginning of a deadlock-prone section we switch to lock-less
> > printk callback, and return back to a default printk implementation at the
> > end; the messages are getting flushed to a logbuf buffer from a safer
> > context.
>
> So I think you're not taking this far enough. You've also missed an
> entire class of deadlocks.
yes. this patch set addresses only printk() recursion. printk deadlocks
in general are much more complicated and basically we don't even know
how many locks we are about to take every time we call printk() (there
may be serial console locks, etc we never know in advance).
> The first is that you still keep the logbuf. Having this global
> serialized thing is a source of fail. It would be much better to only
> keep per cpu stuff. _OR_ at the very least make the logbuf itself
> lockfree. So generate the printk entry local (so we know its size) then
> atomically reserve the logbuf entry and copy it over.
logbuf spin_lock is not the only problematic lock. we _at least_
have console sem spin_lock and rq spin_lock:
https://marc.info/?l=linux-kernel&m=145406977822244
https://marc.info/?l=linux-kernel&m=146853438804534&w=2
> The entire class of deadlocks you've missed is that console->write() is
> a piece of crap too ;-) Even the bog standard 8250 serial console driver
> can do wakeups.
yes. well, I didn't exactly "miss" it. I just don't have any working
solution for those locks yet. so things like
SyS_ioctl
do_vfs_ioctl
tty_ioctl
n_tty_ioctl
tty_mode_ioctl
set_termios
tty_set_termios
uart_set_termios
uart_change_speed
FOO_serial_set_termios
spin_lock_irqsave(&port->lock) // lock the output port
....
!! WARN() or pr_err() or printk()
vprintk_emit()
/* console_trylock() */
console_unlock()
call_console_drivers()
FOO_write()
spin_lock_irqsave(&port->lock) // already locked
are out of this patch set's scope. another class of deadlocks is when we
call printk under one of the locks that printk() can take later in order
to print the message.
so in this particular patch set I make "printk() -> $FOO -> printk()" less
damaging. which is just a first step.
we are also looking at things like might_printk()
https://marc.info/?l=linux-kernel&m=147607261116163
and even extending the lockdep
https://marc.info/?l=linux-kernel&m=147617830511381
not to miss out a DEFERRED_WARN patch set...
//hm, I can't find it online
Subject: [RFC 0/5] printk: Implement WARN_*DEFERRED()
Message-Id: <1474992135-14777-1-git-send-email-pmla...@suse.com>
> See for example:
>
> https://marc.info/?l=linux-kernel&m=132446649823347&w=2
interesting idea.
> I've entirely given up on printk(), I'll post the 3 patches I carry to
> make things work for me.
yes, printk is surprisingly fragile and, some would _probably_
even say, broken.
a) we have numerous scenarios when it can deadlock
b) we have numerous scenarios when it can soft-lockup/hard-lockup the system
c) we don't have a bullet proof panic printk. zap_locks() resets
logbuf and console_sem, but there may be serial console locks/etc.
-ss
