On Tue, 2018-12-25 at 17:47 +0200, Ido Schimmel wrote:
> On Thu, Dec 20, 2018 at 12:56:25PM -0800, Jeff Kirsher wrote:
> > This change is based off of the work and suggestion of Jan Jablonsky
> > <jan.jablon...@thalesgroup.com>.
> >
> > The Watchdog workqueue in igb driver is scheduled every 2s for each
> > network interface. That includes updating a statistics protected by
> > spinlock. Function igb_update_stats in this case will be protected
> > against preemption. According to number of a statistics registers
> > (cca 60), processing this function might cause additional cpu load
> > on CPU0.
> >
> > In case of statistics spinlock may be replaced with mutex, which
> > reduce latency on CPU0.
> >
> > CC: Bernhard Kaindl <bernhard.kai...@thalesgroup.com>
> > CC: Jan Jablonsky <jan.jablon...@thalesgroup.com>
> > Signed-off-by: Jeff Kirsher <jeffrey.t.kirs...@intel.com>
> > Tested-by: Aaron Brown <aaron.f.br...@intel.com>
> > Signed-off-by: Jeff Kirsher <jeffrey.t.kirs...@intel.com>
>
> Hi,
>
> I'm hitting following issue with this commit:
> https://lists.01.org/pipermail/lkp/2018-September/009139.html
Hitting what looks like the same thing with linus tip from today and fedora
rawhide config.
There is some rcu splat that might help.. (ignore the drm,it87 bits)
.....
[ 17.150603] WARNING: suspicious RCU usage
[ 17.150605] 5.0.0-rc1+ #3 Not tainted
[ 17.150606] -----------------------------
[ 17.150607] include/linux/rcupdate.h:281 Illegal context switch in RCU
read-side critical section!
[ 17.150608]
other info that might help us debug this:
[ 17.150609]
rcu_scheduler_active = 2, debug_locks = 1
[ 17.150610] 2 locks held by sadc/819:
[ 17.150610] #0: 000000006764297e (&p->lock){+.+.}, at: seq_read+0x41/0x430
[ 17.150616] #1: 00000000488cf3bc (rcu_read_lock){....}, at:
dev_seq_start+0x5/0x120
[ 17.150621]
stack backtrace:
[ 17.150623] CPU: 3 PID: 819 Comm: sadc Not tainted 5.0.0-rc1+ #3
[ 17.150624] Hardware name: Gigabyte Technology Co., Ltd. X470 AORUS ULTRA
GAMING/X470 AORUS ULTRA GAMING-CF, BIOS F4 12/24/2018
[ 17.150624] Call Trace:
[ 17.150628] dump_stack+0x85/0xc0
[ 17.150632] ___might_sleep+0x100/0x180
[ 17.150635] __mutex_lock+0x55/0x9a0
[ 17.241240] [drm] VCE initialized successfully.
[ 17.244179] ? seq_vprintf+0x30/0x50
[ 17.244181] ? seq_printf+0x53/0x70
[ 17.244185] ? __lock_acquire+0x279/0x1650
[ 17.244193] ? igb_get_stats64+0x29/0x80 [igb]
[ 17.287914] igb_get_stats64+0x29/0x80 [igb]
[ 17.287920] dev_get_stats+0x5b/0xc0
[ 17.298714] dev_seq_printf_stats+0x32/0xe0
[ 17.298721] dev_seq_show+0x10/0x30
[ 17.307934] seq_read+0x2fd/0x430
[ 17.307940] proc_reg_read+0x39/0x60
[ 17.317677] __vfs_read+0x36/0x1a0
[ 17.317683] vfs_read+0x9f/0x160
[ 17.317686] ksys_read+0x52/0xc0
[ 17.330601] do_syscall_64+0x60/0x1f0
[ 17.330604] entry_SYSCALL_64_after_hwframe+0x49/0xbe
[ 17.330606] RIP: 0033:0x7fcce3c5e1b5
[ 17.330608] Code: fe ff ff 50 48 8d 3d 9a d3 09 00 e8 75 00 02 00 0f 1f 44
00 00 f3 0f 1e fa 48 8d 05 65 55 0d 00 8b 00 85 c0 75 0f 31 c0 0f 05 <48> 3d 00
f0 ff ff 77 53 c3 66 90 41 54 49 89 d4 55 48 89 f5 53 89
[ 17.330609] RSP: 002b:00007ffdd6d957e8 EFLAGS: 00000246 ORIG_RAX:
0000000000000000
[ 17.375503] RAX: ffffffffffffffda RBX: 00000000012dd2a0 RCX: 00007fcce3c5e1b5
[ 17.375504] RDX: 0000000000000400 RSI: 00000000012d89e0 RDI: 0000000000000003
[ 17.375505] RBP: 0000000000000d68 R08: 0000000000000001 R09: 0000000000000000
[ 17.375506] R10: 00007fcce3a29740 R11: 0000000000000246 R12: 00007fcce3d2a740
[ 17.375507] R13: 00007fcce3d2b340 R14: 000000000000007f R15: 00000000012dd2a0
[ 17.375577] BUG: sleeping function called from invalid context at
kernel/locking/mutex.c:908
[ 17.428833] in_atomic(): 1, irqs_disabled(): 0, pid: 819, name: sadc
[ 17.428834] 2 locks held by sadc/819:
[ 17.428835] #0: 000000006764297e (&p->lock){+.+.}, at: seq_read+0x41/0x430
[ 17.428841] #1: 00000000488cf3bc (rcu_read_lock){....}, at:
dev_seq_start+0x5/0x120
[ 17.428847] CPU: 3 PID: 819 Comm: sadc Not tainted 5.0.0-rc1+ #3
[ 17.428848] Hardware name: Gigabyte Technology Co., Ltd. X470 AORUS ULTRA
GAMING/X470 AORUS ULTRA GAMING-CF, BIOS F4 12/24/2018
[ 17.428849] Call Trace:
[ 17.428853] dump_stack+0x85/0xc0
[ 17.428857] ___might_sleep.cold.73+0xac/0xbc
[ 17.429616] it87: Found IT8733E chip at 0xa60, revision 3
[ 17.429704] it87: Beeping is supported
[ 17.506837] __mutex_lock+0x55/0x9a0
[ 17.506842] ? seq_vprintf+0x30/0x50
[ 17.516925] ? seq_printf+0x53/0x70
[ 17.516929] ? __lock_acquire+0x279/0x1650
[ 17.516937] ? igb_get_stats64+0x29/0x80 [igb]
[ 17.533393] igb_get_stats64+0x29/0x80 [igb]
[ 17.533397] dev_get_stats+0x5b/0xc0
[ 17.533401] dev_seq_printf_stats+0x32/0xe0
[ 17.549848] dev_seq_show+0x10/0x30
[ 17.549850] seq_read+0x2fd/0x430
[ 17.559502] proc_reg_read+0x39/0x60
[ 17.559505] __vfs_read+0x36/0x1a0
[ 17.559510] vfs_read+0x9f/0x160
[ 17.559513] ksys_read+0x52/0xc0
[ 17.559517] do_syscall_64+0x60/0x1f0
[ 17.559519] entry_SYSCALL_64_after_hwframe+0x49/0xbe
[ 17.559521] RIP: 0033:0x7fcce3c5e1b5
[ 17.559523] Code: fe ff ff 50 48 8d 3d 9a d3 09 00 e8 75 00 02 00 0f 1f 44
00 00 f3 0f 1e fa 48 8d 05 65 55 0d 00 8b 00 85 c0 75 0f 31 c0 0f 05 <48> 3d 00
f0 ff ff 77 53 c3 66 90 41 54 49 89 d4 55 48 89 f5 53 89
[ 17.559524] RSP: 002b:00007ffdd6d957e8 EFLAGS: 00000246 ORIG_RAX:
0000000000000000
[ 17.559526] RAX: ffffffffffffffda RBX: 00000000012dd2a0 RCX: 00007fcce3c5e1b5
[ 17.559527] RDX: 0000000000000400 RSI: 00000000012d89e0 RDI: 0000000000000003
[ 17.559528] RBP: 0000000000000d68 R08: 0000000000000001 R09: 0000000000000000
[ 17.559528] R10: 00007fcce3a29740 R11: 0000000000000246 R12: 00007fcce3d2a740
[ 17.559529] R13: 00007fcce3d2b340 R14: 000000000000007f R15: 00000000012dd2a0
.....
