This bug is missing log files that will aid in diagnosing the problem.
While running an Ubuntu kernel (not a mainline or third-party kernel)
please enter the following command in a terminal window:
apport-collect 1705349
and then change the status of the bug to 'Confirmed'.
If, due to the nature of the issue you have encountered, you are unable
to run this command, please add a comment stating that fact and change
the bug status to 'Confirmed'.
This change has been made by an automated script, maintained by the
Ubuntu Kernel Team.
** Changed in: linux (Ubuntu)
Status: New => Incomplete
** Tags added: trusty
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1705349
Title:
tgtd blocked with hitting hung_task_timeout_secs
Status in linux package in Ubuntu:
New
Bug description:
We see an issue where an IO thread of tgtd violates the hung task timeout and
vmstat reports >20% wait on IO while we are not seeing a issue with the
underlying hardware (iowait and svc time) inside IO stat.
We already limited the kernel parameters
vm.dirty_background_ratio=5
vm.dirty_ratio=10
to reduce the impact of massive dirty blocks hitting the storage which
is still based on spinning rust in a RAID 10 array.
This is based on kernel 3.13.0-96-generic with the deadline IO scheduler and
we got the following stack trace:
[25300644.048029] INFO: task tgtd:27827 blocked for more than 300 seconds.
[25300644.048065] Not tainted 3.13.0-96-generic #143-Ubuntu
[25300644.048100] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
[25300644.048166] tgtd D ffff88103fc13180 0 27827 1
0x00000080
[25300644.048168] ffff88019cc61c78 0000000000000086 ffff880125791800
0000000000013180
[25300644.048171] ffff88019cc61fd8 0000000000013180 ffff880125791800
ffff88103fc13a18
[25300644.048173] ffff88207ffef6e8 ffff88019cc61d00 0000000000000002
ffffffff81151e90
[25300644.048176] Call Trace:
[25300644.048178] [<ffffffff81151e90>] ? wait_on_page_read+0x60/0x60
[25300644.048190] [<ffffffff8172e71d>] io_schedule+0x9d/0x130
[25300644.048192] [<ffffffff81151e9e>] sleep_on_page+0xe/0x20
[25300644.048194] [<ffffffff8172ec98>] __wait_on_bit_lock+0x48/0xb0
[25300644.048196] [<ffffffff81151faa>] __lock_page+0x6a/0x70
[25300644.048198] [<ffffffff810adb10>] ? autoremove_wake_function+0x40/0x40
[25300644.048200] [<ffffffff8115d1c8>] write_cache_pages+0x2f8/0x4b0
[25300644.048202] [<ffffffff8115c740>] ? global_dirtyable_memory+0x50/0x50
[25300644.048205] [<ffffffff810dae48>] ? get_futex_key+0x1d8/0x2c0
[25300644.048207] [<ffffffff8107abfe>] ? send_signal+0x3e/0x80
[25300644.048209] [<ffffffff8115d3c0>] generic_writepages+0x40/0x60
[25300644.048212] [<ffffffff8115e58e>] do_writepages+0x1e/0x40
[25300644.048213] [<ffffffff81153559>] __filemap_fdatawrite_range+0x59/0x60
[25300644.048215] [<ffffffff8115365a>] filemap_write_and_wait_range+0x2a/0x70
[25300644.048217] [<ffffffff811fa79b>] blkdev_fsync+0x1b/0x50
[25300644.048219] [<ffffffff811f1841>] do_fsync+0x51/0x80
[25300644.048221] [<ffffffff811f1af3>] SyS_fdatasync+0x13/0x20
[25300644.048223] [<ffffffff8173aadd>] system_call_fastpath+0x1a/0x1f
UPTIME report
16:30:01 up 292 days, 22:34, 0 users, load average: 9.89, 8.58, 5.33
FREE report
total used free shared buffers cached
Mem: 131777272 116935780 14841492 828 110629740 1123368
-/+ buffers/cache: 5182672 126594600
Swap: 7999484 9004 7990480
VMSTAT report
procs -----------memory---------- ---swap-- -----io---- -system--
------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa
st
0 10 8 14493 108036 1097 0 0 7 227 0 0 0 0 100
0 0
0 10 8 14494 108036 1097 0 0 0 0 690 1149 0 0 78 22
0
0 10 8 14494 108036 1097 0 0 0 8 736 1442 0 0 78 22
0
IOSTAT report
Linux 3.13.0-96-generic (cinder18) 07/19/2017 _x86_64_ (32
CPU)
07/19/2017 04:30:03 PM
avg-cpu: %user %nice %system %iowait %steal %idle
0.35 0.00 0.08 0.07 0.00 99.50
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
sda 0.00 3.97 0.40 10.89 1.75 63.16 11.51
0.00 0.16 0.13 0.16 0.01 0.01
sdb 51.40 107.25 2.20 15.89 215.29 7198.82 819.79
0.07 3.89 3.26 3.97 0.60 1.08
sdc 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 0.11 0.11 0.00 0.11 0.00
dm-0 0.00 0.00 0.00 0.00 0.00 0.01 8.00
0.00 11.76 0.66 14.98 0.17 0.00
dm-1 0.00 0.00 0.01 14.86 0.19 63.13 8.52
0.00 0.15 3.10 0.15 0.00 0.01
dm-2 0.00 0.00 0.00 0.00 0.03 0.02 13.67
0.00 1.07 1.66 0.69 0.47 0.00
dm-18 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 3.51 3.51 4.00 3.51 0.00
dm-10 0.00 0.00 0.01 0.00 0.02 0.00 8.00
0.00 1.76 1.76 0.00 0.41 0.00
dm-28 0.00 0.00 0.00 0.02 0.00 0.06 8.00
0.00 1.39 5.76 1.20 0.06 0.00
dm-29 0.00 0.00 0.00 0.00 0.00 0.01 8.00
0.00 3.62 9.63 2.97 0.02 0.00
dm-30 0.00 0.00 0.00 0.01 0.01 0.03 8.00
0.00 2.99 2.84 3.03 0.03 0.00
dm-24 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 0.04 0.04 0.00 0.04 0.00
dm-25 0.00 0.00 0.00 0.25 0.00 0.99 8.00
0.00 6.27 3.23 6.27 0.00 0.00
dm-35 0.00 0.00 0.00 0.00 0.00 0.01 8.00
0.00 2.47 2.05 2.50 0.01 0.00
dm-36 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 1.98 2.69 1.71 0.09 0.00
dm-37 0.00 0.00 0.10 0.69 0.42 2.75 8.00
0.00 4.26 0.63 4.81 0.00 0.00
dm-38 0.00 0.00 0.00 0.01 0.01 0.85 188.30
0.00 2.40 4.21 1.99 0.89 0.00
dm-39 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 3.19 3.20 0.00 0.60 0.00
dm-40 0.00 0.00 0.02 0.00 0.08 0.00 8.00
0.00 0.20 0.20 0.00 0.01 0.00
dm-59 0.00 0.00 0.06 2.41 0.23 9.62 8.00
0.01 3.63 1.07 3.69 0.01 0.00
dm-60 0.00 0.00 0.00 0.02 0.01 0.09 8.00
0.00 3.72 16.50 1.75 0.16 0.00
dm-62 0.00 0.00 0.00 0.02 0.00 0.08 8.00
0.00 2.05 7.69 1.97 1.82 0.00
dm-63 0.00 0.00 0.03 1.29 0.10 5.16 8.00
0.01 3.96 10.41 3.84 0.20 0.03
dm-65 0.00 0.00 0.06 0.90 0.22 3.61 8.00
0.01 5.40 11.23 5.04 0.12 0.01
dm-104 0.00 0.00 0.00 0.00 0.00 0.01 8.00
0.00 2.86 8.67 2.20 0.02 0.00
dm-105 0.00 0.00 0.00 0.04 0.00 0.14 8.00
0.00 1.11 8.41 1.04 0.04 0.00
dm-106 0.00 0.00 0.00 0.00 0.00 0.01 8.00
0.00 3.85 7.98 3.41 0.02 0.00
dm-107 0.00 0.00 0.00 0.00 0.00 0.01 8.00
0.00 4.83 9.85 4.29 0.02 0.00
dm-108 0.00 0.00 0.00 0.03 0.00 0.12 8.00
0.00 1.95 15.02 1.79 0.08 0.00
dm-109 0.00 0.00 0.00 0.04 0.00 0.14 8.00
0.00 2.20 11.60 2.11 0.08 0.00
dm-116 0.00 0.00 0.00 0.01 0.00 0.04 8.00
0.00 9.27 3.84 9.34 0.01 0.00
dm-117 0.00 0.00 38.85 42.29 155.40 169.18 8.00
0.03 0.35 0.14 0.54 0.01 0.07
dm-118 0.00 0.00 0.00 25.91 0.00 103.63 8.00
0.09 3.31 10.21 3.31 0.00 0.00
dm-122 0.00 0.00 0.00 0.00 0.00 1.04 1006.91
0.00 0.29 2.25 0.26 0.16 0.00
dm-126 0.00 0.00 0.00 0.00 0.00 0.41 982.51
0.00 0.34 2.11 0.26 0.16 0.00
dm-130 0.00 0.00 0.00 0.00 0.00 2.07 1013.70
0.00 0.27 3.68 0.23 0.15 0.00
dm-134 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 2.61 2.61 0.00 0.44 0.00
dm-115 0.00 0.00 0.01 0.03 0.03 0.13 8.00
0.00 81.51 0.78 100.26 0.19 0.00
dm-119 0.00 0.00 0.79 1.82 3.15 7.28 8.00
0.16 60.81 1.02 86.69 0.05 0.01
dm-121 0.00 0.00 2.77 1.59 11.09 6.34 8.00
0.05 11.22 16.24 2.44 0.03 0.01
dm-149 0.00 0.00 0.00 0.01 0.00 0.03 8.00
0.00 36.92 12.62 38.03 0.02 0.00
dm-150 0.00 0.00 0.00 0.00 0.00 0.01 8.00
0.00 10.89 11.99 10.75 0.11 0.00
dm-151 0.00 0.00 0.00 0.01 0.00 0.04 8.00
0.00 78.01 12.63 80.68 0.08 0.00
dm-152 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 2.22 2.90 2.16 0.03 0.00
dm-153 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 71.05 11.97 77.41 1.03 0.00
dm-154 0.00 0.00 0.03 1.29 0.13 5.14 8.00
0.08 59.27 7.21 60.58 0.03 0.00
dm-55 0.00 0.00 4.23 0.87 16.93 3.49 8.00
0.01 1.71 1.68 1.86 0.01 0.00
dm-68 0.00 0.00 0.00 0.00 0.02 0.00 8.00
0.00 15.58 18.88 0.62 1.80 0.00
dm-96 0.00 0.00 0.00 4.94 0.00 19.74 8.00
0.02 3.18 6.69 3.18 0.00 0.00
dm-97 0.00 0.00 0.00 3.95 0.00 15.80 8.00
0.04 10.61 13.14 10.61 0.00 0.00
dm-98 0.00 0.00 0.00 0.08 0.00 0.32 8.00
0.00 51.66 1.12 51.67 0.01 0.00
dm-78 0.00 0.00 0.00 0.01 0.00 0.02 8.00
0.00 1.82 5.50 1.78 1.30 0.00
dm-79 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 3.78 32.34 2.20 0.15 0.00
dm-80 0.00 0.00 0.00 0.35 0.02 1.41 8.00
0.00 5.41 17.35 5.28 0.16 0.01
dm-81 0.00 0.00 0.00 0.00 0.00 0.02 8.00
0.00 7.98 8.75 7.90 0.03 0.00
dm-82 0.00 0.00 0.03 2.04 0.11 8.16 8.00
0.07 35.30 62.02 34.94 0.02 0.00
dm-83 0.00 0.00 0.60 4.38 2.39 17.52 8.00
0.04 8.01 0.38 9.05 0.00 0.00
dm-87 0.00 0.00 0.00 0.23 0.00 0.92 8.00
0.00 2.25 0.83 2.25 0.01 0.00
dm-20 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 1.75 1.75 0.00 1.75 0.00
dm-21 0.00 0.00 0.00 0.28 0.00 1.11 8.00
0.00 4.33 9.42 4.32 0.01 0.00
dm-26 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 10.76 10.76 0.00 10.76 0.00
dm-27 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 1.91 1.96 0.00 1.91 0.00
dm-75 0.00 0.00 0.00 0.16 0.00 0.65 8.00
0.00 3.74 4.58 3.73 0.14 0.00
dm-11 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 1.81 1.81 0.00 1.81 0.00
dm-49 0.00 0.00 0.00 0.23 0.00 0.90 8.00
0.00 4.34 11.38 4.34 0.01 0.00
dm-50 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 6.68 6.68 0.00 6.68 0.00
dm-52 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 2.61 2.63 0.00 2.61 0.00
dm-53 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 2.01 2.01 0.00 2.01 0.00
dm-61 0.00 0.00 0.00 0.03 0.00 0.12 8.00
0.00 2.20 1.83 2.20 0.02 0.00
dm-51 0.00 0.00 0.00 0.00 0.00 0.01 8.00
0.00 2.99 8.71 2.39 0.02 0.00
dm-54 0.00 0.00 0.00 0.01 0.01 0.03 8.00
0.00 3.42 3.13 3.47 0.02 0.00
dm-56 0.00 0.00 0.00 0.00 0.00 0.02 8.00
0.00 3.18 9.73 2.74 0.04 0.00
dm-72 0.00 0.00 0.01 0.07 0.03 0.29 8.00
0.00 12.51 0.40 13.57 0.00 0.00
dm-70 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 2.09 0.91 2.12 0.01 0.00
dm-71 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 2.46 3.10 2.43 0.02 0.00
dm-73 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 2.29 0.82 2.32 0.01 0.00
dm-47 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 1.17 1.17 0.00 1.17 0.00
dm-48 0.00 0.00 0.00 0.01 0.00 0.02 8.00
0.00 2.01 6.15 2.01 0.01 0.00
dm-88 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 4.75 4.76 0.00 4.75 0.00
dm-89 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 0.13 0.13 0.00 0.13 0.00
dm-90 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 3.96 3.97 0.00 3.96 0.00
dm-91 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 3.60 0.17 3.61 0.31 0.00
dm-92 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 0.54 0.54 0.00 0.54 0.00
dm-93 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 0.17 0.17 0.00 0.17 0.00
dm-3 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 0.31 0.31 0.00 0.31 0.00
dm-4 0.00 0.00 0.00 0.00 0.00 0.01 8.00
0.00 1.42 2.29 1.42 0.01 0.00
dm-6 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 0.13 0.13 0.00 0.13 0.00
dm-7 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 0.04 0.04 0.00 0.04 0.00
dm-8 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 0.15 0.15 0.00 0.15 0.00
dm-9 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 8.04 0.00 8.07 0.16 0.00
dm-17 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 0.29 0.29 0.00 0.29 0.00
dm-19 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 0.13 0.13 0.00 0.13 0.00
dm-5 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 0.48 0.48 0.00 0.48 0.00
dm-12 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 0.99 0.22 0.99 0.00 0.00
dm-13 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 0.32 0.32 0.00 0.32 0.00
dm-14 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 0.26 0.26 0.00 0.26 0.00
dm-15 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 0.33 0.33 0.00 0.33 0.00
dm-16 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 0.28 0.04 0.28 0.01 0.00
dm-22 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 0.30 0.30 0.00 0.30 0.00
dm-23 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 0.09 0.09 0.00 0.09 0.00
dm-31 0.00 0.00 0.00 0.00 0.00 0.00 8.00
0.00 0.32 0.31 4.00 0.32 0.00
dm-32 0.00 0.00 0.05 0.00 0.21 1.66 66.75
0.00 0.39 0.10 4.16 0.09 0.00
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1705349/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
