With deduplication disabled, it eventually gets further with the backup, but still stalls, after leaving it, overnight:
[12084.274242] INFO: task z_wr_iss_h:2157 blocked for more than 120 seconds. [12084.281171] Tainted: P C OE 5.4.0-1018-raspi #20-Ubuntu [12084.288126] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [12084.296127] z_wr_iss_h D 0 2157 2 0x00000028 [12084.296142] Call trace: [12084.296163] __switch_to+0x104/0x170 [12084.296174] __schedule+0x30c/0x7c0 [12084.296183] schedule+0x3c/0xb8 [12084.296191] io_schedule+0x20/0x58 [12084.296204] rq_qos_wait+0x100/0x178 [12084.296213] wbt_wait+0xb4/0xf0 [12084.296223] __rq_qos_throttle+0x38/0x50 [12084.296235] blk_mq_make_request+0x128/0x610 [12084.296247] generic_make_request+0xb4/0x2d8 [12084.296258] submit_bio+0x48/0x218 [12084.296496] vdev_disk_io_start+0x670/0x9f8 [zfs] [12084.296716] zio_vdev_io_start+0xdc/0x2b8 [zfs] [12084.296935] zio_nowait+0xd4/0x170 [zfs] [12084.297151] vdev_mirror_io_start+0xa8/0x1b0 [zfs] [12084.297369] zio_vdev_io_start+0x248/0x2b8 [zfs] [12084.297586] zio_execute+0xac/0x110 [zfs] [12084.297626] taskq_thread+0x2f8/0x570 [spl] [12084.297639] kthread+0xfc/0x128 [12084.297650] ret_from_fork+0x10/0x1c [12084.297678] INFO: task txg_quiesce:3077 blocked for more than 120 seconds. [12084.304712] Tainted: P C OE 5.4.0-1018-raspi #20-Ubuntu [12084.311641] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [12084.319632] txg_quiesce D 0 3077 2 0x00000028 [12084.319646] Call trace: [12084.319668] __switch_to+0x104/0x170 [12084.319679] __schedule+0x30c/0x7c0 [12084.319691] schedule+0x3c/0xb8 [12084.319731] cv_wait_common+0x188/0x1b0 [spl] [12084.319767] __cv_wait+0x30/0x40 [spl] [12084.319990] txg_quiesce_thread+0x27c/0x380 [zfs] [12084.320028] thread_generic_wrapper+0x74/0xa0 [spl] [12084.320050] kthread+0xfc/0x128 [12084.320062] ret_from_fork+0x10/0x1c [12084.320133] INFO: task borg:13631 blocked for more than 120 seconds. [12084.326632] Tainted: P C OE 5.4.0-1018-raspi #20-Ubuntu [12084.333562] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [12084.341550] borg D 0 13631 13630 0x00000000 [12084.341564] Call trace: [12084.341578] __switch_to+0x104/0x170 [12084.341588] __schedule+0x30c/0x7c0 [12084.341596] schedule+0x3c/0xb8 [12084.341635] cv_wait_common+0x188/0x1b0 [spl] [12084.341670] __cv_wait+0x30/0x40 [spl] [12084.341895] zil_commit_impl+0x234/0xd30 [zfs] [12084.342112] zil_commit+0x48/0x70 [zfs] [12084.342331] zfs_write+0xa3c/0xb90 [zfs] [12084.342588] zpl_write_common_iovec+0xac/0x120 [zfs] [12084.342817] zpl_iter_write+0xe4/0x150 [zfs] [12084.342830] new_sync_write+0x100/0x1a8 [12084.342839] __vfs_write+0x74/0x90 [12084.342855] vfs_write+0xe4/0x1c8 [12084.342865] ksys_write+0x78/0x100 [12084.342874] __arm64_sys_write+0x24/0x30 [12084.342884] el0_svc_common.constprop.0+0x80/0x218 [12084.342892] el0_svc_handler+0x34/0xa0 [12084.342902] el0_svc+0x10/0x2cc [12084.342927] INFO: task z_wr_iss_h:114156 blocked for more than 120 seconds. [12084.350038] Tainted: P C OE 5.4.0-1018-raspi #20-Ubuntu [12084.356966] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [12084.364950] z_wr_iss_h D 0 114156 2 0x00000028 [12084.364964] Call trace: [12084.364981] __switch_to+0x104/0x170 [12084.364990] __schedule+0x30c/0x7c0 [12084.364998] schedule+0x3c/0xb8 [12084.365006] io_schedule+0x20/0x58 [12084.365019] rq_qos_wait+0x100/0x178 [12084.365027] wbt_wait+0xb4/0xf0 [12084.365037] __rq_qos_throttle+0x38/0x50 [12084.365050] blk_mq_make_request+0x128/0x610 [12084.365061] generic_make_request+0xb4/0x2d8 [12084.365072] submit_bio+0x48/0x218 [12084.365297] vdev_disk_io_start+0x670/0x9f8 [zfs] [12084.365516] zio_vdev_io_start+0xdc/0x2b8 [zfs] [12084.365733] zio_nowait+0xd4/0x170 [zfs] [12084.365949] vdev_mirror_io_start+0xa8/0x1b0 [zfs] [12084.366167] zio_vdev_io_start+0x248/0x2b8 [zfs] [12084.366427] zio_execute+0xac/0x110 [zfs] [12084.366476] taskq_thread+0x2f8/0x570 [spl] [12084.366491] kthread+0xfc/0x128 [12084.366502] ret_from_fork+0x10/0x1c [12084.366514] INFO: task z_wr_iss_h:114161 blocked for more than 120 seconds. [12084.373624] Tainted: P C OE 5.4.0-1018-raspi #20-Ubuntu [12084.380553] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [12084.388541] z_wr_iss_h D 0 114161 2 0x00000028 [12084.388555] Call trace: [12084.388570] __switch_to+0x104/0x170 [12084.388580] __schedule+0x30c/0x7c0 [12084.388588] schedule+0x3c/0xb8 [12084.388596] io_schedule+0x20/0x58 [12084.388608] rq_qos_wait+0x100/0x178 [12084.388616] wbt_wait+0xb4/0xf0 [12084.388627] __rq_qos_throttle+0x38/0x50 [12084.388638] blk_mq_make_request+0x128/0x610 [12084.388650] generic_make_request+0xb4/0x2d8 [12084.388660] submit_bio+0x48/0x218 [12084.388884] vdev_disk_io_start+0x670/0x9f8 [zfs] [12084.389113] zio_vdev_io_start+0xdc/0x2b8 [zfs] [12084.389332] zio_nowait+0xd4/0x170 [zfs] [12084.389548] vdev_mirror_io_start+0xa8/0x1b0 [zfs] [12084.389765] zio_vdev_io_start+0x248/0x2b8 [zfs] [12084.389982] zio_execute+0xac/0x110 [zfs] [12084.390020] taskq_thread+0x2f8/0x570 [spl] [12084.390033] kthread+0xfc/0x128 [12084.390043] ret_from_fork+0x10/0x1c [12084.390136] INFO: task z_wr_iss_h:114162 blocked for more than 120 seconds. [12084.397239] Tainted: P C OE 5.4.0-1018-raspi #20-Ubuntu [12084.404137] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [12084.412094] z_wr_iss_h D 0 114162 2 0x00000028 [12084.412100] Call trace: [12084.412109] __switch_to+0x104/0x170 [12084.412114] __schedule+0x30c/0x7c0 [12084.412118] schedule+0x3c/0xb8 [12084.412122] io_schedule+0x20/0x58 [12084.412127] rq_qos_wait+0x100/0x178 [12084.412130] wbt_wait+0xb4/0xf0 [12084.412135] __rq_qos_throttle+0x38/0x50 [12084.412140] blk_mq_make_request+0x128/0x610 [12084.412145] generic_make_request+0xb4/0x2d8 [12084.412149] submit_bio+0x48/0x218 [12084.412250] vdev_disk_io_start+0x670/0x9f8 [zfs] [12084.412340] zio_vdev_io_start+0xdc/0x2b8 [zfs] [12084.412435] zio_nowait+0xd4/0x170 [zfs] [12084.412521] vdev_mirror_io_start+0xa8/0x1b0 [zfs] [12084.412612] zio_vdev_io_start+0x248/0x2b8 [zfs] [12084.412699] zio_execute+0xac/0x110 [zfs] [12084.412718] taskq_thread+0x2f8/0x570 [spl] [12084.412725] kthread+0xfc/0x128 [12084.412729] ret_from_fork+0x10/0x1c [12084.412734] INFO: task z_wr_iss_h:114163 blocked for more than 120 seconds. [12084.419816] Tainted: P C OE 5.4.0-1018-raspi #20-Ubuntu [12084.426713] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [12084.434673] z_wr_iss_h D 0 114163 2 0x00000028 [12084.434679] Call trace: [12084.434687] __switch_to+0x104/0x170 [12084.434692] __schedule+0x30c/0x7c0 [12084.434695] schedule+0x3c/0xb8 [12084.434698] io_schedule+0x20/0x58 [12084.434703] rq_qos_wait+0x100/0x178 [12084.434707] wbt_wait+0xb4/0xf0 [12084.434711] __rq_qos_throttle+0x38/0x50 [12084.434716] blk_mq_make_request+0x128/0x610 [12084.434721] generic_make_request+0xb4/0x2d8 [12084.434725] submit_bio+0x48/0x218 [12084.434824] vdev_disk_io_start+0x670/0x9f8 [zfs] [12084.434911] zio_vdev_io_start+0xdc/0x2b8 [zfs] [12084.434998] zio_nowait+0xd4/0x170 [zfs] [12084.435084] vdev_mirror_io_start+0xa8/0x1b0 [zfs] [12084.435171] zio_vdev_io_start+0x248/0x2b8 [zfs] [12084.435258] zio_execute+0xac/0x110 [zfs] [12084.435274] taskq_thread+0x2f8/0x570 [spl] [12084.435279] kthread+0xfc/0x128 [12084.435284] ret_from_fork+0x10/0x1c [12205.107010] INFO: task z_wr_iss_h:2157 blocked for more than 241 seconds. [12205.113943] Tainted: P C OE 5.4.0-1018-raspi #20-Ubuntu [12205.120902] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [12205.128906] z_wr_iss_h D 0 2157 2 0x00000028 [12205.128921] Call trace: [12205.128942] __switch_to+0x104/0x170 [12205.128953] __schedule+0x30c/0x7c0 [12205.128962] schedule+0x3c/0xb8 [12205.128970] io_schedule+0x20/0x58 [12205.128982] rq_qos_wait+0x100/0x178 [12205.128991] wbt_wait+0xb4/0xf0 [12205.129002] __rq_qos_throttle+0x38/0x50 [12205.129014] blk_mq_make_request+0x128/0x610 [12205.129026] generic_make_request+0xb4/0x2d8 [12205.129036] submit_bio+0x48/0x218 [12205.129274] vdev_disk_io_start+0x670/0x9f8 [zfs] [12205.129495] zio_vdev_io_start+0xdc/0x2b8 [zfs] [12205.129713] zio_nowait+0xd4/0x170 [zfs] [12205.129930] vdev_mirror_io_start+0xa8/0x1b0 [zfs] [12205.130147] zio_vdev_io_start+0x248/0x2b8 [zfs] [12205.130364] zio_execute+0xac/0x110 [zfs] [12205.130404] taskq_thread+0x2f8/0x570 [spl] [12205.130417] kthread+0xfc/0x128 [12205.130428] ret_from_fork+0x10/0x1c [12205.130456] INFO: task txg_quiesce:3077 blocked for more than 241 seconds. [12205.137492] Tainted: P C OE 5.4.0-1018-raspi #20-Ubuntu [12205.144426] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [12205.152418] txg_quiesce D 0 3077 2 0x00000028 [12205.152432] Call trace: [12205.152454] __switch_to+0x104/0x170 [12205.152464] __schedule+0x30c/0x7c0 [12205.152475] schedule+0x3c/0xb8 [12205.152514] cv_wait_common+0x188/0x1b0 [spl] [12205.152549] __cv_wait+0x30/0x40 [spl] [12205.152770] txg_quiesce_thread+0x27c/0x380 [zfs] [12205.152808] thread_generic_wrapper+0x74/0xa0 [spl] [12205.152831] kthread+0xfc/0x128 [12205.152843] ret_from_fork+0x10/0x1c [12205.152913] INFO: task borg:13631 blocked for more than 241 seconds. [12205.159420] Tainted: P C OE 5.4.0-1018-raspi #20-Ubuntu [12205.166351] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [12205.174340] borg D 0 13631 13630 0x00000000 [12205.174354] Call trace: [12205.174369] __switch_to+0x104/0x170 [12205.174379] __schedule+0x30c/0x7c0 [12205.174387] schedule+0x3c/0xb8 [12205.174426] cv_wait_common+0x188/0x1b0 [spl] [12205.174460] __cv_wait+0x30/0x40 [spl] [12205.174685] zil_commit_impl+0x234/0xd30 [zfs] [12205.174903] zil_commit+0x48/0x70 [zfs] [12205.175162] zfs_write+0xa3c/0xb90 [zfs] [12205.175391] zpl_write_common_iovec+0xac/0x120 [zfs] [12205.175609] zpl_iter_write+0xe4/0x150 [zfs] [12205.175629] new_sync_write+0x100/0x1a8 [12205.175638] __vfs_write+0x74/0x90 [12205.175647] vfs_write+0xe4/0x1c8 [12205.175657] ksys_write+0x78/0x100 [12205.175666] __arm64_sys_write+0x24/0x30 [12205.175676] el0_svc_common.constprop.0+0x80/0x218 [12205.175685] el0_svc_handler+0x34/0xa0 [12205.175694] el0_svc+0x10/0x2cc tyson@ubuntu: -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1899249 Title: OpenZFS writing stalls, under load To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1899249/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
