I have the same issue: kcompactd0 regularity (about every 10 days)
blocks (iowait?) and never recovers. After a few more hours, the system
becomes more or less dead.

I never had this issue until, I would say, 2 month ago. I was running
Ubuntu 18.04 fine until then. No heavy load on the server.

# ps aux|grep compac
root        28  0.0  0.0      0     0 ?        D    Aug27   0:06 [kcompactd0]

btrfs-transaction seems OK though:

# ps aux|grep trans
root       471  0.0  0.0      0     0 ?        S    Aug27   7:01 
[btrfs-transacti]

>From syslog:

Sep  8 16:36:04 nc4 kernel: [1037582.945988] INFO: task kcompactd0:28 blocked 
for more than 120 seconds.
Sep  8 16:36:04 nc4 kernel: [1037582.946216]       Not tainted 
4.15.0-112-generic #113-Ubuntu
Sep  8 16:36:04 nc4 kernel: [1037582.946375] "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep  8 16:36:04 nc4 kernel: [1037582.946557] kcompactd0      D    0    28      
2 0x80000000
Sep  8 16:36:04 nc4 kernel: [1037582.946562] Call Trace:
Sep  8 16:36:04 nc4 kernel: [1037582.946642]  __schedule+0x24e/0x880
Sep  8 16:36:04 nc4 kernel: [1037582.946826]  ? btree_releasepage+0x42/0x50 
[btrfs]
Sep  8 16:36:04 nc4 kernel: [1037582.946831]  schedule+0x2c/0x80
Sep  8 16:36:04 nc4 kernel: [1037582.946833]  io_schedule+0x16/0x40
Sep  8 16:36:04 nc4 kernel: [1037582.946853]  __lock_page+0xff/0x140
Sep  8 16:36:04 nc4 kernel: [1037582.946861]  ? page_cache_tree_insert+0xe0/0xe0
Sep  8 16:36:04 nc4 kernel: [1037582.946869]  migrate_pages+0x91f/0xb80
Sep  8 16:36:04 nc4 kernel: [1037582.946875]  ? __ClearPageMovable+0x10/0x10
Sep  8 16:36:04 nc4 kernel: [1037582.946881]  ? 
isolate_freepages_block+0x3b0/0x3b0
Sep  8 16:36:04 nc4 kernel: [1037582.946884]  compact_zone+0x681/0x950
Sep  8 16:36:04 nc4 kernel: [1037582.946887]  kcompactd_do_work+0xfe/0x2a0
Sep  8 16:36:04 nc4 kernel: [1037582.946890]  ? __switch_to_asm+0x35/0x70
Sep  8 16:36:04 nc4 kernel: [1037582.946892]  ? __switch_to_asm+0x41/0x70
Sep  8 16:36:04 nc4 kernel: [1037582.946895]  kcompactd+0x86/0x1c0
Sep  8 16:36:04 nc4 kernel: [1037582.946897]  ? kcompactd+0x86/0x1c0
Sep  8 16:36:04 nc4 kernel: [1037582.946912]  ? wait_woken+0x80/0x80
Sep  8 16:36:04 nc4 kernel: [1037582.946922]  kthread+0x121/0x140
Sep  8 16:36:04 nc4 kernel: [1037582.946924]  ? kcompactd_do_work+0x2a0/0x2a0
Sep  8 16:36:04 nc4 kernel: [1037582.946926]  ? 
kthread_create_worker_on_cpu+0x70/0x70
Sep  8 16:36:04 nc4 kernel: [1037582.946929]  ret_from_fork+0x35/0x40

Some more infos (all while kcompactd is still blocked):

# uptime
 17:20:25 up 12 days, 57 min,  1 user,  load average: 1.48, 1.19, 1.08

# free -h
              total        used        free      shared  buff/cache   available
Mem:           5.8G        476M        1.6G         72M        3.8G        5.0G
Swap:          6.0G        568M        5.4G

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.5 LTS
Release:        18.04
Codename:       bionic

The root FS is BTRF on a LUKS encrypted volume.

# cryptsetup status sda5_crypt
/dev/mapper/sda5_crypt is active and is in use.
  type:    LUKS1
  cipher:  aes-xts-plain64
  keysize: 512 bits
  key location: dm-crypt
  device:  /dev/sda5
  sector size:  512
  offset:  4096 sectors
  size:    1309712384 sectors
  mode:    read/write
  flags:   discards

# pvs
  PV                     VG     Fmt  Attr PSize    PFree
  /dev/mapper/sda5_crypt nc4-vg lvm2 a--  <624.52g    0

# vgs
  VG     #PV #LV #SN Attr   VSize    VFree
  nc4-vg   1   2   0 wz--n- <624.52g    0

# lvs
  LV     VG     Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync 
Convert
  root   nc4-vg -wi-ao---- <618.52g
  swap_1 nc4-vg -wi-ao----    6.00g

I can still write to disk:

# time dd if=/dev/zero of=io-test bs=16k count=64k conv=fdatasync
65536+0 records in
65536+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.79033 s, 385 MB/s

real    0m2.792s
user    0m0.071s
sys     0m1.858s

# sar
11:05:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
11:15:01 AM     all      1.01      0.00      0.51      0.08      0.02     98.39
11:25:01 AM     all      1.22      0.00      0.49      0.08      0.02     98.19
11:35:01 AM     all      1.01      0.00      0.46      0.06      0.02     98.45
11:45:01 AM     all      1.01      0.00      0.49      0.07      0.02     98.41
11:55:01 AM     all      0.99      0.00      0.49      0.10      0.02     98.41
12:05:01 PM     all      1.30      0.00      0.52      0.09      0.02     98.07
12:15:01 PM     all      1.01      0.00      0.52      0.05      0.02     98.41
12:25:01 PM     all      0.97      0.00      0.49      0.04      0.02     98.48
12:35:01 PM     all      1.23      0.00      0.52      0.10      0.02     98.14
12:45:01 PM     all      0.96      0.00      0.50      0.09      0.02     98.43
12:55:01 PM     all      0.99      0.00      0.46      0.10      0.02     98.43
01:05:01 PM     all      1.23      0.00      0.51      0.15      0.03     98.09
01:15:01 PM     all      0.97      0.00      0.46      0.09      0.02     98.46
01:25:01 PM     all      1.05      0.00      0.47      0.08      0.02     98.39
01:35:01 PM     all      1.08      0.00      0.45      0.08      0.02     98.37
01:45:01 PM     all      0.48      0.00      0.36      0.05      0.02     99.10
01:55:01 PM     all      0.62      0.00      0.42      0.11      0.02     98.83
02:05:01 PM     all      0.56      0.00      0.35      0.14      0.02     98.93
02:15:01 PM     all      0.95      0.00      0.47      0.13      0.02     98.43
02:25:01 PM     all      0.95      0.00      0.46      0.09      0.02     98.47
02:35:01 PM     all      1.10      0.00      0.49      0.13      0.02     98.26
02:45:01 PM     all      1.18      0.00      0.60      0.09      0.02     98.11
02:55:01 PM     all      1.23      0.00      0.59      0.09      0.03     98.06
03:05:01 PM     all      2.38      0.00      0.78      0.13      0.03     96.69
03:15:01 PM     all      1.00      0.00      0.49      0.12      0.02     98.37
03:25:01 PM     all      1.61      0.00      0.68      0.10      0.02     97.60
03:35:01 PM     all      1.33      0.00      0.56      0.07      0.03     98.01
03:45:01 PM     all      1.05      0.00      0.48      0.11      0.02     98.34
03:55:01 PM     all      1.03      0.00      0.51      0.10      0.02     98.35
04:05:01 PM     all      1.31      0.00      0.52      0.10      0.02     98.06
04:15:01 PM     all      0.97      0.00      0.50      0.10      0.02     98.40
04:25:01 PM     all      1.18      0.00      0.56      0.13      0.02     98.11
04:35:01 PM     all      1.23      0.00      0.52     10.06      0.02     88.17 
<-- trouble starts!
04:45:01 PM     all      1.42      0.00      0.89     49.33      0.02     48.33
04:55:01 PM     all      1.28      0.00      0.74     49.12      0.02     48.84
05:05:01 PM     all      1.58      0.00      2.48     50.41      0.04     45.49
05:15:01 PM     all      0.86      0.00      0.39     49.43      0.02     49.29

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1891773

Title:
  kcompactd0 and btrfs-transaction keep deadlocking with each other

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1891773/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

Reply via email to