Hello Alexandre,
I tried to reproduce this bug, and I believe it has been fixed.
I started a i3.4xlarge instance on AWS, with Xenial as the distro:
$ uname -rv
4.4.0-1112-aws #124-Ubuntu SMP Fri Jul 24 11:10:25 UTC 2020
>From there, I checked the NVMe disks:
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 8G 0 disk
`-xvda1 202:1 0 8G 0 part /
nvme0n1 259:0 0 1.7T 0 disk
nvme1n1 259:1 0 1.7T 0 disk
Made a Raid 0 array:
$ sudo mdadm --create --verbose --level=0 /dev/md0 --raid-devices=2
/dev/nvme0n1 /dev/nvme1n1
mdadm: chunk size defaults to 512K
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started
And formatted it:
$ time sudo mkfs.xfs /dev/md0
meta-data=/dev/md0 isize=512 agcount=32, agsize=28989568 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=0
data = bsize=4096 blocks=927666176, imaxpct=5
= sunit=128 swidth=256 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=452968, version=2
= sectsz=512 sunit=8 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
real 0m24.414s
user 0m0.000s
sys 0m7.664s
I also tried to fstrim:
$ sudo mkdir /mnt/disk
$ sudo mount /dev/md0 /mnt/disk
$ sudo fstrim /mnt/disk
$ time sudo fstrim /mnt/disk
real 0m22.083s
user 0m0.000s
sys 0m7.560s
Things seem okay, I think this has been fixed somewhere between 4.4.0-1052-aws
and 4.4.0-1112-aws.
I'm going to mark this bug as fixed released, but let me know if you still have
problems.
** Also affects: linux (Ubuntu)
Importance: Undecided
Status: New
** No longer affects: linux-aws (Ubuntu)
** Also affects: linux (Ubuntu Xenial)
Importance: Undecided
Status: New
** Changed in: linux (Ubuntu)
Status: New => Fix Released
** Changed in: linux (Ubuntu Xenial)
Status: New => Fix Released
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws in Ubuntu.
https://bugs.launchpad.net/bugs/1756315
Title:
fstrim and discard operations take too long to complete - Ubuntu 16.04
Status in linux package in Ubuntu:
Fix Released
Status in linux source package in Xenial:
Fix Released
Bug description:
1-) Ubuntu Release : Ubuntu 16.04 LTS
2-) linux-image-4.4.0.1052.54-aws
3-) mkfs.xfs and fstrim -v on a raid0 array using nvme and md should
not take more than some seconds to complete.
4-) Formating the raid0 array with xfs took around 2 hours. Running
fstrim -v on the mount point mounted on top of the raid array took
around 2 hours.
How to reproduce the issue:
- Launch an i3.4xlarge instance on Amazon AWS using an Ubuntu 16.04 AMI (
ami-78d2be01 on EU-WEST-1 ), this will generate an instance with one 8Gb EBS
root volume and two 1.9T SSD drives that are presented to the instance using
the nvme driver.
- Compose a raid0 array with the following command :
# mdadm --create --verbose --level=0 /dev/md0 --raid-devices=2
/dev/nvme0n1 /dev/nvme1n1
- When trying to format the raid0 array ( /dev/md0 ) using xfs it
takes around 2 hours to complete. I tried other AMIs like RHEL7,
CentOS7 and Ubuntu 18.04 and the time needed was around 2 seconds.
root@ip-172-31-30-133:~# time mkfs.xfs /dev/md0
real 120m45.725s
user 0m0.000s
sys 0m18.248s
- Running fstrim -v on a filesystem mounted on top of /dev/md0 can
take around 2 hours to complete. With other AMIs like RHEL7, CentOS7
and Ubuntu 18.04 and the time needed was around 2 seconds.
- When I try the same with any of the nvme SSD devices alone, let's
say /dev/nvme0n1, the issue doesn't happen.
- I tried to replicate this issue using LVM and striping, fstrim and
mkfs.xfs, the tasks complete without taking hours :
root@ip-172-31-27-69:~# pvcreate /dev/nvme0n1
Physical volume "/dev/nvme0n1" successfully created
root@ip-172-31-27-69:~# pvcreate /dev/nvme1n1
Physical volume "/dev/nvme1n1" successfully created
root@ip-172-31-27-69:~# vgcreate raid0 /dev/nvme0n1 /dev/nvme1n1
Volume group "raid0" successfully created
root@ip-172-31-27-69:~# lvcreate --type striped --stripes 2 --extents
100%FREE raid0 /dev/nvme0n1 /dev/nvme1n1
Using default stripesize 64.00 KiB.
Logical volume "lvol0" created.
root@ip-172-31-27-69:~# vgchange -ay
1 logical volume(s) in volume group "raid0" now active
root@ip-172-31-27-69:~# lvchange -ay /dev/raid0/lvol0
root@ip-172-31-27-69:~# lvs -a /dev/raid0/lvol0
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync
Convert
lvol0 raid0 -wi-a----- 3.46t
htop
root@ip-172-31-27-69:~# time mkfs.xfs /dev/raid0/lvol0
meta-data=/dev/raid0/lvol0 isize=512 agcount=32, agsize=28991664 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=0
data = bsize=4096 blocks=927733248, imaxpct=5
= sunit=16 swidth=32 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=453008, version=2
= sectsz=512 sunit=16 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
real 0m2.926s
user 0m0.180s
sys 0m0.000s
root@ip-172-31-27-69:~# mount /dev/raid0/lvol0 /mnt
root@ip-172-31-27-69:~# time fstrim -v /mnt
/mnt: 3.5 TiB (3798138650624 bytes) trimmed
real 0m1.794s
user 0m0.000s
sys 0m0.000s
So the issue only happens when using nvme and md to compose the raid0
array.
Bellow follows some information that may be useful:
started formating the md array with mkfs.xfs. Process looks hanged.
root@ip-172-31-24-66:~# ps aux | grep -i mkfs.xfs
root 1693 12.0 0.0 12728 968 pts/1 D+ 07:54 0:03 mkfs.xfs
/dev/md0
PID 1693 is in uninterruptible sleep ( D )
Looking at /proc/7965/stack
root@ip-172-31-24-66:~# cat /proc/1693/stack
[<ffffffff8134d8c2>] blkdev_issue_discard+0x232/0x2a0
[<ffffffff813524bd>] blkdev_ioctl+0x61d/0x7d0
[<ffffffff811ff6f1>] block_ioctl+0x41/0x50
[<ffffffff811d89b3>] do_vfs_ioctl+0x2e3/0x4d0
[<ffffffff811d8c21>] SyS_ioctl+0x81/0xa0
[<ffffffff81748030>] system_call_fastpath+0x1a/0x1f
[<ffffffffffffffff>] 0xffffffffffffffff
Looking at the stack, looks like it's hanged on a discard operation
root@ip-172-31-24-66:~# ps -flp 1693
F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME
CMD
4 D root 1693 1682 2 80 0 - 3182 blkdev 07:54 pts/1 00:00:03
mkfs.xfs /dev/md0
root@ip-172-31-24-66:~# cat /proc/1693/wchan
blkdev_issue_discard
Process stuck with function --> blkdev_issue_discard
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1756315/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
