Verification done for zfs-linux on xenial-proposed. The inodes for file, xattr dir, and xattr child are all evicted at file removal time, not making it to any disposal list after file removal.
So the window/scenario for the problem to occur is not present anymore. Log --- $ echo 'deb http://archive.ubuntu.com/ubuntu xenial-proposed main restricted universe' | sudo tee /etc/apt/sources.list.d/xenial-proposed-universe.list $ sudo apt-get update $ apt-cache madison zfs-dkms | grep proposed zfs-dkms | 0.6.5.6-0ubuntu28 | http://archive.ubuntu.com/ubuntu xenial-proposed/universe amd64 Packages $ sudo apt-get install -y spl-dkms zfs-dkms $ modinfo zfs | head filename: /lib/modules/4.4.0-157-generic/updates/dkms/zfs.ko version: 0.6.5.6-0ubuntu28 ... $ sudo ./zfs-mount.sh $ echo 2 | sudo tee /proc/sys/vm/drop_caches 2 $ sudo ./zfs-kprobes.sh $ touch /zfs/file <...>-26587 [001] d... 1196.781161: p_zfs_mknode_0: (zfs_mknode+0x0/0xe00 [zfs]) flag=0x0 dzp=0xffff88004ccfc000 <...>-26587 [001] d... 1196.781306: p_zfs_znode_alloc_0: (zfs_znode_alloc+0x0/0x520 [zfs]) obj=0xa <...>-26587 [001] d... 1196.781336: r_zfs_znode_alloc_0: (zfs_mknode+0x8a3/0xe00 [zfs] <- zfs_znode_alloc) zpp=0xffff88004cd04cc0 $ setfattr -n user.debug -v 1 /zfs/file setfattr-26588 [000] d... 1199.095554: p_zfs_mknode_0: (zfs_mknode+0x0/0xe00 [zfs]) flag=0x2 dzp=0xffff88004cd04cc0 setfattr-26588 [000] d... 1199.095937: p_zfs_znode_alloc_0: (zfs_znode_alloc+0x0/0x520 [zfs]) obj=0xb setfattr-26588 [000] d... 1199.096145: r_zfs_znode_alloc_0: (zfs_mknode+0x8a3/0xe00 [zfs] <- zfs_znode_alloc) zpp=0xffff88004c567740 setfattr-26588 [000] d... 1199.096539: p_zfs_zget_0: (zfs_zget+0x0/0x230 [zfs]) zsb=0xffff88004d1c8000 obj=0xa setfattr-26588 [000] d... 1199.096779: r_zfs_zget_0: (zfs_zaccess+0x12b/0x220 [zfs] <- zfs_zget) setfattr-26588 [000] d... 1199.097139: p_zfs_zget_0: (zfs_zget+0x0/0x230 [zfs]) zsb=0xffff88004d1c8000 obj=0xa setfattr-26588 [000] d... 1199.097352: r_zfs_zget_0: (zfs_zaccess+0x12b/0x220 [zfs] <- zfs_zget) setfattr-26588 [000] d... 1199.097661: p_zfs_zget_0: (zfs_zget+0x0/0x230 [zfs]) zsb=0xffff88004d1c8000 obj=0xa setfattr-26588 [000] d... 1199.097892: r_zfs_zget_0: (zfs_zaccess+0x12b/0x220 [zfs] <- zfs_zget) setfattr-26588 [000] d... 1199.098217: p_zfs_mknode_0: (zfs_mknode+0x0/0xe00 [zfs]) flag=0x0 dzp=0xffff88004c567740 setfattr-26588 [000] d... 1199.098449: p_zfs_znode_alloc_0: (zfs_znode_alloc+0x0/0x520 [zfs]) obj=0xc setfattr-26588 [000] d... 1199.098652: r_zfs_znode_alloc_0: (zfs_mknode+0x8a3/0xe00 [zfs] <- zfs_znode_alloc) zpp=0xffff88004c567300 $ rm /zfs/file <...>-26589 [001] d... 1201.376695: p_zfs_zget_0: (zfs_zget+0x0/0x230 [zfs]) zsb=0xffff88004d1c8000 obj=0xa <...>-26589 [001] d... 1201.376749: r_zfs_zget_0: (zfs_dirent_lock+0x56c/0x6c0 [zfs] <- zfs_zget) rm-26589 [001] d... 1201.376772: p_zfs_zget_0: (zfs_zget+0x0/0x230 [zfs]) zsb=0xffff88004d1c8000 obj=0xb rm-26589 [001] d... 1201.376778: r_zfs_zget_0: (zfs_remove+0x22b/0x4c0 [zfs] <- zfs_zget) rm-26589 [001] d... 1201.376904: p_zpl_evict_inode_0: (zpl_evict_inode+0x0/0x60 [zfs]) inode=0xffff88004cd04ed0 rm-26589 [001] d... 1201.376922: p_zfs_inactive_0: (zfs_inactive+0x0/0x270 [zfs]) inode=0xffff88004cd04ed0 rm-26589 [001] d... 1201.376923: p_zfs_zinactive_0: (zfs_zinactive+0x0/0xe0 [zfs]) znode=0xffff88004cd04cc0 obj=0xa rm-26589 [001] d... 1201.376927: p_zfs_rmnode_0: (zfs_rmnode+0x0/0x360 [zfs]) znode=0xffff88004cd04cc0 rm-26589 [001] d... 1201.376942: p_zfs_zget_0: (zfs_zget+0x0/0x230 [zfs]) zsb=0xffff88004d1c8000 obj=0xb rm-26589 [001] d... 1201.376999: r_zfs_zget_0: (zfs_rmnode+0x25b/0x360 [zfs] <- zfs_zget) rm-26589 [001] d... 1201.377043: p_zfs_iput_async_0: (zfs_iput_async+0x0/0x60 [zfs]) inode=0xffff88004c567950 obj=0x0 z_iput-26241 [001] d... 1201.377065: p_zpl_evict_inode_0: (zpl_evict_inode+0x0/0x60 [zfs]) inode=0xffff88004c567950 z_iput-26241 [001] d... 1201.377067: p_zfs_inactive_0: (zfs_inactive+0x0/0x270 [zfs]) inode=0xffff88004c567950 z_iput-26241 [001] d... 1201.377067: p_zfs_zinactive_0: (zfs_zinactive+0x0/0xe0 [zfs]) znode=0xffff88004c567740 obj=0xb z_iput-26241 [001] d... 1201.377070: p_zfs_rmnode_0: (zfs_rmnode+0x0/0x360 [zfs]) znode=0xffff88004c567740 z_iput-26241 [001] d... 1201.377072: p_zfs_purgedir_0: (zfs_purgedir+0x0/0x230 [zfs]) znode=0xffff88004c567740 z_iput-26241 [001] d... 1201.377079: p_zfs_zget_0: (zfs_zget+0x0/0x230 [zfs]) zsb=0xffff88004d1c8000 obj=0xc z_iput-26241 [001] d... 1201.377085: r_zfs_zget_0: (zfs_purgedir+0xb4/0x230 [zfs] <- zfs_zget) z_iput-26241 [001] d... 1201.377117: p_zfs_iput_async_0: (zfs_iput_async+0x0/0x60 [zfs]) inode=0xffff88004c567510 obj=0x0 z_iput-26241 [001] d... 1201.377123: r_zfs_purgedir_0: (zfs_rmnode+0x272/0x360 [zfs] <- zfs_purgedir) z_iput-26241 [001] d... 1201.377160: p_zpl_evict_inode_0: (zpl_evict_inode+0x0/0x60 [zfs]) inode=0xffff88004c567510 z_iput-26241 [001] d... 1201.377161: p_zfs_inactive_0: (zfs_inactive+0x0/0x270 [zfs]) inode=0xffff88004c567510 z_iput-26241 [001] d... 1201.377162: p_zfs_zinactive_0: (zfs_zinactive+0x0/0xe0 [zfs]) znode=0xffff88004c567300 obj=0xc z_iput-26241 [001] d... 1201.377164: p_zfs_rmnode_0: (zfs_rmnode+0x0/0x360 [zfs]) znode=0xffff88004c567300 $ echo 2 | sudo tee /proc/sys/vm/drop_caches 2 <...>-26592 [001] d... 1204.543205: p_dispose_list_0: (dispose_list+0x0/0x50) tee-26592 [001] dN.. 1204.543357: p_dispose_list_0: (dispose_list+0x0/0x50) tee-26592 [001] d... 1204.543652: p_dispose_list_0: (dispose_list+0x0/0x50) tee-26592 [001] d... 1204.543799: p_dispose_list_0: (dispose_list+0x0/0x50) tee-26592 [001] d... 1204.544250: p_dispose_list_0: (dispose_list+0x0/0x50) tee-26592 [001] d... 1204.544429: p_dispose_list_0: (dispose_list+0x0/0x50) ** Tags added: verification-done-xenial -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1839521 Title: Xenial: ZFS deadlock in shrinker path with xattrs Status in linux package in Ubuntu: Invalid Status in zfs-linux package in Ubuntu: Invalid Status in linux source package in Xenial: Fix Committed Status in zfs-linux source package in Xenial: Fix Committed Status in linux source package in Bionic: Invalid Status in zfs-linux source package in Bionic: Invalid Status in linux source package in Disco: Invalid Status in zfs-linux source package in Disco: Invalid Status in linux source package in Eoan: Invalid Status in zfs-linux source package in Eoan: Invalid Bug description: [Impact] * Xenial's ZFS can deadlock in the memory shrinker path after removing files with extended attributes (xattr). * Extended attributes are enabled by default, but are _not_ used by default, which reduces the likelyhood. * It's very difficult/rare to reproduce this problem, due to file/xattr/remove/shrinker/lru order/timing circumstances required. (weeks for a reporter user) but a synthetic test-case has been found for tests. [Test Case] * A synthetic reproducer is available for this LP, with a few steps to touch/setfattr/rm/drop_caches plus a kernel module to massage the disposal list. (comment #8) * In the original ZFS module: the xattr dir inode is not purged immediately on file removal, but possibly purged _two_ shrinker invocations later. This allows for other thread started before file remove to call zfs_zget() on the xattr child inode and iput() it, so it makes to the same disposal list as the xattr dir inode. (comment #3) * In the modified ZFS module: the xattr dir inode is purged immediately on file removal not possibly later on shrinker invocation, so the problem window above doesn't exist anymore. (comment #12) [Regression Potential] * Low. The patches are confined to extended attributes in ZFS, specifically node removal/purge, and another change how an xattr child inode tracks its xattr dir (parent) inode, so that it can be purged immediately on removal. * The ZFS test-suite has been run on original/modified zfs-dkms package/kernel modules, with no regressions. (comment #11) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1839521/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
