This bug was fixed in the package linux - 4.4.0-134.160 --------------- linux (4.4.0-134.160) xenial; urgency=medium
* linux: 4.4.0-134.160 -proposed tracker (LP: #1787177) * locking sockets broken due to missing AppArmor socket mediation patches (LP: #1780227) - UBUNTU SAUCE: apparmor: fix apparmor mediating locking non-fs, unix sockets * Backport namespaced fscaps to xenial 4.4 (LP: #1778286) - Introduce v3 namespaced file capabilities - commoncap: move assignment of fs_ns to avoid null pointer dereference - capabilities: fix buffer overread on very short xattr - commoncap: Handle memory allocation failure. * Xenial update to 4.4.140 stable release (LP: #1784409) - usb: cdc_acm: Add quirk for Uniden UBC125 scanner - USB: serial: cp210x: add CESINEL device ids - USB: serial: cp210x: add Silicon Labs IDs for Windows Update - n_tty: Fix stall at n_tty_receive_char_special(). - staging: android: ion: Return an ERR_PTR in ion_map_kernel - n_tty: Access echo_* variables carefully. - x86/boot: Fix early command-line parsing when matching at end - ath10k: fix rfc1042 header retrieval in QCA4019 with eth decap mode - i2c: rcar: fix resume by always initializing registers before transfer - ipv4: Fix error return value in fib_convert_metrics() - kprobes/x86: Do not modify singlestep buffer while resuming - nvme-pci: initialize queue memory before interrupts - netfilter: nf_tables: use WARN_ON_ONCE instead of BUG_ON in nft_do_chain() - ARM: dts: imx6q: Use correct SDMA script for SPI5 core - ubi: fastmap: Correctly handle interrupted erasures in EBA - mm: hugetlb: yield when prepping struct pages - tracing: Fix missing return symbol in function_graph output - scsi: sg: mitigate read/write abuse - s390: Correct register corruption in critical section cleanup - drbd: fix access after free - cifs: Fix infinite loop when using hard mount option - jbd2: don't mark block as modified if the handle is out of credits - ext4: make sure bitmaps and the inode table don't overlap with bg descriptors - ext4: always check block group bounds in ext4_init_block_bitmap() - ext4: only look at the bg_flags field if it is valid - ext4: verify the depth of extent tree in ext4_find_extent() - ext4: include the illegal physical block in the bad map ext4_error msg - ext4: clear i_data in ext4_inode_info when removing inline data - ext4: add more inode number paranoia checks - ext4: add more mount time checks of the superblock - ext4: check superblock mapped prior to committing - HID: i2c-hid: Fix "incomplete report" noise - HID: hiddev: fix potential Spectre v1 - HID: debug: check length before copy_to_user() - x86/mce: Detect local MCEs properly - x86/mce: Fix incorrect "Machine check from unknown source" message - media: cx25840: Use subdev host data for PLL override - mm, page_alloc: do not break __GFP_THISNODE by zonelist reset - dm bufio: avoid sleeping while holding the dm_bufio lock - dm bufio: drop the lock when doing GFP_NOIO allocation - mtd: rawnand: mxc: set spare area size register explicitly - dm bufio: don't take the lock in dm_bufio_shrink_count - mtd: cfi_cmdset_0002: Change definition naming to retry write operation - mtd: cfi_cmdset_0002: Change erase functions to retry for error - mtd: cfi_cmdset_0002: Change erase functions to check chip good only - netfilter: nf_log: don't hold nf_log_mutex during user access - staging: comedi: quatech_daqp_cs: fix no-op loop daqp_ao_insn_write() - Linux 4.4.140 * Xenial update to 4.4.139 stable release (LP: #1784382) - xfrm6: avoid potential infinite loop in _decode_session6() - netfilter: ebtables: handle string from userspace with care - ipvs: fix buffer overflow with sync daemon and service - atm: zatm: fix memcmp casting - net: qmi_wwan: Add Netgear Aircard 779S - net/sonic: Use dma_mapping_error() - Revert "Btrfs: fix scrub to repair raid6 corruption" - tcp: do not overshoot window_clamp in tcp_rcv_space_adjust() - Btrfs: make raid6 rebuild retry more - usb: musb: fix remote wakeup racing with suspend - bonding: re-evaluate force_primary when the primary slave name changes - tcp: verify the checksum of the first data segment in a new connection - ext4: update mtime in ext4_punch_hole even if no blocks are released - ext4: fix fencepost error in check for inode count overflow during resize - driver core: Don't ignore class_dir_create_and_add() failure. - btrfs: scrub: Don't use inode pages for device replace - ALSA: hda - Handle kzalloc() failure in snd_hda_attach_pcm_stream() - ALSA: hda: add dock and led support for HP EliteBook 830 G5 - ALSA: hda: add dock and led support for HP ProBook 640 G4 - cpufreq: Fix new policy initialization during limits updates via sysfs - libata: zpodd: make arrays cdb static, reduces object code size - libata: zpodd: small read overflow in eject_tray() - libata: Drop SanDisk SD7UB3Q*G1001 NOLPM quirk - w1: mxc_w1: Enable clock before calling clk_get_rate() on it - x86/spectre_v1: Disable compiler optimizations over array_index_mask_nospec() - m68k/mm: Adjust VM area to be unmapped by gap size for __iounmap() - serial: sh-sci: Use spin_{try}lock_irqsave instead of open coding version - signal/xtensa: Consistenly use SIGBUS in do_unaligned_user - usb: do not reset if a low-speed or full-speed device timed out - 1wire: family module autoload fails because of upper/lower case mismatch. - ASoC: dapm: delete dapm_kcontrol_data paths list before freeing it - ASoC: cirrus: i2s: Fix LRCLK configuration - ASoC: cirrus: i2s: Fix {TX|RX}LinCtrlData setup - lib/vsprintf: Remove atomic-unsafe support for %pCr - mips: ftrace: fix static function graph tracing - branch-check: fix long->int truncation when profiling branches - ipmi:bt: Set the timeout before doing a capabilities check - Bluetooth: hci_qca: Avoid missing rampatch failure with userspace fw loader - fuse: atomic_o_trunc should truncate pagecache - fuse: don't keep dead fuse_conn at fuse_fill_super(). - fuse: fix control dir setup and teardown - powerpc/mm/hash: Add missing isync prior to kernel stack SLB switch - powerpc/ptrace: Fix setting 512B aligned breakpoints with PTRACE_SET_DEBUGREG - powerpc/ptrace: Fix enforcement of DAWR constraints - cpuidle: powernv: Fix promotion from snooze if next state disabled - powerpc/fadump: Unregister fadump on kexec down path. - ARM: 8764/1: kgdb: fix NUMREGBYTES so that gdb_regs[] is the correct size - of: unittest: for strings, account for trailing \0 in property length field - IB/qib: Fix DMA api warning with debug kernel - RDMA/mlx4: Discard unknown SQP work requests - mtd: cfi_cmdset_0002: Change write buffer to check correct value - mtd: cfi_cmdset_0002: Use right chip in do_ppb_xxlock() - mtd: cfi_cmdset_0002: fix SEGV unlocking multiple chips - mtd: cfi_cmdset_0002: Fix unlocking requests crossing a chip boudary - mtd: cfi_cmdset_0002: Avoid walking all chips when unlocking. - MIPS: BCM47XX: Enable 74K Core ExternalSync for PCIe erratum - PCI: pciehp: Clear Presence Detect and Data Link Layer Status Changed on resume - MIPS: io: Add barrier after register read in inX() - time: Make sure jiffies_to_msecs() preserves non-zero time periods - Btrfs: fix clone vs chattr NODATASUM race - iio:buffer: make length types match kfifo types - scsi: qla2xxx: Fix setting lower transfer speed if GPSC fails - scsi: zfcp: fix missing SCSI trace for result of eh_host_reset_handler - scsi: zfcp: fix missing SCSI trace for retry of abort / scsi_eh TMF - scsi: zfcp: fix misleading REC trigger trace where erp_action setup failed - scsi: zfcp: fix missing REC trigger trace on terminate_rport_io early return - scsi: zfcp: fix missing REC trigger trace on terminate_rport_io for ERP_FAILED - scsi: zfcp: fix missing REC trigger trace for all objects in ERP_FAILED - scsi: zfcp: fix missing REC trigger trace on enqueue without ERP thread - linvdimm, pmem: Preserve read-only setting for pmem devices - md: fix two problems with setting the "re-add" device state. - ubi: fastmap: Cancel work upon detach - UBIFS: Fix potential integer overflow in allocation - xfrm: skip policies marked as dead while rehashing - backlight: as3711_bl: Fix Device Tree node lookup - backlight: max8925_bl: Fix Device Tree node lookup - backlight: tps65217_bl: Fix Device Tree node lookup - mfd: intel-lpss: Program REMAP register in PIO mode - perf tools: Fix symbol and object code resolution for vdso32 and vdsox32 - perf intel-pt: Fix sync_switch INTEL_PT_SS_NOT_TRACING - perf intel-pt: Fix decoding to accept CBR between FUP and corresponding TIP - perf intel-pt: Fix MTC timing after overflow - perf intel-pt: Fix "Unexpected indirect branch" error - perf intel-pt: Fix packet decoding of CYC packets - media: v4l2-compat-ioctl32: prevent go past max size - media: dvb_frontend: fix locking issues at dvb_frontend_get_event() - nfsd: restrict rd_maxcount to svc_max_payload in nfsd_encode_readdir - NFSv4: Fix possible 1-byte stack overflow in nfs_idmap_read_and_verify_message - video: uvesafb: Fix integer overflow in allocation - Input: elan_i2c - add ELAN0618 (Lenovo v330 15IKB) ACPI ID - xen: Remove unnecessary BUG_ON from __unbind_from_irq() - udf: Detect incorrect directory size - Input: elan_i2c_smbus - fix more potential stack buffer overflows - Input: elantech - enable middle button of touchpads on ThinkPad P52 - Input: elantech - fix V4 report decoding for module with middle key - ALSA: hda/realtek - Add a quirk for FSC ESPRIMO U9210 - Btrfs: fix unexpected cow in run_delalloc_nocow - spi: Fix scatterlist elements size in spi_map_buf - block: Fix transfer when chunk sectors exceeds max - dm thin: handle running out of data space vs concurrent discard - cdc_ncm: avoid padding beyond end of skb - Bluetooth: Fix connection if directed advertising and privacy is used - Linux 4.4.139 * Support AverMedia DVD EZMaker 7 USB video capture dongle (LP: #1620762) // Xenial update to 4.4.139 stable release (LP: #1784382) - media: cx231xx: Add support for AverMedia DVD EZMaker 7 * vfio/pci: cannot assign a i40e pf device to a vm using vfio-pci (LP: #1779830) - vfio/pci: Hide broken INTx support from user * Kernel error "task zfs:pid blocked for more than 120 seconds" (LP: #1781364) - SAUCE: (noup) zfs to 0.6.5.6-0ubuntu25 * Allow multiple mounts of zfs datasets (LP: #1759848) - SAUCE: Allow mounting datasets more than once (LP: #1759848) * CVE-2018-12233 - jfs: Fix inconsistency between memory allocation and ea_buf->max_size * Redpine: Observed kernel panic while running wireless tests in regression mode (LP: #1773410) // Redpine: Observed kernel panic while running soft-ap tests (LP: #1777850) - SAUCE: Redpine: improve cancel_hw_scan handling to fix kernel panic * [HMS] Upgrades to Support SocketCAN over USB on Dell IoT 300x Gateways (LP: #1783241) - SAUCE: (no-up) upgrade IXXAT USB SocketCAN driver * CVE-2018-13094 - xfs: don't call xfs_da_shrink_inode with NULL bp * other users' coredumps can be read via setgid directory and killpriv bypass (LP: #1779923) // CVE-2018-13405 - Fix up non-directory creation in SGID directories * snapcraft.yaml: missing ubuntu-retpoline-extract-one script breaks the build (LP: #1782116) - snapcraft.yaml: copy retpoline-extract-one to scripts before build * Enable basic support for Solarflare 8000 series NIC (LP: #1783152) - sfc: make TSO version a per-queue parameter - sfc: Add PCI ID for Solarflare 8000 series 10/40G NIC * Redpine: Observed kernel panic while running wireless regressions tests (LP: #1777858) - SAUCE: Redpine: improve kernel thread handling to fix kernel panic * Xenial update to 4.4.138 stable release (LP: #1777389) - x86: Remove unused function cpu_has_ht_siblings() - x86/cpufeature: Remove unused and seldomly used cpu_has_xx macros - x86/fpu: Disable AVX when eagerfpu is off - x86/fpu: Revert ("x86/fpu: Disable AVX when eagerfpu is off") - x86/fpu: Hard-disable lazy FPU mode - af_key: Always verify length of provided sadb_key - x86/crypto, x86/fpu: Remove X86_FEATURE_EAGER_FPU #ifdef from the crc32c code - gpio: No NULL owner - Clarify (and fix) MAX_LFS_FILESIZE macros - serial: samsung: fix maxburst parameter for DMA transactions - vmw_balloon: fixing double free when batching mode is off - Input: goodix - add new ACPI id for GPD Win 2 touch screen - crypto: vmx - Remove overly verbose printk from AES init routines - Linux 4.4.138 * Redpine: wifi-ap stopped working after restart (LP: #1773400) - SAUCE: Redpine: fix soft-ap invisible issue * Xenial update to 4.4.137 stable release (LP: #1777063) - tpm: do not suspend/resume if power stays on - tpm: self test failure should not cause suspend to fail - mmap: introduce sane default mmap limits - mmap: relax file size limit for regular files - kconfig: Avoid format overflow warning from GCC 8.1 - xfs: fix incorrect log_flushed on fsync - drm: set FMODE_UNSIGNED_OFFSET for drm files - brcmfmac: Fix check for ISO3166 code - bnx2x: use the right constant - dccp: don't free ccid2_hc_tx_sock struct in dccp_disconnect() - enic: set DMA mask to 47 bit - ip6mr: only set ip6mr_table from setsockopt when ip6mr_new_table succeeds - ipv4: remove warning in ip_recv_error - isdn: eicon: fix a missing-check bug - netdev-FAQ: clarify DaveM's position for stable backports - net/packet: refine check for priv area size - net: usb: cdc_mbim: add flag FLAG_SEND_ZLP - packet: fix reserve calculation - qed: Fix mask for physical address in ILT entry - net/mlx4: Fix irq-unsafe spinlock usage - team: use netdev_features_t instead of u32 - rtnetlink: validate attributes in do_setlink() - net: phy: broadcom: Fix bcm_write_exp() - net: metrics: add proper netlink validation - Linux 4.4.137 * Xenial update to 4.4.136 stable release (LP: #1776177) - arm64: lse: Add early clobbers to some input/output asm operands - powerpc/64s: Clear PCR on boot - USB: serial: cp210x: use tcflag_t to fix incompatible pointer type - sh: New gcc support - xfs: detect agfl count corruption and reset agfl - Input: elan_i2c_smbus - fix corrupted stack - tracing: Fix crash when freeing instances with event triggers - selinux: KASAN: slab-out-of-bounds in xattr_getsecurity - cfg80211: further limit wiphy names to 64 bytes - rtlwifi: rtl8192cu: Remove variable self-assignment in rf.c - ASoC: Intel: sst: remove redundant variable dma_dev_name - irda: fix overly long udelay() - tcp: avoid integer overflows in tcp_rcv_space_adjust() - i2c: rcar: make sure clocks are on when doing clock calculation - i2c: rcar: rework hw init - i2c: rcar: remove unused IOERROR state - i2c: rcar: remove spinlock - i2c: rcar: refactor setup of a msg - i2c: rcar: init new messages in irq - i2c: rcar: don't issue stop when HW does it automatically - i2c: rcar: check master irqs before slave irqs - i2c: rcar: revoke START request early - dmaengine: usb-dmac: fix endless loop in usb_dmac_chan_terminate_all() - iio:kfifo_buf: check for uint overflow - MIPS: ptrace: Fix PTRACE_PEEKUSR requests for 64-bit FGRs - MIPS: prctl: Disallow FRE without FR with PR_SET_FP_MODE requests - scsi: scsi_transport_srp: Fix shost to rport translation - stm class: Use vmalloc for the master map - hwtracing: stm: fix build error on some arches - drm/i915: Disable LVDS on Radiant P845 - Kbuild: change CC_OPTIMIZE_FOR_SIZE definition - [Config] Add CONFIG_CC_OPTIMIZE_FOR_PERFORMANCE=y - fix io_destroy()/aio_complete() race - mm: fix the NULL mapping case in __isolate_lru_page() - sparc64: Fix build warnings with gcc 7. - Linux 4.4.136 * Xenial update to 4.4.135 stable release (LP: #1776158) - Revert "vti4: Don't override MTU passed on link creation via IFLA_MTU" - Linux 4.4.135 -- Kleber Sacilotto de Souza <kleber.so...@canonical.com> Wed, 15 Aug 2018 13:51:11 +0000 ** Changed in: linux (Ubuntu Xenial) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to zfs-linux in Ubuntu. https://bugs.launchpad.net/bugs/1781364 Title: Kernel error "task zfs:pid blocked for more than 120 seconds" Status in Linux: Fix Released Status in linux package in Ubuntu: Fix Released Status in zfs-linux package in Ubuntu: Fix Released Status in linux source package in Xenial: Fix Released Status in zfs-linux source package in Xenial: Confirmed Status in linux source package in Bionic: Fix Released Status in zfs-linux source package in Bionic: Fix Released Status in linux source package in Cosmic: Fix Released Status in zfs-linux source package in Cosmic: Fix Released Bug description: == SRU Justification, XENIAL, BIONIC == Exercising ZFS with lxd with many mount/umounts can cause lockups and 120 second timeout messages. == How to reproduce bug == In a VM, 2 CPUs, 16GB of memory running Bionic: sudo apt update sudo apt install lxd lxd-client lxd-tools zfsutils-linux sudo lxd init (and with the default init options) then run: lxd-benchmark launch --count 96 --parallel 96 This will reliably show the lockup every time without the fix. With the fix (detailed below) one cannot reproduce the lockup. == Fix == Upstream ZFS commit commit ac09630d8b0bf6c92084a30fdaefd03fd0adbdc1 Author: Brian Behlendorf <behlendo...@llnl.gov> Date: Wed Jul 11 15:49:10 2018 -0700 Fix zpl_mount() deadlock == Regression Potential == This just changes the locking in the mount path of ZFS and will only affect ZFS mount/unmounts. The regression potential is small as this touches a very small code path that has been exhaustively exercises this code under multiple thread/CPU contention and shown not to break. ------------------ ZFS bug report: https://github.com/zfsonlinux/zfs/issues/7691 "I am using LXD containers that are configured to use a ZFS storage backend. I create many containers using a benchmark tool, which probably stresses the use of ZFS. In two out of four attempts, I got [ 725.970508] INFO: task lxd:4455 blocked for more than 120 seconds. [ 725.976730] Tainted: P O 4.15.0-20-generic #21-Ubuntu [ 725.983551] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 725.991624] INFO: task txg_sync:4202 blocked for more than 120 seconds. [ 725.998264] Tainted: P O 4.15.0-20-generic #21-Ubuntu [ 726.005071] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 726.013313] INFO: task lxd:99919 blocked for more than 120 seconds. [ 726.019609] Tainted: P O 4.15.0-20-generic #21-Ubuntu [ 726.026418] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 726.034560] INFO: task zfs:100513 blocked for more than 120 seconds. [ 726.040936] Tainted: P O 4.15.0-20-generic #21-Ubuntu [ 726.047746] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 726.055791] INFO: task zfs:100584 blocked for more than 120 seconds. [ 726.062170] Tainted: P O 4.15.0-20-generic #21-Ubuntu [ 726.068979] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Describe how to reproduce the problem Start an Ubuntu 18.04 LTS server. Install LXD if not already installed. sudo apt update sudo apt install lxd lxd-client lxd-tools zfsutils-linux Configure LXD with sudo lxd init. When prompted for the storage backend, select ZFS and specify an empty disk. $ sudo lxd init Would you like to use LXD clustering? (yes/no) [default=no]: Do you want to configure a new storage pool? (yes/no) [default=yes]: Name of the new storage pool [default=default]: Name of the storage backend to use (dir, zfs) [default=zfs]: Create a new ZFS pool? (yes/no) [default=yes]: Would you like to use an existing block device? (yes/no) [default=no]: yes Path to the existing block device: /dev/sdb Would you like to connect to a MAAS server? (yes/no) [default=no]: Would you like to create a new local network bridge? (yes/no) [default=yes]: no Would you like to configure LXD to use an existing bridge or host interface? (yes/no) [default=no]: no Would you like LXD to be available over the network? (yes/no) [default=no]: Would you like stale cached images to be updated automatically? (yes/no) [default=yes] Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: Now run the following to launch 48 containers in batches of 12. lxd-benchmark launch --count 48 --parallel 12 In two out of four attempts, I got the kernel errors. I also tried echo 1 >/sys/module/spl/parameters/spl_taskq_kick but did not manage to continue. Include any warning/errors/backtraces from the system logs dmesg output [ 725.970508] INFO: task lxd:4455 blocked for more than 120 seconds. [ 725.976730] Tainted: P O 4.15.0-20-generic #21-Ubuntu [ 725.983551] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 725.991408] lxd D 0 4455 1 0x00000000 [ 725.991412] Call Trace: [ 725.991424] __schedule+0x297/0x8b0 [ 725.991428] schedule+0x2c/0x80 [ 725.991429] rwsem_down_write_failed+0x162/0x360 [ 725.991460] ? dbuf_rele_and_unlock+0x1a8/0x4b0 [zfs] [ 725.991465] call_rwsem_down_write_failed+0x17/0x30 [ 725.991468] ? call_rwsem_down_write_failed+0x17/0x30 [ 725.991469] down_write+0x2d/0x40 [ 725.991472] grab_super+0x30/0x90 [ 725.991501] ? zpl_create+0x160/0x160 [zfs] [ 725.991504] sget_userns+0x91/0x490 [ 725.991507] ? get_anon_bdev+0x100/0x100 [ 725.991534] ? zpl_create+0x160/0x160 [zfs] [ 725.991537] sget+0x7d/0xa0 [ 725.991540] ? get_anon_bdev+0x100/0x100 [ 725.991567] zpl_mount+0xa8/0x160 [zfs] [ 725.991570] mount_fs+0x37/0x150 [ 725.991574] vfs_kern_mount.part.23+0x5d/0x110 [ 725.991576] do_mount+0x5ed/0xce0 [ 725.991577] ? copy_mount_options+0x2c/0x220 [ 725.991578] SyS_mount+0x98/0xe0 [ 725.991582] do_syscall_64+0x73/0x130 [ 725.991583] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [ 725.991585] RIP: 0033:0x4dbd5a [ 725.991586] RSP: 002b:000000c428be6890 EFLAGS: 00000206 ORIG_RAX: 00000000000000a5 [ 725.991588] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00000000004dbd5a [ 725.991589] RDX: 000000c421a04b7c RSI: 000000c426f94f40 RDI: 000000c4274ceaa0 [ 725.991590] RBP: 000000c428be6930 R08: 000000c425521a90 R09: 0000000000000000 [ 725.991590] R10: 0000000000000000 R11: 0000000000000206 R12: ffffffffffffffff [ 725.991591] R13: 000000000000003e R14: 000000000000003d R15: 0000000000000080 [ 725.991624] INFO: task txg_sync:4202 blocked for more than 120 seconds. [ 725.998264] Tainted: P O 4.15.0-20-generic #21-Ubuntu [ 726.005071] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 726.012928] txg_sync D 0 4202 2 0x80000000 [ 726.012930] Call Trace: [ 726.012933] __schedule+0x297/0x8b0 [ 726.012939] schedule+0x2c/0x80 [ 726.012945] cv_wait_common+0x11e/0x140 [spl] [ 726.012948] ? wait_woken+0x80/0x80 [ 726.012954] __cv_wait+0x15/0x20 [spl] [ 726.012981] rrw_enter_write+0x3c/0xa0 [zfs] [ 726.013006] rrw_enter+0x13/0x20 [zfs] [ 726.013033] spa_sync+0x7c9/0xd80 [zfs] [ 726.013062] txg_sync_thread+0x2cd/0x4a0 [zfs] [ 726.013089] ? txg_quiesce_thread+0x3d0/0x3d0 [zfs] [ 726.013093] thread_generic_wrapper+0x74/0x90 [spl] [ 726.013098] kthread+0x121/0x140 [ 726.013101] ? __thread_exit+0x20/0x20 [spl] [ 726.013103] ? kthread_create_worker_on_cpu+0x70/0x70 [ 726.013104] ret_from_fork+0x22/0x40 [ 726.013313] INFO: task lxd:99919 blocked for more than 120 seconds. [ 726.019609] Tainted: P O 4.15.0-20-generic #21-Ubuntu [ 726.026418] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 726.034272] lxd D 0 99919 99626 0x00000000 [ 726.034274] Call Trace: [ 726.034277] __schedule+0x297/0x8b0 [ 726.034283] ? __wake_up_common+0x73/0x130 [ 726.034286] schedule+0x2c/0x80 [ 726.034290] cv_wait_common+0x11e/0x140 [spl] [ 726.034293] ? wait_woken+0x80/0x80 [ 726.034297] __cv_wait+0x15/0x20 [spl] [ 726.034322] txg_wait_synced+0xdd/0x130 [zfs] [ 726.034349] zil_create+0x249/0x300 [zfs] [ 726.034376] zil_commit_writer+0x6d1/0x7c0 [zfs] [ 726.034401] zil_commit.part.14+0x93/0x100 [zfs] [ 726.034429] zil_commit+0x17/0x20 [zfs] [ 726.034457] zfs_sync+0x6e/0xb0 [zfs] [ 726.034484] zpl_sync_fs+0x50/0x80 [zfs] [ 726.034490] __sync_filesystem+0x33/0x60 [ 726.034493] sync_filesystem+0x39/0x40 [ 726.034495] generic_shutdown_super+0x27/0x120 [ 726.034496] kill_anon_super+0x12/0x20 [ 726.034518] zpl_kill_sb+0x1a/0x20 [zfs] [ 726.034524] deactivate_locked_super+0x48/0x80 [ 726.034529] deactivate_super+0x40/0x60 [ 726.034532] cleanup_mnt+0x3f/0x80 [ 726.034534] __cleanup_mnt+0x12/0x20 [ 726.034535] task_work_run+0x9d/0xc0 [ 726.034537] exit_to_usermode_loop+0xc0/0xd0 [ 726.034539] do_syscall_64+0x115/0x130 [ 726.034542] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [ 726.034550] RIP: 0033:0x7fb553b3e8c7 [ 726.034551] RSP: 002b:00007fff426c5388 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 726.034553] RAX: 0000000000000000 RBX: 000000000000000f RCX: 00007fb553b3e8c7 [ 726.034553] RDX: 00007fb55476eb9f RSI: 0000000000000002 RDI: 00007fb554770b6b [ 726.034554] RBP: 000000000000000c R08: 0000000000000000 R09: 00007fb553b8ae67 [ 726.034555] R10: 0000000000084000 R11: 0000000000000246 R12: 00007fff426c64a0 [ 726.034555] R13: 0000000003176690 R14: 0000000003177810 R15: 00000000031741f0 [ 726.034560] INFO: task zfs:100513 blocked for more than 120 seconds. [ 726.040936] Tainted: P O 4.15.0-20-generic #21-Ubuntu [ 726.047746] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 726.055600] zfs D 0 100513 2626 0x00000000 [ 726.055602] Call Trace: [ 726.055606] __schedule+0x297/0x8b0 [ 726.055609] schedule+0x2c/0x80 [ 726.055613] cv_wait_common+0x11e/0x140 [spl] [ 726.055615] ? wait_woken+0x80/0x80 [ 726.055619] __cv_wait+0x15/0x20 [spl] [ 726.055642] rrw_enter_read_impl+0x4e/0x160 [zfs] [ 726.055666] rrw_enter+0x1c/0x20 [zfs] [ 726.055691] dsl_pool_hold+0x5a/0x80 [zfs] [ 726.055713] dmu_objset_hold+0x33/0xa0 [zfs] [ 726.055740] zfs_ioc_objset_stats+0x32/0xa0 [zfs] [ 726.055766] zfsdev_ioctl+0x1e0/0x610 [zfs] [ 726.055771] do_vfs_ioctl+0xa8/0x630 [ 726.055774] ? handle_mm_fault+0xb1/0x1f0 [ 726.055776] ? __do_page_fault+0x270/0x4d0 [ 726.055777] SyS_ioctl+0x79/0x90 [ 726.055779] do_syscall_64+0x73/0x130 [ 726.055781] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [ 726.055782] RIP: 0033:0x7fd4adc795d7 [ 726.055782] RSP: 002b:00007ffe356b8718 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [ 726.055783] RAX: ffffffffffffffda RBX: 00007ffe356b8740 RCX: 00007fd4adc795d7 [ 726.055784] RDX: 00007ffe356b8740 RSI: 0000000000005a12 RDI: 0000000000000003 [ 726.055785] RBP: 000055632e278660 R08: 000000000000ffff R09: 00007fd4adcd1ed0 [ 726.055785] R10: 2f746c7561666564 R11: 0000000000000246 R12: 000055632e278660 [ 726.055788] R13: 00007ffe356beec0 R14: 00007fd4af1756e0 R15: 00007ffe356bbe30 [ 726.055791] INFO: task zfs:100584 blocked for more than 120 seconds. [ 726.062170] Tainted: P O 4.15.0-20-generic #21-Ubuntu [ 726.068979] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 726.076835] zfs D 0 100584 15513 0x00000000 [ 726.076837] Call Trace: [ 726.076840] __schedule+0x297/0x8b0 [ 726.076845] schedule+0x2c/0x80 [ 726.076851] cv_wait_common+0x11e/0x140 [spl] [ 726.076854] ? wait_woken+0x80/0x80 [ 726.076859] __cv_wait+0x15/0x20 [spl] [ 726.076881] rrw_enter_read_impl+0x4e/0x160 [zfs] [ 726.076905] rrw_enter+0x1c/0x20 [zfs] [ 726.076929] dsl_pool_hold+0x5a/0x80 [zfs] [ 726.076951] dmu_objset_hold+0x33/0xa0 [zfs] [ 726.076977] zfs_ioc_objset_stats+0x32/0xa0 [zfs] [ 726.077001] zfsdev_ioctl+0x1e0/0x610 [zfs] [ 726.077005] do_vfs_ioctl+0xa8/0x630 [ 726.077006] ? handle_mm_fault+0xb1/0x1f0 [ 726.077008] ? __do_page_fault+0x270/0x4d0 [ 726.077010] SyS_ioctl+0x79/0x90 [ 726.077011] do_syscall_64+0x73/0x130 [ 726.077013] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [ 726.077014] RIP: 0033:0x7fc2734075d7 [ 726.077014] RSP: 002b:00007fff653a4b08 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [ 726.077015] RAX: ffffffffffffffda RBX: 00007fff653a4b30 RCX: 00007fc2734075d7 [ 726.077016] RDX: 00007fff653a4b30 RSI: 0000000000005a12 RDI: 0000000000000003 [ 726.077017] RBP: 000055f3576e9660 R08: 000000000000ffff R09: 00007fc27345fed0 [ 726.077017] R10: 2f746c7561666564 R11: 0000000000000246 R12: 000055f3576e9660 [ 726.077018] R13: 00007fff653aaec0 R14: 00007fc2749036e0 R15: 00007fff653a8220 [ 846.801124] INFO: task lxd:4455 blocked for more than 120 seconds. [ 846.807352] Tainted: P O 4.15.0-20-generic #21-Ubuntu [ 846.814170] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 846.822028] lxd D 0 4455 1 0x00000000 [ 846.822031] Call Trace: [ 846.822042] __schedule+0x297/0x8b0 [ 846.822045] schedule+0x2c/0x80 [ 846.822047] rwsem_down_write_failed+0x162/0x360 [ 846.822078] ? dbuf_rele_and_unlock+0x1a8/0x4b0 [zfs] [ 846.822083] call_rwsem_down_write_failed+0x17/0x30 [ 846.822086] ? call_rwsem_down_write_failed+0x17/0x30 [ 846.822087] down_write+0x2d/0x40 [ 846.822091] grab_super+0x30/0x90 [ 846.822118] ? zpl_create+0x160/0x160 [zfs] [ 846.822121] sget_userns+0x91/0x490 [ 846.822123] ? get_anon_bdev+0x100/0x100 [ 846.822150] ? zpl_create+0x160/0x160 [zfs] [ 846.822153] sget+0x7d/0xa0 [ 846.822156] ? get_anon_bdev+0x100/0x100 [ 846.822181] zpl_mount+0xa8/0x160 [zfs] [ 846.822183] mount_fs+0x37/0x150 [ 846.822188] vfs_kern_mount.part.23+0x5d/0x110 [ 846.822189] do_mount+0x5ed/0xce0 [ 846.822190] ? copy_mount_options+0x2c/0x220 [ 846.822192] SyS_mount+0x98/0xe0 [ 846.822195] do_syscall_64+0x73/0x130 [ 846.822196] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [ 846.822198] RIP: 0033:0x4dbd5a [ 846.822199] RSP: 002b:000000c428be6890 EFLAGS: 00000206 ORIG_RAX: 00000000000000a5 [ 846.822201] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00000000004dbd5a [ 846.822203] RDX: 000000c421a04b7c RSI: 000000c426f94f40 RDI: 000000c4274ceaa0 [ 846.822205] RBP: 000000c428be6930 R08: 000000c425521a90 R09: 0000000000000000 [ 846.822206] R10: 0000000000000000 R11: 0000000000000206 R12: ffffffffffffffff [ 846.822206] R13: 000000000000003e R14: 000000000000003d R15: 0000000000000080 [ 846.822239] INFO: task txg_sync:4202 blocked for more than 120 seconds. [ 846.828882] Tainted: P O 4.15.0-20-generic #21-Ubuntu [ 846.835692] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 846.843549] txg_sync D 0 4202 2 0x80000000 [ 846.843551] Call Trace: [ 846.843554] __schedule+0x297/0x8b0 [ 846.843560] schedule+0x2c/0x80 [ 846.843566] cv_wait_common+0x11e/0x140 [spl] [ 846.843570] ? wait_woken+0x80/0x80 [ 846.843574] __cv_wait+0x15/0x20 [spl] [ 846.843603] rrw_enter_write+0x3c/0xa0 [zfs] [ 846.843629] rrw_enter+0x13/0x20 [zfs] [ 846.843654] spa_sync+0x7c9/0xd80 [zfs] [ 846.843682] txg_sync_thread+0x2cd/0x4a0 [zfs] [ 846.843708] ? txg_quiesce_thread+0x3d0/0x3d0 [zfs] [ 846.843713] thread_generic_wrapper+0x74/0x90 [spl] [ 846.843717] kthread+0x121/0x140 [ 846.843720] ? __thread_exit+0x20/0x20 [spl] [ 846.843721] ? kthread_create_worker_on_cpu+0x70/0x70 [ 846.843723] ret_from_fork+0x22/0x40 [ 846.843931] INFO: task lxd:99919 blocked for more than 120 seconds. [ 846.850227] Tainted: P O 4.15.0-20-generic #21-Ubuntu [ 846.857040] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 846.864892] lxd D 0 99919 99626 0x00000000 [ 846.864894] Call Trace: [ 846.864897] __schedule+0x297/0x8b0 [ 846.864903] ? __wake_up_common+0x73/0x130 [ 846.864906] schedule+0x2c/0x80 [ 846.864910] cv_wait_common+0x11e/0x140 [spl] [ 846.864912] ? wait_woken+0x80/0x80 [ 846.864917] __cv_wait+0x15/0x20 [spl] [ 846.864942] txg_wait_synced+0xdd/0x130 [zfs] [ 846.864971] zil_create+0x249/0x300 [zfs] [ 846.864998] zil_commit_writer+0x6d1/0x7c0 [zfs] [ 846.865023] zil_commit.part.14+0x93/0x100 [zfs] [ 846.865051] zil_commit+0x17/0x20 [zfs] [ 846.865080] zfs_sync+0x6e/0xb0 [zfs] [ 846.865107] zpl_sync_fs+0x50/0x80 [zfs] [ 846.865111] __sync_filesystem+0x33/0x60 [ 846.865113] sync_filesystem+0x39/0x40 [ 846.865114] generic_shutdown_super+0x27/0x120 [ 846.865116] kill_anon_super+0x12/0x20 [ 846.865138] zpl_kill_sb+0x1a/0x20 [zfs] [ 846.865140] deactivate_locked_super+0x48/0x80 [ 846.865143] deactivate_super+0x40/0x60 [ 846.865145] cleanup_mnt+0x3f/0x80 [ 846.865147] __cleanup_mnt+0x12/0x20 [ 846.865148] task_work_run+0x9d/0xc0 [ 846.865150] exit_to_usermode_loop+0xc0/0xd0 [ 846.865152] do_syscall_64+0x115/0x130 [ 846.865153] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [ 846.865154] RIP: 0033:0x7fb553b3e8c7 [ 846.865155] RSP: 002b:00007fff426c5388 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 846.865156] RAX: 0000000000000000 RBX: 000000000000000f RCX: 00007fb553b3e8c7 [ 846.865158] RDX: 00007fb55476eb9f RSI: 0000000000000002 RDI: 00007fb554770b6b [ 846.865159] RBP: 000000000000000c R08: 0000000000000000 R09: 00007fb553b8ae67 [ 846.865162] R10: 0000000000084000 R11: 0000000000000246 R12: 00007fff426c64a0 [ 846.865162] R13: 0000000003176690 R14: 0000000003177810 R15: 00000000031741f0 [ 846.865167] INFO: task zfs:100513 blocked for more than 120 seconds. [ 846.871546] Tainted: P O 4.15.0-20-generic #21-Ubuntu [ 846.878357] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 846.886214] zfs D 0 100513 2626 0x00000000 [ 846.886215] Call Trace: [ 846.886218] __schedule+0x297/0x8b0 [ 846.886223] schedule+0x2c/0x80 [ 846.886230] cv_wait_common+0x11e/0x140 [spl] [ 846.886232] ? wait_woken+0x80/0x80 [ 846.886237] __cv_wait+0x15/0x20 [spl] [ 846.886261] rrw_enter_read_impl+0x4e/0x160 [zfs] [ 846.886284] rrw_enter+0x1c/0x20 [zfs] [ 846.886309] dsl_pool_hold+0x5a/0x80 [zfs] [ 846.886331] dmu_objset_hold+0x33/0xa0 [zfs] [ 846.886357] zfs_ioc_objset_stats+0x32/0xa0 [zfs] [ 846.886383] zfsdev_ioctl+0x1e0/0x610 [zfs] [ 846.886388] do_vfs_ioctl+0xa8/0x630 [ 846.886391] ? handle_mm_fault+0xb1/0x1f0 [ 846.886394] ? __do_page_fault+0x270/0x4d0 [ 846.886396] SyS_ioctl+0x79/0x90 [ 846.886397] do_syscall_64+0x73/0x130 [ 846.886399] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [ 846.886400] RIP: 0033:0x7fd4adc795d7 [ 846.886402] RSP: 002b:00007ffe356b8718 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [ 846.886405] RAX: ffffffffffffffda RBX: 00007ffe356b8740 RCX: 00007fd4adc795d7 [ 846.886406] RDX: 00007ffe356b8740 RSI: 0000000000005a12 RDI: 0000000000000003 [ 846.886406] RBP: 000055632e278660 R08: 000000000000ffff R09: 00007fd4adcd1ed0 [ 846.886407] R10: 2f746c7561666564 R11: 0000000000000246 R12: 000055632e278660 [ 846.886408] R13: 00007ffe356beec0 R14: 00007fd4af1756e0 R15: 00007ffe356bbe30 [ 846.886410] INFO: task zfs:100584 blocked for more than 120 seconds. [ 846.892790] Tainted: P O 4.15.0-20-generic #21-Ubuntu [ 846.899598] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 846.907454] zfs D 0 100584 15513 0x00000000 [ 846.907456] Call Trace: [ 846.907459] __schedule+0x297/0x8b0 [ 846.907465] schedule+0x2c/0x80 [ 846.907470] cv_wait_common+0x11e/0x140 [spl] [ 846.907471] ? wait_woken+0x80/0x80 [ 846.907477] __cv_wait+0x15/0x20 [spl] [ 846.907499] rrw_enter_read_impl+0x4e/0x160 [zfs] [ 846.907523] rrw_enter+0x1c/0x20 [zfs] [ 846.907547] dsl_pool_hold+0x5a/0x80 [zfs] [ 846.907568] dmu_objset_hold+0x33/0xa0 [zfs] [ 846.907594] zfs_ioc_objset_stats+0x32/0xa0 [zfs] [ 846.907618] zfsdev_ioctl+0x1e0/0x610 [zfs] [ 846.907622] do_vfs_ioctl+0xa8/0x630 [ 846.907624] ? handle_mm_fault+0xb1/0x1f0 [ 846.907625] ? __do_page_fault+0x270/0x4d0 [ 846.907627] SyS_ioctl+0x79/0x90 [ 846.907628] do_syscall_64+0x73/0x130 [ 846.907630] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [ 846.907631] RIP: 0033:0x7fc2734075d7 [ 846.907631] RSP: 002b:00007fff653a4b08 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [ 846.907633] RAX: ffffffffffffffda RBX: 00007fff653a4b30 RCX: 00007fc2734075d7 [ 846.907633] RDX: 00007fff653a4b30 RSI: 0000000000005a12 RDI: 0000000000000003 [ 846.907634] RBP: 000055f3576e9660 R08: 000000000000ffff R09: 00007fc27345fed0 [ 846.907635] R10: 2f746c7561666564 R11: 0000000000000246 R12: 000055f3576e9660 [ 846.907637] R13: 00007fff653aaec0 R14: 00007fc2749036e0 R15: 00007fff653a8220 Contents of "/proc/spl/kstat/zfs/arcstats" 13 1 0x01 96 4608 11300672527 1808059980062 name type data hits 4 44186496 misses 4 1247761 demand_data_hits 4 3327097 demand_data_misses 4 17953 demand_metadata_hits 4 39832270 demand_metadata_misses 4 1076950 prefetch_data_hits 4 1357 prefetch_data_misses 4 23 prefetch_metadata_hits 4 1025772 prefetch_metadata_misses 4 152835 mru_hits 4 18193851 mru_ghost_hits 4 0 mfu_hits 4 24976976 mfu_ghost_hits 4 0 deleted 4 10 mutex_miss 4 0 access_skip 4 68 evict_skip 4 1 evict_not_enough 4 0 evict_l2_cached 4 0 evict_l2_eligible 4 101376 evict_l2_ineligible 4 2048 evict_l2_skip 4 0 hash_elements 4 38683 hash_elements_max 4 57741 hash_collisions 4 1520 hash_chains 4 5 hash_chain_max 4 1 p 4 16815604736 c 4 33631209472 c_min 4 2101950592 c_max 4 33631209472 size 4 833199872 compressed_size 4 286609408 uncompressed_size 4 831992320 overhead_size 4 330568192 hdr_size 4 14660144 data_size 4 193579520 metadata_size 4 423598080 dbuf_size 4 46065424 dnode_size 4 112384064 bonus_size 4 42912640 anon_size 4 36502016 anon_evictable_data 4 0 anon_evictable_metadata 4 0 mru_size 4 310932480 mru_evictable_data 4 27623424 mru_evictable_metadata 4 136801792 mru_ghost_size 4 0 mru_ghost_evictable_data 4 0 mru_ghost_evictable_metadata 4 0 mfu_size 4 269743104 mfu_evictable_data 4 53612032 mfu_evictable_metadata 4 3195392 mfu_ghost_size 4 0 mfu_ghost_evictable_data 4 0 mfu_ghost_evictable_metadata 4 0 l2_hits 4 0 l2_misses 4 0 l2_feeds 4 0 l2_rw_clash 4 0 l2_read_bytes 4 0 l2_write_bytes 4 0 l2_writes_sent 4 0 l2_writes_done 4 0 l2_writes_error 4 0 l2_writes_lock_retry 4 0 l2_evict_lock_retry 4 0 l2_evict_reading 4 0 l2_evict_l1cached 4 0 l2_free_on_write 4 0 l2_abort_lowmem 4 0 l2_cksum_bad 4 0 l2_io_error 4 0 l2_size 4 0 l2_asize 4 0 l2_hdr_size 4 0 memory_throttle_count 4 0 memory_direct_count 4 0 memory_indirect_count 4 0 memory_all_bytes 4 67262418944 memory_free_bytes 4 57157578752 memory_available_bytes 3 56106606592 arc_no_grow 4 0 arc_tempreserve 4 0 arc_loaned_bytes 4 0 arc_prune 4 0 arc_meta_used 4 639620352 arc_meta_limit 4 25223407104 arc_dnode_limit 4 2522340710 arc_meta_max 4 965175896 arc_meta_min 4 16777216 sync_wait_for_async 4 168 demand_hit_predictive_prefetch 4 999 arc_need_free 4 0 arc_sys_free 4 1050975296 Command "slabtop -o" Active / Total Objects (% used) : 28354235 / 29140626 (97.3%) Active / Total Slabs (% used) : 382017 / 382017 (100.0%) Active / Total Caches (% used) : 95 / 128 (74.2%) Active / Total Size (% used) : 4580408.56K / 4743868.00K (96.6%) Minimum / Average / Maximum Object : 0.01K / 0.16K / 21.81K OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 17206144 17204917 0% 0.03K 134423 128 537692K kmalloc-32 1141392 966790 0% 0.19K 27176 42 217408K dentry 1114496 1004537 0% 0.06K 17414 64 69656K kmalloc-64 1088192 1078186 0% 0.50K 17003 64 544096K kmalloc-512 1038300 714247 0% 0.13K 17305 60 138440K kernfs_node_cache 937536 931296 0% 0.25K 14649 64 234384K filp 684160 682244 0% 0.06K 10690 64 42760K pid 582099 569877 0% 0.59K 10983 53 351456K inode_cache 520104 518529 0% 0.20K 13336 39 106688K vm_area_struct 404334 388371 0% 0.09K 9627 42 38508K kmalloc-96 342286 341596 0% 0.09K 7441 46 29764K anon_vma 342016 338948 0% 0.25K 5344 64 85504K kmalloc-256 277032 276822 0% 0.19K 6596 42 52768K cred_jar 248352 241634 0% 0.66K 5174 48 165568K proc_inode_cache 248320 233052 0% 0.01K 485 512 1940K kmalloc-8 214984 143177 0% 0.57K 3839 56 122848K radix_tree_node To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/1781364/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp