Hi folks, On 12/13/18 11:49 PM, Igor Cicimov wrote:
On Fri, Dec 14, 2018 at 2:57 AM Lars Ellenberg <[email protected] <mailto:[email protected]>> wrote:Unlikely to have anything to do with DRBD. since you apparently can reproduce, monitor grep -e Dirty -e Writeback /proc/meminfo and slabtop before/during/after umount. Also check sysctl settings sysctl vm | grep dirty
Attached. Hope this helps.
Good point, people running servers with huge amount of ram should understand there is also a huge amount of cache that needs to get flushed to the device before it gets removed.
I agree that the problem might be unrelated to drbd, but isn't sync supposed to flush page cache to the block device? The sample test I ran took 47 secs *after* the sync; see output.txt. sync itself took just a few millisecs. Is sync broken for drbd? Regards Harri
# mount | grep /data1 /dev/drbd1 on /data1 type ext4 (rw,noatime,stripe=256) # grep -e Dirty -e Writeback /proc/meminfo Dirty: 7488 kB Writeback: 0 kB WritebackTmp: 0 kB # sysctl vm | grep dirty vm.dirty_background_bytes = 0 vm.dirty_background_ratio = 10 vm.dirty_bytes = 0 vm.dirty_expire_centisecs = 3000 vm.dirty_ratio = 20 vm.dirty_writeback_centisecs = 500 vm.dirtytime_expire_seconds = 43200 # slabtop -o Active / Total Objects (% used) : 72522435 / 72526157 (100.0%) Active / Total Slabs (% used) : 1884685 / 1884685 (100.0%) Active / Total Caches (% used) : 97 / 147 (66.0%) Active / Total Size (% used) : 28241385.36K / 28242758.19K (100.0%) Minimum / Average / Maximum Object : 0.01K / 0.39K / 23.00K OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 20200656 20200648 0% 0.19K 480968 42 3847744K dentry 20089200 20089140 0% 1.06K 669640 30 21428480K ext4_inode_cache 13065156 13065156 100% 0.10K 335004 39 1340016K buffer_head 10687866 10687826 0% 0.09K 254473 42 1017892K kmalloc-96 6182976 6182913 0% 0.06K 96609 64 386436K kmalloc-64 829260 828753 0% 0.04K 8130 102 32520K ext4_extent_status 362816 362816 100% 0.12K 11338 32 45352K kmalloc-128 287140 287055 0% 0.57K 10255 28 164080K radix_tree_node 125776 125776 100% 0.07K 2246 56 8984K Acpi-Operand 90972 90972 100% 0.14K 3249 28 12996K ext4_groupinfo_4k 75310 75310 100% 0.05K 886 85 3544K ftrace_event_field 72450 72450 100% 0.13K 2415 30 9660K kernfs_node_cache 38912 38912 100% 0.06K 608 64 2432K pid 33920 33802 0% 0.03K 265 128 1060K kmalloc-32 28160 28160 100% 0.01K 55 512 220K kmalloc-8 27948 27948 100% 0.04K 274 102 1096K Acpi-Namespace 26082 26082 100% 0.09K 567 46 2268K anon_vma 25545 25545 100% 0.20K 655 39 5240K vm_area_struct 21504 21504 100% 0.02K 84 256 336K kmalloc-16 18285 18285 100% 0.59K 345 53 11040K inode_cache 18250 17520 0% 0.05K 250 73 1000K mbcache 16290 16290 100% 0.13K 543 30 2172K kvm_async_pf 16002 16002 100% 0.19K 381 42 3048K kmalloc-192 15680 15247 0% 0.25K 490 32 3920K filp 12624 12624 100% 0.66K 263 48 8416K proc_inode_cache 11648 11616 0% 0.50K 364 32 5824K kmalloc-512 11556 11360 0% 0.22K 321 36 2568K drbd_req 10336 10058 0% 0.25K 323 32 2584K pool_workqueue 10038 10038 100% 0.19K 239 42 1912K cred_jar 7888 7752 0% 0.12K 232 34 928K jbd2_journal_head 5920 5920 100% 0.12K 185 32 740K scsi_sense_cache 4992 4352 0% 0.06K 78 64 312K ext4_io_end 4896 4896 100% 0.04K 48 102 192K pde_opener 4448 4321 0% 1.00K 139 32 4448K kmalloc-1024 4224 4224 100% 0.03K 33 128 132K fscrypt_info 4080 4080 100% 0.05K 48 85 192K fscrypt_ctx 4048 4048 100% 0.70K 88 46 2816K shmem_inode_cache 3680 3680 100% 0.25K 115 32 920K skbuff_head_cache 3431 3431 100% 0.05K 47 73 188K Acpi-Parse 2890 2890 100% 0.02K 17 170 68K numa_policy 2688 2624 0% 0.50K 84 32 1344K skbuff_fclone_cache 2688 2688 100% 0.07K 48 56 192K eventpoll_pwq 2624 2624 100% 0.06K 41 64 164K dmaengine-unmap-2 2528 2528 100% 0.25K 79 32 632K proc_dir_entry 2499 2499 100% 0.08K 49 51 196K Acpi-State 2448 2448 100% 0.16K 48 51 384K sigqueue 2432 2432 100% 1.00K 76 32 2432K signal_cache 2397 2397 100% 0.62K 47 51 1504K task_group 2346 2346 100% 0.69K 51 46 1632K files_cache 2346 2346 100% 0.69K 51 46 1632K sock_inode_cache 2208 2208 100% 2.00K 138 16 4416K kmalloc-2048 2176 2176 100% 0.25K 68 32 544K kmalloc-256 2142 2142 100% 0.19K 51 42 408K dmaengine-unmap-16 2080 2080 100% 0.12K 65 32 260K seq_file 1886 1886 100% 0.09K 41 46 164K trace_event_file 1880 1880 100% 0.20K 47 40 376K file_lock_cache 1872 1872 100% 0.10K 48 39 192K blkdev_ioc 1728 1728 100% 0.12K 54 32 216K secpath_cache 1590 1590 100% 1.06K 53 30 1696K mm_struct 1536 1536 100% 0.25K 48 32 384K dquot 1536 1536 100% 0.12K 48 32 192K ext4_allocation_context # time sync real 0m0.014s user 0m0.002s sys 0m0.000s # grep -e Dirty -e Writeback /proc/meminfo Dirty: 820 kB Writeback: 0 kB WritebackTmp: 0 kB # sysctl vm | grep dirty vm.dirty_background_bytes = 0 vm.dirty_background_ratio = 10 vm.dirty_bytes = 0 vm.dirty_expire_centisecs = 3000 vm.dirty_ratio = 20 vm.dirty_writeback_centisecs = 500 vm.dirtytime_expire_seconds = 43200 # slabtop -o Active / Total Objects (% used) : 72522200 / 72525840 (100.0%) Active / Total Slabs (% used) : 1884676 / 1884676 (100.0%) Active / Total Caches (% used) : 97 / 147 (66.0%) Active / Total Size (% used) : 28241349.16K / 28242684.23K (100.0%) Minimum / Average / Maximum Object : 0.01K / 0.39K / 23.00K OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 20200656 20200648 0% 0.19K 480968 42 3847744K dentry 20089200 20089140 0% 1.06K 669640 30 21428480K ext4_inode_cache 13065156 13065156 100% 0.10K 335004 39 1340016K buffer_head 10687866 10687826 0% 0.09K 254473 42 1017892K kmalloc-96 6182976 6182913 0% 0.06K 96609 64 386436K kmalloc-64 829260 828753 0% 0.04K 8130 102 32520K ext4_extent_status 362816 362816 100% 0.12K 11338 32 45352K kmalloc-128 287140 287055 0% 0.57K 10255 28 164080K radix_tree_node 125776 125776 100% 0.07K 2246 56 8984K Acpi-Operand 90972 90972 100% 0.14K 3249 28 12996K ext4_groupinfo_4k 75310 75310 100% 0.05K 886 85 3544K ftrace_event_field 72450 72450 100% 0.13K 2415 30 9660K kernfs_node_cache 38976 38976 100% 0.06K 609 64 2436K pid 33920 33802 0% 0.03K 265 128 1060K kmalloc-32 28160 28160 100% 0.01K 55 512 220K kmalloc-8 27948 27948 100% 0.04K 274 102 1096K Acpi-Namespace 26082 26082 100% 0.09K 567 46 2268K anon_vma 25584 25584 100% 0.20K 656 39 5248K vm_area_struct 21504 21504 100% 0.02K 84 256 336K kmalloc-16 18285 18285 100% 0.59K 345 53 11040K inode_cache 18250 17520 0% 0.05K 250 73 1000K mbcache 16290 16290 100% 0.13K 543 30 2172K kvm_async_pf 16002 16002 100% 0.19K 381 42 3048K kmalloc-192 15680 15247 0% 0.25K 490 32 3920K filp 12624 12624 100% 0.66K 263 48 8416K proc_inode_cache 11648 11616 0% 0.50K 364 32 5824K kmalloc-512 10944 10640 0% 0.22K 304 36 2432K drbd_req 10528 10372 0% 0.25K 329 32 2632K pool_workqueue 10038 10038 100% 0.19K 239 42 1912K cred_jar 7888 7752 0% 0.12K 232 34 928K jbd2_journal_head 5920 5920 100% 0.12K 185 32 740K scsi_sense_cache 4992 4416 0% 0.06K 78 64 312K ext4_io_end 4896 4896 100% 0.04K 48 102 192K pde_opener 4448 4321 0% 1.00K 139 32 4448K kmalloc-1024 4224 4224 100% 0.03K 33 128 132K fscrypt_info 4080 4080 100% 0.05K 48 85 192K fscrypt_ctx 4048 4048 100% 0.70K 88 46 2816K shmem_inode_cache 3680 3680 100% 0.25K 115 32 920K skbuff_head_cache 3431 3431 100% 0.05K 47 73 188K Acpi-Parse 2890 2890 100% 0.02K 17 170 68K numa_policy 2688 2624 0% 0.50K 84 32 1344K skbuff_fclone_cache 2688 2688 100% 0.07K 48 56 192K eventpoll_pwq 2624 2624 100% 0.06K 41 64 164K dmaengine-unmap-2 2528 2528 100% 0.25K 79 32 632K proc_dir_entry 2499 2499 100% 0.08K 49 51 196K Acpi-State 2448 2448 100% 0.16K 48 51 384K sigqueue 2432 2432 100% 1.00K 76 32 2432K signal_cache 2397 2397 100% 0.62K 47 51 1504K task_group 2346 2346 100% 0.69K 51 46 1632K files_cache 2346 2346 100% 0.69K 51 46 1632K sock_inode_cache 2208 2208 100% 2.00K 138 16 4416K kmalloc-2048 2176 2176 100% 0.25K 68 32 544K kmalloc-256 2142 2142 100% 0.19K 51 42 408K dmaengine-unmap-16 2080 2080 100% 0.12K 65 32 260K seq_file 1886 1886 100% 0.09K 41 46 164K trace_event_file 1880 1880 100% 0.20K 47 40 376K file_lock_cache 1872 1872 100% 0.10K 48 39 192K blkdev_ioc 1728 1728 100% 0.12K 54 32 216K secpath_cache 1590 1590 100% 1.06K 53 30 1696K mm_struct 1536 1536 100% 0.25K 48 32 384K dquot 1536 1536 100% 0.12K 48 32 192K ext4_allocation_context # time umount /data1 real 0m47.821s user 0m0.000s sys 0m47.793s # grep -e Dirty -e Writeback /proc/meminfo Dirty: 12 kB Writeback: 0 kB WritebackTmp: 0 kB # sysctl vm | grep dirty vm.dirty_background_bytes = 0 vm.dirty_background_ratio = 10 vm.dirty_bytes = 0 vm.dirty_expire_centisecs = 3000 vm.dirty_ratio = 20 vm.dirty_writeback_centisecs = 500 vm.dirtytime_expire_seconds = 43200 # slabtop -o Active / Total Objects (% used) : 1153666 / 1203726 (95.8%) Active / Total Slabs (% used) : 26750 / 26750 (100.0%) Active / Total Caches (% used) : 97 / 147 (66.0%) Active / Total Size (% used) : 292668.49K / 304711.85K (96.0%) Minimum / Average / Maximum Object : 0.01K / 0.25K / 23.00K OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 133014 120791 0% 0.19K 3167 42 25336K dentry 125776 125776 100% 0.07K 2246 56 8984K Acpi-Operand 106692 98365 0% 0.04K 1046 102 4184K ext4_extent_status 87360 85658 0% 1.06K 2912 30 93184K ext4_inode_cache 72450 72450 100% 0.13K 2415 30 9660K kernfs_node_cache 61035 56132 0% 0.10K 1565 39 6260K buffer_head 57664 53163 0% 0.06K 901 64 3604K kmalloc-64 56532 47839 0% 0.57K 2019 28 32304K radix_tree_node 39872 39872 100% 0.06K 623 64 2492K pid 33920 33800 0% 0.03K 265 128 1060K kmalloc-32 28160 28160 100% 0.01K 55 512 220K kmalloc-8 26542 26542 100% 0.09K 577 46 2308K anon_vma 26208 26208 100% 0.20K 672 39 5376K vm_area_struct 25398 24971 0% 0.04K 249 102 996K Acpi-Namespace 21504 21504 100% 0.02K 84 256 336K kmalloc-16 19295 17659 0% 0.05K 227 85 908K ftrace_event_field 18285 18285 100% 0.59K 345 53 11040K inode_cache 18250 17520 0% 0.05K 250 73 1000K mbcache 17094 13482 0% 0.09K 407 42 1628K kmalloc-96 16290 16290 100% 0.13K 543 30 2172K kvm_async_pf 15680 15247 0% 0.25K 490 32 3920K filp 14910 14813 0% 0.19K 355 42 2840K kmalloc-192 12624 12624 100% 0.66K 263 48 8416K proc_inode_cache 10944 10640 0% 0.22K 304 36 2432K drbd_req 10688 10399 0% 0.50K 334 32 5344K kmalloc-512 10528 9893 0% 0.12K 329 32 1316K kmalloc-128 10528 10372 0% 0.25K 329 32 2632K pool_workqueue 10038 10038 100% 0.19K 239 42 1912K cred_jar 7888 7752 0% 0.12K 232 34 928K jbd2_journal_head 5920 5920 100% 0.12K 185 32 740K scsi_sense_cache 4992 4416 0% 0.06K 78 64 312K ext4_io_end 4896 4896 100% 0.04K 48 102 192K pde_opener 4448 4321 0% 1.00K 139 32 4448K kmalloc-1024 4224 4224 100% 0.03K 33 128 132K fscrypt_info 4080 4080 100% 0.05K 48 85 192K fscrypt_ctx 4048 4048 100% 0.70K 88 46 2816K shmem_inode_cache 3680 3680 100% 0.25K 115 32 920K skbuff_head_cache 3431 3431 100% 0.05K 47 73 188K Acpi-Parse 2890 2890 100% 0.02K 17 170 68K numa_policy 2688 2624 0% 0.50K 84 32 1344K skbuff_fclone_cache 2688 2688 100% 0.07K 48 56 192K eventpoll_pwq 2624 2624 100% 0.06K 41 64 164K dmaengine-unmap-2 2604 2437 0% 0.14K 93 28 372K ext4_groupinfo_4k 2528 2528 100% 0.25K 79 32 632K proc_dir_entry 2499 2499 100% 0.08K 49 51 196K Acpi-State 2448 2448 100% 0.16K 48 51 384K sigqueue 2432 2432 100% 1.00K 76 32 2432K signal_cache 2397 2397 100% 0.62K 47 51 1504K task_group 2346 2346 100% 0.69K 51 46 1632K files_cache 2346 2346 100% 0.69K 51 46 1632K sock_inode_cache 2208 2208 100% 2.00K 138 16 4416K kmalloc-2048 2184 2184 100% 0.19K 52 42 416K dmaengine-unmap-16 2176 2176 100% 0.25K 68 32 544K kmalloc-256 2080 2080 100% 0.12K 65 32 260K seq_file 1886 1886 100% 0.09K 41 46 164K trace_event_file 1880 1880 100% 0.20K 47 40 376K file_lock_cache 1872 1872 100% 0.10K 48 39 192K blkdev_ioc 1728 1728 100% 0.12K 54 32 216K secpath_cache 1590 1590 100% 1.06K 53 30 1696K mm_struct 1536 1536 100% 0.25K 48 32 384K dquot 1536 1536 100% 0.12K 48 32 192K ext4_allocation_context
_______________________________________________ drbd-user mailing list [email protected] http://lists.linbit.com/mailman/listinfo/drbd-user
