As suggested privately by riastradh@, I ran the tar command under dtrace
dtrace -n 'profile:::profile-97 { @[stack()] = count() }' -c ...
and got output I can't make sense of. I attach the last (top-count) lines.
The first thing I don't understand is why the top (or the two top) hits
are in idle_loop. Is that because the other fives cores are idle?
The second question is what the single number without a backtrace means.
And the third question is that I would have expected a large amount of
hits in the write system call if the problem really was stdio to issue
small writes.
netbsd`0xffffffff8021ff62
netbsd`intr_biglock_wrapper+0x16
netbsd`Xhandle_ioapic_level5+0x6a
netbsd`acpicpu_cstate_idle_enter+0xdb
netbsd`acpicpu_cstate_idle+0xdb
netbsd`idle_loop+0x111
netbsd`lwp_trampoline+0x17
287
netbsd`0xffffffff8022008f
netbsd`AcpiHwReadPort+0xc1
netbsd`AcpiHwRead+0x11f
netbsd`acpitimer_read_fast+0x11
netbsd`acpicpu_cstate_idle_enter+0x82
netbsd`acpicpu_cstate_idle+0xdb
netbsd`idle_loop+0x111
netbsd`lwp_trampoline+0x17
292
netbsd`0xffffffff8021ff62
netbsd`intr_biglock_wrapper+0x16
netbsd`Xhandle_ioapic_level5+0x6a
netbsd`biodone2+0x6d
netbsd`biointr+0x33
netbsd`softint_dispatch+0x91
netbsd`Xsoftintr+0x4f
297
netbsd`0xffffffff8022008f
netbsd`AcpiHwReadPort+0xc1
netbsd`AcpiHwRead+0x11f
netbsd`acpitimer_read_fast+0x11
netbsd`acpicpu_cstate_idle_enter+0x1c
netbsd`acpicpu_cstate_idle+0xdb
netbsd`idle_loop+0x111
netbsd`lwp_trampoline+0x17
314
netbsd`ufs_lookup+0x2f7
netbsd`VOP_LOOKUP+0x4d
netbsd`lookup_once+0x18d
netbsd`namei_tryemulroot.constprop.9+0x3c5
netbsd`namei+0x43
netbsd`fd_nameiat.isra.2+0x54
netbsd`do_sys_renameat.isra.6+0x155
netbsd`syscall+0x1ec
netbsd`handle_syscall+0x2d
431
netbsd`ufs_lookup+0x2f7
netbsd`VOP_LOOKUP+0x4d
netbsd`relookup+0x42
netbsd`do_sys_renameat.isra.6+0x2ef
netbsd`syscall+0x1ec
netbsd`handle_syscall+0x2d
436
netbsd`ufs_lookup+0x2f7
netbsd`VOP_LOOKUP+0x4d
netbsd`relookup+0x42
netbsd`ufs_gro_lookup+0x3c
netbsd`genfs_sane_rename+0x1e7
netbsd`ufs_sane_rename+0x3a
netbsd`genfs_insane_rename+0x93
netbsd`VOP_RENAME+0x7c
netbsd`do_sys_renameat.isra.6+0x4d1
netbsd`syscall+0x1ec
netbsd`handle_syscall+0x2d
443
netbsd`ufs_lookup+0x2f7
netbsd`VOP_LOOKUP+0x4d
netbsd`lookup_once+0x18d
netbsd`namei_tryemulroot.constprop.9+0x3c5
netbsd`namei+0x43
netbsd`vn_open+0xa3
netbsd`do_open+0x112
netbsd`do_sys_openat+0x68
netbsd`sys_open+0x24
netbsd`syscall+0x1ec
netbsd`handle_syscall+0x2d
486
netbsd`ufs_lookup+0x2f7
netbsd`VOP_LOOKUP+0x4d
netbsd`relookup+0x42
netbsd`ufs_gro_lookup+0x3c
netbsd`genfs_sane_rename+0x1a5
netbsd`ufs_sane_rename+0x3a
netbsd`genfs_insane_rename+0x93
netbsd`VOP_RENAME+0x7c
netbsd`do_sys_renameat.isra.6+0x4d1
netbsd`syscall+0x1ec
netbsd`handle_syscall+0x2d
512
netbsd`ufs_lookup+0x2f7
netbsd`VOP_LOOKUP+0x4d
netbsd`lookup_once+0x18d
netbsd`namei_tryemulroot.constprop.9+0x3c5
netbsd`namei+0x43
netbsd`fd_nameiat.isra.2+0x54
netbsd`do_sys_renameat.isra.6+0xc3
netbsd`syscall+0x1ec
netbsd`handle_syscall+0x2d
556
netbsd`0xffffffff8021ff62
netbsd`biodone2+0x6d
netbsd`biointr+0x33
netbsd`softint_dispatch+0x91
netbsd`Xsoftintr+0x4f
560
611
netbsd`0xffffffff8021ff3e
netbsd`acpicpu_cstate_idle_enter+0xdb
netbsd`acpicpu_cstate_idle+0xdb
netbsd`idle_loop+0x111
netbsd`lwp_trampoline+0x17
86750
netbsd`Xspllower+0xe
netbsd`md_root_setconf
203
netbsd`uvm_map_space_avail.constprop.0+0x5c
netbsd`uvm_map+0xd8
netbsd`uvm_pagermapin+0x97
netbsd`genfs_gop_write+0x2d
netbsd`genfs_do_putpages+0x661
netbsd`VOP_PUTPAGES+0x43
netbsd`ffs_full_fsync+0x146
netbsd`VOP_FSYNC+0x47
netbsd`sched_sync+0x15b
netbsd`lwp_trampoline+0x17
362
netbsd`_kernel_lock+0xca
netbsd`cv_wait+0xb7
netbsd`DAGExecutionThread+0x136
netbsd`lwp_trampoline+0x17
507
netbsd`_kernel_lock+0xca
netbsd`spec_strategy+0x58
netbsd`VOP_STRATEGY+0x3c
netbsd`dkstart+0x13e
netbsd`bdev_strategy+0x81
netbsd`spec_strategy+0x58
netbsd`VOP_STRATEGY+0x3c
netbsd`genfs_do_io+0x16a
netbsd`genfs_gop_write+0x52
netbsd`genfs_do_putpages+0x661
netbsd`VOP_PUTPAGES+0x43
netbsd`ffs_full_fsync+0x146
netbsd`VOP_FSYNC+0x47
netbsd`sched_sync+0x15b
netbsd`lwp_trampoline+0x17
926
netbsd`_kernel_lock+0xca
netbsd`cv_wait+0xb7
netbsd`rf_RaidIOThread+0x14e
netbsd`lwp_trampoline+0x17
1691
2124
netbsd`x86_stihlt+0x6
netbsd`idle_loop+0xb7
netbsd`lwp_trampoline+0x17
3894
netbsd`x86_stihlt+0x6
netbsd`idle_loop+0xb7
netbsd`md_root_setconf
25933
