freebsd-update 12.3 to 14.0RC1 takes 12-24 hours (block cloning regression)

2023-10-17 Thread Kevin Bowling
Hi,

I have two systems with a zpool 2x2 mirror on 7.2k RPM disks.  One
system also has a flash SLOG.

The flash SLOG system took around 12 hours to complete freebsd-update
from 13.2 to 14.0-RC1.  The system without the SLOG took nearly 24
hours.  This was the result of ~50k patches, and ~10k files from
freebsd-update and a very pathological 'install' command performance.

'ps auxww | grep install':
root   52225   0.0  0.0  12852   2504  0  D+   20:55  0:00.00
install -S -o 0 -g 0 -m 0644
b6850914127c27fe192a41387f5cec04a1d927e6605ff09e8fd88dcd74fdec9d
///usr/src/sys/netgraph/ng_vlan.h
root   68042   0.0  0.0  13580   3648  0  I+   02:24  0:01.14
/bin/sh /usr/sbin/freebsd-update install  root   69946
0.0  0.0  13580   3632  0  S+   02:24  0:15.65 /bin/sh
/usr/sbin/freebsd-update install

'control+t on freebsd-update':

load: 0.16  cmd: install 97128 [tx->tx_sync_done_cv] 0.67r 0.00u 0.00s
0% 2440k
mi_switch+0xc2 _cv_wait+0x113 txg_wait_synced_impl+0xb9
txg_wait_synced+0xb dmu_offset_next+0x77 zfs_holey+0x137 zfs_fre
ebsd_ioctl+0x4f vn_generic_copy_file_range+0x64b
kern_copy_file_range+0x327 sys_copy_file_range+0x78
amd64_syscall+0x10c
 fast_syscall_common+0xf8

I spoke with mjg about this and because my pools do not have block
cloning enabled, copy_file_range turns into a massive pessimization in
'install'.  He suggested a workaround of 'sysctl
vfs.zfs.dmu_offset_next_sync=0' but we should probably sort this out
for 14.0-RELEASE.

Regards,
Kevin



RE: freebsd-update 12.3 to 14.0RC1 takes 12-24 hours (block cloning regression)

2023-10-17 Thread Mark Millard
Kevin Bowling  wrote on
Date: Tue, 17 Oct 2023 16:40:37 UTC :

> I have two systems with a zpool 2x2 mirror on 7.2k RPM disks. One
> system also has a flash SLOG.
> 
> The flash SLOG system took around 12 hours to complete freebsd-update
> from 13.2 to 14.0-RC1. The system without the SLOG took nearly 24
> hours. This was the result of ~50k patches, and ~10k files from
> freebsd-update and a very pathological 'install' command performance.
> 
> 'ps auxww | grep install':
> root 52225 0.0 0.0 12852 2504 0 D+ 20:55 0:00.00
> install -S -o 0 -g 0 -m 0644
> b6850914127c27fe192a41387f5cec04a1d927e6605ff09e8fd88dcd74fdec9d
> ///usr/src/sys/netgraph/ng_vlan.h
> root 68042 0.0 0.0 13580 3648 0 I+ 02:24 0:01.14
> /bin/sh /usr/sbin/freebsd-update install root 69946
> 0.0 0.0 13580 3632 0 S+ 02:24 0:15.65 /bin/sh
> /usr/sbin/freebsd-update install
> 
> 'control+t on freebsd-update':
> 
> load: 0.16 cmd: install 97128 [tx->tx_sync_done_cv] 0.67r 0.00u 0.00s
> 0% 2440k
> mi_switch+0xc2 _cv_wait+0x113 txg_wait_synced_impl+0xb9
> txg_wait_synced+0xb dmu_offset_next+0x77 zfs_holey+0x137 zfs_fre
> ebsd_ioctl+0x4f vn_generic_copy_file_range+0x64b
> kern_copy_file_range+0x327 sys_copy_file_range+0x78
> amd64_syscall+0x10c
> fast_syscall_common+0xf8
> 
> I spoke with mjg about this and because my pools do not have block
> cloning enabled, copy_file_range turns into a massive pessimization in
> 'install'.

Block cloning is new. So the past is sort of like Block cloning not being
enabled now.

This leads me to wonder: prior to block cloning existing, what would
analogous times have been like instead of 12 hrs and 24 hrs? (Not that
analogous would be easy to identify in history or test now.)

Depending on the results, my next question might have been: "What
happened for block cloning being disabled now to make it worse than
before block cloning existed?"

> He suggested a workaround of 'sysctl
> vfs.zfs.dmu_offset_next_sync=0' but we should probably sort this out
> for 14.0-RELEASE.

===
Mark Millard
marklmi at yahoo.com