On 06.07.20 20:21, Jan Bramkamp wrote:
On 03.07.20 21:30, Alan Somers wrote:
I'm using geli, gmultipath, and ZFS on a large system, with hundreds of
drives. What I'm seeing is that under at least some workloads, the
overall
performance is limited by the single geom kernel process. procstat and
kgdb aren't much help in telling exactly why this process is using so
much
CPU, but it certainly must be related to the fact that over 15,000
IOPs are
going through that thread. What can I do to improve this situation?
Would
it make sense to enable direct dispatch for geli? That would hurt
single-threaded performance, but probably improve performance for highly
multithreaded workloads like mine.
Example top output:
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
13 root -8 - 0B 96K CPU46 46 82.7H 70.54%
geom{g_down}
13 root -8 - 0B 96K - 9 35.5H 25.32%
geom{g_up}
-Alan
The problem isn't GELI. It's the problem is that gmultipath lacks
direct dispatch support. Last one and a half years ago I ran into the
same problem. Because I needed the performance I looked at what
gmultipath did and found now reason why it has run in the GEOM up and
down threads. So i patched in the flags claiming direct dispatch
support. It improved my read performance from 2.2GB/s to 3.4GB/s and
write performance from 750MB/s to 1.5GB/s the system worked for a few
days under high load (saturated a 2 x 10Gb/s lagg(4) as read only
WebDAV server and while receiving uploads via SFTP). It worked until I
attempted to shutdown the system. It hung on shutdown an never powered
off. I had to power cycle the box via IPMI to recover. I never found
the time to debug this problem.
The server in question was an old Westmere dual hexacore with 96GB RAM
and had 42 (+ 3 spares) 7200rpm SAS2 disks in a dual ported JBOD
connected via 4 x 4 lanes to two LSI2?08 HBAs with IT firmware
configured as 3way mirrored ZFS pool with GELI underneath.
_______________________________________________
freebsd-geom@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-geom
To unsubscribe, send any mail to "freebsd-geom-unsubscr...@freebsd.org"
