> On Mon, Oct 30, 2017 at 09:23:51PM +0200, Mihai Popescu wrote: .. > Back in a former life, I often had to transfer terabytes of information > between hosts in my network. The goal was to reduce the overhead of the > transfer so that more of the data would get transferred. I had been > using netcat/nc and pushing data through the connection as fast as I > could. This worked great, provided I could verify that things were being > transferred without errors. Any connection problems and I'd be screwed. .. > Essentially, he's compressing files and then sending them over to fit > more data down the pipe. I also learned of this other program called .. > Check those out, and I'm very curious if anyone else on the list has > more OpenBSD-centric techniques!
For file transfers today, I presume speed is limited not by the networking subsystem or even by compression speed, but instead by the block device/file IO subsystem, which has a cap last time I checked of around 120MB/sec system-wide. (Faster for retrieval from the buffer cache, however that boost drops very quickly.) A SATA SSD gives you ~~450MB/sec, and an M.2 NVME SSD gives you ~~900MB/sec, sequential and random. You get multipliers of that in a RAID. You get some of this performance in OpenBSD today if you make your accesses to /dev/rsd* only, and of 16KB-aligned 16KB multiples, that makes OpenBSD yield ~600MB/sec or so, however this access mode has tons of limitations, for instance /dev/rsd* doesn't support mmap(). My best understanding of OpenBSD's block device/file IO subsystem is that it has a "biglock" and that all underlying disk/file access is serialized / totally synchronous, with no support for or use of multiqueuing or any other parallellization. Implementing asynchronous logics is tedious and from a security perspective I would understand the charm of keeping an implementation that is cemented to be synchronous only. Is this what's going on? All this leads me to the perception that of all areas in OpenBSD, disk/file IO is the weakest/the one with the least strengths, compared to many other areas where obviously OpenBSD is the very best. (Way lower priority topics, however related, are unified buffer cache https://marc.info/?t=143228266700001&r=1&w=2 and the buffer cache 4GB limit https://marc.info/?t=145538710500002&r=1&w=2 , https://marc.info/?l=openbsd-tech&m=147905745617957&w=2 , https://marc.info/?t=146824436600004&r=1&w=2 , https://unix.stackexchange.com/questions/61459/does-sysctl-kern-bufcachepercent-not-work-in-openbsd-5-2-above-1-7gb .) Is there any interest among developers to speed up the disk/file IO? What would it actually involve? What's needed from the community, donations? Would directed donations be welcome? Thank you very much and sorry for the buzz.
