On 31/7/21 8:21 pm, Rich Freeman wrote: > On Fri, Jul 30, 2021 at 11:50 PM Wols Lists <antli...@youngman.org.uk> wrote: >> btw, you're scrubbing over USB? Are you running a raid over USB? Bad >> things are likely to happen ... > So, USB hosts vary in quality I'm sure, but I've been running USB3 > drives on lizardfs for a while now with zero issues. > > At first I was shucking them and using LSI HBAs. That was a pain for > a bunch of reasons, and I would have issues probably due to the HBAs > being old or maybe cheap cable issues (and new SAS hardware carries a > hefty price tag). > > Then I decided to just try running a drive on USB3 and it worked fine. > This isn't for heavy use, but it basically performs identically to > SATA. I did the math and for spinning disks you can get 2 drives per > host before the data rate starts to become a concern. This is for a > distributed filesystem and I'm just using gigabit ethernet, and the > cluster is needed more for capacity than IOPS, so USB3 isn't the > bottleneck anyway. > > I have yet to have a USB drive have any sort of issue, or drop a > connection. And they're running on cheap Pi4s for the most part > (which have two USB3 hosts). If for some reason a drive or host > dropped the filesystem is redundant at the host level, and it also > gracefully recovers data if a host shows back up, but I have yet to > see that even happen due to a USB issue. I've had far more issues > when I was trying to use LSI HBAs on RockPro64 SBCs (which have a PCIe > slot - I had to also use a powered riser). > > Now, if you want to do something where you're going to be pulling > closer to max bandwidth out of all your disks at once and you have > more than a few disks and you have it on 10GbE or faster, then USB3 > could be a bottleneck unless you have a lot of hosts (though even then > adding USB3 hosts to the motherboard might not be any harder than > adding SATA hosts). > I'll generally agree with your USB3 comments - besides the backup disk I am running moosefs on 5 odroid HC2's (one old WD red or green on each, the HC2 is a 32 bit BIG.little arm system and uses a built in USB sata connection - excellent on a 5.12 kernel, just ok on 4.x series) and an Odroid C4 (arm64) with 2 asmedia USB3 adaptors from ebay - the adaptors are crap but do work somewhat with the right tweaks! and a single sata ssd on the master (intel). I tried using moosefs with a rpi3B in the mix and it didn't go well once I started adding data - rpi 4's were not available when I set it up. I think that SMR disks will work quite well on moosefs or lizardfs - I don't see long continuous writes to one disk but a random distribution of writes across the cluster with gaps between on each disk (1G network).
With a good adaptor, USB3 is great ... otherwise its been quite frustrating :( I do suspect linux and its pedantic correctness trying to deal with hardware that isn't truly standardised (as in the manufacturer probably supplies a windows driver that covers it up) is part of the problem. These adaptors are quite common and I needed to apply the ATA command filter and turn off UAS using the usb tweaks mechanism to stop the crashes and data corruption. The comments in the kernel driver code for these adaptors are illuminating! BillK
