https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=244356
--- Comment #75 from Olivier Certner <olivier.free...@free.fr> --- I've tested again SD_128G, more thoroughly this time. When copying '/usr' to some UFS partition, I have lots of stalls, and long ones (~ 30s). Copying to an exFAT partition gives a similar iostat profile, but with the difference that duration of stalls is ~74% of the total test duration, whereas it is ~83% for the UFS test. Also, practically all exFAT accesses are through 128KiB transactions, whereas for UFS median and average of accesses is around 70KiB. Not taking into account stalls, average bandwidth is approximately at 48MiB/s for exFAT and 25MiB/s for UFS (12.3MiB/s vs 4.3MiB/s including stalls). For these two metrics, ratios between the two FSes are close (1.82 and 1.89). A priori, I would have expected that the second ratio would be lower than the first, because smaller transactions should be compensated in part, I thought, by more transactions per second. But TPS for both tests is not that different (median: 440 vs. 466, average: 381 vs. 566, stalls not counted). More precisely, for a transaction average close to 128KiB/s for UFS, TPS is around 220, wereas it is around 224 for exFAT (so no significant difference), whereas bandwidth degrades a lot for smaller transactions, because there TPS does not compensate at all (the computed equivalent bandwidth-preserving TPS for 128KiB transactions becomes ~34 on average). The worse performance of UFS cannot be attributed to possible stick "degradation" since the UFS test was actually performed *before* the exFAT one. Following up on Hans and Gary's responses, I was wondering how the controller could know which blocks of the sticks are actually used or not. For the record, I tried using `camcontrol sanitize` and `camcontrol security`, but none worked (don't know whether it's because umass SCSI's XPT doesn't allow/implement it, or because they are explicitly not supported in the UAS standard, or because the sticks don't implement them). So what else is possible? Some ideas: The controller could note when sectors full of zeros are written, keep track of that and free the underlying storage. Or, which I find dubious, the controller could interpret (ex)FAT metadata and thus know which blocks are actually used. Without going that far, it could interpret writes to the MBR (or GPT) area to know which partitions are currently allocated (but semantically, this is more dangerous, so even more dubious, although simpler to implement). From additional experiments, I observe behavior compatible with the first possibility (or some variant), and can also infirm the third one. For the first one, I simply `dd` the whole stick with zeros. After that, a similar UFS `cp` experiment shows an average bandwidth of 7.8MiB/s, so almost twice higher than the previously obtained 4.3MiB/s above. Transfer is stalled "only" for 71% of the whole test duration. This also infirms the third, since I had done other UFS tests before the one reported above, and no performance jump was observed then. What I'm working on now is to somehow make FreeBSD access the stick with bigger transactions, because of the results reported initially in this comment, which are compatible with the fact that the stick is handling badly small transactions, and because MacOS uses 1MiB transactions (so manufacturers may want to optimize for that; Linux does as well). Also, flash memory is structured in blocks of several pages, and erasure can only happens at block granularity, with current blocks being usually greater than 128KiB (I'm not a specialist, just read Wikipedia), so partial write may lead to full erasures and rewrites, degrading performance. I did a first experiment with UFS block size increased to 64KiB (which happens to be the maximum value UFS currently accepts), and thus fragment size to 8KiB, which I expect will increase the minimum (and average) transaction size. And indeed, results are a bit better. I get 9.8MiB/s on average (vs. 7.8MiB/s with default block size as reported above), and transfers are stalled "only" 56% of the time. Indeed, the minimum transaction size observed is now 8KiB instead of 4KiB, but to my surprise, average and median transaction sizes actually decrease a bit (51.5 and 25.6KiB vs. 49 and 22.8KiB; stalls omitted). So it seems that it's the minimum transaction size that really matters, and somehow increasing it reduces stalls, which could hint indeed to limited/poor caching when writing to non-full blocks. Still, I observe transaction sizes that can be any multiple of 8KiB up to 64KiB. I ran the same test a second time, after a full `dd if=/dev/zero` on the whole stick, and got worse results (8.3MiB/s on average, stalls ~63% of the time, 49.4KiB and 25.7KiB for average and median transaction sizes), which are not much of a clear improvement over UFS with default block size. So the 64KiB block size tests don't prove decisive to me at this point. MAXPHYS limits us to 128KiB/s, but without touching it I tought I could at least try to have all transactions be 128KiB, as happens with exFAT. So I turned to GEOM_NOP and then GEOM_CACHE, thinking they could eventually do the job out of the box, but in fact they can't. GEOM_NOP allows to set the block size, but then every I/O request's size must be a multiple of this block size, so rising the block size simply makes 'newfs' to fail. GEOM_CACHE is closer to what I would like, but it only caches reads, not writes, plus, while testing it, it deadlocked the whole GEOM subsystem. So I'm now understanding, fixing and extending it, in the hope that I can finally force 128KiB transactions (at the price of read-modify-write) and run more tests... All this, because I want to see whether the speed of UFS on these sticks can significantly be improved with software tuning to workaround potentially limited hardware. For the RMA, guys from SanDisk want proof that copying is indeed slow when using exFAT. Copying '/usr' to an exFAT partition happened at 12.3MiB/s (see first paragraph), not very much superior to latest UFS tests, but this was before `dd`ing the whole stick with zeros. So I guess I now have to do this experiment again, to see how much I can get from exFAT. Not sure if the SanDisk guys will consider ~12MiB/s slow enough to warrant a replacement. I could couple the argument with the "sticks don't respect standards" angle (prior `usbtest`), but I doubt it will add much weight. Anyway, I find it amazing that they can market write speeds up to 100MiB/s if we can reach only ~1/10th of it with (not so) small files. Maurizio, Sébastien, if you're interested, you can try to `dd if=/dev/zero` your sticks and then retry your experiments (with UFS+SU), to see if you can get any significant speed improvement. Also, it may be important to transfer a large quantity of data to see whether degradation/more frequent stalls indeed happen. Ideally, try to fill your sticks almost completely (which means very long tests). In fact, I usually stopped UFS tests after around ~18GiB had been transferred (around 30min duration). I realize this may have biased most results so far, so I'll try some bigger transfers later on. Anyway, I haven't arguably done enough tests for the results to be statistically significant, so more testing may change current interpretations. Assuming the behavior of SD_128G can indeed be understood of course. I've kept all the raw iostat files used to produce the statistics in this post, but have not attached them to the PR because there are several of them again and I'm not sure that would be useful at this point. Feel free to ask if you want to see them. -- You are receiving this mail because: You are the assignee for the bug. _______________________________________________ freebsd-usb@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to "freebsd-usb-unsubscr...@freebsd.org"
