https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=244356
--- Comment #26 from Olivier Certner <olivier.free...@free.fr> --- I did some tests yesterday on a very recent machine (MB Asus X570) with 3 different USB sticks: - SanDisk Extreme GO USB 3.1 64GB (abbrev: SD_64G) - SanDisk Extreme GO USB 3.1 128GB (abbrev: SD_128G) - Kingston DataTraveler 100 G3 32GB (abbrev: KT_32G) They confirm and extend what Sébastien has already reported. ********************* * Executive Summary * ********************* A. For tests performed yesterday and whose reports are below: - Transfers to USB 3.0 mass storage appears to stall very frequently, for at least 2 or 3s, and frequently for several runs of that. Duration of periods of non-zero bandwidth may become as low as 2 or 3s. - There is an initial grace period where bandwidth is steady and at arguably normal numbers. It lasted between 10 and 20min. But once the stalls start, bandwidth is never steady again, even when changing USB hubs, USB sticks, filesystems and method of writing (for the precise extent of the variations, please see the reports below). The exact conditions for the grace period to manifest are not clear yet; I suspect a reboot should reenable it, but the machine was never rebooted during the tests. - Using FreeBSD 12.1-STABLE at r355888, unmodified. B. For related experience during the last 2 years: - I observed similar overall behavior on very old laptops (> 10 years) that only have USB 2.0 hubs, with some set of USB 3.0 sticks (including KT_32G for sure). - Same on another old desktop machine with USB 3.0 hubs (MB Asus P5P43TD). - Same also with other USB 3.0 sticks (from yet another manufacturer, Corsair). - Using continuously updated FreeBSD versions (10 to 12; but never CURRENT up to now). - No such behavior is exhibited on Linux (USB key is ejected properly, and the data on it is checked after replug and found correct). C. Wrap-up and follow-up: Currently, I thus have 5 USB 3.0 sticks, none of which can sustain acceptable steady bandwidth when transferring data to filesystems, except probably for some limited time after machine boot-up. Unless I'm particularly unlucky, I'm suspecting the extent of the problem is such that a significant number of USB 3.0 sticks may actually not work with decent bandwidth for a non-trivial duration at all. Having positive reports might help in narrowing the problem down. I really want this problem to be fixed, because I don't see myself buying more and more USB sticks in the hope that one is finally going to work, or having to change motherboards or laptops just because of this, especially when it seems the hardware really can work as expected (which might need quirks, workarounds, etc.). If knowledgeable people could kindly look into the reports and attachments, it would be much appreciated. I'm more than willing to run more tests based on feedbacks. I'm probably yet too new to USB and CAM to make fast debugging progress on my own, and I may not have the proper hardware background, so I would appreciate any guidance as to what to test and which code to try tinkering with. ***************************************** * Hardware/software, common test traits * ***************************************** All sticks are MBR-partitioned, with a single 4K-aligned partition occupying all available space. FS type varies and is mentioned in the reports below. FreeBSD version is 12.1-STABLE at r355888. For most tests, I grabbed iostat statistics every second (iostat -w 1). Inside the iostat output, you'll want to look for da0 (last columns) for the current USB stick numbers. /usr partition is on a M.2 NVMe SSD, appearing as nvd0 (first disk columns). Other active disks during tests were ada3 and ada4, with unrelated traffic (none of their FSes were used for the tests below). All hard disks on these machines are ZFS pools. Tests are reported on chronological order. The machine was never rebooted in the course of these tests. *********** * Reports * *********** I. Initial tests with SD_64G For these initial tests, I did not put in place instrumentation as advised earlier in this PR, but I wish I had, since I observed a different behavior (the grace period), which is why I'm reporting about them. First, I used a freshly created exFAT partition. exFAT was created with package `exfat-utils` and then mounted thanks to package `fusefs-exfat`; both are at version 1.3.0. I copied /usr/local/bin (~661M) using `cp -a`. This completed without errors, at a rate of ~66MB/s, rate that was steady according to iostat. Then, I replaced the exFAT partition by a standard UFS+SU one (default, so 32K blocks, 8K frags), and did the same test. Got same result. Finally, I reformatted the partition with UFS (without SU) and did the test again. Got steady albeit lower performance (around 50/55MB/s if I recall correctly). So far so good. II. Iostat-instrumented tests with SD_64G Done on a freshly created UFS+SU partition. This time I launched the copy of whole `usr` (reported to be 46GB by `du`). I stopped it after about 35GB were copied. The corresponding attachment is: SD_64G_ufs_SU_usr_cp.txz. Just to repeat some information given in introduction in order for the context to be very clear: `usr`'s files are read from the VM cache (previous tests) or a ZFS partition on the NVMe SSD. As can be seen from the iostat file, transfer bandwidth to the stick varies, being below 10MB/s at start, and then most of the time being at several tenths of MB/s, with ranges of up to 2 minutes where the bandwidth is mostly between 30 and 60 MB/s. The transfer proceeds well for approximately 10min and a half (again, one line per second in the iostat file). But then, it starts to exhibit a very curious pattern, where bandwidth alternatively goes to 0 during 2 to 10s (except for one (small) transaction every 2 or 3s), and then rises to reasonable values (10s of MB/s) with very high TPS. Traffic never goes back to the first phase's pattern. I stopped iostat collection relatively quickly afterwards, but let the copy proceed more, and stopped it after ~35GB were transferred. In total, the transfer took a little more than 3 hours (see trace in attachment), which gives little more than 3MB/s for the whole run. I relaunched this test but this time replacing the UFS+SU partition by an exFAT one, hoping that the initial correct behavior would return (see I.). I did this without unplugging/replugging the stick after the UFS+SU test. The corresponding attachment is: SD_64G_exFAT_usr_cp.txz. It contains only the iostat trace. Behavior is similar to the second phase of the previous run. There was no "normal" phase (with decent steady bandwidth) first. Finally, I reran the test on a fresh UFS+SU partition, with again the same result. I did collect only a couple of seconds of iostat. Transfer stopped after 8 hours with a device full, meaning an overall transfer rate of around ~2.25MB/s. The corresponding attachment is: SD_64G_no_unplug_ufs_SU_usr_cp.txz. III. USB and umass instrumented tests, with variations From this point, I systematically set sysctl `hw.usb.umass.debug` to -1, and run usbdump as prescribed by HPS earlier in the PR. I also grabbed iostat output, except for A. A. Using a different stick: SD_128G Corresponding attachment: SD_128G_ufs_SU_usr_cp.txz. This is a bigger stick from the same manufacturer. Test is the "same" as for SD_64G: I ran the `cp -a` test to a fresh UFS+SU MBR partition covering the whole stick. Result was essentially the same as previous tests, with stalls and writes alternating. B. Using `dd if=/dev/zero` instead of `cp -a /usr` Corresponding attachment: SD_64G_ufs_SU_dd.txz. I used this to rule out any influence of the source filesystem (ZFS on NVMe SSD), as well as some metadata operations on the destination FS. `dd` writes to a fresh new partition, into a single file. For this one, I collected only 2 minutes of iostat, but stalls are clearly visible, and the problem seems worse: There are several runs of 2 to 3s blocks delimited by a single transaction before bandwidth can really take off, and for no more than 2 or 3s. The full test ran for 6 minutes, and kernel logs as well as usbdump output are provided in the attachment. C. Using `dd if=/dev/zero`, yet another stick: KT_23G, plugged on rear USB hub Corresponding attachment: KT_32G_ufs_SU_dd.txz. The plugs I had used up to now were facade ones on the case, and using them KT_32G stick is not recognized as plugged. So I switched to using plugs solded to the motherboard. This is a smaller stick from another manufacturer. Test was done as for B., and ran 5 minutes. Overall same results. D. Using `dd if=/dev/zero`, original stick, plugged on rear USB hub Corresponding attachment: SD_64G_2nd_hub_ufs_SU_dd.txz. This test was done just to ensure that results were not influenced by the hub. Indeed, KT_32G not recognized on first hub could leave the impression that something was wrong on these plugs. But results were again the same. Traces inside the attachment as usual. -- You are receiving this mail because: You are the assignee for the bug. _______________________________________________ freebsd-usb@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to "freebsd-usb-unsubscr...@freebsd.org"
