Awesome, Jared - many thanks for this! On Fri, 28 Jun 2019 at 08:08, Jared D. McNeill <jmcne...@netbsd.org> wrote:
> Module Name: src > Committed By: jmcneill > Date: Fri Jun 28 15:08:47 UTC 2019 > > Modified Files: > src/sys/dev/ic: nvme.c nvmevar.h > > Log Message: > Fix a performance issue where one busy queue can starve all other queues. > > In normal operations with multiple queues, the nvme driver will attempt > to schedule I/O requests on the submitting CPU. This breaks down when any > one of the queues becomes full; the driver returns EAGAIN to the disk > layer, which causes the disk layer to stop submitting more requests until > the blocked request is consumed. When space becomes available in the full > queue, it pulls the next buffer from the bufq and fills the queue again, > until finally hitting EAGAIN and preventing other queues from processing > requests. > > Two changes here to fix the problem: > > - When processing requests from the bufq, attempt to assign them to the > queue associated with the CPU that originated the request. > - If that queue is busy, try to find another queue with available space > before returning EAGAIN. This way, only when all queues are full will > the disk layer stop submitting more requests. > > Now for some real numbers. On a Rockchip RK3399 board (6 CPUs), with 6 > concurrent readers: > > Old code: > 4294967296 bytes transferred in 52.420 secs (81933752 bytes/sec) > 4294967296 bytes transferred in 53.969 secs (79582117 bytes/sec) > 4294967296 bytes transferred in 55.391 secs (77539082 bytes/sec) > 4294967296 bytes transferred in 55.649 secs (77179595 bytes/sec) > 4294967296 bytes transferred in 56.102 secs (76556402 bytes/sec) > 4294967296 bytes transferred in 72.901 secs (58915066 bytes/sec) > > New code: > 4294967296 bytes transferred in 37.171 secs (115546186 bytes/sec) > 4294967296 bytes transferred in 37.611 secs (114194445 bytes/sec) > 4294967296 bytes transferred in 37.655 secs (114061009 bytes/sec) > 4294967296 bytes transferred in 38.247 secs (112295534 bytes/sec) > 4294967296 bytes transferred in 38.496 secs (111569183 bytes/sec) > 4294967296 bytes transferred in 38.595 secs (111282997 bytes/sec) > > > To generate a diff of this commit: > cvs rdiff -u -r1.43 -r1.44 src/sys/dev/ic/nvme.c > cvs rdiff -u -r1.19 -r1.20 src/sys/dev/ic/nvmevar.h > > Please note that diffs are not public domain; they are subject to the > copyright notices on the relevant files. > >
