On 11/19/2013 02:30 PM, Dave Chinner wrote:
> On Tue, Nov 19, 2013 at 09:05:09AM -0700, Jens Axboe wrote:
>> On Tue, Nov 19 2013, Christoph Hellwig wrote:
>>> On Tue, Nov 19, 2013 at 07:02:18PM +1100, Dave Chinner wrote:
>>>> I have no idea if it's related to the above hang, but either way
>>>> breaking iostat is a major regression....
>>>
>>> Both of them are most likely due to the conversion of virtio_blk
>>> to the blk-mq code.
>>>
>>> I've not seen the hang in my heavy xfstests testing, but that was a
>>> slightly different codebase than what finally got in, so I'll try
>>> to reproduce it once I get some spare QA cycles.
>>
>> Thanks! Dave, if you still have it in that state, can you dump the
>> contents of /sys/block/<devs>/mq/ for the device/devices that are hung?
>
> # find /sys/block/vdb/mq -type f -print -exec cat {} \;
> /sys/block/vdb/mq/0/run
> 1313835
> /sys/block/vdb/mq/0/cpu0/completed
> 546857 207203
> /sys/block/vdb/mq/0/cpu0/rq_list
> CTX pending:
> /sys/block/vdb/mq/0/cpu0/merged
> 0
> /sys/block/vdb/mq/0/cpu0/dispatched
> 257733 496352
> /sys/block/vdb/mq/0/cpu1/completed
> 547714 200741
> /sys/block/vdb/mq/0/cpu1/rq_list
> CTX pending:
> /sys/block/vdb/mq/0/cpu1/merged
> 0
> /sys/block/vdb/mq/0/cpu1/dispatched
> 262114 486381
> /sys/block/vdb/mq/0/tags
> nr_tags=66, reserved_tags=2, batch_move=16, max_cache=32
> nr_free=0, nr_reserved=1
> cpu00: nr_free=0
> cpu01: nr_free=0
> /sys/block/vdb/mq/0/ipi_redirect
> 0
> /sys/block/vdb/mq/0/queued
> 1384795
> /sys/block/vdb/mq/0/dispatched
> 0 194596
> 1 1022691
> 2 68540
> 4 10138
> 8 7686
> 16 10234
> 32 0
> 64 0
> 128 0
> 256 0
> /sys/block/vdb/mq/0/pending
> HCTX pending:
> #
>
> It looks like it's hung because of this:
>
> /sys/block/vdb/mq/0/tags
> nr_tags=66, reserved_tags=2, batch_move=16, max_cache=32
> nr_free=0, nr_reserved=1
> cpu00: nr_free=0
> cpu01: nr_free=0
>
> No free tags in teh pool, nor any free tags in the per-cpu
> magazines. Perhaps there is a leak somewhere?
It's hung because all tags and requests are allocated an inflight. In
the above, you have 65 allocated, which must be 64 reads+writes and a
flush out of the reserved pool. Which matches the free stats here near
the bottom, 0 normal tags free and 1 reserved free.
The end_io path for virtio-blk is pretty darn simple. I'm trying to
debug this now...
--
Jens Axboe
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
