Re: blk-mq + bfq IO hangs after writing partition table

[Ming Lei]

On Fri, Dec 01, 2017 at 02:46:39PM +0000, Ivan Kozik wrote:
> On Fri, Dec 1, 2017 at 1:35 PM, Ming Lei <ming....@redhat.com> wrote:
> > Basically all IO hang happen inside get_request(), and seems this issue
> > isn't related with recent change in V4.15, could you run V4.14 to see if
> > there is such issue?
> 
> I tried several times and I could not reproduce the problem on linus's v4.14.
> 
> Also, while testing, I noticed that udev rules can switch out the IO
> scheduler after gdisk writes partitions, but this appears to be
> unrelated to the problem, just something to remember when testing.


Now I can reproduce the issue on usb-storage.

After applying the attached debug patch, the log[1] can be generated.
>From the log, there are requests inserted to BFQ queue, but these requests
can't be retrieved by e->type->ops.mq.dispatch_request() in
blk_mq_do_dispatch_sched(), then IO hang is caused.

Not see this issue on mq-deadline, looks a BFQ specific issue.


[1] http://people.redhat.com/minlei/tests/logs/bfq-trace


-- 
Ming

diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
index bcb6d21baf12..2db8ac732fba 100644
--- a/block/bfq-iosched.c
+++ b/block/bfq-iosched.c
@@ -4362,6 +4362,10 @@ static void bfq_insert_requests(struct blk_mq_hw_ctx 
*hctx,
 
                rq = list_first_entry(list, struct request, queuelist);
                list_del_init(&rq->queuelist);
+
+               if (hctx->queue->tag_set->flags & 0x8)
+                       trace_printk("insert rq->%d\n", rq->internal_tag);
+
                bfq_insert_request(hctx, rq, at_head);
        }
 }
diff --git a/block/blk-mq-sched.c b/block/blk-mq-sched.c
index c117bd8fd1f6..62efca331797 100644
--- a/block/blk-mq-sched.c
+++ b/block/blk-mq-sched.c
@@ -107,8 +107,16 @@ static void blk_mq_do_dispatch_sched(struct blk_mq_hw_ctx 
*hctx)
 
                rq = e->type->ops.mq.dispatch_request(hctx);
                if (!rq) {
+                       if (q->tag_set->flags & 0x8)
+                               trace_printk("not get rq, %d\n",
+                                               e->type->ops.mq.has_work(hctx));
                        blk_mq_put_dispatch_budget(hctx);
                        break;
+               } else {
+                       if (q->tag_set->flags & 0x8)
+                               trace_printk("get rq->%d, %d\n",
+                                               rq->internal_tag,
+                                               e->type->ops.mq.has_work(hctx));
                }
 
                /*
diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
index 1cbc497e00bd..db9556662e27 100644
--- a/drivers/scsi/scsi_lib.c
+++ b/drivers/scsi/scsi_lib.c
@@ -2268,6 +2268,8 @@ int scsi_mq_setup_tags(struct Scsi_Host *shost)
        shost->tag_set.cmd_size = cmd_size;
        shost->tag_set.numa_node = NUMA_NO_NODE;
        shost->tag_set.flags = BLK_MQ_F_SHOULD_MERGE | BLK_MQ_F_SG_MERGE;
+       if (!strcmp(shost->hostt->name, "usb-storage"))
+               shost->tag_set.flags |= 0x8;
        shost->tag_set.flags |=
                BLK_ALLOC_POLICY_TO_MQ_FLAG(shost->hostt->tag_alloc_policy);
        shost->tag_set.driver_data = shost;