From: Zhu Yangyang <zhuyangyan...@huawei.com>
In the disable branch of qmp_block_set_io_throttle(), we call
bdrv_drained_begin().
We know that bdrv_drained_begin() is a blocking interface used to wait for all
submitted
I/O operations to complete, i.e., to wait until bs->in_flight becomes zero.
Theoretically, once we stop submitting I/O operations, bs->in_flight should
quickly
become zero(regardless of success or failure). However, if we are using network
storage
and a network link failure occurs at that moment, things become different.
The underlying storage driver will then retry the operation, which may take 1
to 2 minutes
before responding with an I/O error to QEMU. As a result, bdrv_drained_begin()
will be blocked
for 1 to 2 minutes, leading to two issues:
1. qmp_block_set_io_throttle() gets blocked, which is an external interface,
and this can provide a poor user experience.
2. More seriously, qmp_block_set_io_throttle() holds the BQL during its
execution, which could
lead to the blocking of vcpu thread, further causing guest os softlockup and
potentially a panic.
The stack when qmp_block_set_io_throttle() is blocked is as follows:
At this point, there are no IO submit events or IO response events.
aio_poll() will remain blocked until the network storage reports an EIO to QEMU.
#0 0x00007f54877fc39e in ppoll () from target:/usr/lib64/libc.so.6
#1 0x0000556dced07b7d in ppoll (__ss=0x0, __timeout=0x0, __nfds=<optimized
out>, __fds=<optimized out>) at /usr/include/bits/poll2.h:81
#2 0x0000556dcecee599 in fdmon_poll_wait (ctx=0x556de95f8e40,
ready_list=0x7ffcc6378b18, timeout=-1) at ../util/fdmon-poll.c:79
#3 0x0000556dceceda9e in aio_poll (ctx=0x556de95f8e40,
blocking=blocking@entry=true) at ../util/aio-posix.c:671
#4 0x0000556dcebe654a in bdrv_do_drained_begin (poll=<optimized out>,
parent=0x0, bs=0x556de9896420) at ../block/io.c:378
#5 bdrv_do_drained_begin (bs=0x556de9896420, parent=0x0, poll=<optimized out>)
at ../block/io.c:347
#6 0x0000556dcebdc5a1 in blk_io_limits_disable (blk=0x556dea739470) at
../block/block-backend.c:2701
#7 0x0000556dce917584 in qmp_block_set_io_throttle
(arg=arg@entry=0x7ffcc6378d30, errp=errp@entry=0x7ffcc6378d18) at
../block/qapi-system.c:505
#8 0x0000556dcec5a8d2 in qmp_marshal_block_set_io_throttle (args=<optimized
out>, ret=<optimized out>, errp=0x7f5486395ea0) at
qapi/qapi-commands-block.c:368
#9 0x0000556dcece3ed9 in do_qmp_dispatch_bh (opaque=0x7f5486395eb0) at
../qapi/qmp-dispatch.c:128
#10 0x0000556dced03c55 in aio_bh_poll (ctx=ctx@entry=0x556de95e07c0) at
../util/async.c:219
#11 0x0000556dceced93e in aio_dispatch (ctx=0x556de95e07c0) at
../util/aio-posix.c:424
#12 0x0000556dced038fe in aio_ctx_dispatch (source=<optimized out>,
callback=<optimized out>, user_data=<optimized out>) at ../util/async.c:361
#13 0x00007f5487ec606e in g_main_context_dispatch () from
target:/usr/lib64/libglib-2.0.so.0
#14 0x0000556dced04fd8 in glib_pollfds_poll () at ../util/main-loop.c:287
#15 os_host_main_loop_wait (timeout=0) at ../util/main-loop.c:310
#16 main_loop_wait (nonblocking=nonblocking@entry=0) at ../util/main-loop.c:589
#17 0x0000556dce925431 in qemu_main_loop () at ../system/runstate.c:835
#18 0x0000556dcec565c7 in qemu_default_main (opaque=opaque@entry=0x0) at
../system/main.c:48
#19 0x0000556dce6f7848 in main (argc=<optimized out>, argv=<optimized out>) at
../system/main.c:76
Zhu Yangyang (2):
io/block: Refactoring the bdrv_drained_begin() function and implement
a timeout mechanism.
qapi: Fix qmp_block_set_io_throttle blocked for too long
block/block-backend.c | 14 ++++-
block/io.c | 55 +++++++++++++++----
block/qapi-system.c | 7 ++-
include/block/aio-wait.h | 58 +++++++++++++++++++++
include/block/block-io.h | 7 +++
include/system/block-backend-global-state.h | 1 +
util/aio-wait.c | 7 +++
7 files changed, 137 insertions(+), 12 deletions(-)
--
2.33.0
