Re: [PATCH v3 1/2] vhost: destroy device when all vqs are inactive

[Xuan Zhuo]

On Mon, 29 Apr 2024 09:27:42 -0700, Stephen Hemminger 
 wrote:
> On Tue, 11 Oct 2022 18:44:28 +0200
> Maxime Coquelin  wrote:
>
> > On 9/12/22 05:36, Kangjie Xu wrote:
> > > We change the behavior of vhost_user_get_vring_base(). Previosly,
> > > destroying a virtqueue will cause the whole device to be destroyed.
> > > The behavior is not specified in the vhost-user protocol.
> > >
> > > Thus, we refactor this part. The device will be destroyed only when
> > > all virtqueues in the device are going to be destroyed.
> > >
> > > This helps us to simplify the implementation when resetting a virtqueue.
> > >
> > > Signed-off-by: Kangjie Xu 
> > > Signed-off-by: Xuan Zhuo 
> > > ---
> > >   lib/vhost/vhost_user.c | 10 --
> > >   1 file changed, 8 insertions(+), 2 deletions(-)
> > >
> > > diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
> > > index 4ad28bac45..a9f0709f94 100644
> > > --- a/lib/vhost/vhost_user.c
> > > +++ b/lib/vhost/vhost_user.c
> > > @@ -2088,10 +2088,16 @@ vhost_user_get_vring_base(struct virtio_net 
> > > **pdev,
> > >   {
> > >   struct virtio_net *dev = *pdev;
> > >   struct vhost_virtqueue *vq = 
> > > dev->virtqueue[ctx->msg.payload.state.index];
> > > + uint32_t i, num_live_vring = 0;
> > >   uint64_t val;
> > >
> > > - /* We have to stop the queue (virtio) if it is running. */
> > > - vhost_destroy_device_notify(dev);
> > > + /* Stop the device when vq is the last active queue */
> > > + for (i = 0; i < dev->nr_vring; i++)
> > > + if (dev->virtqueue[i]->access_ok)
> > > + num_live_vring++;
> > > +
> > > + if (num_live_vring == 1 && vq->access_ok)
> > > + vhost_destroy_device_notify(dev);
> > >
> > >   dev->flags &= ~VIRTIO_DEV_READY;
> > >   dev->flags &= ~VIRTIO_DEV_VDPA_CONFIGURED;
> >
> > I think we are missing something here.
> >
> > We used to send the device destroy notification before getting the ring
> > indexes, in order to ensure that the application has stopped processing
> > the rings.
> >
> > With this patch, the application may still be polling the ring while we
> > get the ring indexes (e.g. a thread in the application may be in the
> > middle of rte_vhost_dequeue_burst() on that ring). So at best the ring
> > indexes returned to the Vhost-user master will be outdated. At worst, it
> > will crash the application because we call vring_invalidate() without
> > the vq's lock being taken.
> >
> > I think you should protect all the VQ indexes fetching and VQ deinit
> > using its access_lock.
> >
> > Maxime
> >
>
> Please address Maxime's feedback.


Kangjie has already resigned.

Sorry, we don't have anyone in charge of this.

If you need it, you can propose a new patch to solve this problem.

Thanks.

Re: RE: [PATCH v2 2/2] vhost: introduce VHOST_USER_RESET_VRING

[Xuan Zhuo]

On Thu, 22 Sep 2022 09:35:35 +, "Xia, Chenbo"  wrote:
> > -Original Message-
> > From: Kangjie Xu 
> > Sent: Monday, September 5, 2022 11:48 AM
> > To: maxime.coque...@redhat.com; Xia, Chenbo 
> > Cc: dev@dpdk.org; xuanz...@linux.alibaba.com; hen...@linux.alibaba.com;
> > jasonw...@redhat.com; m...@redhat.com
> > Subject: [PATCH v2 2/2] vhost: introduce VHOST_USER_RESET_VRING
> >
> > To support the reset operation for an individual virtqueue, we
> > introduce a new message VHOST_USER_RESET_VRING. When the feature
> > VIRTIO_F_RING_RESET feature has been successfully negotiated, This
> > message is submitted by the front-end to reset an individual
> > virtqueue to initial states in the back-end. The reply is needed
> > to ensure that the reset operation is complete.
>
> completed
>
> >
> > Signed-off-by: Kangjie Xu 
> > Signed-off-by: Xuan Zhuo 
> > ---
> >  lib/vhost/vhost.c  |  2 +-
> >  lib/vhost/vhost.h  |  1 +
> >  lib/vhost/vhost_user.c | 27 ++-
> >  lib/vhost/vhost_user.h |  1 +
> >  4 files changed, 29 insertions(+), 2 deletions(-)
> >
> > diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c
> > index 60cb05a0ff..215a1ca355 100644
> > --- a/lib/vhost/vhost.c
> > +++ b/lib/vhost/vhost.c
> > @@ -610,7 +610,7 @@ init_vring_queue(struct virtio_net *dev, uint32_t
> > vring_idx)
> > vhost_user_iotlb_init(dev, vring_idx);
> >  }
> >
> > -static void
> > +void
> >  reset_vring_queue(struct virtio_net *dev, uint32_t vring_idx)
> >  {
> > struct vhost_virtqueue *vq;
> > diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h
> > index 76461a3406..eccb52842d 100644
> > --- a/lib/vhost/vhost.h
> > +++ b/lib/vhost/vhost.h
> > @@ -791,6 +791,7 @@ get_device(int vid)
> >
> >  int vhost_new_device(void);
> >  void cleanup_device(struct virtio_net *dev, int destroy);
> > +void reset_vring_queue(struct virtio_net *dev, uint32_t vring_idx);
> >  void reset_device(struct virtio_net *dev);
> >  void vhost_destroy_device(int);
> >  void vhost_destroy_device_notify(struct virtio_net *dev);
> > diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
> > index 4ad28bac45..5f7743d9d9 100644
> > --- a/lib/vhost/vhost_user.c
> > +++ b/lib/vhost/vhost_user.c
> > @@ -2771,6 +2771,30 @@ vhost_user_set_status(struct virtio_net **pdev,
> > return RTE_VHOST_MSG_RESULT_OK;
> >  }
> >
> > +static int
> > +vhost_user_reset_vring(struct virtio_net **pdev,
> > +   struct vhu_msg_context *ctx __rte_unused,
> > +   int main_fd __rte_unused)
> > +{
> > +   struct virtio_net *dev = *pdev;
> > +   int index = (int)ctx->msg.payload.state.index;
>
> Why not just use unsigned int?
>
> > +
> > +   VHOST_LOG_CONFIG(dev->ifname, INFO, "reset queue: queue idx: %d\n",
> > index);
> > +
> > +   if (!(dev->features & (1ULL << VIRTIO_F_RING_RESET))) {
> > +   return RTE_VHOST_MSG_RESULT_ERR;
> > +   }
>
> braces {} are not necessary for single statement blocks
>
> > +
> > +   dev->virtqueue[index]->enabled = false;
> > +   reset_vring_queue(dev, index);
> > +
> > +   ctx->msg.payload.state.num = 0;
> > +   ctx->msg.size = sizeof(ctx->msg.payload.u64);
> > +   ctx->fd_num = 0;
> > +
> > +   return RTE_VHOST_MSG_RESULT_REPLY;
> > +}
>
> IIUC, before this handler, we need to lock the queue? Using 
> vhost_user_lock_all_queue_pairs
>
> BTW, is this support merged in QEMU now? I remember for similar cases,
> we wait for QEMU to merge first and then merge in DPDK.
>
> Maxime, do I remember this correctly?


Yes, we are simultaneously pushing this feature to QEMU.

We have a patch for v3, maybe you missed it.

Thanks.


>
> Thanks,
> Chenbo
>
> > +
> >  #define VHOST_MESSAGE_HANDLERS \
> >  VHOST_MESSAGE_HANDLER(VHOST_USER_NONE, NULL, false) \
> >  VHOST_MESSAGE_HANDLER(VHOST_USER_GET_FEATURES, vhost_user_get_features,
> > false) \
> > @@ -2803,7 +2827,8 @@ VHOST_MESSAGE_HANDLER(VHOST_USER_POSTCOPY_END,
> > vhost_user_postcopy_end, false) \
> >  VHOST_MESSAGE_HANDLER(VHOST_USER_GET_INFLIGHT_FD,
> > vhost_user_get_inflight_fd, false) \
> >  VHOST_MESSAGE_HANDLER(VHOST_USER_SET_INFLIGHT_FD,
> > vhost_user_set_inflight_fd, true) \
> >  VHOST_MESSAGE_HANDLER(VHOST_USER_SET_STATUS, vhost_user_set_status, false)
> > \
> > -VHOST_MESSAGE_HANDLER(VHOST_USER_GET_STATUS, vhost_user_get