On 2023/11/15 7:09, Yuri Benditovich wrote:
On Tue, Nov 14, 2023 at 9:03 AM Akihiko Odaki <akihiko.od...@daynix.com
<mailto:akihiko.od...@daynix.com>> wrote:
On 2023/11/14 2:26, Yuri Benditovich wrote:
>
>
> On Mon, Nov 13, 2023 at 2:44 PM Akihiko Odaki
<akihiko.od...@daynix.com <mailto:akihiko.od...@daynix.com>
> <mailto:akihiko.od...@daynix.com
<mailto:akihiko.od...@daynix.com>>> wrote:
>
> On 2023/11/13 20:44, Yuri Benditovich wrote:
> >
> >
> > On Sat, Nov 11, 2023 at 5:28 PM Akihiko Odaki
> <akihiko.od...@daynix.com <mailto:akihiko.od...@daynix.com>
<mailto:akihiko.od...@daynix.com <mailto:akihiko.od...@daynix.com>>
> > <mailto:akihiko.od...@daynix.com
<mailto:akihiko.od...@daynix.com>
> <mailto:akihiko.od...@daynix.com
<mailto:akihiko.od...@daynix.com>>>> wrote:
> >
> > On 2023/11/03 22:14, Yuri Benditovich wrote:
> > >
> > >
> > > On Fri, Nov 3, 2023 at 11:55 AM Akihiko Odaki
> > <akihiko.od...@daynix.com
<mailto:akihiko.od...@daynix.com> <mailto:akihiko.od...@daynix.com
<mailto:akihiko.od...@daynix.com>>
> <mailto:akihiko.od...@daynix.com
<mailto:akihiko.od...@daynix.com> <mailto:akihiko.od...@daynix.com
<mailto:akihiko.od...@daynix.com>>>
> > > <mailto:akihiko.od...@daynix.com
<mailto:akihiko.od...@daynix.com>
> <mailto:akihiko.od...@daynix.com
<mailto:akihiko.od...@daynix.com>>
> > <mailto:akihiko.od...@daynix.com
<mailto:akihiko.od...@daynix.com>
> <mailto:akihiko.od...@daynix.com
<mailto:akihiko.od...@daynix.com>>>>> wrote:
> > >
> > > On 2023/11/03 18:35, Yuri Benditovich wrote:
> > > >
> > > >
> > > > On Thu, Nov 2, 2023 at 4:56 PM Akihiko Odaki
> > > <akihiko.od...@daynix.com
<mailto:akihiko.od...@daynix.com>
> <mailto:akihiko.od...@daynix.com
<mailto:akihiko.od...@daynix.com>> <mailto:akihiko.od...@daynix.com
<mailto:akihiko.od...@daynix.com>
> <mailto:akihiko.od...@daynix.com
<mailto:akihiko.od...@daynix.com>>>
> > <mailto:akihiko.od...@daynix.com
<mailto:akihiko.od...@daynix.com>
> <mailto:akihiko.od...@daynix.com
<mailto:akihiko.od...@daynix.com>> <mailto:akihiko.od...@daynix.com
<mailto:akihiko.od...@daynix.com>
> <mailto:akihiko.od...@daynix.com
<mailto:akihiko.od...@daynix.com>>>>
> > > > <mailto:akihiko.od...@daynix.com
<mailto:akihiko.od...@daynix.com>
> <mailto:akihiko.od...@daynix.com
<mailto:akihiko.od...@daynix.com>>
> > <mailto:akihiko.od...@daynix.com
<mailto:akihiko.od...@daynix.com>
> <mailto:akihiko.od...@daynix.com
<mailto:akihiko.od...@daynix.com>>>
> > > <mailto:akihiko.od...@daynix.com
<mailto:akihiko.od...@daynix.com>
> <mailto:akihiko.od...@daynix.com
<mailto:akihiko.od...@daynix.com>>
> > <mailto:akihiko.od...@daynix.com
<mailto:akihiko.od...@daynix.com>
> <mailto:akihiko.od...@daynix.com
<mailto:akihiko.od...@daynix.com>>>>>> wrote:
> > > >
> > > > On 2023/11/02 19:20, Yuri Benditovich wrote:
> > > > >
> > > > >
> > > > > On Thu, Nov 2, 2023 at 11:33 AM
Michael S.
> Tsirkin
> > > > <m...@redhat.com <mailto:m...@redhat.com>
<mailto:m...@redhat.com <mailto:m...@redhat.com>>
> <mailto:m...@redhat.com <mailto:m...@redhat.com>
<mailto:m...@redhat.com <mailto:m...@redhat.com>>>
> > <mailto:m...@redhat.com <mailto:m...@redhat.com>
<mailto:m...@redhat.com <mailto:m...@redhat.com>>
> <mailto:m...@redhat.com <mailto:m...@redhat.com>
<mailto:m...@redhat.com <mailto:m...@redhat.com>>>>
> > > <mailto:m...@redhat.com <mailto:m...@redhat.com>
<mailto:m...@redhat.com <mailto:m...@redhat.com>>
> <mailto:m...@redhat.com <mailto:m...@redhat.com>
<mailto:m...@redhat.com <mailto:m...@redhat.com>>>
> > <mailto:m...@redhat.com <mailto:m...@redhat.com>
<mailto:m...@redhat.com <mailto:m...@redhat.com>>
> <mailto:m...@redhat.com <mailto:m...@redhat.com>
<mailto:m...@redhat.com <mailto:m...@redhat.com>>>>>
> > > > > <mailto:m...@redhat.com
<mailto:m...@redhat.com>
> <mailto:m...@redhat.com <mailto:m...@redhat.com>>
<mailto:m...@redhat.com <mailto:m...@redhat.com>
<mailto:m...@redhat.com <mailto:m...@redhat.com>>>
> > <mailto:m...@redhat.com <mailto:m...@redhat.com>
<mailto:m...@redhat.com <mailto:m...@redhat.com>>
> <mailto:m...@redhat.com <mailto:m...@redhat.com>
<mailto:m...@redhat.com <mailto:m...@redhat.com>>>>
> > > <mailto:m...@redhat.com <mailto:m...@redhat.com>
<mailto:m...@redhat.com <mailto:m...@redhat.com>>
> <mailto:m...@redhat.com <mailto:m...@redhat.com>
<mailto:m...@redhat.com <mailto:m...@redhat.com>>>
> > <mailto:m...@redhat.com <mailto:m...@redhat.com>
<mailto:m...@redhat.com <mailto:m...@redhat.com>>
> <mailto:m...@redhat.com <mailto:m...@redhat.com>
<mailto:m...@redhat.com <mailto:m...@redhat.com>>>>>>> wrote:
> > > > >
> > > > > On Thu, Nov 02, 2023 at 11:09:27AM
> +0200, Yuri
> > > Benditovich wrote:
> > > > > > Probably we mix two different
patches
> in this
> > > discussion.
> > > > > > Focusing on the patch in the
e-mail
> header:
> > > > > >
> > > > > > IMO it is not acceptable to
fail QEMU run
> > for one
> > > feature
> > > > that we
> > > > > can't make
> > > > > > active when we silently drop
all other
> > features in
> > > such a
> > > > case.
> > > > >
> > > > > If the feature is off by default
then it
> seems more
> > > reasonable
> > > > > and silent masking can be seen as
a bug.
> > > > > Most virtio features are on by
default
> this is
> > why it's
> > > > > reasonable to mask them.
> > > > >
> > > > >
> > > > > If we are talking about RSS: setting it
> initially
> > off is the
> > > > development
> > > > > time decision.
> > > > > When it will be completely stable
there is
> no reason to
> > > keep it
> > > > off by
> > > > > default, so this is more a question
of time
> and of a
> > > readiness of
> > > > libvirt.
> > > >
> > > > It is not ok to make "on" the default;
that will
> > enable RSS
> > > even when
> > > > eBPF steering support is not present and can
> result in
> > > performance
> > > > degradation.
> > > >
> > > >
> > > > Exactly as it is today - with vhost=on the host
> does not
> > suggest RSS
> > > > without eBPF.
> > > > I do not understand what you call "performance
> > degradation", can you
> > > > describe the scenario?
> > >
> > > I was not clear, but I was talking about the
case of
> > vhost=off or peers
> > > other than tap (e.g., user). rss=on employs
in-qemu RSS,
> > which incurs
> > > overheads for such configurations.
> > >
> > >
> > > So, vhost=off OR peers other than tap:
> > >
> > > In the case of peers other than tap (IMO) we're not
> talking about
> > > performance at all.
> > > Backends like "user" (without vnet_hdr) do not
support _many_
> > > performance-oriented features.
> > > If RSS is somehow "supported" for such backends this is
> rather a
> > > misunderstanding (IMO again).
> >
> > We do not need to ensure good performance when RSS is
enabled
> by the
> > guest for backends without eBPF steering program as
you say.
> In-QEMU
> > RSS
> > is only useful for testing and not meant to improve the
> performance.
> >
> > However, if you set rss=on, QEMU will advertise the
> availability of RSS
> > feature. The guest will have no mean to know if it's
> implemented in a
> > way not performance-wise so it may decide to use the
feature
> to improve
> > the performance, which can result in performance
degradation.
> > Therefore,
> > it's better not to set rss=on for such backends.
> >
> >
> > I still do not understand what is the scenario where you
see or
> suspect
> > the mentioned "performance degradation".
> > We can discuss whether such a problem exists as soon as you
> explain it.
>
> The scenario is that:
> - rss=on,
> - A backend without eBPF steering support is in use, and
> - The guest expects VIRTIO_NET_F_RSS has little overheads as
hardware
> RSS implementations do.
>
> I consider the risk of the performance degradation in such a
situation
> is the reason why virtio-net emits a warning ("Can't load
eBPF RSS -
> fallback to software RSS") when in-QEMU RSS is in use.
>
>
> In a described scenario (vhost=off) I do not see why the performance
> degradation should happen:
> the SW RSS (if activated) will place each packet into proper
queue (even
> if the auto_mq in kernel is not able to do that) and such a way the
> guest will not need to reschedule the packet to proper CPU
>
The scenario I'm concerned is that the guest has its own packet
steering
mechanism which is feature-wise superior to RSS. For example, Linux has
such a mechanism called RPS, which has some advantages due to its
extensible nature according to:
https://www.kernel.org/doc/html/v6.6/networking/scaling.html#rps-receive-packet-steering
<https://www.kernel.org/doc/html/v6.6/networking/scaling.html#rps-receive-packet-steering>
Such a guest may still prefer hardware RSS if available since hardware
RSS is expected to have less overheads. However, it is not true for
in-qemu RSS, and using in-QEMU RSS instead of the guest-side steering
mechanism may just hide useful features the guest-side steering
mechanism has and result in performance degradation.
Note that in terms of per-packet computation for RSS the in-QEMU RSS
does exactly the same operations in native code that the eBPF does in
the emulation.
So, I wouldn't say that SW RSS brings some "performance degradation".
We prefer eBPF as it can serve both vhost and non-vhost setups.
I see the eBPF steering program in a different way.
tuntap can have several queues, and the assignment of packets is
automatically done by tuntap by default. However, the default assignment
policy may not be good for all applications, and some may need a
different policy. Such an application can still re-assign packets to
application-internal queues that reside on different CPUs and that's
what 'in-qemu' RSS does. However, that incurs communication overheads
across CPUs. The eBPF steering program can eliminate the overheads by
allowing the userspace to override the packet assignment policy with eBPF.
The eBPF steering program feature of tuntap and its benefit are
independent of the vhost usage. In fact, the feature predates the usage
in QEMU that combines vhost and eBPF steering program. The initial
proposal of the eBPF steering program I found is submitted by Jason and
available at:
https://lore.kernel.org/lkml/1506500637-13881-1-git-send-email-jasow...@redhat.com/
Jason, please point out if you find something wrong in my understanding
with the eBPF steering program feature or something to add.
