* Daniel P. Berrangé (berra...@redhat.com) wrote:
> On Wed, Sep 01, 2021 at 11:35:33AM -0400, Peter Xu wrote:
> > On Wed, Sep 01, 2021 at 09:53:07AM +0100, Daniel P. Berrangé wrote:
> > > On Tue, Aug 31, 2021 at 04:29:09PM -0400, Peter Xu wrote:
> > > > On Tue, Aug 31, 2021 at 02:16:42PM +0100, Daniel P. Berrangé wrote:
> > > > > On Tue, Aug 31, 2021 at 08:02:39AM -0300, Leonardo Bras wrote:
> > > > > > Call qio_channel_set_zerocopy(true) in the start of every multifd
> > > > > > thread.
> > > > > >
> > > > > > Change the send_write() interface of multifd, allowing it to pass
> > > > > > down
> > > > > > flags for qio_channel_write*().
> > > > > >
> > > > > > Pass down MSG_ZEROCOPY flag for sending memory pages, while keeping
> > > > > > the
> > > > > > other data being sent at the default copying approach.
> > > > > >
> > > > > > Signed-off-by: Leonardo Bras <leob...@redhat.com>
> > > > > > ---
> > > > > > migration/multifd-zlib.c | 7 ++++---
> > > > > > migration/multifd-zstd.c | 7 ++++---
> > > > > > migration/multifd.c | 9 ++++++---
> > > > > > migration/multifd.h | 3 ++-
> > > > > > 4 files changed, 16 insertions(+), 10 deletions(-)
> > > > >
> > > > > > @@ -675,7 +676,8 @@ static void *multifd_send_thread(void *opaque)
> > > > > > }
> > > > > >
> > > > > > if (used) {
> > > > > > - ret = multifd_send_state->ops->send_write(p, used,
> > > > > > &local_err);
> > > > > > + ret = multifd_send_state->ops->send_write(p, used,
> > > > > > MSG_ZEROCOPY,
> > > > > > +
> > > > > > &local_err);
> > > > >
> > > > > I don't think it is valid to unconditionally enable this feature due
> > > > > to the
> > > > > resource usage implications
> > > > >
> > > > > https://www.kernel.org/doc/html/v5.4/networking/msg_zerocopy.html
> > > > >
> > > > > "A zerocopy failure will return -1 with errno ENOBUFS. This happens
> > > > > if the socket option was not set, the socket exceeds its optmem
> > > > > limit or the user exceeds its ulimit on locked pages."
> > > > >
> > > > > The limit on locked pages is something that looks very likely to be
> > > > > exceeded unless you happen to be running a QEMU config that already
> > > > > implies locked memory (eg PCI assignment)
> > > >
> > > > Yes it would be great to be a migration capability in parallel to
> > > > multifd. At
> > > > initial phase if it's easy to be implemented on multi-fd only, we can
> > > > add a
> > > > dependency between the caps. In the future we can remove that
> > > > dependency when
> > > > the code is ready to go without multifd. Thanks,
> > >
> > > Also, I'm wondering how zerocopy support interacts with kernel support
> > > for kTLS and multipath-TCP, both of which we want to be able to use
> > > with migration.
> >
> > Copying Jason Wang for net implications between these features on kernel
> > side
> > and whether they can be enabled together (MSG_ZEROCOPY, mptcp, kTLS).
> >
> > From the safe side we may want to only enable one of them until we prove
> > they'll work together I guess..
>
> MPTCP is good when we're network limited for migration
>
> KTLS will be good when we're CPU limited on AES for migration,
> which is essentially always when TLS is used.
>
> ZEROCOPY will be good when we're CPU limited for data copy
> on migration, or to reduce the impact on other concurrent
> VMs on the same CPUs.
>
> Ultimately we woudld benefit from all of them at the same
> time, if it were technically possible todo.
I think last time I spoke to Paolo Abeni there were some interactions
between them; I can't remember what though (I think mptcp and ktls
didn't play at the time).
Dave
> > Not a immediate concern as I don't really think any of them is really
> > explicitly supported in qemu.
>
> QEMU has mptcp support already:
>
> commit 8bd1078aebcec5eac196a83ef1a7e74be0ba67b7
> Author: Dr. David Alan Gilbert <dgilb...@redhat.com>
> Date: Wed Apr 21 12:28:34 2021 +0100
>
> sockets: Support multipath TCP
>
> Multipath TCP allows combining multiple interfaces/routes into a single
> socket, with very little work for the user/admin.
>
> It's enabled by 'mptcp' on most socket addresses:
>
> ./qemu-system-x86_64 -nographic -incoming tcp:0:4444,mptcp
>
> > KTLS may be implicitly included by a new gnutls, but we need to mark TLS and
> > ZEROCOPY mutual exclusive anyway because at least the userspace TLS code of
> > gnutls won't has a way to maintain the tls buffers used by zerocopy. So at
> > least we need some knob to detect whether kTLS is enabled in gnutls.
>
> It isn't possible for gnutls to transparently enable KTLS, because
> GNUTLS doesn't get to see the actual socket directly - it'll need
> some work in QEMU to enable it. We know MPTCP and KTLS are currently
> mutually exclusive as they both use the same kernel network hooks
> framework.
>
> Regards,
> Daniel
> --
> |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org -o- https://fstop138.berrange.com :|
> |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
>
--
Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK
