On Thu, Jan 13, 2022 at 7:34 AM Peter Xu <pet...@redhat.com> wrote:
>
> On Thu, Jan 13, 2022 at 10:06:14AM +0000, Daniel P. Berrangé wrote:
> > On Thu, Jan 13, 2022 at 02:48:15PM +0800, Peter Xu wrote:
> > > On Thu, Jan 06, 2022 at 07:13:39PM -0300, Leonardo Bras wrote:
> > > > @@ -558,15 +575,26 @@ static ssize_t
> > > > qio_channel_socket_writev(QIOChannel *ioc,
> > > > memcpy(CMSG_DATA(cmsg), fds, fdsize);
> > > > }
> > > >
> > > > + if (flags & QIO_CHANNEL_WRITE_FLAG_ZERO_COPY) {
> > > > + sflags = MSG_ZEROCOPY;
> > > > + }
> > > > +
> > > > retry:
> > > > - ret = sendmsg(sioc->fd, &msg, 0);
> > > > + ret = sendmsg(sioc->fd, &msg, sflags);
> > > > if (ret <= 0) {
> > > > - if (errno == EAGAIN) {
> > > > + switch (errno) {
> > > > + case EAGAIN:
> > > > return QIO_CHANNEL_ERR_BLOCK;
> > > > - }
> > > > - if (errno == EINTR) {
> > > > + case EINTR:
> > > > goto retry;
> > > > + case ENOBUFS:
> > > > + if (sflags & MSG_ZEROCOPY) {
> > > > + error_setg_errno(errp, errno,
> > > > + "Process can't lock enough memory for
> > > > using MSG_ZEROCOPY");
> > > > + return -1;
> > > > + }
> > >
> > > I have no idea whether it'll make a real differnece, but - should we
> > > better add
> > > a "break" here? If you agree and with that fixed, feel free to add:
> > >
> > > Reviewed-by: Peter Xu <pet...@redhat.com>
> > >
> > > I also wonder whether you hit ENOBUFS in any of the environments. On
> > > Fedora
> > > here it's by default unlimited, but just curious when we should keep an
> > > eye.
> >
> > Fedora doesn't allow unlimited locked memory by default
> >
> > $ grep "locked memory" /proc/self/limits
> > Max locked memory 65536 65536 bytes
> >
> > And regardless of Fedora defaults, libvirt will set a limit
> > for the guest. It will only be unlimited if requiring certain
> > things like VFIO.
>
> Thanks, I obviously checked up the wrong host..
>
> Leo, do you know how much locked memory will be needed by zero copy? Will
> there be a limit? Is it linear to the number of sockets/channels?
It depends on the number of channels, of course, but there are
influencing factors, like network bandwidth & usage, and cpu speed &
usage, network queue size, VM pagesize and so on.
A simple exemple:
If the cpu is free/fast, but there are other applications using the
network, we may enqueue a lot of stuff for sending, and end up needing
a lot of locked memory.
I don't think it's easy to calculate a good reference value for locked
memory here.
>
> It'll be better if we can fail at enabling the feature when we detected that
> the specified locked memory limit may not be suffice.
I agree it's a good idea. But having this reference value calculated
is not much simple, IIUC.
>
> --
> Peter Xu
>
