On Tue, Feb 08, 2022 at 01:23:29PM +0000, Dr. David Alan Gilbert wrote: > * Peter Xu (pet...@redhat.com) wrote: > > On Tue, Feb 08, 2022 at 11:24:14AM +0000, Dr. David Alan Gilbert wrote: > > > > The current model is we only have 1 postcopy channel and 1 precopy > > > > channel, but > > > > it should be easier if we want to make it N post + 1 pre base on this > > > > series. > > > > > > It's not clear to me if we need to be able to do N post + M pre, or > > > whether we have a rule like always at least 1 post, but if there's more > > > pagefaults in the queue then you can steal all of the pre channels. > > > > Right, >1 queue length should easily happen with workloads in real cloud > > environment. Though even with only 1post channel we can already hit at > > least > > <~1ms with this series even if there're 16 pending requests per my test. I > > think that may cover quite some real workloads. > > > > One thing to mention is that we should always assume the pre-channels are > > filled up with tons of pages already in the NIC send buffer, so they won't > > be > > good candidate for postcopy requests, IMHO. So I'm not sure whether we can > > mixly use the pre/post channels - we may need to leave the post channels > > idle. > > No I'm not sure either; even with separate channels do we have problems > with contention on the NIC?
Not on the NIC, but on the same socket; assuming each multifd thread is working only with one socket. For example, we have N multifd threads/sockets. If we find some of them are "free" from multifd POV, it only means we can write() to those sockets, but it does not mean that these sockets have empty send buffer. IMHO that's the major problem we're facing: as long as a socket is shared between pre and post purposes, then the post pages can be after some pre pages. I think what we could to do to achieve provisioning of multi-sockets is to keep M out of N multifd sockets to be idle (M<N), servicing postcopy faults only. Though those M sockets/threads will work mostly different from the rest multifd threads, hence we probably can't call it multifd anymore.. -- Peter Xu
