On Tue, May 07, 2024 at 01:50:43AM +0000, Gonglei (Arei) wrote: > Hello, > > > -----Original Message----- > > From: Peter Xu [mailto:pet...@redhat.com] > > Sent: Monday, May 6, 2024 11:18 PM > > To: Gonglei (Arei) <arei.gong...@huawei.com> > > Cc: Daniel P. Berrangé <berra...@redhat.com>; Markus Armbruster > > <arm...@redhat.com>; Michael Galaxy <mgal...@akamai.com>; Yu Zhang > > <yu.zh...@ionos.com>; Zhijian Li (Fujitsu) <lizhij...@fujitsu.com>; Jinpu > > Wang > > <jinpu.w...@ionos.com>; Elmar Gerdes <elmar.ger...@ionos.com>; > > qemu-devel@nongnu.org; Yuval Shaia <yuval.shaia...@gmail.com>; Kevin Wolf > > <kw...@redhat.com>; Prasanna Kumar Kalever > > <prasanna.kale...@redhat.com>; Cornelia Huck <coh...@redhat.com>; > > Michael Roth <michael.r...@amd.com>; Prasanna Kumar Kalever > > <prasanna4...@gmail.com>; integrat...@gluster.org; Paolo Bonzini > > <pbonz...@redhat.com>; qemu-bl...@nongnu.org; de...@lists.libvirt.org; > > Hanna Reitz <hre...@redhat.com>; Michael S. Tsirkin <m...@redhat.com>; > > Thomas Huth <th...@redhat.com>; Eric Blake <ebl...@redhat.com>; Song > > Gao <gaos...@loongson.cn>; Marc-André Lureau > > <marcandre.lur...@redhat.com>; Alex Bennée <alex.ben...@linaro.org>; > > Wainer dos Santos Moschetta <waine...@redhat.com>; Beraldo Leal > > <bl...@redhat.com>; Pannengyuan <pannengy...@huawei.com>; > > Xiexiangyou <xiexiang...@huawei.com> > > Subject: Re: [PATCH-for-9.1 v2 2/3] migration: Remove RDMA protocol handling > > > > On Mon, May 06, 2024 at 02:06:28AM +0000, Gonglei (Arei) wrote: > > > Hi, Peter > > > > Hey, Lei, > > > > Happy to see you around again after years. > > > Haha, me too. > > > > RDMA features high bandwidth, low latency (in non-blocking lossless > > > network), and direct remote memory access by bypassing the CPU (As you > > > know, CPU resources are expensive for cloud vendors, which is one of > > > the reasons why we introduced offload cards.), which TCP does not have. > > > > It's another cost to use offload cards, v.s. preparing more cpu resources? > > > Software and hardware offload converged architecture is the way to go for all > cloud vendors > (Including comprehensive benefits in terms of performance, cost, security, > and innovation speed), > it's not just a matter of adding the resource of a DPU card. > > > > In some scenarios where fast live migration is needed (extremely short > > > interruption duration and migration duration) is very useful. To this > > > end, we have also developed RDMA support for multifd. > > > > Will any of you upstream that work? I'm curious how intrusive would it be > > when adding it to multifd, if it can keep only 5 exported functions like > > what > > rdma.h does right now it'll be pretty nice. We also want to make sure it > > works > > with arbitrary sized loads and buffers, e.g. vfio is considering to add IO > > loads to > > multifd channels too. > > > > In fact, we sent the patchset to the community in 2021. Pls see: > https://lore.kernel.org/all/20210203185906.GT2950@work-vm/T/
I wasn't aware of that for sure in the past.. Multifd has changed quite a bit in the last 9.0 release, that may not apply anymore. One thing to mention is please look at Dan's comment on possible use of rsocket.h: https://lore.kernel.org/all/zjjm6rcqs5eho...@redhat.com/ And Jinpu did help provide an initial test result over the library: https://lore.kernel.org/qemu-devel/camgffek8wiknqmouyxcathgtiem2dwocf_w7t0vmcd-i30t...@mail.gmail.com/ It looks like we have a chance to apply that in QEMU. > > > > One thing to note that the question here is not about a pure performance > > comparison between rdma and nics only. It's about help us make a decision > > on whether to drop rdma, iow, even if rdma performs well, the community > > still > > has the right to drop it if nobody can actively work and maintain it. > > It's just that if nics can perform as good it's more a reason to drop, > > unless > > companies can help to provide good support and work together. > > > > We are happy to provide the necessary review and maintenance work for RDMA > if the community needs it. > > CC'ing Chuan Zheng. I'm not sure whether you and Jinpu's team would like to work together and provide a final solution for rdma over multifd. It could be much simpler than the original 2021 proposal if the rsocket API will work out. Thanks, -- Peter Xu
