Hi, I've been playing with getting an RDMA setup for COLO and have something that mostly works, but it is very new and quite hacky; but I thought I'd share my work so far.
You can find it at: https://github.com/orbitfp7/qemu/commits/orbit-wp4-colo-dec What I've done is: a) Wire up a partner TCP connection by the side of the RDMA connection. b) Use the TCP connection just for the responses from secondary->primary c) Make the RDMA connection write to the colo-cache after the first migrate d) Make the RDMA connection notify the secondary when it sends writes, so that the secondary can know that it needs to flush those pages in the colo-cache. e) Add a shutdown function and fix some other bugs I've had that working on both your current world (which is what that tree is based off) and your older COLO world from July (with a bit more hacking to make it take the newer patches). Looking at the speed: a) The CPU load on the incoming thread is much lower - maybe only 10-11% instead of 30-40%. b) The performance of guest code is a little slower (~10% slower?) on RDMA rather than TCP (on both 10Gbps and 40Gbps links) I've not worked out why yet. (My guess is it could be to do with RDMA dynamic registration) Things I know I need to do: 1) Tidy it up - it's very messy! 2) Try and get rid of the TCP connection and use an RDMA channel for the backwards connection 3) Make sure the shutdown really can cope with the other host being dead. 4) It only deals with the dynamic registration mode of RDMA; setting pin-all will probably break it. 5) Figure out why it's slower! 6) Test failover more. My work on this is part of the EU Orbit project ( http://www.orbitproject.eu/ ) Dave -- Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK
