Thank you for such a detailed reply! Currently what's in my mind is to let memory also be migrated when a vcpu or a vm has to be migrated to some other node. In that case let's suppose there would be a simple load balancing mechanism. And I think a NUMA machine can be some sort of seen as small symmetric 'machines' connected together. So in my mind I think migrating both vcpu and memory will be similar to a live migration. Yes that's the 'local migration' in my mind. But as most time I am in Xen4.1.2, I think I need to learn about the cpu affinity now and perhaps disscus with you later, if you're convinent. (But I'm still wondering if it's posdible to add some code to specify the target node....)
A local migration will be the best if we have a way to specify the target node. However my plan B is to follow the migration procedures to handle the memory . In such way maybe I could be able to bypass some tricky part. :) And if we see it a little further, migrating all memory pages is not economic as (supposing that) the load is heavy and perhaps a short time little you have to move a vcpu again. Then given the locality theory maybe we should only migrate the recent used pages and then others if needed. However that's my illusions as there's no base for the whole building now.LOL. Thank you, I'm really appreciated. (And it's really weird to type on a ipad screen) Dario Faggioli <dario.faggi...@citrix.com>于2015年7月31日 周五20:46写道： > On Fri, 2015-07-31 at 02:32 +0000, Kun Cheng wrote: > > Hi all, > > > Hi, > > > > I'm sorry for taking your time and I'd like to make an enquery about > > the status of VM migration support on a NUMA server. > > > Status is: it's not there, and won't happen soon. I've started working > on it, but then got preempted by other issues, and concentrated on > making Xen do the best possible _without_ moving the memory (e.g., with > NUMA-aware scheduling, now achieved through per-vcpu soft affinities). > > Moving memory around is really really tricky. It's probably at least > doable for HVM guests, while, for PV, I'm not even so sure it can be > done! :-/ > > > Currently It looks like when a vm is migrated only its vcpus are moved > > to the other node but not its memory. So, is anyone trying to fix that > > issue? > > > What do you mean with "when a vm is migrated"? If soft affinity for a VM > is specified in the config file (or afterwards, but I'd recommend to do > it in the config file, if you're interested in NUMA effects), memory is > allocated from the NUMA node that such affinity spans, and the Xen > scheduler (provided you're using Credit1, our default scheduler), will > try as hard as it can to schedule the vcpus of the vm on the pcpus of > that same node (or set of nodes). > > If it's not possible, because all those pcpus are busy, the vcpus are > allowed to run on some other pcpu, outside of the NUMA node(s) the vm > has affinity with, on the basis that some execution, even with slow > memory access, is better than no execution at all. > > If you're interested in having the vcpus of the vm _only_ running on the > pcpus of the node to which the memory is attached, I'd suggest using > hard affinity, instead than soft (still specifying it in the config > file). > > Support for soft affinity in Credit2 is being worked on. For other > schedulers (ARINC and RTDS), it's not that useful. > > > If I want to do it myself, it seems like two major problems are ahead > > of me: > > > > 1) How to specify the target node for memory migration? I'll be > > grateful if anyone can give me some hints. > > > Not sure I get. In my mind, if we will have this in place at some point, > migration will happen, either: > - automatically, upon load balancing considerations > - manually, with dedicated libxl interfaces and xl command > > at that point, for the latter case, there will be a way of specifying a > target node, and that will most likely be an integer, or a list of > integers... > > > 2) Memory Migration. Looks like it can be done by leveraging the > > existing migration related functions on Xen. > > > Mmmm... Maybe I see what you mean now. So, you want to perform a local > migration, and use that as a way of actually moving the guest to another > node, is this correct? If yes, it did work, last time I checked. > > If doing this like that, it's true that you don't have any way for > specifying a target node. Therefore, what happens is, either: > - if no soft or hard affinity is specified in the config file, the > automatic NUMA placement code will run, and it most likely will > choose a different node for the target vm, but not in a way that you > can control easily > - if any affinity is set, the vm will be re-created in the same exact > node. > > That is why, a way to workaround this, and actually use local migration > as a memory-migration mechanism, is to leverage `xl config-update'. In > fact, you can do as follows: > > # xl create vm.cfg 'cpus_soft="node:1'" > # xl config-update <domid> 'cpus_soft="node:0"' > # <do a local migration> > > As I said, this all worked last time I tried... Is it not working for > you? Or was it something else you were after? > > Regards, > Dario > > -- > <<This happens because I choose it to happen!>> (Raistlin Majere) > ----------------------------------------------------------------- > Dario Faggioli, Ph.D, http://about.me/dario.faggioli > Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) >
_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
