Awesome Jay! Do you think this is something that can be backporting to Liberty w/o other dependencies? We're running Liberty on our system right now.
On Tue, Jun 14, 2016 at 4:10 PM Jay Pipes <jaypi...@gmail.com> wrote: > On 06/14/2016 12:30 PM, Paul Michali wrote: > > Well, looks like we figured out what is going on - maybe folks have some > > ideas on how we could handle this issue. > > > > What I see is that for each VM create (small flavor), 1024 huge pages > > are used and NUMA node 0 used. It appears that, when there is no longer > > enough huge pages on that NUMA node, Nova with then schedule to the > > other NUMA node and use those huge pages. > > > > In our case, we happen to have a special container running on the > > compute nodes, that uses 512 huge pages. As a result, when there are 768 > > huge pages left, Nova thinks there are 1280 pages left and thinks one > > more VM can be create. It tries, but the create fails. > > > > Some questions... > > > > 1) Is there some way to "reserve" huge pages in Nova? > > Yes. Code merged recently from Sahid does this: > > https://review.openstack.org/#/c/277422/ > > Best, > -jay > > > 2) If the create fails, should Nova try the other NUMA node (or is this > > because it doesn't know why it failed)? > > 3) Any ideas on how we can deal with this - without changing Nova? > > > > Thanks! > > > > PCM > > > > > > > > On Tue, Jun 14, 2016 at 1:09 PM Paul Michali <p...@michali.net > > <mailto:p...@michali.net>> wrote: > > > > Great info Chris and thanks for confirming the assignment of blocks > > of pages to a numa node. > > > > I'm still struggling with why each VM is being assigned to NUMA node > > 0. Any ideas on where I should look to see why Nova is not using > > NUMA id 1? > > > > Thanks! > > > > > > PCM > > > > > > On Tue, Jun 14, 2016 at 10:29 AM Chris Friesen > > <chris.frie...@windriver.com <mailto:chris.frie...@windriver.com>> > > wrote: > > > > On 06/13/2016 02:17 PM, Paul Michali wrote: > > > Hmm... I tried Friday and again today, and I'm not seeing the > > VMs being evenly > > > created on the NUMA nodes. Every Cirros VM is created on > > nodeid 0. > > > > > > I have the m1/small flavor (@GB) selected and am using > > hw:numa_nodes=1 and > > > hw:mem_page_size=2048 flavor-key settings. Each VM is > > consuming 1024 huge pages > > > (of size 2MB), but is on nodeid 0 always. Also, it seems that > > when I reach 1/2 > > > of the total number of huge pages used, libvirt gives an > > error saying there is > > > not enough memory to create the VM. Is it expected that the > > huge pages are > > > "allocated" to each NUMA node? > > > > Yes, any given memory page exists on one NUMA node, and a > > single-NUMA-node VM > > will be constrained to a single host NUMA node and will use > > memory from that > > host NUMA node. > > > > You can see and/or adjust how many hugepages are available on > > each NUMA node via > > /sys/devices/system/node/nodeX/hugepages/hugepages-2048kB/* > > where X is the host > > NUMA node number. > > > > Chris > > > > > > > __________________________________________________________________________ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: > > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > < > http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe> > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > > > > __________________________________________________________________________ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
