Resolving this one, but please reopen if you disagree. I have filed
https://bugs.launchpad.net/nova/+bug/1154162 to track adding caching to
our calls to "qemu-img info".
** Changed in: nova
Importance: High => Undecided
** Changed in: nova
Status: Confirmed => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1078811
Title:
nova-compute instance creation time increases as instance count
increases
Status in OpenStack Compute (Nova):
Invalid
Status in OpenStack Manuals:
New
Bug description:
I'm seeing some odd behavior with nova-compute taking exponentially
longer to fully create/spawn instances as the number of instances on
any given node increases.
This doesn't seem to be load/CPU utilization related as I'm still
fairly idle on that front (my two nova-compute nodes have 40 cores
each....4x 10 core sockets).
Occasionally the instance spawn seems to get caught in some audit task
where nova-compute will run 'qemu-img info' against each of the
instance disks. This process typically inputs a very large delay
between the instance spawn AMQP message to nova-compute (see timestamp
2012-11-13 05:59:54 in the attached logfile), and nova-compute
generating the XML for the instance to send to libvirt/KVM (see
timestamp 2012-11-13 06:19:26 in the attached logfile). While in this
example it took about 20 minutes, this can cause the instances to take
as much as an hour to spawn, depending on how many instances are
hosted on that nova-compute node.
In other occasions (even with a similarly high instance count, and
sometimes right after an instance that took significantly longer to
create), there is no delay between the instance creation task start
(see timestamp 2012-11-13 06:30:09 in the attached logfile) and the
XML generation for libvirt (see timestamp 2012-11-13 06:30:21 in the
attached logfile). Likewise, these instances create predictably and
consistently (usually about 2 or 3 minutes after sending the XML to
libvirt/KVM on a busy host).
The inconsistent and sometimes excessively long instance creation time
is definitely causing me some headaches with automation and general
expectations to set around performance in our OpenStack environment.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1078811/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
