Re: [Openstack] launching multiple VMs takes very long time

2013-05-01 Thread Mark Lehrer
where should I start looking to find out why things are so slow? Check the glance node. The first prototype I set up had an incredibly slow glance node, which led to this kind of behavior. Are the compute nodes downloading at the same time, or sequentially? Mark ___

Re: [Openstack] launching multiple VMs takes very long time

2013-05-01 Thread Steve Heistand
so I dont think there is a network or disk bottle neck now. in watching instances launch I see that the disk image itself gets copied out to the compute node in < 1 minute, the xml file for libvirt gets created. then a pause of ~ 20 minutes then the kvm process starts up and off the VM goes just

Re: [Openstack] launching multiple VMs takes very long time

2013-05-01 Thread Rick Jones
On 05/01/2013 10:43 AM, Steve Heistand wrote: there may be some network issues going on here, trying to shove some amount of data bigger then a few Gig seems to start slowing things down. If you think there is an actual network problem, you could I suppose try exploring that with netperf or i

Re: [Openstack] launching multiple VMs takes very long time

2013-05-01 Thread Eric_E_Smith
12:43 PM To: openstack@lists.launchpad.net Subject: Re: [Openstack] launching multiple VMs takes very long time -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 ok a followup to this. its not any sort of disk access issue. I created a new disk to hold the glance directory and running atop while

Re: [Openstack] launching multiple VMs takes very long time

2013-05-01 Thread Steve Heistand
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 ok a followup to this. its not any sort of disk access issue. I created a new disk to hold the glance directory and running atop while launching shows all the activity on the original root drive nothing using glance. Normally at idle the root drive d

Re: [Openstack] launching multiple VMs takes very long time

2013-04-30 Thread Rick Jones
On 04/30/2013 01:36 PM, Melanie Witt wrote: This presentation from the summit might be of interest to you: http://www.openstack.org/summit/portland-2013/session-videos/presentation/scaling-the-boot-barrier-identifying-and-eliminating-contention-in-openstack A nice presentation. Based on his c

Re: [Openstack] launching multiple VMs takes very long time

2013-04-30 Thread Steve Heistand
this certainly looks promising.. thanks s On 04/30/2013 01:36 PM, Melanie Witt wrote: > This presentation from the summit might be of interest to you: > > http://www.openstack.org/summit/portland-2013/session-videos/presentation/scaling-the-boot-barrier-identifying-and-eliminating-contention-i

Re: [Openstack] launching multiple VMs takes very long time

2013-04-30 Thread Melanie Witt
This presentation from the summit might be of interest to you: http://www.openstack.org/summit/portland-2013/session-videos/presentation/scaling-the-boot-barrier-identifying-and-eliminating-contention-in-openstack I couldn't find just the slide deck anywhere so far. Your issue is different bein

Re: [Openstack] launching multiple VMs takes very long time

2013-04-30 Thread Steve Heistand
They are all booting the same image. there still may be a random read issues if they arent mostly synced in copying the file to the compute node. But there isnt much of a load on the disk while the instances are being launched. Its just a sata drive at this point so the performance isnt great but

Re: [Openstack] launching multiple VMs takes very long time

2013-04-30 Thread Eric_E_Smith
Subject: [Openstack] launching multiple VMs takes very long time if I launch one vm at a time its doesnt very long to start up the instance. maybe a minute. if I launch 4 instances (of the same snapshot as before) it takes 30 minutes. they are all launching to different compute nodes, the controllers

Re: [Openstack] launching multiple VMs takes very long time

2013-04-30 Thread Rick Jones
On 04/30/2013 11:42 AM, Steve Heistand wrote: if I launch one vm at a time its doesnt very long to start up the instance. maybe a minute. if I launch 4 instances (of the same snapshot as before) it takes 30 minutes. they are all launching to different compute nodes, the controllers are all mul

[Openstack] launching multiple VMs takes very long time

2013-04-30 Thread Steve Heistand
if I launch one vm at a time its doesnt very long to start up the instance. maybe a minute. if I launch 4 instances (of the same snapshot as before) it takes 30 minutes. they are all launching to different compute nodes, the controllers are all multicore, I dont see any processes on the compute