I found the problem.
disconnectPhysicalDiskByPath is being passed in (in my situation) the
following:
/var/lib/libvirt/images/9887d511-8dc7-4cb4-96f9-01230fe4bbb6
Due to the name of the method, my code was expecting data such as the
following:
/dev/disk/by-path/ip-192.168.233.10:3260-iscsi-iqn.2012-03.com.solidfire:volume1-lun-0
Was it intentional to send the data into this method in the current way?
On Wed, Oct 23, 2013 at 1:57 PM, Mike Tutkowski <
mike.tutkow...@solidfire.com> wrote:
> You know, I forgot we supposed to be doing that! :) Multi-tasking too much
> today, I guess.
>
> Anyways, it must not be working because I still had a hypervisor
> connection after I shut down the VM.
>
> Let me investigate.
>
>
> On Wed, Oct 23, 2013 at 1:48 PM, Marcus Sorensen <shadow...@gmail.com>wrote:
>
>> Are we not disconnecting when we stop the vm? There's a method for it, we
>> should be. disconnectPhysicalDiskViaVmSpec
>> On Oct 23, 2013 1:28 PM, "Mike Tutkowski" <mike.tutkow...@solidfire.com>
>> wrote:
>>
>>> I see one problem for us now, Marcus.
>>>
>>> * You have a running VM that you attach a volume to.
>>> * You stop the VM.
>>> * You detach the volume.
>>> * You start up the VM.
>>>
>>> The VM will not be connected to the volume (which is good), but the
>>> hypervisor will still be connected to the volume.
>>>
>>> It would be great if we actually sent a command to the last host ID of
>>> the stopped VM when detaching a volume (to have the hypervisor disconnect
>>> from the volume).
>>>
>>> What do you think about that?
>>>
>>>
>>> On Wed, Oct 23, 2013 at 1:15 PM, Mike Tutkowski <
>>> mike.tutkow...@solidfire.com> wrote:
>>>
>>>> OK, whatever way you prefer then, Marcus (createVdb first or second).
>>>>
>>>> If I leave createVdb first and return 0, it does seem to work.
>>>>
>>>>
>>>> On Wed, Oct 23, 2013 at 11:13 AM, Marcus Sorensen
>>>> <shadow...@gmail.com>wrote:
>>>>
>>>>> I think we could flip-flop these two lines if necessary:
>>>>>
>>>>> createVbd(conn, vmSpec, vmName, vm);
>>>>>
>>>>> _storagePoolMgr.connectPhysicalDisksViaVmSpec(vmSpec);
>>>>>
>>>>> I haven't actually tried it though. But in general I don't see the
>>>>> Libvirt DiskDef using size at all, which is what createVbd does
>>>>> (creates XML definitions for disks to attach to the VM definition). It
>>>>> just takes the device at it's native advertised size when it actually
>>>>> goes to use it.
>>>>>
>>>>> On Wed, Oct 23, 2013 at 10:52 AM, Mike Tutkowski
>>>>> <mike.tutkow...@solidfire.com> wrote:
>>>>> > Little problem that I wanted to get your take on, Marcus.
>>>>> >
>>>>> > When a VM is being started, we call createVdb before calling
>>>>> > connectPhysicalDisksViaVmSpec.
>>>>> >
>>>>> > The problem is that createVdb calls getPhysicalDisk and my volume
>>>>> has not
>>>>> > yet been connected because connectPhysicalDisksViaVmSpec has not yet
>>>>> been
>>>>> > called.
>>>>> >
>>>>> > When I try to read up the size of the disk to populate a
>>>>> PhysicalDisk, I get
>>>>> > an error, of course, because the path does not yet exist.
>>>>> >
>>>>> > I could populate a 0 for the size of the physical disk and then the
>>>>> next
>>>>> > time getPhysicalDisk is called, it should be filled in with a proper
>>>>> size.
>>>>> >
>>>>> > Do you see a problem with that approach?
>>>>> >
>>>>> >
>>>>> > On Tue, Oct 22, 2013 at 6:40 PM, Marcus Sorensen <
>>>>> shadow...@gmail.com>
>>>>> > wrote:
>>>>> >>
>>>>> >> That's right. All should be well.
>>>>> >>
>>>>> >> On Oct 22, 2013 6:03 PM, "Mike Tutkowski" <
>>>>> mike.tutkow...@solidfire.com>
>>>>> >> wrote:
>>>>> >>>
>>>>> >>> Looks like we disconnect physical disks when the VM is stopped.
>>>>> >>>
>>>>> >>> I didn't see that before.
>>>>> >>>
>>>>> >>> I suppose that means the disks are physically disconnected when
>>>>> the VM is
>>>>> >>> stopped, but the CloudStack DB still has the VM associated with
>>>>> the disks
>>>>> >>> for the next time the VM may be started up (unless someone does a
>>>>> disconnect
>>>>> >>> while the VM is in the Stopped State).
>>>>> >>>
>>>>> >>>
>>>>> >>> On Tue, Oct 22, 2013 at 4:19 PM, Mike Tutkowski
>>>>> >>> <mike.tutkow...@solidfire.com> wrote:
>>>>> >>>>
>>>>> >>>> Hey Marcus,
>>>>> >>>>
>>>>> >>>> Quick question for you related to attaching/detaching volumes
>>>>> when the
>>>>> >>>> VM is in the Stopped State.
>>>>> >>>>
>>>>> >>>> If I detach a volume from a VM that is in the Stopped State, the
>>>>> DB
>>>>> >>>> seems to get updated, but I don't see a command going to the KVM
>>>>> hypervisor
>>>>> >>>> that leads to the removal of the iSCSI target.
>>>>> >>>>
>>>>> >>>> It seems the iSCSI target is only removed the next time the VM is
>>>>> >>>> started.
>>>>> >>>>
>>>>> >>>> Do you know if this is true?
>>>>> >>>>
>>>>> >>>> If it is, I'm concerned that the volume could be attached to
>>>>> another VM
>>>>> >>>> before the Stopped VM is re-started and when the Stopped VM gets
>>>>> restarted
>>>>> >>>> that it would disconnect the iSCSI volume from underneath the VM
>>>>> that now
>>>>> >>>> has the volume attached.
>>>>> >>>>
>>>>> >>>> I still want to perform some tests on this, but am first trying
>>>>> to get a
>>>>> >>>> VM to start after I've attached a volume to it when it was in the
>>>>> Stopped
>>>>> >>>> State.
>>>>> >>>>
>>>>> >>>> Thanks,
>>>>> >>>> Mike
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> On Mon, Oct 21, 2013 at 10:57 PM, Mike Tutkowski
>>>>> >>>> <mike.tutkow...@solidfire.com> wrote:
>>>>> >>>>>
>>>>> >>>>> Thanks for that info, Marcus.
>>>>> >>>>>
>>>>> >>>>> By the way, I wanted to see if I could attach my volume to a VM
>>>>> in the
>>>>> >>>>> Stopped State.
>>>>> >>>>>
>>>>> >>>>> The attach logic didn't trigger any exceptions; however, when I
>>>>> started
>>>>> >>>>> the VM, I received an Insufficient Capacity exception.
>>>>> >>>>>
>>>>> >>>>> If I detach the volume and then start the VM, the VM starts just
>>>>> fine.
>>>>> >>>>>
>>>>> >>>>> I noticed a problem here (in StoragePoolHostDaoImpl):
>>>>> >>>>>
>>>>> >>>>> @Override
>>>>> >>>>>
>>>>> >>>>> public StoragePoolHostVO findByPoolHost(long poolId, long
>>>>> hostId) {
>>>>> >>>>>
>>>>> >>>>> SearchCriteria<StoragePoolHostVO> sc =
>>>>> PoolHostSearch.create();
>>>>> >>>>>
>>>>> >>>>> sc.setParameters("pool_id", poolId);
>>>>> >>>>>
>>>>> >>>>> sc.setParameters("host_id", hostId);
>>>>> >>>>>
>>>>> >>>>> return findOneIncludingRemovedBy(sc);
>>>>> >>>>>
>>>>> >>>>> }
>>>>> >>>>>
>>>>> >>>>> The findOneIncludingRemovedBy method returns null (the poolId is
>>>>> my
>>>>> >>>>> storage pool's ID and the hostId is the expected host ID).
>>>>> >>>>>
>>>>> >>>>> I'm not sure what this method is trying to do.
>>>>> >>>>>
>>>>> >>>>> I looked in the storage_pool_host_ref table (is that the correct
>>>>> >>>>> table?) and it only has one row, which maps the local storage
>>>>> pool of the
>>>>> >>>>> KVM host to the KVM host (which explains why no match is found
>>>>> for my
>>>>> >>>>> situation).
>>>>> >>>>>
>>>>> >>>>> Do you understand what this logic is trying to do?
>>>>> >>>>>
>>>>> >>>>> Thanks!
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>> On Mon, Oct 21, 2013 at 8:08 PM, Marcus Sorensen <
>>>>> shadow...@gmail.com>
>>>>> >>>>> wrote:
>>>>> >>>>>>
>>>>> >>>>>> Do you have the capability to clone the root disk? Normally the
>>>>> >>>>>> template is installed to primary, and then cloned for each root
>>>>> disk.
>>>>> >>>>>> In some cases (such as CLVM), this isn't efficient and so the
>>>>> template
>>>>> >>>>>> is copied fresh to populate each root disk.
>>>>> >>>>>>
>>>>> >>>>>> I'm actually not 100% sure how this works in the new code. It
>>>>> used to
>>>>> >>>>>> be handled by copyPhysicalDisk in the storage adaptor, called by
>>>>> >>>>>> copyTemplateToPrimaryStorage, which runs on the agent. It would
>>>>> pass
>>>>> >>>>>> template/secondary storage info, and the destination
>>>>> volume/primary
>>>>> >>>>>> storage info, and copyPhysicalDisk would do the work of
>>>>> installing the
>>>>> >>>>>> image to the destination. Then subsequent root disks would be
>>>>> cloned
>>>>> >>>>>> in CreateCommand by calling createDiskFromTemplate.
>>>>> >>>>>>
>>>>> >>>>>> In master it looks like this was moved to KVMStorageProcessor
>>>>> >>>>>> 'cloneVolumeFromBaseTemplate', although I think this just takes
>>>>> over
>>>>> >>>>>> as default, and there's something in your storage driver that
>>>>> should
>>>>> >>>>>> be capable of cloning templates on the mgmt server side. I'm
>>>>> less sure
>>>>> >>>>>> about how the template gets to primary storage in the first
>>>>> place, I
>>>>> >>>>>> assume copyTemplateToPrimaryStorage in KVMStorageProcessor
>>>>> calling
>>>>> >>>>>> copyPhysicalDisk in your adaptor. It's a bit tough for me to
>>>>> tell
>>>>> >>>>>> since our earlier storage adaptor did everything on the host it
>>>>> mostly
>>>>> >>>>>> just worked with the default stuff.
>>>>> >>>>>>
>>>>> >>>>>> On Mon, Oct 21, 2013 at 4:49 PM, Mike Tutkowski
>>>>> >>>>>> <mike.tutkow...@solidfire.com> wrote:
>>>>> >>>>>> > Hey Marcus,
>>>>> >>>>>> >
>>>>> >>>>>> > So...now that this works well for data disks, I was wondering
>>>>> what
>>>>> >>>>>> > might be
>>>>> >>>>>> > involved in getting this process to work for root disks.
>>>>> >>>>>> >
>>>>> >>>>>> > Can you point me in the right direction as far as what gets
>>>>> invoked
>>>>> >>>>>> > when a
>>>>> >>>>>> > VM is being created on KVM (so that its root disk can be
>>>>> created and
>>>>> >>>>>> > the
>>>>> >>>>>> > necessary template laid down or ISO installed)?
>>>>> >>>>>> >
>>>>> >>>>>> > Thanks!
>>>>> >>>>>> >
>>>>> >>>>>> >
>>>>> >>>>>> > On Mon, Oct 21, 2013 at 1:14 PM, Mike Tutkowski
>>>>> >>>>>> > <mike.tutkow...@solidfire.com> wrote:
>>>>> >>>>>> >>
>>>>> >>>>>> >> Hey Marcus,
>>>>> >>>>>> >>
>>>>> >>>>>> >> Just wanted to let you know the branch of mine that has your
>>>>> code
>>>>> >>>>>> >> and mine
>>>>> >>>>>> >> appears to work well with regards to attaching a data disk
>>>>> to a
>>>>> >>>>>> >> running VM:
>>>>> >>>>>> >>
>>>>> >>>>>> >> fdisk -l from hypervisor:
>>>>> >>>>>> >>
>>>>> >>>>>> >> http://i.imgur.com/NkP5fo0.png
>>>>> >>>>>> >>
>>>>> >>>>>> >> fdisk -l from within VM:
>>>>> >>>>>> >>
>>>>> >>>>>> >> http://i.imgur.com/8YwiiC7.png
>>>>> >>>>>> >>
>>>>> >>>>>> >> I plan to do more testing on this over the coming days.
>>>>> >>>>>> >>
>>>>> >>>>>> >> If all goes well, perhaps we can check this code in by the
>>>>> end of
>>>>> >>>>>> >> the
>>>>> >>>>>> >> week?
>>>>> >>>>>> >>
>>>>> >>>>>> >> Talk to you later,
>>>>> >>>>>> >> Mike
>>>>> >>>>>> >>
>>>>> >>>>>> >>
>>>>> >>>>>> >> On Sun, Oct 20, 2013 at 10:23 PM, Mike Tutkowski
>>>>> >>>>>> >> <mike.tutkow...@solidfire.com> wrote:
>>>>> >>>>>> >>>
>>>>> >>>>>> >>> Don't ask me, but it works now (I've been having this
>>>>> trouble
>>>>> >>>>>> >>> quite a
>>>>> >>>>>> >>> while today).
>>>>> >>>>>> >>>
>>>>> >>>>>> >>> I guess the trick is to send you an e-mail. :)
>>>>> >>>>>> >>>
>>>>> >>>>>> >>>
>>>>> >>>>>> >>> On Sun, Oct 20, 2013 at 10:05 PM, Marcus Sorensen
>>>>> >>>>>> >>> <shadow...@gmail.com>
>>>>> >>>>>> >>> wrote:
>>>>> >>>>>> >>>>
>>>>> >>>>>> >>>> Did you create a service offering that uses local storage,
>>>>> or add
>>>>> >>>>>> >>>> a
>>>>> >>>>>> >>>> shared primary storage? By default there is no storage that
>>>>> >>>>>> >>>> matches the
>>>>> >>>>>> >>>> built in offerings.
>>>>> >>>>>> >>>>
>>>>> >>>>>> >>>> On Oct 20, 2013 9:39 PM, "Mike Tutkowski"
>>>>> >>>>>> >>>> <mike.tutkow...@solidfire.com>
>>>>> >>>>>> >>>> wrote:
>>>>> >>>>>> >>>>>
>>>>> >>>>>> >>>>> Hey Marcus,
>>>>> >>>>>> >>>>>
>>>>> >>>>>> >>>>> So, I went back to the branch of mine that has your code
>>>>> and
>>>>> >>>>>> >>>>> mine and
>>>>> >>>>>> >>>>> was able to create a new CloudStack install from scratch
>>>>> with it
>>>>> >>>>>> >>>>> (once
>>>>> >>>>>> >>>>> again, after manually deleting what was in
>>>>> >>>>>> >>>>> /var/lib/libvirt/images to the
>>>>> >>>>>> >>>>> get system VMs to start).
>>>>> >>>>>> >>>>>
>>>>> >>>>>> >>>>> Anyways, my system VMs are running now and I tried to
>>>>> kick off a
>>>>> >>>>>> >>>>> VM
>>>>> >>>>>> >>>>> using the CentOS 6.3 image you provided me a while back.
>>>>> >>>>>> >>>>>
>>>>> >>>>>> >>>>> The virtual router has a Status of Running; however, my
>>>>> VM fails
>>>>> >>>>>> >>>>> to
>>>>> >>>>>> >>>>> start (with the generic message of Insufficient Capacity).
>>>>> >>>>>> >>>>>
>>>>> >>>>>> >>>>> I've not seen this exception before (related to the VR).
>>>>> Do you
>>>>> >>>>>> >>>>> have
>>>>> >>>>>> >>>>> any insight into this?:
>>>>> >>>>>> >>>>>
>>>>> >>>>>> >>>>> com.cloud.exception.ResourceUnavailableException: Resource
>>>>> >>>>>> >>>>> [Pod:1] is
>>>>> >>>>>> >>>>> unreachable: Unable to apply userdata and password entry
>>>>> on
>>>>> >>>>>> >>>>> router
>>>>> >>>>>> >>>>> at
>>>>> >>>>>> >>>>>
>>>>> >>>>>> >>>>>
>>>>> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:3793)
>>>>> >>>>>> >>>>> at
>>>>> >>>>>> >>>>>
>>>>> >>>>>> >>>>>
>>>>> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyUserData(VirtualNetworkApplianceManagerImpl.java:3017)
>>>>> >>>>>> >>>>> at
>>>>> >>>>>> >>>>>
>>>>> >>>>>> >>>>>
>>>>> com.cloud.network.element.VirtualRouterElement.addPasswordAndUserdata(VirtualRouterElement.java:933)
>>>>> >>>>>> >>>>> at
>>>>> >>>>>> >>>>>
>>>>> >>>>>> >>>>>
>>>>> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareElement(NetworkOrchestrator.java:1172)
>>>>> >>>>>> >>>>> at
>>>>> >>>>>> >>>>>
>>>>> >>>>>> >>>>>
>>>>> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareNic(NetworkOrchestrator.java:1288)
>>>>> >>>>>> >>>>> at
>>>>> >>>>>> >>>>>
>>>>> >>>>>> >>>>>
>>>>> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(NetworkOrchestrator.java:1224)
>>>>> >>>>>> >>>>> at
>>>>> >>>>>> >>>>>
>>>>> >>>>>> >>>>>
>>>>> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:826)
>>>>> >>>>>> >>>>> at
>>>>> >>>>>> >>>>>
>>>>> >>>>>> >>>>>
>>>>> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:508)
>>>>> >>>>>> >>>>> at
>>>>> >>>>>> >>>>>
>>>>> >>>>>> >>>>>
>>>>> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:227)
>>>>> >>>>>> >>>>> at
>>>>> >>>>>> >>>>>
>>>>> >>>>>> >>>>>
>>>>> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
>>>>> >>>>>> >>>>> at
>>>>> >>>>>> >>>>>
>>>>> >>>>>> >>>>>
>>>>> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3338)
>>>>> >>>>>> >>>>> at
>>>>> >>>>>> >>>>>
>>>>> >>>>>> >>>>>
>>>>> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2919)
>>>>> >>>>>> >>>>> at
>>>>> >>>>>> >>>>>
>>>>> >>>>>> >>>>>
>>>>> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2905)
>>>>> >>>>>> >>>>> at
>>>>> >>>>>> >>>>>
>>>>> >>>>>> >>>>>
>>>>> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>> >>>>>> >>>>> at
>>>>> >>>>>> >>>>>
>>>>> >>>>>> >>>>>
>>>>> org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:421)
>>>>> >>>>>> >>>>> at
>>>>> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
>>>>> >>>>>> >>>>> at
>>>>> >>>>>> >>>>>
>>>>> >>>>>> >>>>>
>>>>> com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
>>>>> >>>>>> >>>>> at
>>>>> >>>>>> >>>>>
>>>>> >>>>>> >>>>>
>>>>> com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
>>>>> >>>>>> >>>>> at
>>>>> >>>>>> >>>>>
>>>>> >>>>>> >>>>>
>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>> >>>>>> >>>>> at
>>>>> >>>>>> >>>>>
>>>>> >>>>>> >>>>>
>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>> >>>>>> >>>>> at
>>>>> >>>>>> >>>>>
>>>>> >>>>>> >>>>>
>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>> >>>>>> >>>>> at
>>>>> >>>>>> >>>>>
>>>>> >>>>>> >>>>>
>>>>> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
>>>>> >>>>>> >>>>> at
>>>>> >>>>>> >>>>>
>>>>> >>>>>> >>>>>
>>>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.runInContext(AsyncJobManagerImpl.java:532)
>>>>> >>>>>> >>>>> at
>>>>> >>>>>> >>>>>
>>>>> >>>>>> >>>>>
>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>> >>>>>> >>>>> at
>>>>> >>>>>> >>>>>
>>>>> >>>>>> >>>>>
>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>> >>>>>> >>>>> at
>>>>> >>>>>> >>>>>
>>>>> >>>>>> >>>>>
>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>> >>>>>> >>>>> at
>>>>> >>>>>> >>>>>
>>>>> >>>>>> >>>>>
>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>> >>>>>> >>>>> at
>>>>> >>>>>> >>>>>
>>>>> >>>>>> >>>>>
>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>> >>>>>> >>>>> at
>>>>> >>>>>> >>>>>
>>>>> >>>>>> >>>>>
>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>> >>>>>> >>>>> at
>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>> >>>>>> >>>>> at
>>>>> >>>>>> >>>>>
>>>>> >>>>>> >>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>> >>>>>> >>>>> at
>>>>> >>>>>> >>>>>
>>>>> >>>>>> >>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>> >>>>>> >>>>> at java.lang.Thread.run(Thread.java:724)
>>>>> >>>>>> >>>>>
>>>>> >>>>>> >>>>> Thanks!
>>>>> >>>>>> >>>
>>>>> >>>>>> >>>
>>>>> >>>>>> >>>
>>>>> >>>>>> >>>
>>>>> >>>>>> >>> --
>>>>> >>>>>> >>> Mike Tutkowski
>>>>> >>>>>> >>> Senior CloudStack Developer, SolidFire Inc.
>>>>> >>>>>> >>> e: mike.tutkow...@solidfire.com
>>>>> >>>>>> >>> o: 303.746.7302
>>>>> >>>>>> >>> Advancing the way the world uses the cloud™
>>>>> >>>>>> >>
>>>>> >>>>>> >>
>>>>> >>>>>> >>
>>>>> >>>>>> >>
>>>>> >>>>>> >> --
>>>>> >>>>>> >> Mike Tutkowski
>>>>> >>>>>> >> Senior CloudStack Developer, SolidFire Inc.
>>>>> >>>>>> >> e: mike.tutkow...@solidfire.com
>>>>> >>>>>> >> o: 303.746.7302
>>>>> >>>>>> >> Advancing the way the world uses the cloud™
>>>>> >>>>>> >
>>>>> >>>>>> >
>>>>> >>>>>> >
>>>>> >>>>>> >
>>>>> >>>>>> > --
>>>>> >>>>>> > Mike Tutkowski
>>>>> >>>>>> > Senior CloudStack Developer, SolidFire Inc.
>>>>> >>>>>> > e: mike.tutkow...@solidfire.com
>>>>> >>>>>> > o: 303.746.7302
>>>>> >>>>>> > Advancing the way the world uses the cloud™
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>> --
>>>>> >>>>> Mike Tutkowski
>>>>> >>>>> Senior CloudStack Developer, SolidFire Inc.
>>>>> >>>>> e: mike.tutkow...@solidfire.com
>>>>> >>>>> o: 303.746.7302
>>>>> >>>>> Advancing the way the world uses the cloud™
>>>>> >>>>
>>>>> >>>>
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> --
>>>>> >>>> Mike Tutkowski
>>>>> >>>> Senior CloudStack Developer, SolidFire Inc.
>>>>> >>>> e: mike.tutkow...@solidfire.com
>>>>> >>>> o: 303.746.7302
>>>>> >>>> Advancing the way the world uses the cloud™
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>> --
>>>>> >>> Mike Tutkowski
>>>>> >>> Senior CloudStack Developer, SolidFire Inc.
>>>>> >>> e: mike.tutkow...@solidfire.com
>>>>> >>> o: 303.746.7302
>>>>> >>> Advancing the way the world uses the cloud™
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > --
>>>>> > Mike Tutkowski
>>>>> > Senior CloudStack Developer, SolidFire Inc.
>>>>> > e: mike.tutkow...@solidfire.com
>>>>> > o: 303.746.7302
>>>>> > Advancing the way the world uses the cloud™
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Mike Tutkowski*
>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>> e: mike.tutkow...@solidfire.com
>>>> o: 303.746.7302
>>>> Advancing the way the world uses the
>>>> cloud<http://solidfire.com/solution/overview/?video=play>
>>>> *™*
>>>>
>>>
>>>
>>>
>>> --
>>> *Mike Tutkowski*
>>> *Senior CloudStack Developer, SolidFire Inc.*
>>> e: mike.tutkow...@solidfire.com
>>> o: 303.746.7302
>>> Advancing the way the world uses the
>>> cloud<http://solidfire.com/solution/overview/?video=play>
>>> *™*
>>>
>>
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkow...@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the
> cloud<http://solidfire.com/solution/overview/?video=play>
> *™*
>
--
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkow...@solidfire.com
o: 303.746.7302
Advancing the way the world uses the
cloud<http://solidfire.com/solution/overview/?video=play>
*™*
