You know, I forgot we supposed to be doing that! :) Multi-tasking too much
today, I guess.
Anyways, it must not be working because I still had a hypervisor connection
after I shut down the VM.
Let me investigate.
On Wed, Oct 23, 2013 at 1:48 PM, Marcus Sorensen <shadow...@gmail.com>wrote:
> Are we not disconnecting when we stop the vm? There's a method for it, we
> should be. disconnectPhysicalDiskViaVmSpec
> On Oct 23, 2013 1:28 PM, "Mike Tutkowski" <mike.tutkow...@solidfire.com>
> wrote:
>
>> I see one problem for us now, Marcus.
>>
>> * You have a running VM that you attach a volume to.
>> * You stop the VM.
>> * You detach the volume.
>> * You start up the VM.
>>
>> The VM will not be connected to the volume (which is good), but the
>> hypervisor will still be connected to the volume.
>>
>> It would be great if we actually sent a command to the last host ID of
>> the stopped VM when detaching a volume (to have the hypervisor disconnect
>> from the volume).
>>
>> What do you think about that?
>>
>>
>> On Wed, Oct 23, 2013 at 1:15 PM, Mike Tutkowski <
>> mike.tutkow...@solidfire.com> wrote:
>>
>>> OK, whatever way you prefer then, Marcus (createVdb first or second).
>>>
>>> If I leave createVdb first and return 0, it does seem to work.
>>>
>>>
>>> On Wed, Oct 23, 2013 at 11:13 AM, Marcus Sorensen
>>> <shadow...@gmail.com>wrote:
>>>
>>>> I think we could flip-flop these two lines if necessary:
>>>>
>>>> createVbd(conn, vmSpec, vmName, vm);
>>>>
>>>> _storagePoolMgr.connectPhysicalDisksViaVmSpec(vmSpec);
>>>>
>>>> I haven't actually tried it though. But in general I don't see the
>>>> Libvirt DiskDef using size at all, which is what createVbd does
>>>> (creates XML definitions for disks to attach to the VM definition). It
>>>> just takes the device at it's native advertised size when it actually
>>>> goes to use it.
>>>>
>>>> On Wed, Oct 23, 2013 at 10:52 AM, Mike Tutkowski
>>>> <mike.tutkow...@solidfire.com> wrote:
>>>> > Little problem that I wanted to get your take on, Marcus.
>>>> >
>>>> > When a VM is being started, we call createVdb before calling
>>>> > connectPhysicalDisksViaVmSpec.
>>>> >
>>>> > The problem is that createVdb calls getPhysicalDisk and my volume has
>>>> not
>>>> > yet been connected because connectPhysicalDisksViaVmSpec has not yet
>>>> been
>>>> > called.
>>>> >
>>>> > When I try to read up the size of the disk to populate a
>>>> PhysicalDisk, I get
>>>> > an error, of course, because the path does not yet exist.
>>>> >
>>>> > I could populate a 0 for the size of the physical disk and then the
>>>> next
>>>> > time getPhysicalDisk is called, it should be filled in with a proper
>>>> size.
>>>> >
>>>> > Do you see a problem with that approach?
>>>> >
>>>> >
>>>> > On Tue, Oct 22, 2013 at 6:40 PM, Marcus Sorensen <shadow...@gmail.com
>>>> >
>>>> > wrote:
>>>> >>
>>>> >> That's right. All should be well.
>>>> >>
>>>> >> On Oct 22, 2013 6:03 PM, "Mike Tutkowski" <
>>>> mike.tutkow...@solidfire.com>
>>>> >> wrote:
>>>> >>>
>>>> >>> Looks like we disconnect physical disks when the VM is stopped.
>>>> >>>
>>>> >>> I didn't see that before.
>>>> >>>
>>>> >>> I suppose that means the disks are physically disconnected when the
>>>> VM is
>>>> >>> stopped, but the CloudStack DB still has the VM associated with the
>>>> disks
>>>> >>> for the next time the VM may be started up (unless someone does a
>>>> disconnect
>>>> >>> while the VM is in the Stopped State).
>>>> >>>
>>>> >>>
>>>> >>> On Tue, Oct 22, 2013 at 4:19 PM, Mike Tutkowski
>>>> >>> <mike.tutkow...@solidfire.com> wrote:
>>>> >>>>
>>>> >>>> Hey Marcus,
>>>> >>>>
>>>> >>>> Quick question for you related to attaching/detaching volumes when
>>>> the
>>>> >>>> VM is in the Stopped State.
>>>> >>>>
>>>> >>>> If I detach a volume from a VM that is in the Stopped State, the DB
>>>> >>>> seems to get updated, but I don't see a command going to the KVM
>>>> hypervisor
>>>> >>>> that leads to the removal of the iSCSI target.
>>>> >>>>
>>>> >>>> It seems the iSCSI target is only removed the next time the VM is
>>>> >>>> started.
>>>> >>>>
>>>> >>>> Do you know if this is true?
>>>> >>>>
>>>> >>>> If it is, I'm concerned that the volume could be attached to
>>>> another VM
>>>> >>>> before the Stopped VM is re-started and when the Stopped VM gets
>>>> restarted
>>>> >>>> that it would disconnect the iSCSI volume from underneath the VM
>>>> that now
>>>> >>>> has the volume attached.
>>>> >>>>
>>>> >>>> I still want to perform some tests on this, but am first trying to
>>>> get a
>>>> >>>> VM to start after I've attached a volume to it when it was in the
>>>> Stopped
>>>> >>>> State.
>>>> >>>>
>>>> >>>> Thanks,
>>>> >>>> Mike
>>>> >>>>
>>>> >>>>
>>>> >>>> On Mon, Oct 21, 2013 at 10:57 PM, Mike Tutkowski
>>>> >>>> <mike.tutkow...@solidfire.com> wrote:
>>>> >>>>>
>>>> >>>>> Thanks for that info, Marcus.
>>>> >>>>>
>>>> >>>>> By the way, I wanted to see if I could attach my volume to a VM
>>>> in the
>>>> >>>>> Stopped State.
>>>> >>>>>
>>>> >>>>> The attach logic didn't trigger any exceptions; however, when I
>>>> started
>>>> >>>>> the VM, I received an Insufficient Capacity exception.
>>>> >>>>>
>>>> >>>>> If I detach the volume and then start the VM, the VM starts just
>>>> fine.
>>>> >>>>>
>>>> >>>>> I noticed a problem here (in StoragePoolHostDaoImpl):
>>>> >>>>>
>>>> >>>>> @Override
>>>> >>>>>
>>>> >>>>> public StoragePoolHostVO findByPoolHost(long poolId, long
>>>> hostId) {
>>>> >>>>>
>>>> >>>>> SearchCriteria<StoragePoolHostVO> sc =
>>>> PoolHostSearch.create();
>>>> >>>>>
>>>> >>>>> sc.setParameters("pool_id", poolId);
>>>> >>>>>
>>>> >>>>> sc.setParameters("host_id", hostId);
>>>> >>>>>
>>>> >>>>> return findOneIncludingRemovedBy(sc);
>>>> >>>>>
>>>> >>>>> }
>>>> >>>>>
>>>> >>>>> The findOneIncludingRemovedBy method returns null (the poolId is
>>>> my
>>>> >>>>> storage pool's ID and the hostId is the expected host ID).
>>>> >>>>>
>>>> >>>>> I'm not sure what this method is trying to do.
>>>> >>>>>
>>>> >>>>> I looked in the storage_pool_host_ref table (is that the correct
>>>> >>>>> table?) and it only has one row, which maps the local storage
>>>> pool of the
>>>> >>>>> KVM host to the KVM host (which explains why no match is found
>>>> for my
>>>> >>>>> situation).
>>>> >>>>>
>>>> >>>>> Do you understand what this logic is trying to do?
>>>> >>>>>
>>>> >>>>> Thanks!
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> On Mon, Oct 21, 2013 at 8:08 PM, Marcus Sorensen <
>>>> shadow...@gmail.com>
>>>> >>>>> wrote:
>>>> >>>>>>
>>>> >>>>>> Do you have the capability to clone the root disk? Normally the
>>>> >>>>>> template is installed to primary, and then cloned for each root
>>>> disk.
>>>> >>>>>> In some cases (such as CLVM), this isn't efficient and so the
>>>> template
>>>> >>>>>> is copied fresh to populate each root disk.
>>>> >>>>>>
>>>> >>>>>> I'm actually not 100% sure how this works in the new code. It
>>>> used to
>>>> >>>>>> be handled by copyPhysicalDisk in the storage adaptor, called by
>>>> >>>>>> copyTemplateToPrimaryStorage, which runs on the agent. It would
>>>> pass
>>>> >>>>>> template/secondary storage info, and the destination
>>>> volume/primary
>>>> >>>>>> storage info, and copyPhysicalDisk would do the work of
>>>> installing the
>>>> >>>>>> image to the destination. Then subsequent root disks would be
>>>> cloned
>>>> >>>>>> in CreateCommand by calling createDiskFromTemplate.
>>>> >>>>>>
>>>> >>>>>> In master it looks like this was moved to KVMStorageProcessor
>>>> >>>>>> 'cloneVolumeFromBaseTemplate', although I think this just takes
>>>> over
>>>> >>>>>> as default, and there's something in your storage driver that
>>>> should
>>>> >>>>>> be capable of cloning templates on the mgmt server side. I'm
>>>> less sure
>>>> >>>>>> about how the template gets to primary storage in the first
>>>> place, I
>>>> >>>>>> assume copyTemplateToPrimaryStorage in KVMStorageProcessor
>>>> calling
>>>> >>>>>> copyPhysicalDisk in your adaptor. It's a bit tough for me to tell
>>>> >>>>>> since our earlier storage adaptor did everything on the host it
>>>> mostly
>>>> >>>>>> just worked with the default stuff.
>>>> >>>>>>
>>>> >>>>>> On Mon, Oct 21, 2013 at 4:49 PM, Mike Tutkowski
>>>> >>>>>> <mike.tutkow...@solidfire.com> wrote:
>>>> >>>>>> > Hey Marcus,
>>>> >>>>>> >
>>>> >>>>>> > So...now that this works well for data disks, I was wondering
>>>> what
>>>> >>>>>> > might be
>>>> >>>>>> > involved in getting this process to work for root disks.
>>>> >>>>>> >
>>>> >>>>>> > Can you point me in the right direction as far as what gets
>>>> invoked
>>>> >>>>>> > when a
>>>> >>>>>> > VM is being created on KVM (so that its root disk can be
>>>> created and
>>>> >>>>>> > the
>>>> >>>>>> > necessary template laid down or ISO installed)?
>>>> >>>>>> >
>>>> >>>>>> > Thanks!
>>>> >>>>>> >
>>>> >>>>>> >
>>>> >>>>>> > On Mon, Oct 21, 2013 at 1:14 PM, Mike Tutkowski
>>>> >>>>>> > <mike.tutkow...@solidfire.com> wrote:
>>>> >>>>>> >>
>>>> >>>>>> >> Hey Marcus,
>>>> >>>>>> >>
>>>> >>>>>> >> Just wanted to let you know the branch of mine that has your
>>>> code
>>>> >>>>>> >> and mine
>>>> >>>>>> >> appears to work well with regards to attaching a data disk to
>>>> a
>>>> >>>>>> >> running VM:
>>>> >>>>>> >>
>>>> >>>>>> >> fdisk -l from hypervisor:
>>>> >>>>>> >>
>>>> >>>>>> >> http://i.imgur.com/NkP5fo0.png
>>>> >>>>>> >>
>>>> >>>>>> >> fdisk -l from within VM:
>>>> >>>>>> >>
>>>> >>>>>> >> http://i.imgur.com/8YwiiC7.png
>>>> >>>>>> >>
>>>> >>>>>> >> I plan to do more testing on this over the coming days.
>>>> >>>>>> >>
>>>> >>>>>> >> If all goes well, perhaps we can check this code in by the
>>>> end of
>>>> >>>>>> >> the
>>>> >>>>>> >> week?
>>>> >>>>>> >>
>>>> >>>>>> >> Talk to you later,
>>>> >>>>>> >> Mike
>>>> >>>>>> >>
>>>> >>>>>> >>
>>>> >>>>>> >> On Sun, Oct 20, 2013 at 10:23 PM, Mike Tutkowski
>>>> >>>>>> >> <mike.tutkow...@solidfire.com> wrote:
>>>> >>>>>> >>>
>>>> >>>>>> >>> Don't ask me, but it works now (I've been having this trouble
>>>> >>>>>> >>> quite a
>>>> >>>>>> >>> while today).
>>>> >>>>>> >>>
>>>> >>>>>> >>> I guess the trick is to send you an e-mail. :)
>>>> >>>>>> >>>
>>>> >>>>>> >>>
>>>> >>>>>> >>> On Sun, Oct 20, 2013 at 10:05 PM, Marcus Sorensen
>>>> >>>>>> >>> <shadow...@gmail.com>
>>>> >>>>>> >>> wrote:
>>>> >>>>>> >>>>
>>>> >>>>>> >>>> Did you create a service offering that uses local storage,
>>>> or add
>>>> >>>>>> >>>> a
>>>> >>>>>> >>>> shared primary storage? By default there is no storage that
>>>> >>>>>> >>>> matches the
>>>> >>>>>> >>>> built in offerings.
>>>> >>>>>> >>>>
>>>> >>>>>> >>>> On Oct 20, 2013 9:39 PM, "Mike Tutkowski"
>>>> >>>>>> >>>> <mike.tutkow...@solidfire.com>
>>>> >>>>>> >>>> wrote:
>>>> >>>>>> >>>>>
>>>> >>>>>> >>>>> Hey Marcus,
>>>> >>>>>> >>>>>
>>>> >>>>>> >>>>> So, I went back to the branch of mine that has your code
>>>> and
>>>> >>>>>> >>>>> mine and
>>>> >>>>>> >>>>> was able to create a new CloudStack install from scratch
>>>> with it
>>>> >>>>>> >>>>> (once
>>>> >>>>>> >>>>> again, after manually deleting what was in
>>>> >>>>>> >>>>> /var/lib/libvirt/images to the
>>>> >>>>>> >>>>> get system VMs to start).
>>>> >>>>>> >>>>>
>>>> >>>>>> >>>>> Anyways, my system VMs are running now and I tried to kick
>>>> off a
>>>> >>>>>> >>>>> VM
>>>> >>>>>> >>>>> using the CentOS 6.3 image you provided me a while back.
>>>> >>>>>> >>>>>
>>>> >>>>>> >>>>> The virtual router has a Status of Running; however, my VM
>>>> fails
>>>> >>>>>> >>>>> to
>>>> >>>>>> >>>>> start (with the generic message of Insufficient Capacity).
>>>> >>>>>> >>>>>
>>>> >>>>>> >>>>> I've not seen this exception before (related to the VR).
>>>> Do you
>>>> >>>>>> >>>>> have
>>>> >>>>>> >>>>> any insight into this?:
>>>> >>>>>> >>>>>
>>>> >>>>>> >>>>> com.cloud.exception.ResourceUnavailableException: Resource
>>>> >>>>>> >>>>> [Pod:1] is
>>>> >>>>>> >>>>> unreachable: Unable to apply userdata and password entry on
>>>> >>>>>> >>>>> router
>>>> >>>>>> >>>>> at
>>>> >>>>>> >>>>>
>>>> >>>>>> >>>>>
>>>> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:3793)
>>>> >>>>>> >>>>> at
>>>> >>>>>> >>>>>
>>>> >>>>>> >>>>>
>>>> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyUserData(VirtualNetworkApplianceManagerImpl.java:3017)
>>>> >>>>>> >>>>> at
>>>> >>>>>> >>>>>
>>>> >>>>>> >>>>>
>>>> com.cloud.network.element.VirtualRouterElement.addPasswordAndUserdata(VirtualRouterElement.java:933)
>>>> >>>>>> >>>>> at
>>>> >>>>>> >>>>>
>>>> >>>>>> >>>>>
>>>> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareElement(NetworkOrchestrator.java:1172)
>>>> >>>>>> >>>>> at
>>>> >>>>>> >>>>>
>>>> >>>>>> >>>>>
>>>> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareNic(NetworkOrchestrator.java:1288)
>>>> >>>>>> >>>>> at
>>>> >>>>>> >>>>>
>>>> >>>>>> >>>>>
>>>> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(NetworkOrchestrator.java:1224)
>>>> >>>>>> >>>>> at
>>>> >>>>>> >>>>>
>>>> >>>>>> >>>>>
>>>> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:826)
>>>> >>>>>> >>>>> at
>>>> >>>>>> >>>>>
>>>> >>>>>> >>>>>
>>>> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:508)
>>>> >>>>>> >>>>> at
>>>> >>>>>> >>>>>
>>>> >>>>>> >>>>>
>>>> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:227)
>>>> >>>>>> >>>>> at
>>>> >>>>>> >>>>>
>>>> >>>>>> >>>>>
>>>> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
>>>> >>>>>> >>>>> at
>>>> >>>>>> >>>>>
>>>> >>>>>> >>>>>
>>>> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3338)
>>>> >>>>>> >>>>> at
>>>> >>>>>> >>>>>
>>>> >>>>>> >>>>>
>>>> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2919)
>>>> >>>>>> >>>>> at
>>>> >>>>>> >>>>>
>>>> >>>>>> >>>>>
>>>> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2905)
>>>> >>>>>> >>>>> at
>>>> >>>>>> >>>>>
>>>> >>>>>> >>>>>
>>>> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>> >>>>>> >>>>> at
>>>> >>>>>> >>>>>
>>>> >>>>>> >>>>>
>>>> org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:421)
>>>> >>>>>> >>>>> at
>>>> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
>>>> >>>>>> >>>>> at
>>>> >>>>>> >>>>>
>>>> >>>>>> >>>>>
>>>> com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
>>>> >>>>>> >>>>> at
>>>> >>>>>> >>>>>
>>>> >>>>>> >>>>>
>>>> com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
>>>> >>>>>> >>>>> at
>>>> >>>>>> >>>>>
>>>> >>>>>> >>>>>
>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>> >>>>>> >>>>> at
>>>> >>>>>> >>>>>
>>>> >>>>>> >>>>>
>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>> >>>>>> >>>>> at
>>>> >>>>>> >>>>>
>>>> >>>>>> >>>>>
>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>> >>>>>> >>>>> at
>>>> >>>>>> >>>>>
>>>> >>>>>> >>>>>
>>>> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
>>>> >>>>>> >>>>> at
>>>> >>>>>> >>>>>
>>>> >>>>>> >>>>>
>>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.runInContext(AsyncJobManagerImpl.java:532)
>>>> >>>>>> >>>>> at
>>>> >>>>>> >>>>>
>>>> >>>>>> >>>>>
>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>> >>>>>> >>>>> at
>>>> >>>>>> >>>>>
>>>> >>>>>> >>>>>
>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>> >>>>>> >>>>> at
>>>> >>>>>> >>>>>
>>>> >>>>>> >>>>>
>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>> >>>>>> >>>>> at
>>>> >>>>>> >>>>>
>>>> >>>>>> >>>>>
>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>> >>>>>> >>>>> at
>>>> >>>>>> >>>>>
>>>> >>>>>> >>>>>
>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>> >>>>>> >>>>> at
>>>> >>>>>> >>>>>
>>>> >>>>>> >>>>>
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>> >>>>>> >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>> >>>>>> >>>>> at
>>>> >>>>>> >>>>>
>>>> >>>>>> >>>>>
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> >>>>>> >>>>> at
>>>> >>>>>> >>>>>
>>>> >>>>>> >>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> >>>>>> >>>>> at java.lang.Thread.run(Thread.java:724)
>>>> >>>>>> >>>>>
>>>> >>>>>> >>>>> Thanks!
>>>> >>>>>> >>>
>>>> >>>>>> >>>
>>>> >>>>>> >>>
>>>> >>>>>> >>>
>>>> >>>>>> >>> --
>>>> >>>>>> >>> Mike Tutkowski
>>>> >>>>>> >>> Senior CloudStack Developer, SolidFire Inc.
>>>> >>>>>> >>> e: mike.tutkow...@solidfire.com
>>>> >>>>>> >>> o: 303.746.7302
>>>> >>>>>> >>> Advancing the way the world uses the cloud™
>>>> >>>>>> >>
>>>> >>>>>> >>
>>>> >>>>>> >>
>>>> >>>>>> >>
>>>> >>>>>> >> --
>>>> >>>>>> >> Mike Tutkowski
>>>> >>>>>> >> Senior CloudStack Developer, SolidFire Inc.
>>>> >>>>>> >> e: mike.tutkow...@solidfire.com
>>>> >>>>>> >> o: 303.746.7302
>>>> >>>>>> >> Advancing the way the world uses the cloud™
>>>> >>>>>> >
>>>> >>>>>> >
>>>> >>>>>> >
>>>> >>>>>> >
>>>> >>>>>> > --
>>>> >>>>>> > Mike Tutkowski
>>>> >>>>>> > Senior CloudStack Developer, SolidFire Inc.
>>>> >>>>>> > e: mike.tutkow...@solidfire.com
>>>> >>>>>> > o: 303.746.7302
>>>> >>>>>> > Advancing the way the world uses the cloud™
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> --
>>>> >>>>> Mike Tutkowski
>>>> >>>>> Senior CloudStack Developer, SolidFire Inc.
>>>> >>>>> e: mike.tutkow...@solidfire.com
>>>> >>>>> o: 303.746.7302
>>>> >>>>> Advancing the way the world uses the cloud™
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>> --
>>>> >>>> Mike Tutkowski
>>>> >>>> Senior CloudStack Developer, SolidFire Inc.
>>>> >>>> e: mike.tutkow...@solidfire.com
>>>> >>>> o: 303.746.7302
>>>> >>>> Advancing the way the world uses the cloud™
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> --
>>>> >>> Mike Tutkowski
>>>> >>> Senior CloudStack Developer, SolidFire Inc.
>>>> >>> e: mike.tutkow...@solidfire.com
>>>> >>> o: 303.746.7302
>>>> >>> Advancing the way the world uses the cloud™
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Mike Tutkowski
>>>> > Senior CloudStack Developer, SolidFire Inc.
>>>> > e: mike.tutkow...@solidfire.com
>>>> > o: 303.746.7302
>>>> > Advancing the way the world uses the cloud™
>>>>
>>>
>>>
>>>
>>> --
>>> *Mike Tutkowski*
>>> *Senior CloudStack Developer, SolidFire Inc.*
>>> e: mike.tutkow...@solidfire.com
>>> o: 303.746.7302
>>> Advancing the way the world uses the
>>> cloud<http://solidfire.com/solution/overview/?video=play>
>>> *™*
>>>
>>
>>
>>
>> --
>> *Mike Tutkowski*
>> *Senior CloudStack Developer, SolidFire Inc.*
>> e: mike.tutkow...@solidfire.com
>> o: 303.746.7302
>> Advancing the way the world uses the
>> cloud<http://solidfire.com/solution/overview/?video=play>
>> *™*
>>
>
--
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkow...@solidfire.com
o: 303.746.7302
Advancing the way the world uses the
cloud<http://solidfire.com/solution/overview/?video=play>
*™*
