Weird...no such file exists.
On Mon, Sep 23, 2013 at 4:54 PM, Marcus Sorensen <shadow...@gmail.com>wrote: > maybe cloudstack-agent.out > > On Mon, Sep 23, 2013 at 4:44 PM, Mike Tutkowski > <mike.tutkow...@solidfire.com> wrote: > > OK, so, nothing is screaming out in the logs. I did notice the following: > > > > From setup.log: > > > > DEBUG:root:execute:apparmor_status |grep libvirt > > > > DEBUG:root:Failed to execute: > > > > > > DEBUG:root:execute:sudo /usr/sbin/service cloudstack-agent status > > > > DEBUG:root:Failed to execute: * could not access PID file for > > cloudstack-agent > > > > > > This is the final line in this log file: > > > > DEBUG:root:execute:sudo /usr/sbin/service cloudstack-agent start > > > > > > This is from agent.log: > > > > 2013-09-23 15:30:55,549 DEBUG [cloud.agent.AgentShell] (main:null) > Checking > > to see if agent.pid exists. > > > > 2013-09-23 15:30:55,655 DEBUG [cloud.utils.ProcessUtil] (main:null) > > Executing: bash -c echo $PPID > > > > 2013-09-23 15:30:55,742 DEBUG [cloud.utils.ProcessUtil] (main:null) > > Execution is successful. > > > > 2013-09-23 15:30:56,000 INFO [cloud.agent.Agent] (main:null) id is > > > > 2013-09-23 15:30:56,000 DEBUG [cloud.resource.ServerResourceBase] > > (main:null) Retrieving network interface: cloudbr0 > > > > 2013-09-23 15:30:56,016 DEBUG [cloud.resource.ServerResourceBase] > > (main:null) Retrieving network interface: cloudbr0 > > > > 2013-09-23 15:30:56,016 DEBUG [cloud.resource.ServerResourceBase] > > (main:null) Retrieving network interface: null > > > > 2013-09-23 15:30:56,017 DEBUG [cloud.resource.ServerResourceBase] > > (main:null) Retrieving network interface: null > > > > > > The following kinds of lines are repeated for a bunch of different .sh > > files. I think they often end up being found here: > > /usr/share/cloudstack-common/scripts/network/domr, so this is probably > not > > an issue. > > > > > > 2013-09-23 15:30:56,111 DEBUG [utils.script.Script] (main:null) Looking > for > > call_firewall.sh in the classpath > > > > 2013-09-23 15:30:56,112 DEBUG [utils.script.Script] (main:null) System > > resource: null > > > > 2013-09-23 15:30:56,113 DEBUG [utils.script.Script] (main:null) Classpath > > resource: null > > > > 2013-09-23 15:30:56,123 DEBUG [utils.script.Script] (main:null) Looking > for > > call_firewall.sh > > > > > > Is there a log file for the Java code that I could write stuff out to and > > see how far we get? > > > > > > On Mon, Sep 23, 2013 at 3:17 PM, Mike Tutkowski < > > mike.tutkow...@solidfire.com> wrote: > > > >> Thanks, Marcus > >> > >> I've been developing on Windows for most of my time, so a bunch of these > >> Linux-type commands are new to me and I don't always interpret the > output > >> correctly. Getting there. :) > >> > >> > >> On Mon, Sep 23, 2013 at 2:37 PM, Marcus Sorensen <shadow...@gmail.com > >wrote: > >> > >>> Nope, not running. That's just your grep process. It would look like: > >>> > >>> root 24429 24428 1 14:25 ? 00:00:08 jsvc.exec -cp > >>> > >>> > /usr/share/java/commons-daemon.jar:/usr/share/java/jna.jar:/usr/share/cloudstack-agent/lib/activation-1.1.jar:/usr/share/cloudstack-agent/lib/antisamy-1.4.3.jar:/usr/share/cloudstack-agent/lib/aopalliance-1.0.jar:/usr/share/cloudstack-agent/lib/apache-log4j-extras-1.1.jar:/usr/share/cloudstack-agent/lib/aspectjrt-1.7. > >>> > >>> Your agent log should tell you why it failed to start if you set it in > >>> debug and try to start... or maybe cloudstack-agent.out if it doesn't > >>> get far enough (say it's missing a class or something and can't > >>> start). > >>> > >>> On Mon, Sep 23, 2013 at 2:33 PM, Mike Tutkowski > >>> <mike.tutkow...@solidfire.com> wrote: > >>> > Looks like it's running, though: > >>> > > >>> > mtutkowski@ubuntu:~$ ps -ef | grep jsvc > >>> > 1000 7097 7013 0 14:32 pts/1 00:00:00 grep --color=auto > jsvc > >>> > > >>> > > >>> > > >>> > On Mon, Sep 23, 2013 at 2:31 PM, Mike Tutkowski < > >>> > mike.tutkow...@solidfire.com> wrote: > >>> > > >>> >> Hey Marcus, > >>> >> > >>> >> Maybe you could give me a better idea of what the "flow" is when > >>> adding a > >>> >> KVM host. > >>> >> > >>> >> It looks like we SSH into the potential KVM host and execute a > startup > >>> >> script (giving it necessary info about the cloud and the management > >>> server > >>> >> it should talk to). > >>> >> > >>> >> After this, is the Java VM started? > >>> >> > >>> >> After a reboot, I assume the JVM is started automatically? > >>> >> > >>> >> How do you debug your KVM-side Java code? > >>> >> > >>> >> Been looking through the logs and nothing obvious sticks out. I will > >>> have > >>> >> another look. > >>> >> > >>> >> Thanks > >>> >> > >>> >> > >>> >> On Mon, Sep 23, 2013 at 2:15 PM, Mike Tutkowski < > >>> >> mike.tutkow...@solidfire.com> wrote: > >>> >> > >>> >>> Hey Marcus, > >>> >>> > >>> >>> I've been investigating my issue with not being able to add a KVM > >>> host to > >>> >>> CS. > >>> >>> > >>> >>> For what it's worth, this comes back successful: > >>> >>> > >>> >>> SSHCmdHelper.sshExecuteCmd(sshConnection, "cloudstack-setup-agent > " + > >>> >>> parameters, 3); > >>> >>> > >>> >>> This is what the command looks like: > >>> >>> > >>> >>> cloudstack-setup-agent -m 192.168.233.1 -z 1 -p 1 -c 1 -g > >>> >>> 6b4aa1c2-2ac9-3c60-aabe-704aed40c684 -a --pubNic=cloudbr0 > >>> --prvNic=cloudbr0 > >>> >>> --guestNic=cloudbr0 > >>> >>> > >>> >>> The problem is this method in LibvirtServerDiscoverer never finds a > >>> >>> matching host in the DB: > >>> >>> > >>> >>> waitForHostConnect(long dcId, long podId, long clusterId, String > guid) > >>> >>> > >>> >>> I assume once the KVM host is up and running that it's supposed to > >>> call > >>> >>> into the CS MS so the DB can be updated as such? > >>> >>> > >>> >>> If so, the problem must be on the KVM side. > >>> >>> > >>> >>> I did run this again (from the KVM host) to see if the connection > was > >>> in > >>> >>> place: > >>> >>> > >>> >>> mtutkowski@ubuntu:~$ telnet 192.168.233.1 8250 > >>> >>> > >>> >>> Trying 192.168.233.1... > >>> >>> > >>> >>> Connected to 192.168.233.1. > >>> >>> > >>> >>> Escape character is '^]'. > >>> >>> So that looks good. > >>> >>> > >>> >>> I turned on more info in the debug log, but nothing obvious jumps > out > >>> as > >>> >>> of yet. > >>> >>> > >>> >>> If you have any thoughts on this, please shoot them my way. :) > >>> >>> > >>> >>> Thanks! > >>> >>> > >>> >>> > >>> >>> On Sun, Sep 22, 2013 at 12:11 AM, Mike Tutkowski < > >>> >>> mike.tutkow...@solidfire.com> wrote: > >>> >>> > >>> >>>> First step is for me to get this working for KVM, though. :) > >>> >>>> > >>> >>>> Once I do that, I can perhaps make modifications to the storage > >>> >>>> framework and hypervisor plug-ins to refactor the logic and such. > >>> >>>> > >>> >>>> > >>> >>>> On Sun, Sep 22, 2013 at 12:09 AM, Mike Tutkowski < > >>> >>>> mike.tutkow...@solidfire.com> wrote: > >>> >>>> > >>> >>>>> Same would work for KVM. > >>> >>>>> > >>> >>>>> If CreateCommand and DestroyCommand were called at the > appropriate > >>> >>>>> times by the storage framework, I could move my connect and > >>> disconnect > >>> >>>>> logic out of the attach/detach logic. > >>> >>>>> > >>> >>>>> > >>> >>>>> On Sun, Sep 22, 2013 at 12:08 AM, Mike Tutkowski < > >>> >>>>> mike.tutkow...@solidfire.com> wrote: > >>> >>>>> > >>> >>>>>> Conversely, if the storage framework called the DestroyCommand > for > >>> >>>>>> managed storage after the DetachCommand, then I could have had > my > >>> remove > >>> >>>>>> SR/datastore logic placed in the DestroyCommand handling rather > >>> than in the > >>> >>>>>> DetachCommand handling. > >>> >>>>>> > >>> >>>>>> > >>> >>>>>> On Sun, Sep 22, 2013 at 12:06 AM, Mike Tutkowski < > >>> >>>>>> mike.tutkow...@solidfire.com> wrote: > >>> >>>>>> > >>> >>>>>>> Edison's plug-in calls the CreateCommand. Mine does not. > >>> >>>>>>> > >>> >>>>>>> The initial approach that was discussed during 4.2 was for me > to > >>> >>>>>>> modify the attach/detach logic only in the XenServer and VMware > >>> hypervisor > >>> >>>>>>> plug-ins. > >>> >>>>>>> > >>> >>>>>>> Now that I think about it more, though, I kind of would have > >>> liked to > >>> >>>>>>> have the storage framework send a CreateCommand to the > hypervisor > >>> before > >>> >>>>>>> sending the AttachCommand if the storage in question was > managed. > >>> >>>>>>> > >>> >>>>>>> Then I could have created my SR/datastore in the CreateCommand > and > >>> >>>>>>> the AttachCommand would have had the SR/datastore that it was > >>> always > >>> >>>>>>> expecting (and I wouldn't have had to create the SR/datastore > in > >>> the > >>> >>>>>>> AttachCommand). > >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> On Sat, Sep 21, 2013 at 11:56 PM, Marcus Sorensen < > >>> >>>>>>> shadow...@gmail.com> wrote: > >>> >>>>>>> > >>> >>>>>>>> Yeah, I think it probably is as well, but I figured you'd be > in a > >>> >>>>>>>> better position to tell. > >>> >>>>>>>> > >>> >>>>>>>> I see that copyAsync is unsupported in your current 4.2 > driver, > >>> does > >>> >>>>>>>> that mean that there's no template support? Or is it some > other > >>> call > >>> >>>>>>>> that does templating now? I'm still getting up to speed on all > >>> of the > >>> >>>>>>>> 4.2 changes. I was just looking at CreateCommand in > >>> >>>>>>>> LibvirtComputingResource, since that's the only place > >>> >>>>>>>> createPhysicalDisk is called, and it occurred to me that > >>> >>>>>>>> CreateCommand > >>> >>>>>>>> might be skipped altogether when utilizing storage plugins. > >>> >>>>>>>> > >>> >>>>>>>> On Sat, Sep 21, 2013 at 11:38 PM, Mike Tutkowski > >>> >>>>>>>> <mike.tutkow...@solidfire.com> wrote: > >>> >>>>>>>> > That's an interesting comment, Marcus. > >>> >>>>>>>> > > >>> >>>>>>>> > It was my intent that it should work with any CloudStack > >>> "managed" > >>> >>>>>>>> storage > >>> >>>>>>>> > that uses an iSCSI target. Even though I'm using CHAP, I > wrote > >>> the > >>> >>>>>>>> code so > >>> >>>>>>>> > CHAP didn't have to be used. > >>> >>>>>>>> > > >>> >>>>>>>> > As I'm doing my testing, I can try to think about whether > it is > >>> >>>>>>>> generic > >>> >>>>>>>> > enough to keep those names or not. > >>> >>>>>>>> > > >>> >>>>>>>> > My expectation is that it is generic enough. > >>> >>>>>>>> > > >>> >>>>>>>> > > >>> >>>>>>>> > On Sat, Sep 21, 2013 at 11:32 PM, Marcus Sorensen < > >>> >>>>>>>> shadow...@gmail.com>wrote: > >>> >>>>>>>> > > >>> >>>>>>>> >> I added a comment to your diff. In general I think it looks > >>> good, > >>> >>>>>>>> >> though I obviously can't vouch for whether or not it will > >>> work. > >>> >>>>>>>> One > >>> >>>>>>>> >> thing I do have reservations about is the adaptor/pool > >>> naming. If > >>> >>>>>>>> you > >>> >>>>>>>> >> think the code is generic enough that it will work for > anyone > >>> who > >>> >>>>>>>> does > >>> >>>>>>>> >> an iscsi LUN-per-volume plugin, then it's OK, but if > there's > >>> >>>>>>>> anything > >>> >>>>>>>> >> about it that's specific to YOUR iscsi target or how it > likes > >>> to > >>> >>>>>>>> be > >>> >>>>>>>> >> treated then I'd say that they should be named something > less > >>> >>>>>>>> generic > >>> >>>>>>>> >> than iScsiAdmStorage. > >>> >>>>>>>> >> > >>> >>>>>>>> >> On Sat, Sep 21, 2013 at 7:23 PM, Mike Tutkowski > >>> >>>>>>>> >> <mike.tutkow...@solidfire.com> wrote: > >>> >>>>>>>> >> > Great - thanks! > >>> >>>>>>>> >> > > >>> >>>>>>>> >> > Just to give you an overview of what my code does (for > when > >>> you > >>> >>>>>>>> get a > >>> >>>>>>>> >> > chance to review it): > >>> >>>>>>>> >> > > >>> >>>>>>>> >> > SolidFireHostListener is registered in > >>> >>>>>>>> SolidfirePrimaryDataStoreProvider. > >>> >>>>>>>> >> > Its hostConnect method is invoked when a host connects > with > >>> the > >>> >>>>>>>> CS MS. If > >>> >>>>>>>> >> > the host is running KVM, the listener sends a > >>> >>>>>>>> ModifyStoragePoolCommand to > >>> >>>>>>>> >> > the host. This logic was based off of > DefaultHostListener. > >>> >>>>>>>> >> > > >>> >>>>>>>> >> > The handling of ModifyStoragePoolCommand is unchanged. It > >>> >>>>>>>> invokes > >>> >>>>>>>> >> > createStoragePool on the KVMStoragePoolManager. The > >>> >>>>>>>> KVMStoragePoolManager > >>> >>>>>>>> >> > asks for an adaptor and finds my new one: > >>> >>>>>>>> iScsiAdmStorageAdaptor (which > >>> >>>>>>>> >> was > >>> >>>>>>>> >> > registered in the constructor for KVMStoragePoolManager > >>> under > >>> >>>>>>>> the key of > >>> >>>>>>>> >> > StoragePoolType.Iscsi.toString()). > >>> >>>>>>>> >> > > >>> >>>>>>>> >> > iScsiAdmStorageAdaptor.createStoragePool just makes an > >>> instance > >>> >>>>>>>> of > >>> >>>>>>>> >> > iScsiAdmStoragePool, adds it to a map, and returns the > >>> pointer > >>> >>>>>>>> to the > >>> >>>>>>>> >> > iScsiAdmStoragePool object. The key of the map is the > UUID > >>> of > >>> >>>>>>>> the storage > >>> >>>>>>>> >> > pool. > >>> >>>>>>>> >> > > >>> >>>>>>>> >> > When a volume is attached, createPhysicalDisk is invoked > for > >>> >>>>>>>> managed > >>> >>>>>>>> >> > storage rather than getPhysicalDisk. createPhysicalDisk > uses > >>> >>>>>>>> iscsiadm to > >>> >>>>>>>> >> > establish the iSCSI connection to the volume on the SAN > and > >>> a > >>> >>>>>>>> >> > KVMPhysicalDisk is returned to be used in the attach > logic > >>> that > >>> >>>>>>>> follows. > >>> >>>>>>>> >> > > >>> >>>>>>>> >> > When a volume is detached, getPhysicalDisk is invoked > with > >>> the > >>> >>>>>>>> IQN of the > >>> >>>>>>>> >> > volume if the storage pool in question is managed > storage. > >>> >>>>>>>> Otherwise, the > >>> >>>>>>>> >> > normal vol.getPath() is used. > >>> >>>>>>>> iScsiAdmStorageAdaptor.getPhysicalDisk just > >>> >>>>>>>> >> > returns a new instance of KVMPhysicalDisk to be used in > the > >>> >>>>>>>> detach logic. > >>> >>>>>>>> >> > > >>> >>>>>>>> >> > Once the volume has been detached, > >>> >>>>>>>> iScsiAdmStoragePool.deletePhysicalDisk > >>> >>>>>>>> >> > is invoked if the storage pool is managed. > >>> deletePhysicalDisk > >>> >>>>>>>> removes the > >>> >>>>>>>> >> > iSCSI connection to the volume using iscsiadm. > >>> >>>>>>>> >> > > >>> >>>>>>>> >> > > >>> >>>>>>>> >> > On Sat, Sep 21, 2013 at 5:46 PM, Marcus Sorensen < > >>> >>>>>>>> shadow...@gmail.com > >>> >>>>>>>> >> >wrote: > >>> >>>>>>>> >> > > >>> >>>>>>>> >> >> Its the log4j properties file in /etc/cloudstack/agent > >>> change > >>> >>>>>>>> all INFO > >>> >>>>>>>> >> to > >>> >>>>>>>> >> >> DEBUG. I imagine the agent just isn't starting, you can > >>> tail > >>> >>>>>>>> the log > >>> >>>>>>>> >> when > >>> >>>>>>>> >> >> you try to start the service, or maybe it will spit > >>> something > >>> >>>>>>>> out into > >>> >>>>>>>> >> one > >>> >>>>>>>> >> >> of the other files in /var/log/cloudstack/agent > >>> >>>>>>>> >> >> On Sep 21, 2013 5:19 PM, "Mike Tutkowski" < > >>> >>>>>>>> mike.tutkow...@solidfire.com > >>> >>>>>>>> >> > > >>> >>>>>>>> >> >> wrote: > >>> >>>>>>>> >> >> > >>> >>>>>>>> >> >> > This is how I've been trying to query for the status > of > >>> the > >>> >>>>>>>> service (I > >>> >>>>>>>> >> >> > assume it could be started this way, as well, by > changing > >>> >>>>>>>> "status" to > >>> >>>>>>>> >> >> > "start" or "restart"?): > >>> >>>>>>>> >> >> > > >>> >>>>>>>> >> >> > mtutkowski@ubuntu:/etc/cloudstack/agent$ sudo > >>> >>>>>>>> /usr/sbin/service > >>> >>>>>>>> >> >> > cloudstack-agent status > >>> >>>>>>>> >> >> > > >>> >>>>>>>> >> >> > I get this back: > >>> >>>>>>>> >> >> > > >>> >>>>>>>> >> >> > Failed to execute: * could not access PID file for > >>> >>>>>>>> cloudstack-agent > >>> >>>>>>>> >> >> > > >>> >>>>>>>> >> >> > I've made a bunch of code changes recently, though, > so I > >>> >>>>>>>> think I'm > >>> >>>>>>>> >> going > >>> >>>>>>>> >> >> to > >>> >>>>>>>> >> >> > rebuild and redeploy everything. > >>> >>>>>>>> >> >> > > >>> >>>>>>>> >> >> > The debug info sounds helpful. Where can I set > >>> enable.debug? > >>> >>>>>>>> >> >> > > >>> >>>>>>>> >> >> > Thanks, Marcus! > >>> >>>>>>>> >> >> > > >>> >>>>>>>> >> >> > > >>> >>>>>>>> >> >> > On Sat, Sep 21, 2013 at 4:11 PM, Marcus Sorensen < > >>> >>>>>>>> shadow...@gmail.com > >>> >>>>>>>> >> >> > >wrote: > >>> >>>>>>>> >> >> > > >>> >>>>>>>> >> >> > > OK, will check it out in the next few days. As > >>> mentioned, > >>> >>>>>>>> you can > >>> >>>>>>>> >> set > >>> >>>>>>>> >> >> up > >>> >>>>>>>> >> >> > > your Ubuntu vm as the management server as well if > all > >>> >>>>>>>> else fails. > >>> >>>>>>>> >> If > >>> >>>>>>>> >> >> > you > >>> >>>>>>>> >> >> > > can get to the mgmt server on 8250 from the KVM > host, > >>> then > >>> >>>>>>>> you need > >>> >>>>>>>> >> to > >>> >>>>>>>> >> >> > > enable.debug on the agent. It won't run without > >>> >>>>>>>> complaining loudly > >>> >>>>>>>> >> if > >>> >>>>>>>> >> >> it > >>> >>>>>>>> >> >> > > can't get to the mgmt server, and I didn't see that > in > >>> >>>>>>>> your agent > >>> >>>>>>>> >> log, > >>> >>>>>>>> >> >> so > >>> >>>>>>>> >> >> > > perhaps its not running. I assume you know how to > >>> >>>>>>>> stop/start the > >>> >>>>>>>> >> agent > >>> >>>>>>>> >> >> on > >>> >>>>>>>> >> >> > > KVM via 'service cloud stacks agent'. > >>> >>>>>>>> >> >> > > On Sep 21, 2013 3:02 PM, "Mike Tutkowski" < > >>> >>>>>>>> >> >> mike.tutkow...@solidfire.com> > >>> >>>>>>>> >> >> > > wrote: > >>> >>>>>>>> >> >> > > > >>> >>>>>>>> >> >> > > > Hey Marcus, > >>> >>>>>>>> >> >> > > > > >>> >>>>>>>> >> >> > > > I haven't yet been able to test my new code, but I > >>> >>>>>>>> thought you > >>> >>>>>>>> >> would > >>> >>>>>>>> >> >> > be a > >>> >>>>>>>> >> >> > > > good person to ask to review it: > >>> >>>>>>>> >> >> > > > > >>> >>>>>>>> >> >> > > > > >>> >>>>>>>> >> >> > > > > >>> >>>>>>>> >> >> > > > >>> >>>>>>>> >> >> > > >>> >>>>>>>> >> >> > >>> >>>>>>>> >> > >>> >>>>>>>> > >>> > https://github.com/mike-tutkowski/incubator-cloudstack/commit/ea74b312a8a36801994500407fd54f0cdda55e37 > >>> >>>>>>>> >> >> > > > > >>> >>>>>>>> >> >> > > > All it is supposed to do is attach and detach a > data > >>> >>>>>>>> disk (that > >>> >>>>>>>> >> has > >>> >>>>>>>> >> >> > > > guaranteed IOPS) with KVM as the hypervisor. The > data > >>> >>>>>>>> disk > >>> >>>>>>>> >> happens to > >>> >>>>>>>> >> >> > be > >>> >>>>>>>> >> >> > > > from SolidFire-backed storage - where we have a > 1:1 > >>> >>>>>>>> mapping > >>> >>>>>>>> >> between a > >>> >>>>>>>> >> >> > > > CloudStack volume and a data disk. > >>> >>>>>>>> >> >> > > > > >>> >>>>>>>> >> >> > > > There is no support for hypervisor snapshots or > stuff > >>> >>>>>>>> like that > >>> >>>>>>>> >> >> > (likely a > >>> >>>>>>>> >> >> > > > future release)...just attaching and detaching a > data > >>> >>>>>>>> disk in 4.3. > >>> >>>>>>>> >> >> > > > > >>> >>>>>>>> >> >> > > > Thanks! > >>> >>>>>>>> >> >> > > > > >>> >>>>>>>> >> >> > > > > >>> >>>>>>>> >> >> > > > On Fri, Sep 20, 2013 at 11:05 PM, Mike Tutkowski < > >>> >>>>>>>> >> >> > > > mike.tutkow...@solidfire.com> wrote: > >>> >>>>>>>> >> >> > > > > >>> >>>>>>>> >> >> > > > > When I re-deployed the DEBs, I didn't remove > >>> >>>>>>>> cloudstack-agent > >>> >>>>>>>> >> >> first. > >>> >>>>>>>> >> >> > > > Would > >>> >>>>>>>> >> >> > > > > that be a problem? I just did a sudo apt-get > >>> install > >>> >>>>>>>> >> >> > cloudstack-agent. > >>> >>>>>>>> >> >> > > > > > >>> >>>>>>>> >> >> > > > > > >>> >>>>>>>> >> >> > > > > On Fri, Sep 20, 2013 at 11:03 PM, Mike > Tutkowski < > >>> >>>>>>>> >> >> > > > > mike.tutkow...@solidfire.com> wrote: > >>> >>>>>>>> >> >> > > > > > >>> >>>>>>>> >> >> > > > >> I get the same error running the command > manually: > >>> >>>>>>>> >> >> > > > >> > >>> >>>>>>>> >> >> > > > >> mtutkowski@ubuntu:/etc/cloudstack/agent$ sudo > >>> >>>>>>>> >> /usr/sbin/service > >>> >>>>>>>> >> >> > > > >> cloudstack-agent status > >>> >>>>>>>> >> >> > > > >> * could not access PID file for > cloudstack-agent > >>> >>>>>>>> >> >> > > > >> > >>> >>>>>>>> >> >> > > > >> > >>> >>>>>>>> >> >> > > > >> On Fri, Sep 20, 2013 at 11:00 PM, Mike > Tutkowski < > >>> >>>>>>>> >> >> > > > >> mike.tutkow...@solidfire.com> wrote: > >>> >>>>>>>> >> >> > > > >> > >>> >>>>>>>> >> >> > > > >>> agent.log looks OK to me: > >>> >>>>>>>> >> >> > > > >>> > >>> >>>>>>>> >> >> > > > >>> 2013-09-20 19:35:39,010 INFO > >>> >>>>>>>> [cloud.agent.AgentShell] > >>> >>>>>>>> >> >> (main:null) > >>> >>>>>>>> >> >> > > > Agent > >>> >>>>>>>> >> >> > > > >>> started > >>> >>>>>>>> >> >> > > > >>> 2013-09-20 19:35:39,011 INFO > >>> >>>>>>>> [cloud.agent.AgentShell] > >>> >>>>>>>> >> >> (main:null) > >>> >>>>>>>> >> >> > > > >>> Implementation Version is 4.3.0-SNAPSHOT > >>> >>>>>>>> >> >> > > > >>> 2013-09-20 19:35:39,015 INFO > >>> >>>>>>>> [cloud.agent.AgentShell] > >>> >>>>>>>> >> >> (main:null) > >>> >>>>>>>> >> >> > > > >>> agent.properties found at > >>> >>>>>>>> >> /etc/cloudstack/agent/agent.properties > >>> >>>>>>>> >> >> > > > >>> 2013-09-20 19:35:39,023 INFO > >>> >>>>>>>> [cloud.agent.AgentShell] > >>> >>>>>>>> >> >> (main:null) > >>> >>>>>>>> >> >> > > > >>> Defaulting to using properties file for > storage > >>> >>>>>>>> >> >> > > > >>> 2013-09-20 19:35:39,024 INFO > >>> >>>>>>>> [cloud.agent.AgentShell] > >>> >>>>>>>> >> >> (main:null) > >>> >>>>>>>> >> >> > > > >>> Defaulting to the constant time backoff > algorithm > >>> >>>>>>>> >> >> > > > >>> 2013-09-20 19:35:39,029 INFO > >>> [cloud.utils.LogUtils] > >>> >>>>>>>> >> (main:null) > >>> >>>>>>>> >> >> > > log4j > >>> >>>>>>>> >> >> > > > >>> configuration found at > >>> >>>>>>>> /etc/cloudstack/agent/log4j-cloud.xml > >>> >>>>>>>> >> >> > > > >>> 2013-09-20 19:35:39,163 INFO > [cloud.agent.Agent] > >>> >>>>>>>> (main:null) > >>> >>>>>>>> >> id > >>> >>>>>>>> >> >> > is 3 > >>> >>>>>>>> >> >> > > > >>> 2013-09-20 19:35:39,197 INFO > >>> >>>>>>>> >> >> > > > >>> > [resource.virtualnetwork.VirtualRoutingResource] > >>> >>>>>>>> (main:null) > >>> >>>>>>>> >> >> > > > >>> VirtualRoutingResource _scriptDir to use: > >>> >>>>>>>> >> >> scripts/network/domr/kvm > >>> >>>>>>>> >> >> > > > >>> > >>> >>>>>>>> >> >> > > > >>> However, I wasn't aware that setup.log was > >>> >>>>>>>> important. This > >>> >>>>>>>> >> seems > >>> >>>>>>>> >> >> to > >>> >>>>>>>> >> >> > > be > >>> >>>>>>>> >> >> > > > a > >>> >>>>>>>> >> >> > > > >>> problem, but I'm not sure what it might > indicate: > >>> >>>>>>>> >> >> > > > >>> > >>> >>>>>>>> >> >> > > > >>> DEBUG:root:execute:sudo /usr/sbin/service > >>> >>>>>>>> cloudstack-agent > >>> >>>>>>>> >> status > >>> >>>>>>>> >> >> > > > >>> DEBUG:root:Failed to execute: * could not > access > >>> PID > >>> >>>>>>>> file for > >>> >>>>>>>> >> >> > > > >>> cloudstack-agent > >>> >>>>>>>> >> >> > > > >>> DEBUG:root:execute:sudo /usr/sbin/service > >>> >>>>>>>> cloudstack-agent > >>> >>>>>>>> >> start > >>> >>>>>>>> >> >> > > > >>> > >>> >>>>>>>> >> >> > > > >>> > >>> >>>>>>>> >> >> > > > >>> On Fri, Sep 20, 2013 at 10:53 PM, Marcus > >>> Sorensen < > >>> >>>>>>>> >> >> > > shadow...@gmail.com > >>> >>>>>>>> >> >> > > > >wrote: > >>> >>>>>>>> >> >> > > > >>> > >>> >>>>>>>> >> >> > > > >>>> Sorry, I saw that in the log, I thought it > was > >>> the > >>> >>>>>>>> agent log > >>> >>>>>>>> >> for > >>> >>>>>>>> >> >> > > some > >>> >>>>>>>> >> >> > > > >>>> reason. Is the agent started? That might be > the > >>> >>>>>>>> place to > >>> >>>>>>>> >> look. > >>> >>>>>>>> >> >> > There > >>> >>>>>>>> >> >> > > > is > >>> >>>>>>>> >> >> > > > >>>> an > >>> >>>>>>>> >> >> > > > >>>> agent log for the agent and one for the setup > >>> when > >>> >>>>>>>> it adds > >>> >>>>>>>> >> the > >>> >>>>>>>> >> >> > host, > >>> >>>>>>>> >> >> > > > >>>> both > >>> >>>>>>>> >> >> > > > >>>> in /var/log > >>> >>>>>>>> >> >> > > > >>>> On Sep 20, 2013 10:42 PM, "Mike Tutkowski" < > >>> >>>>>>>> >> >> > > > >>>> mike.tutkow...@solidfire.com> > >>> >>>>>>>> >> >> > > > >>>> wrote: > >>> >>>>>>>> >> >> > > > >>>> > >>> >>>>>>>> >> >> > > > >>>> > Is it saying that the MS is at the IP > address > >>> or > >>> >>>>>>>> the KVM > >>> >>>>>>>> >> host? > >>> >>>>>>>> >> >> > > > >>>> > > >>> >>>>>>>> >> >> > > > >>>> > The KVM host is at 192.168.233.10. > >>> >>>>>>>> >> >> > > > >>>> > The MS host is at 192.168.233.1. > >>> >>>>>>>> >> >> > > > >>>> > > >>> >>>>>>>> >> >> > > > >>>> > I see this for my host Global Settings > >>> parameter: > >>> >>>>>>>> >> >> > > > >>>> > hostThe ip address of management > >>> >>>>>>>> server192.168.233.1 > >>> >>>>>>>> >> >> > > > >>>> > > >>> >>>>>>>> >> >> > > > >>>> > > >>> >>>>>>>> >> >> > > > >>>> > /etc/cloudstack/agent/agent.properties has > a > >>> >>>>>>>> >> >> host=192.168.233.1 > >>> >>>>>>>> >> >> > > > value. > >>> >>>>>>>> >> >> > > > >>>> > > >>> >>>>>>>> >> >> > > > >>>> > > >>> >>>>>>>> >> >> > > > >>>> > On Fri, Sep 20, 2013 at 10:21 PM, Marcus > >>> Sorensen > >>> >>>>>>>> < > >>> >>>>>>>> >> >> > > > >>>> shadow...@gmail.com > >>> >>>>>>>> >> >> > > > >>>> > >wrote: > >>> >>>>>>>> >> >> > > > >>>> > > >>> >>>>>>>> >> >> > > > >>>> > > The log says your mgmt server is > >>> >>>>>>>> 192.168.233.10? But you > >>> >>>>>>>> >> >> tried > >>> >>>>>>>> >> >> > > to > >>> >>>>>>>> >> >> > > > >>>> telnet > >>> >>>>>>>> >> >> > > > >>>> > to > >>> >>>>>>>> >> >> > > > >>>> > > 192.168.233.1? It might be enough to > change > >>> >>>>>>>> that in > >>> >>>>>>>> >> >> > > > >>>> > > /etc/cloudstack/agent/agent.properties, > but > >>> you > >>> >>>>>>>> may want > >>> >>>>>>>> >> to > >>> >>>>>>>> >> >> > edit > >>> >>>>>>>> >> >> > > > the > >>> >>>>>>>> >> >> > > > >>>> > config > >>> >>>>>>>> >> >> > > > >>>> > > as well to tell it the real ms IP. > >>> >>>>>>>> >> >> > > > >>>> > > On Sep 20, 2013 10:12 PM, "Mike > Tutkowski" < > >>> >>>>>>>> >> >> > > > >>>> mike.tutkow...@solidfire.com > >>> >>>>>>>> >> >> > > > >>>> > > > >>> >>>>>>>> >> >> > > > >>>> > > wrote: > >>> >>>>>>>> >> >> > > > >>>> > > > >>> >>>>>>>> >> >> > > > >>>> > > > Here's what my /etc/network/interfaces > >>> file > >>> >>>>>>>> looks > >>> >>>>>>>> >> like, if > >>> >>>>>>>> >> >> > > that > >>> >>>>>>>> >> >> > > > >>>> is of > >>> >>>>>>>> >> >> > > > >>>> > > > interest (the 192.168.233.0 network is > the > >>> >>>>>>>> NAT network > >>> >>>>>>>> >> >> > VMware > >>> >>>>>>>> >> >> > > > >>>> Fusion > >>> >>>>>>>> >> >> > > > >>>> > set > >>> >>>>>>>> >> >> > > > >>>> > > > up): > >>> >>>>>>>> >> >> > > > >>>> > > > > >>> >>>>>>>> >> >> > > > >>>> > > > auto lo > >>> >>>>>>>> >> >> > > > >>>> > > > iface lo inet loopback > >>> >>>>>>>> >> >> > > > >>>> > > > > >>> >>>>>>>> >> >> > > > >>>> > > > auto eth0 > >>> >>>>>>>> >> >> > > > >>>> > > > iface eth0 inet manual > >>> >>>>>>>> >> >> > > > >>>> > > > > >>> >>>>>>>> >> >> > > > >>>> > > > auto cloudbr0 > >>> >>>>>>>> >> >> > > > >>>> > > > iface cloudbr0 inet static > >>> >>>>>>>> >> >> > > > >>>> > > > address 192.168.233.10 > >>> >>>>>>>> >> >> > > > >>>> > > > netmask 255.255.255.0 > >>> >>>>>>>> >> >> > > > >>>> > > > network 192.168.233.0 > >>> >>>>>>>> >> >> > > > >>>> > > > broadcast 192.168.233.255 > >>> >>>>>>>> >> >> > > > >>>> > > > dns-nameservers 8.8.8.8 > >>> >>>>>>>> >> >> > > > >>>> > > > bridge_ports eth0 > >>> >>>>>>>> >> >> > > > >>>> > > > bridge_fd 5 > >>> >>>>>>>> >> >> > > > >>>> > > > bridge_stp off > >>> >>>>>>>> >> >> > > > >>>> > > > bridge_maxwait 1 > >>> >>>>>>>> >> >> > > > >>>> > > > post-up route add default gw > >>> >>>>>>>> 192.168.233.2 metric 1 > >>> >>>>>>>> >> >> > > > >>>> > > > pre-down route del default gw > >>> >>>>>>>> 192.168.233.2 > >>> >>>>>>>> >> >> > > > >>>> > > > > >>> >>>>>>>> >> >> > > > >>>> > > > > >>> >>>>>>>> >> >> > > > >>>> > > > On Fri, Sep 20, 2013 at 10:08 PM, Mike > >>> >>>>>>>> Tutkowski < > >>> >>>>>>>> >> >> > > > >>>> > > > mike.tutkow...@solidfire.com> wrote: > >>> >>>>>>>> >> >> > > > >>>> > > > > >>> >>>>>>>> >> >> > > > >>>> > > > > You appear to be correct. This is > from > >>> the > >>> >>>>>>>> MS log > >>> >>>>>>>> >> >> (below). > >>> >>>>>>>> >> >> > > > >>>> Discovery > >>> >>>>>>>> >> >> > > > >>>> > > > timed > >>> >>>>>>>> >> >> > > > >>>> > > > > out. > >>> >>>>>>>> >> >> > > > >>>> > > > > > >>> >>>>>>>> >> >> > > > >>>> > > > > I'm not sure why this would be. My > >>> network > >>> >>>>>>>> settings > >>> >>>>>>>> >> >> > > shouldn't > >>> >>>>>>>> >> >> > > > >>>> have > >>> >>>>>>>> >> >> > > > >>>> > > > changed > >>> >>>>>>>> >> >> > > > >>>> > > > > since the last time I tried this. > >>> >>>>>>>> >> >> > > > >>>> > > > > > >>> >>>>>>>> >> >> > > > >>>> > > > > I am able to ping the KVM host from > the > >>> MS > >>> >>>>>>>> host and > >>> >>>>>>>> >> vice > >>> >>>>>>>> >> >> > > > versa. > >>> >>>>>>>> >> >> > > > >>>> > > > > > >>> >>>>>>>> >> >> > > > >>>> > > > > I'm even able to manually kick off a > VM > >>> on > >>> >>>>>>>> the KVM > >>> >>>>>>>> >> host > >>> >>>>>>>> >> >> > and > >>> >>>>>>>> >> >> > > > >>>> ping from > >>> >>>>>>>> >> >> > > > >>>> > > it > >>> >>>>>>>> >> >> > > > >>>> > > > > to the MS host. > >>> >>>>>>>> >> >> > > > >>>> > > > > > >>> >>>>>>>> >> >> > > > >>>> > > > > I am using NAT from my Mac OS X host > >>> (also > >>> >>>>>>>> running > >>> >>>>>>>> >> the > >>> >>>>>>>> >> >> CS > >>> >>>>>>>> >> >> > > MS) > >>> >>>>>>>> >> >> > > > >>>> to the > >>> >>>>>>>> >> >> > > > >>>> > VM > >>> >>>>>>>> >> >> > > > >>>> > > > > running KVM (VMware Fusion). > >>> >>>>>>>> >> >> > > > >>>> > > > > > >>> >>>>>>>> >> >> > > > >>>> > > > > 2013-09-20 19:40:40,141 DEBUG > >>> >>>>>>>> >> >> > > > >>>> [c.c.h.k.d.LibvirtServerDiscoverer] > >>> >>>>>>>> >> >> > > > >>>> > > > > (613487991@qtp-1659933291-3 > >>> :ctx-6b28dc48) > >>> >>>>>>>> Timeout, > >>> >>>>>>>> >> to > >>> >>>>>>>> >> >> > wait > >>> >>>>>>>> >> >> > > > for > >>> >>>>>>>> >> >> > > > >>>> the > >>> >>>>>>>> >> >> > > > >>>> > > host > >>> >>>>>>>> >> >> > > > >>>> > > > > connecting to mgt svr, assuming it is > >>> failed > >>> >>>>>>>> >> >> > > > >>>> > > > > 2013-09-20 19:40:40,144 WARN > >>> >>>>>>>> >> >> [c.c.r.ResourceManagerImpl] > >>> >>>>>>>> >> >> > > > >>>> > > > > (613487991@qtp-1659933291-3 > >>> :ctx-6b28dc48) > >>> >>>>>>>> Unable to > >>> >>>>>>>> >> >> find > >>> >>>>>>>> >> >> > > the > >>> >>>>>>>> >> >> > > > >>>> server > >>> >>>>>>>> >> >> > > > >>>> > > > > resources at http://192.168.233.10 > >>> >>>>>>>> >> >> > > > >>>> > > > > 2013-09-20 19:40:40,145 INFO > >>> >>>>>>>> >> >> > [c.c.u.e.CSExceptionErrorCode] > >>> >>>>>>>> >> >> > > > >>>> > > > > (613487991@qtp-1659933291-3 > >>> :ctx-6b28dc48) > >>> >>>>>>>> Could not > >>> >>>>>>>> >> >> find > >>> >>>>>>>> >> >> > > > >>>> exception: > >>> >>>>>>>> >> >> > > > >>>> > > > > > com.cloud.exception.DiscoveryException > >>> in > >>> >>>>>>>> error code > >>> >>>>>>>> >> >> list > >>> >>>>>>>> >> >> > > for > >>> >>>>>>>> >> >> > > > >>>> > > exceptions > >>> >>>>>>>> >> >> > > > >>>> > > > > 2013-09-20 19:40:40,147 WARN > >>> >>>>>>>> >> [o.a.c.a.c.a.h.AddHostCmd] > >>> >>>>>>>> >> >> > > > >>>> > > > > (613487991@qtp-1659933291-3 > >>> :ctx-6b28dc48) > >>> >>>>>>>> Exception: > >>> >>>>>>>> >> >> > > > >>>> > > > > > com.cloud.exception.DiscoveryException: > >>> >>>>>>>> Unable to add > >>> >>>>>>>> >> >> the > >>> >>>>>>>> >> >> > > host > >>> >>>>>>>> >> >> > > > >>>> > > > > at > >>> >>>>>>>> >> >> > > > >>>> > > > > > >>> >>>>>>>> >> >> > > > >>>> > > > > >>> >>>>>>>> >> >> > > > >>>> > > > >>> >>>>>>>> >> >> > > > >>>> > > >>> >>>>>>>> >> >> > > > >>>> > >>> >>>>>>>> >> >> > > > > >>> >>>>>>>> >> >> > > > >>> >>>>>>>> >> >> > > >>> >>>>>>>> >> >> > >>> >>>>>>>> >> > >>> >>>>>>>> > >>> > com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:778) > >>> >>>>>>>> >> >> > > > >>>> > > > > > >>> >>>>>>>> >> >> > > > >>>> > > > > I do seem to be able to telnet in > from > >>> my > >>> >>>>>>>> KVM host to > >>> >>>>>>>> >> >> the > >>> >>>>>>>> >> >> > MS > >>> >>>>>>>> >> >> > > > >>>> host's > >>> >>>>>>>> >> >> > > > >>>> > > 8250 > >>> >>>>>>>> >> >> > > > >>>> > > > > port: > >>> >>>>>>>> >> >> > > > >>>> > > > > > >>> >>>>>>>> >> >> > > > >>>> > > > > mtutkowski@ubuntu:~$ telnet > >>> 192.168.233.1 > >>> >>>>>>>> 8250 > >>> >>>>>>>> >> >> > > > >>>> > > > > Trying 192.168.233.1... > >>> >>>>>>>> >> >> > > > >>>> > > > > Connected to 192.168.233.1. > >>> >>>>>>>> >> >> > > > >>>> > > > > Escape character is '^]'. > >>> >>>>>>>> >> >> > > > >>>> > > > > > >>> >>>>>>>> >> >> > > > >>>> > > > > >>> >>>>>>>> >> >> > > > >>>> > > > > >>> >>>>>>>> >> >> > > > >>>> > > > > >>> >>>>>>>> >> >> > > > >>>> > > > -- > >>> >>>>>>>> >> >> > > > >>>> > > > *Mike Tutkowski* > >>> >>>>>>>> >> >> > > > >>>> > > > *Senior CloudStack Developer, SolidFire > >>> Inc.* > >>> >>>>>>>> >> >> > > > >>>> > > > e: mike.tutkow...@solidfire.com > >>> >>>>>>>> >> >> > > > >>>> > > > o: 303.746.7302 > >>> >>>>>>>> >> >> > > > >>>> > > > Advancing the way the world uses the > >>> >>>>>>>> >> >> > > > >>>> > > > cloud< > >>> >>>>>>>> >> http://solidfire.com/solution/overview/?video=play> > >>> >>>>>>>> >> >> > > > >>>> > > > *™* > >>> >>>>>>>> >> >> > > > >>>> > > > > >>> >>>>>>>> >> >> > > > >>>> > > > >>> >>>>>>>> >> >> > > > >>>> > > >>> >>>>>>>> >> >> > > > >>>> > > >>> >>>>>>>> >> >> > > > >>>> > > >>> >>>>>>>> >> >> > > > >>>> > -- > >>> >>>>>>>> >> >> > > > >>>> > *Mike Tutkowski* > >>> >>>>>>>> >> >> > > > >>>> > *Senior CloudStack Developer, SolidFire > Inc.* > >>> >>>>>>>> >> >> > > > >>>> > e: mike.tutkow...@solidfire.com > >>> >>>>>>>> >> >> > > > >>>> > o: 303.746.7302 > >>> >>>>>>>> >> >> > > > >>>> > Advancing the way the world uses the > >>> >>>>>>>> >> >> > > > >>>> > cloud< > >>> >>>>>>>> http://solidfire.com/solution/overview/?video=play> > >>> >>>>>>>> >> >> > > > >>>> > *™* > >>> >>>>>>>> >> >> > > > >>>> > > >>> >>>>>>>> >> >> > > > >>>> > >>> >>>>>>>> >> >> > > > >>> > >>> >>>>>>>> >> >> > > > >>> > >>> >>>>>>>> >> >> > > > >>> > >>> >>>>>>>> >> >> > > > >>> -- > >>> >>>>>>>> >> >> > > > >>> *Mike Tutkowski* > >>> >>>>>>>> >> >> > > > >>> *Senior CloudStack Developer, SolidFire Inc.* > >>> >>>>>>>> >> >> > > > >>> e: mike.tutkow...@solidfire.com > >>> >>>>>>>> >> >> > > > >>> o: 303.746.7302 > >>> >>>>>>>> >> >> > > > >>> Advancing the way the world uses the cloud< > >>> >>>>>>>> >> >> > > > > http://solidfire.com/solution/overview/?video=play> > >>> >>>>>>>> >> >> > > > >>> *™* > >>> >>>>>>>> >> >> > > > >>> > >>> >>>>>>>> >> >> > > > >> > >>> >>>>>>>> >> >> > > > >> > >>> >>>>>>>> >> >> > > > >> > >>> >>>>>>>> >> >> > > > >> -- > >>> >>>>>>>> >> >> > > > >> *Mike Tutkowski* > >>> >>>>>>>> >> >> > > > >> *Senior CloudStack Developer, SolidFire Inc.* > >>> >>>>>>>> >> >> > > > >> e: mike.tutkow...@solidfire.com > >>> >>>>>>>> >> >> > > > >> o: 303.746.7302 > >>> >>>>>>>> >> >> > > > >> Advancing the way the world uses the cloud< > >>> >>>>>>>> >> >> > > > > http://solidfire.com/solution/overview/?video=play> > >>> >>>>>>>> >> >> > > > >> *™* > >>> >>>>>>>> >> >> > > > >> > >>> >>>>>>>> >> >> > > > > > >>> >>>>>>>> >> >> > > > > > >>> >>>>>>>> >> >> > > > > > >>> >>>>>>>> >> >> > > > > -- > >>> >>>>>>>> >> >> > > > > *Mike Tutkowski* > >>> >>>>>>>> >> >> > > > > *Senior CloudStack Developer, SolidFire Inc.* > >>> >>>>>>>> >> >> > > > > e: mike.tutkow...@solidfire.com > >>> >>>>>>>> >> >> > > > > o: 303.746.7302 > >>> >>>>>>>> >> >> > > > > Advancing the way the world uses the cloud< > >>> >>>>>>>> >> >> > > > > http://solidfire.com/solution/overview/?video=play> > >>> >>>>>>>> >> >> > > > > *™* > >>> >>>>>>>> >> >> > > > > > >>> >>>>>>>> >> >> > > > > >>> >>>>>>>> >> >> > > > > >>> >>>>>>>> >> >> > > > > >>> >>>>>>>> >> >> > > > -- > >>> >>>>>>>> >> >> > > > *Mike Tutkowski* > >>> >>>>>>>> >> >> > > > *Senior CloudStack Developer, SolidFire Inc.* > >>> >>>>>>>> >> >> > > > e: mike.tutkow...@solidfire.com > >>> >>>>>>>> >> >> > > > o: 303.746.7302 > >>> >>>>>>>> >> >> > > > Advancing the way the world uses the > >>> >>>>>>>> >> >> > > > cloud< > >>> http://solidfire.com/solution/overview/?video=play > >>> >>>>>>>> > > >>> >>>>>>>> >> >> > > > *™* > >>> >>>>>>>> >> >> > > > > >>> >>>>>>>> >> >> > > > >>> >>>>>>>> >> >> > > >>> >>>>>>>> >> >> > > >>> >>>>>>>> >> >> > > >>> >>>>>>>> >> >> > -- > >>> >>>>>>>> >> >> > *Mike Tutkowski* > >>> >>>>>>>> >> >> > *Senior CloudStack Developer, SolidFire Inc.* > >>> >>>>>>>> >> >> > e: mike.tutkow...@solidfire.com > >>> >>>>>>>> >> >> > o: 303.746.7302 > >>> >>>>>>>> >> >> > Advancing the way the world uses the > >>> >>>>>>>> >> >> > cloud< > http://solidfire.com/solution/overview/?video=play > >>> > > >>> >>>>>>>> >> >> > *™* > >>> >>>>>>>> >> >> > > >>> >>>>>>>> >> >> > >>> >>>>>>>> >> > > >>> >>>>>>>> >> > > >>> >>>>>>>> >> > > >>> >>>>>>>> >> > -- > >>> >>>>>>>> >> > *Mike Tutkowski* > >>> >>>>>>>> >> > *Senior CloudStack Developer, SolidFire Inc.* > >>> >>>>>>>> >> > e: mike.tutkow...@solidfire.com > >>> >>>>>>>> >> > o: 303.746.7302 > >>> >>>>>>>> >> > Advancing the way the world uses the > >>> >>>>>>>> >> > cloud<http://solidfire.com/solution/overview/?video=play > > > >>> >>>>>>>> >> > *™* > >>> >>>>>>>> >> > >>> >>>>>>>> > > >>> >>>>>>>> > > >>> >>>>>>>> > > >>> >>>>>>>> > -- > >>> >>>>>>>> > *Mike Tutkowski* > >>> >>>>>>>> > *Senior CloudStack Developer, SolidFire Inc.* > >>> >>>>>>>> > e: mike.tutkow...@solidfire.com > >>> >>>>>>>> > o: 303.746.7302 > >>> >>>>>>>> > Advancing the way the world uses the > >>> >>>>>>>> > cloud<http://solidfire.com/solution/overview/?video=play> > >>> >>>>>>>> > *™* > >>> >>>>>>>> > >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> -- > >>> >>>>>>> *Mike Tutkowski* > >>> >>>>>>> *Senior CloudStack Developer, SolidFire Inc.* > >>> >>>>>>> e: mike.tutkow...@solidfire.com > >>> >>>>>>> o: 303.746.7302 > >>> >>>>>>> Advancing the way the world uses the cloud< > >>> http://solidfire.com/solution/overview/?video=play> > >>> >>>>>>> *™* > >>> >>>>>>> > >>> >>>>>> > >>> >>>>>> > >>> >>>>>> > >>> >>>>>> -- > >>> >>>>>> *Mike Tutkowski* > >>> >>>>>> *Senior CloudStack Developer, SolidFire Inc.* > >>> >>>>>> e: mike.tutkow...@solidfire.com > >>> >>>>>> o: 303.746.7302 > >>> >>>>>> Advancing the way the world uses the cloud< > >>> http://solidfire.com/solution/overview/?video=play> > >>> >>>>>> *™* > >>> >>>>>> > >>> >>>>> > >>> >>>>> > >>> >>>>> > >>> >>>>> -- > >>> >>>>> *Mike Tutkowski* > >>> >>>>> *Senior CloudStack Developer, SolidFire Inc.* > >>> >>>>> e: mike.tutkow...@solidfire.com > >>> >>>>> o: 303.746.7302 > >>> >>>>> Advancing the way the world uses the cloud< > >>> http://solidfire.com/solution/overview/?video=play> > >>> >>>>> *™* > >>> >>>>> > >>> >>>> > >>> >>>> > >>> >>>> > >>> >>>> -- > >>> >>>> *Mike Tutkowski* > >>> >>>> *Senior CloudStack Developer, SolidFire Inc.* > >>> >>>> e: mike.tutkow...@solidfire.com > >>> >>>> o: 303.746.7302 > >>> >>>> Advancing the way the world uses the cloud< > >>> http://solidfire.com/solution/overview/?video=play> > >>> >>>> *™* > >>> >>>> > >>> >>> > >>> >>> > >>> >>> > >>> >>> -- > >>> >>> *Mike Tutkowski* > >>> >>> *Senior CloudStack Developer, SolidFire Inc.* > >>> >>> e: mike.tutkow...@solidfire.com > >>> >>> o: 303.746.7302 > >>> >>> Advancing the way the world uses the cloud< > >>> http://solidfire.com/solution/overview/?video=play> > >>> >>> *™* > >>> >>> > >>> >> > >>> >> > >>> >> > >>> >> -- > >>> >> *Mike Tutkowski* > >>> >> *Senior CloudStack Developer, SolidFire Inc.* > >>> >> e: mike.tutkow...@solidfire.com > >>> >> o: 303.746.7302 > >>> >> Advancing the way the world uses the cloud< > >>> http://solidfire.com/solution/overview/?video=play> > >>> >> *™* > >>> >> > >>> > > >>> > > >>> > > >>> > -- > >>> > *Mike Tutkowski* > >>> > *Senior CloudStack Developer, SolidFire Inc.* > >>> > e: mike.tutkow...@solidfire.com > >>> > o: 303.746.7302 > >>> > Advancing the way the world uses the > >>> > cloud<http://solidfire.com/solution/overview/?video=play> > >>> > *™* > >>> > >> > >> > >> > >> -- > >> *Mike Tutkowski* > >> *Senior CloudStack Developer, SolidFire Inc.* > >> e: mike.tutkow...@solidfire.com > >> o: 303.746.7302 > >> Advancing the way the world uses the cloud< > http://solidfire.com/solution/overview/?video=play> > >> *™* > >> > > > > > > > > -- > > *Mike Tutkowski* > > *Senior CloudStack Developer, SolidFire Inc.* > > e: mike.tutkow...@solidfire.com > > o: 303.746.7302 > > Advancing the way the world uses the > > cloud<http://solidfire.com/solution/overview/?video=play> > > *™* > -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloud<http://solidfire.com/solution/overview/?video=play> *™*
