Why is my DEBUG show uuid of 8a4039f2-bb71-11e4-8c76-0050569b1662 but below in
my catch exception it shows uuid ab6f3bcd-4c3e-4a7a-9f8b-45a822dbaaaf.
See below.
2017-06-21 10:46:16,431 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-45:ctx-7cdfe536 job-342/job-147412 ctx-39b2bc63) Seq
1-6981705322331112844:
Sending { Cmd , MgmtId: 345050411715, via: 1(Flex-Xen2.flexhost.local), Ver:
v1,
Flags: 100111,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":
{"path":"ab6f3bcd-4c3e-4a7a-9f8b-45a822dbaaaf","origUrl":"http://download.cloud.com/templates/4.5/systemvm64template-4.5-xen.vhd.bz2";,
"uuid":"8a4039f2-bb71-11e4-8c76-0050569b1662","id":1,"format":"VHD","accountId":1,"checksum":"2b15ab4401c2d655264732d3fc600241","hvm":false,
"displayText":"SystemVM Template
(XenServer)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"FlexSAN2-LUN0",
"id":5,"poolType":"PreSetup","host":"localhost","path":"/FlexSAN2-LUN0","port":0,"url":"PreSetup://localhost/FlexSAN2-LUN0/?ROLE=Primary&STOREUUID=FlexSAN2-LUN0"}},
"name":"routing-1","hypervisorType":"XenServer"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{
"uuid":"a2456229-2942-4d9b-9bff-c6d9ea004fbd","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{
"uuid":"FlexSAN2-LUN0","id":5,"poolType":"PreSetup","host":"localhost","path":"/FlexSAN2-LUN0","port":0,"url":
"PreSetup://localhost/FlexSAN2-LUN0/?ROLE=Primary&STOREUUID=FlexSAN2-LUN0"}},"name":"ROOT-22411","size":2689602048,"volumeId":27330,"vmName":
"v-22411-VM","accountId":1,"format":"VHD","provisioningType":"THIN","id":27330,"deviceId":0,"hypervisorType":"XenServer"}},"executeInSequence":true,"options":{},"wait":0}}]
}
2017-06-21 10:46:16,431 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-45:ctx-7cdfe536 job-342/job-147412 ctx-39b2bc63) Seq
1-6981705322331112844:
2017-06-21 10:46:16,444 WARN [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-152:ctx-385c99e9)
Unable to create volume;
Pool=volumeTO[uuid=a2456229-2942-4d9b-9bff-c6d9ea004fbd|path=null|datastore=PrimaryDataStoreTO[uuid=FlexSAN2-LUN0|name=null|id=5|pooltype=PreSetup]];
Disk:com.cloud.utils.exception.CloudRuntimeException: Catch Exception
com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for
uuid: ab6f3bcd-4c3e-4a7a-9f8b-45a822dbaaaf failed due to The uuid you supplied
was invalid.
Now if I check the DB to find out what my templates uuid should be.
SELECT * FROM cloud.vm_template where type='SYSTEM';
1 routing-1 SystemVM Template (XenServer)
8a4039f2-bb71-11e4-8c76-0050569b1662 0 0 SYSTEM 0 64
http://download.cloud.com/templates/4.5/systemvm64template-4.5-xen.vhd.bz2
VHD 2015-02-23 09:35:05 1
2b15ab4401c2d655264732d3fc600241 SystemVM Template (XenServer) 0
0 184 1 0 1 0 XenServer
0 2689602048 Active 0 0
3 routing-3 SystemVM Template (KVM)
8a46062a-bb71-11e4-8c76-0050569b1662 0 0 SYSTEM 0 64
http://download.cloud.com/templates/4.5/systemvm64template-4.5-kvm.qcow2.bz2
QCOW2 2015-02-23 09:35:05 1
aa9f501fecd3de1daeb9e2f357f6f002 SystemVM Template (KVM) 0 0
15 1 0 1 0 KVM 0
Active 0 0
8 routing-8 SystemVM Template (vSphere)
8a4e70c6-bb71-11e4-8c76-0050569b1662 0 0 SYSTEM 0 64
http://download.cloud.com/templates/4.5/systemvm64template-4.5-vmware.ova
OVA 2015-02-23 09:35:05 1
3106a79a4ce66cd7f6a7c50e93f2db57 SystemVM Template (vSphere) 0
0 15 1 0 1 0 VMware 0
Active 0 1
9 routing-9 SystemVM Template (HyperV)
8a5184e6-bb71-11e4-8c76-0050569b1662 0 0 SYSTEM 0 64
http://download.cloud.com/templates/4.5/systemvm64template-4.5-hyperv.vhd.zip
VHD 2015-02-23 09:35:05 1
70bd30ea02ee9ed67d2c6b85c179cee9 SystemVM Template (HyperV) 0
0 15 1 0 1 0 Hyperv 0
Active 0 0
10 routing-10 SystemVM Template (LXC)
5bb9e71c-bb72-11e4-8c76-0050569b1662 0 0 SYSTEM 0 64
http://download.cloud.com/templates/4.5/systemvm64template-4.5-kvm.qcow2.bz2
QCOW2 2015-02-23 09:40:56 1
aa9f501fecd3de1daeb9e2f357f6f002 SystemVM Template (LXC) 0 0
15 1 0 1 0 LXC 0
Active 0 0
Ok so that shows me my system templates UUID is
8a4039f2-bb71-11e4-8c76-0050569b1662 and that lines up correctly with my debug
command uuid 8a4039f2-bb71-11e4-8c76-0050569b1662.
Suggestions ? Ideas? Thoughts?
Thank you.
Jeremy
-----Original Message-----
From: Jeremy Peterson [mailto:[email protected]]
Sent: Wednesday, June 21, 2017 11:58 AM
To: [email protected]; S. Brüseke - proIO GmbH <[email protected]>
Subject: RE: Recreating SystemVM's
You are correct I had 2 hosts disabled when I tried to launch that VM. But my
hosts all show state Up.
Heres Flex-Xen1.flexhost.local
http://prntscr.com/fmi0tw
Heres the info page of the host
http://prntscr.com/fmi16g
Resource state: Enabled
State up: Up
I did a force reconnect on all hosts and that cleared the avoid set error.
But now I am getting UUID invalid when trying to launch a VM. This is whats
happening to the system VM's.
https://pastebin.com/2DhzFVDZ
You see it errors : The uuid you supplied was invalid.
Now I see the above command declared the host and storage but the UUID is
"uuid":"8a4039f2-bb71-11e4-8c76-0050569b1662"
How can I see what that ties to ?
I redeployed systemcl64template-5.6-xen.vhd.bz2 last week does that not
recreated the uuid ?
Jeremy
-----Original Message-----
From: Dag Sonstebo [mailto:[email protected]]
Sent: Wednesday, June 21, 2017 11:12 AM
To: [email protected]; S. Brüseke - proIO GmbH <[email protected]>
Subject: Re: Recreating SystemVM's
Hi Jeremy,
You have 6 hosts: “List of hosts in ascending order of number of VMs: [15, 17,
19, 1, 16, 18]” – my guess is you have disabled hosts 16+18 for their reboot.
You immediately have the rest of the hosts in an avoid set: “Deploy avoids
pods: [], clusters: [], hosts: [17, 1, 19, 15]”.
So you need to work out why those hosts are considered non-valid. Do they show
up as live in your CloudStack GUI? Are they all enabled as well as out of
maintenance mode?
Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue
On 21/06/2017, 15:13, "Jeremy Peterson" <[email protected]> wrote:
So this morning I reconnected all hosts.
I also disabled my two hosts that need to reboot and powered on a VM and
now I am getting a Insufficient Resources.
Whats odd is that Host Allocator returning 0 suitable hosts?
2017-06-21 08:43:53,695 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-71:ctx-f01a90b9 job-146764/job-146768 ctx-66c78348) Root
volume is ready, need to place VM in volume's cluster
2017-06-21 08:43:53,695 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-71:ctx-f01a90b9 job-146764/job-146768 ctx-66c78348)
Vol[537|vm=446|ROOT] is READY, changing deployment plan to use this pool's
dcId: 1 , podId: 1 , and clusterId: 1
2017-06-21 08:43:53,702 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-71:ctx-f01a90b9 job-146764/job-146768 ctx-66c78348) Deploy
avoids pods: [], clusters: [], hosts: [17, 1, 19, 15]
2017-06-21 08:43:53,703 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-71:ctx-f01a90b9 job-146764/job-146768 ctx-66c78348)
DeploymentPlanner allocation algorithm:
com.cloud.deploy.UserDispersingPlanner@4cafa203
2017-06-21 08:43:53,703 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-71:ctx-f01a90b9 job-146764/job-146768 ctx-66c78348) Trying
to allocate a host and storage pools from dc:1, pod:1,cluster:1, requested cpu:
8000, requested ram: 12884901888
2017-06-21 08:43:53,703 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-71:ctx-f01a90b9 job-146764/job-146768 ctx-66c78348) Is ROOT
volume READY (pool already allocated)?: Yes
2017-06-21 08:43:53,703 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-71:ctx-f01a90b9 job-146764/job-146768 ctx-66c78348) This VM
has last host_id specified, trying to choose the same host: 1
2017-06-21 08:43:53,704 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-71:ctx-f01a90b9 job-146764/job-146768 ctx-66c78348) The last
host of this VM is in avoid set
2017-06-21 08:43:53,704 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-71:ctx-f01a90b9 job-146764/job-146768 ctx-66c78348) Cannot
choose the last host to deploy this VM
2017-06-21 08:43:53,704 DEBUG [c.c.d.FirstFitPlanner]
(Work-Job-Executor-71:ctx-f01a90b9 job-146764/job-146768 ctx-66c78348)
Searching resources only under specified Cluster: 1
2017-06-21 08:43:53,714 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-71:ctx-f01a90b9 job-146764/job-146768 ctx-66c78348) Checking
resources in Cluster: 1 under Pod: 1
2017-06-21 08:43:53,714 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-71:ctx-f01a90b9 job-146764/job-146768 ctx-66c78348
FirstFitRoutingAllocator) Looking for hosts in dc: 1 pod:1 cluster:1
2017-06-21 08:43:53,718 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-71:ctx-f01a90b9 job-146764/job-146768 ctx-66c78348
FirstFitRoutingAllocator) List of hosts in ascending order of number of VMs:
[15, 17, 19, 1, 16, 18]
2017-06-21 08:43:53,718 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-71:ctx-f01a90b9 job-146764/job-146768 ctx-66c78348
FirstFitRoutingAllocator) FirstFitAllocator has 4 hosts to check for
allocation: [Host[-15-Routing], Host[-17-Routing], Host[-19-Routing],
Host[-1-Routing]]
2017-06-21 08:43:53,727 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-71:ctx-f01a90b9 job-146764/job-146768 ctx-66c78348
FirstFitRoutingAllocator) Found 4 hosts for allocation after prioritization:
[Host[-15-Routing], Host[-17-Routing], Host[-19-Routing], Host[-1-Routing]]
2017-06-21 08:43:53,727 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-71:ctx-f01a90b9 job-146764/job-146768 ctx-66c78348
FirstFitRoutingAllocator) Looking for speed=8000Mhz, Ram=12288
2017-06-21 08:43:53,727 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-71:ctx-f01a90b9 job-146764/job-146768 ctx-66c78348
FirstFitRoutingAllocator) Host name: Flex-Xen3.flexhost.local, hostId: 15 is in
avoid set, skipping this and trying other available hosts
2017-06-21 08:43:53,727 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-71:ctx-f01a90b9 job-146764/job-146768 ctx-66c78348
FirstFitRoutingAllocator) Host name: Flex-Xen4.flexhost.local, hostId: 17 is in
avoid set, skipping this and trying other available hosts
2017-06-21 08:43:53,727 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-71:ctx-f01a90b9 job-146764/job-146768 ctx-66c78348
FirstFitRoutingAllocator) Host name: Flex-Xen1.flexhost.local, hostId: 19 is in
avoid set, skipping this and trying other available hosts
2017-06-21 08:43:53,727 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-71:ctx-f01a90b9 job-146764/job-146768 ctx-66c78348
FirstFitRoutingAllocator) Host name: Flex-Xen2.flexhost.local, hostId: 1 is in
avoid set, skipping this and trying other available hosts
2017-06-21 08:43:53,727 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-71:ctx-f01a90b9 job-146764/job-146768 ctx-66c78348
FirstFitRoutingAllocator) Host Allocator returning 0 suitable hosts
2017-06-21 08:43:53,727 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-71:ctx-f01a90b9 job-146764/job-146768 ctx-66c78348) No
suitable hosts found
2017-06-21 08:43:53,727 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-71:ctx-f01a90b9 job-146764/job-146768 ctx-66c78348) No
suitable hosts found under this Cluster: 1
2017-06-21 08:43:53,728 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-71:ctx-f01a90b9 job-146764/job-146768 ctx-66c78348) Could
not find suitable Deployment Destination for this VM under any clusters,
returning.
2017-06-21 08:43:53,728 DEBUG [c.c.d.FirstFitPlanner]
(Work-Job-Executor-71:ctx-f01a90b9 job-146764/job-146768 ctx-66c78348)
Searching resources only under specified Cluster: 1
2017-06-21 08:43:53,729 DEBUG [c.c.d.FirstFitPlanner]
(Work-Job-Executor-71:ctx-f01a90b9 job-146764/job-146768 ctx-66c78348) The
specified cluster is in avoid set, returning.
2017-06-21 08:43:53,736 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-71:ctx-f01a90b9 job-146764/job-146768 ctx-66c78348) Deploy
avoids pods: [], clusters: [1], hosts: [17, 1, 19, 15]
2017-06-21 08:43:53,737 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-71:ctx-f01a90b9 job-146764/job-146768 ctx-66c78348)
DeploymentPlanner allocation algorithm:
com.cloud.deploy.UserDispersingPlanner@4cafa203
2017-06-21 08:43:53,737 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-71:ctx-f01a90b9 job-146764/job-146768 ctx-66c78348) Trying
to allocate a host and storage pools from dc:1, pod:1,cluster:null, requested
cpu: 8000, requested ram: 12884901888
2017-06-21 08:43:53,737 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-71:ctx-f01a90b9 job-146764/job-146768 ctx-66c78348) Is ROOT
volume READY (pool already allocated)?: No
2017-06-21 08:43:53,737 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-71:ctx-f01a90b9 job-146764/job-146768 ctx-66c78348) This VM
has last host_id specified, trying to choose the same host: 1
2017-06-21 08:43:53,739 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-71:ctx-f01a90b9 job-146764/job-146768 ctx-66c78348) The last
host of this VM is in avoid set
All oddities.
So I did a force reconnect on all 6 hosts and enabled the two hosts that
were pending updates.
Jeremy
-----Original Message-----
From: Jeremy Peterson [mailto:[email protected]]
Sent: Tuesday, June 20, 2017 12:33 PM
To: [email protected]; S. Brüseke - proIO GmbH
<[email protected]>
Subject: RE: Recreating SystemVM's
Ok so my issues have not gone away.
I have two hosts that have not rebooted yet tonight I will be maintenancing
those hosts out and migrating vm's away from those hosts and then performing a
reboot of the host and installing a couple xenserver updates.
One thing is I am not getting the CANNOT ATTACH NETWORK error anymore which
is cool but.
https://drive.google.com/open?id=0B5IXhrpPAT9qQ0FFUmRyRjN4NlE
Take a look at creation of VM 20685
2017-06-20 12:15:48,083 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-82:ctx-c39fa1f8 job-1042/job-138603 ctx-c17ce6fc) Found a
potential host id: 1 name: Flex-Xen2.flexhost.local and associated storage
pools for this VM
2017-06-20 12:15:48,084 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-82:ctx-c39fa1f8 job-1042/job-138603 ctx-c17ce6fc) Returning
Deployment Destination:
Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))]
: Dest[Zone(1)-Pod(1)-Cluster(1)-Host(1)-Storage(Volume(25604|ROOT-->Pool(5))]
2017-06-20 12:15:48,084 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-82:ctx-c39fa1f8 job-1042/job-138603 ctx-c17ce6fc) Deployment
found - P0=VM[SecondaryStorageVm|s-20685-VM],
P0=Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))]
: Dest[Zone(1)-Pod(1)-Cluster(1)-Host(1)-Storage(Volume(25604|ROOT-->Pool(5))]
So it found a host and storage pool
Networks were already created on line 482-484
But then look it fails on create volume UUID is invalid???
2017-06-20 12:15:48,262 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-88:ctx-c51dafa0 job-342/job-138604 ctx-75edebb0) VM is being
created in podId: 1
2017-06-20 12:15:48,264 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Work-Job-Executor-88:ctx-c51dafa0 job-342/job-138604 ctx-75edebb0) Network
id=200 is already implemented
2017-06-20 12:15:48,269 DEBUG [c.c.n.g.PodBasedNetworkGuru]
(Work-Job-Executor-82:ctx-c39fa1f8 job-1042/job-138603 ctx-c17ce6fc) Allocated
a nic
NicProfile[81905-20685-0493941d-d193-4325-84bc-d325a8900332-10.90.2.207-null
for VM[SecondaryStorageVm|s-20685-VM]
2017-06-20 12:15:48,280 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Work-Job-Executor-82:ctx-c39fa1f8 job-1042/job-138603 ctx-c17ce6fc) Network
id=203 is already implemented
2017-06-20 12:15:48,290 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Work-Job-Executor-88:ctx-c51dafa0 job-342/job-138604 ctx-75edebb0) Network
id=202 is already implemented
2017-06-20 12:15:48,316 DEBUG [c.c.n.g.StorageNetworkGuru]
(Work-Job-Executor-82:ctx-c39fa1f8 job-1042/job-138603 ctx-c17ce6fc) Allocated
a storage nic
NicProfile[81906-20685-0493941d-d193-4325-84bc-d325a8900332-10.83.2.205-null
for VM[SecondaryStorageVm|s-20685-VM]
2017-06-20 12:15:48,336 DEBUG [o.a.c.e.o.VolumeOrchestrator]
(Work-Job-Executor-82:ctx-c39fa1f8 job-1042/job-138603 ctx-c17ce6fc) Checking
if we need to prepare 1 volumes for VM[SecondaryStorageVm|s-20685-VM]
2017-06-20 12:15:48,342 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl]
(Work-Job-Executor-82:ctx-c39fa1f8 job-1042/job-138603 ctx-c17ce6fc) template 1
is already in store:5, type:Image
2017-06-20 12:15:48,344 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl]
(Work-Job-Executor-82:ctx-c39fa1f8 job-1042/job-138603 ctx-c17ce6fc) template 1
is already in store:5, type:Primary
2017-06-20 12:15:48,346 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Work-Job-Executor-88:ctx-c51dafa0 job-342/job-138604 ctx-75edebb0) Network
id=201 is already implemented
2017-06-20 12:15:48,372 DEBUG [c.c.d.d.DataCenterIpAddressDaoImpl]
(Work-Job-Executor-88:ctx-c51dafa0 job-342/job-138604 ctx-75edebb0) Releasing
ip address for instance=49817
2017-06-20 12:15:48,381 DEBUG [o.a.c.s.m.AncientDataMotionStrategy]
(Work-Job-Executor-82:ctx-c39fa1f8 job-1042/job-138603 ctx-c17ce6fc) copyAsync
inspecting src type TEMPLATE copyAsync inspecting dest type VOLUME
2017-06-20 12:15:48,386 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-82:ctx-c39fa1f8 job-1042/job-138603 ctx-c17ce6fc) Seq
16-3622864425242874354: Sending { Cmd , MgmtId: 345050411715, via:
16(Flex-Xen6.flexhost.local), Ver: v1, Flags: 100111,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"ab6f3bcd-4c3e-4a7a-9f8b-45a822dbaaaf","origUrl":"http://download.cloud.com/templates/4.5/systemvm64template-4.5-xen.vhd.bz2","uuid":"8a4039f2-bb71-11e4-8c76-0050569b1662","id":1,"format":"VHD","accountId":1,"checksum":"2b15ab4401c2d655264732d3fc600241","hvm":false,"displayText":"SystemVM
Template
(XenServer)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"FlexSAN2-LUN0","id":5,"poolType":"PreSetup","host":"localhost","path":"/FlexSAN2-LUN0","port":0,"url":"PreSetup://localhost/FlexSAN2-LUN0/?ROLE=Primary&STOREUUID=FlexSAN2-LUN0"}},"name":"routing-1","hypervisorType":"XenServer"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"4dba9def-2657-430e-8cd8-9369aebcaa25","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"FlexSAN2-LUN0","id":5,"poolType":"PreSetup","host":"localhost","path":"/FlexSAN2-LUN0","port":0,"url":"PreSetup://localhost/FlexSAN2-LUN0/?ROLE=Primary&STOREUUID=FlexSAN2-LUN0"}},"name":"ROOT-20685","size":2689602048,"volumeId":25604,"vmName":"s-20685-VM","accountId":1,"format":"VHD","provisioningType":"THIN","id":25604,"deviceId":0,"hypervisorType":"XenServer"}},"executeInSequence":true,"options":{},"wait":0}}]
}
2017-06-20 12:15:48,386 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-82:ctx-c39fa1f8 job-1042/job-138603 ctx-c17ce6fc) Seq
16-3622864425242874354: Executing: { Cmd , MgmtId: 345050411715, via:
16(Flex-Xen6.flexhost.local), Ver: v1, Flags: 100111,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"ab6f3bcd-4c3e-4a7a-9f8b-45a822dbaaaf","origUrl":"http://download.cloud.com/templates/4.5/systemvm64template-4.5-xen.vhd.bz2","uuid":"8a4039f2-bb71-11e4-8c76-0050569b1662","id":1,"format":"VHD","accountId":1,"checksum":"2b15ab4401c2d655264732d3fc600241","hvm":false,"displayText":"SystemVM
Template
(XenServer)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"FlexSAN2-LUN0","id":5,"poolType":"PreSetup","host":"localhost","path":"/FlexSAN2-LUN0","port":0,"url":"PreSetup://localhost/FlexSAN2-LUN0/?ROLE=Primary&STOREUUID=FlexSAN2-LUN0"}},"name":"routing-1","hypervisorType":"XenServer"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"4dba9def-2657-430e-8cd8-9369aebcaa25","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"FlexSAN2-LUN0","id":5,"poolType":"PreSetup","host":"localhost","path":"/FlexSAN2-LUN0","port":0,"url":"PreSetup://localhost/FlexSAN2-LUN0/?ROLE=Primary&STOREUUID=FlexSAN2-LUN0"}},"name":"ROOT-20685","size":2689602048,"volumeId":25604,"vmName":"s-20685-VM","accountId":1,"format":"VHD","provisioningType":"THIN","id":25604,"deviceId":0,"hypervisorType":"XenServer"}},"executeInSequence":true,"options":{},"wait":0}}]
}
2017-06-20 12:15:48,386 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-74:ctx-0acdd419) Seq 16-3622864425242874354: Executing request
2017-06-20 12:15:48,387 DEBUG [c.c.n.g.PodBasedNetworkGuru]
(Work-Job-Executor-88:ctx-c51dafa0 job-342/job-138604 ctx-75edebb0) Allocated a
nic
NicProfile[49817-12662-629b85e7-ce19-4568-9df7-143c76d24300-10.90.2.204-null
for VM[ConsoleProxy|v-12662-VM]
So how do I check UUID's to validate that they are correct ?
2017-06-20 12:15:48,391 DEBUG [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-74:ctx-0acdd419) Catch Exception
com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid:
ab6f3bcd-4c3e-4a7a-9f8b-45a822dbaaaf failed due to The uuid you supplied was
invalid.
2017-06-20 12:15:48,391 WARN [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-74:ctx-0acdd419) Unable to create volume;
Pool=volumeTO[uuid=4dba9def-2657-430e-8cd8-9369aebcaa25|path=null|datastore=PrimaryDataStoreTO[uuid=FlexSAN2-LUN0|name=null|id=5|pooltype=PreSetup]];
Disk:
com.cloud.utils.exception.CloudRuntimeException: Catch Exception
com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid:
ab6f3bcd-4c3e-4a7a-9f8b-45a822dbaaaf failed due to The uuid you supplied was
invalid.
Jeremy
-----Original Message-----
From: Jeremy Peterson [mailto:[email protected]]
Sent: Thursday, June 15, 2017 4:20 PM
To: [email protected]; S. Brüseke - proIO GmbH
<[email protected]>
Subject: RE: Recreating SystemVM's
What type of networking are you using on the XenServers?
XenServers are connected with 6 nic's per host connected to separate
nexus 5k switches
NIC 0 and NIC 1 are Bond 0+1 10Gb nics
NIC 2 and NIC 3 are Bond 2+3 10Gb nics
NIC 4 and NIC 5 are Bond 4+5 2Gb nics
Cloudstack is running Advanced networking
Bond 0+1 is primary storage
Bond 2+3 is secondary storage
Bond 4+5 is Management
What version of os does the ms run on?
CentOS release 6.9 (Final)
What are the systemvm templates defined in your env?
http://cloudstack.apt-get.eu/systemvm/4.5/systemvm64template-4.5-xen.vhd.bz2
What is the version of the systemvm.iso?
Successfully installed system VM template to
/secondary/template/tmpl/1/1/
I just reinstalled systemvm's from the above 4.5-xen.vhd What is the
capacity you have in your (test) environment?
This is a production enviroment and currently cloudstack shows the
following.
Public IP Addresses 61%
VLAN 35%
Management IP Addresses 20%
Primary Storage 44%
CPU 21%
Memory 5%
Of cource Secondary Storage shows 0%
What is the host os version for the hypervisors?
XenServer 6.5 SP1
What is the management network range?
management.network.cidr 10.90.1.0/24
What are the other physical networks?
?? Not sure what more you need
What storage do you use?
Primary - ISCSI
Secondary - NFS
Is it reachable from the systemvm?
All of my CS management servers have internet access Is the big bad
internet reachable for your SSVM’s public interface?
My SSVM does not go online but yes the public network is the same as
the VR public vlan and all instances behind VR's are connected to the internet
at this time
Jeremy
-----Original Message-----
From: Daan Hoogland [mailto:[email protected]]
Sent: Thursday, June 15, 2017 9:34 AM
To: [email protected]; S. Brüseke - proIO GmbH
<[email protected]>
Subject: Re: Recreating SystemVM's
Your problem might be like what Swen says, Jeremy but also a wrong systemvm
offering or a fault in your management network definition.
I am going to sum up some trivialities so bear with me;
What type of networking are you using on the XenServers?
What version of os does the ms run on?
What are the systemvm templates defined in your env?
What is the version of the systemvm.iso?
What is the capacity you have in your (test) environment?
What is the host os version for the hypervisors?
What is the management network range?
What are the other physical networks?
What storage do you use?
Is it reachable from the systemvm?
Is the big bad internet reachable for your SSVM’s public interface?
And of course,
How is the weather, where you are at?
I am not sure any of these question is going to lead you in the right
direction but one of them should.
On 15/06/17 13:56, "S. Brüseke - proIO GmbH" <[email protected]> wrote:
I once did have some similar problem with my systemvms and my root
cause was that in the global settings it referred to the wrong systemvm
template. I am not sure if this helps you, but wanted to tell you.
Mit freundlichen Grüßen / With kind regards,
Swen
-----Ursprüngliche Nachricht-----
Von: Jeremy Peterson [mailto:[email protected]]
Gesendet: Donnerstag, 15. Juni 2017 01:55
An: [email protected]
Betreff: RE: Recreating SystemVM's
Hahaha. The best response ever.
I dug through these emails and someone had soft of the same log
messages cannot attach network and blamed xenserver. Ok I'm cool with that but
why oh why is it only system vms?
Jeremy
________________________________________
From: Imran Ahmed [[email protected]]
Sent: Wednesday, June 14, 2017 6:22 PM
To: [email protected]
Subject: RE: Recreating SystemVM's
Yes,
-----Original Message-----
From: Jeremy Peterson [mailto:[email protected]]
Sent: Wednesday, June 14, 2017 9:59 PM
To: [email protected]
Subject: RE: Recreating SystemVM's
Is there anyone out there reading these messages?
Am I just not seeing responses?
Jeremy
-----Original Message-----
From: Jeremy Peterson [mailto:[email protected]]
Sent: Wednesday, June 14, 2017 8:12 AM
To: [email protected]
Subject: RE: Recreating SystemVM's
I opened an issue since this is still an issue. CLOUDSTACK-9960
Jeremy
-----Original Message-----
From: Jeremy Peterson [mailto:[email protected]]
Sent: Sunday, June 11, 2017 9:10 AM
To: [email protected]
Subject: Re: Recreating SystemVM's
Any other suggestions?
I am going to be scheduling to run XenServer updates. But this all
points back to CANNOT_ATTACH_NETWORk.
I've verified nothing is active on the Public IP space that those two
VM's were living on.
Jeremy
________________________________________
From: Jeremy Peterson <[email protected]>
Sent: Friday, June 9, 2017 9:58 AM
To: [email protected]
Subject: RE: Recreating SystemVM's
I see the vm's try to create on a host that I just removed from
maintenance mode to install updates and here are the logs
I don't see anything that sticks out to me as a failure message.
Jun 9 09:53:54 Xen3 SM: [13068] ['ip', 'route', 'del',
'169.254.0.0/16']
Jun 9 09:53:54 Xen3 SM: [13068] pread SUCCESS
Jun 9 09:53:54 Xen3 SM: [13068] ['ifconfig', 'xapi12', '169.254.0.1',
'netmask', '255.255.0.0']
Jun 9 09:53:54 Xen3 SM: [13068] pread SUCCESS
Jun 9 09:53:54 Xen3 SM: [13068] ['ip', 'route', 'add',
'169.254.0.0/16', 'dev', 'xapi12', 'src', '169.254.0.1']
Jun 9 09:53:54 Xen3 SM: [13068] pread SUCCESS
Jun 9 09:53:54 Xen3 SM: [13071] ['ip', 'route', 'del',
'169.254.0.0/16']
Jun 9 09:53:54 Xen3 SM: [13071] pread SUCCESS
Jun 9 09:53:54 Xen3 SM: [13071] ['ifconfig', 'xapi12', '169.254.0.1',
'netmask', '255.255.0.0']
Jun 9 09:53:54 Xen3 SM: [13071] pread SUCCESS
Jun 9 09:53:54 Xen3 SM: [13071] ['ip', 'route', 'add',
'169.254.0.0/16', 'dev', 'xapi12', 'src', '169.254.0.1']
Jun 9 09:53:54 Xen3 SM: [13071] pread SUCCESS
Jun 9 09:54:00 Xen3 SM: [13115] on-slave.multi: {'vgName':
'VG_XenStorage-469b6dcd-8466-3d03-de0e-cc3983e1b6e2', 'lvName1':
'VHD-633338a7-6c40-4aa6-b88e-c798b6fdc04d', 'action1':
'deactivateNoRefcount', 'action2': 'cleanupLock', 'uuid2':
'633338a7-6c40-4aa6-b88e-c798b6fdc04d', 'ns2':
'lvm-469b6dcd-8466-3d03-de0e-cc3983e1b6e2'}
Jun 9 09:54:00 Xen3 SM: [13115] LVMCache created for
VG_XenStorage-469b6dcd-8466-3d03-de0e-cc3983e1b6e2
Jun 9 09:54:00 Xen3 SM: [13115] on-slave.action 1:
deactivateNoRefcount Jun
9 09:54:00 Xen3 SM: [13115] LVMCache: will initialize now Jun 9
09:54:00
Xen3 SM: [13115] LVMCache: refreshing Jun 9 09:54:00 Xen3 SM: [13115]
['/usr/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags',
'/dev/VG_XenStorage-469b6dcd-8466-3d03-de0e-cc3983e1b6e2']
Jun 9 09:54:00 Xen3 SM: [13115] pread SUCCESS
Jun 9 09:54:00 Xen3 SM: [13115] ['/usr/sbin/lvchange', '-an',
'/dev/VG_XenStorage-469b6dcd-8466-3d03-de0e-cc3983e1b6e2/VHD-633338a7-6c40-4
aa6-b88e-c798b6fdc04d']
Jun 9 09:54:00 Xen3 SM: [13115] pread SUCCESS
Jun 9 09:54:00 Xen3 SM: [13115] ['/sbin/dmsetup', 'status',
'VG_XenStorage--469b6dcd--8466--3d03--de0e--cc3983e1b6e2-VHD--633338a7--6c40
--4aa6--b88e--c798b6fdc04d']
Jun 9 09:54:00 Xen3 SM: [13115] pread SUCCESS
Jun 9 09:54:00 Xen3 SM: [13115] on-slave.action 2: cleanupLock
Jun 9 09:54:16 Xen3 SM: [13230] ['ip', 'route', 'del',
'169.254.0.0/16']
Jun 9 09:54:16 Xen3 SM: [13230] pread SUCCESS
Jun 9 09:54:16 Xen3 SM: [13230] ['ifconfig', 'xapi12', '169.254.0.1',
'netmask', '255.255.0.0']
Jun 9 09:54:16 Xen3 SM: [13230] pread SUCCESS
Jun 9 09:54:16 Xen3 SM: [13230] ['ip', 'route', 'add',
'169.254.0.0/16', 'dev', 'xapi12', 'src', '169.254.0.1']
Jun 9 09:54:16 Xen3 SM: [13230] pread SUCCESS
Jun 9 09:54:19 Xen3 updatempppathd: [15446] The garbage collection
routine
returned: 0 Jun 9 09:54:23 Xen3 SM: [13277] ['ip', 'route', 'del',
'169.254.0.0/16']
Jun 9 09:54:23 Xen3 SM: [13277] pread SUCCESS
Jun 9 09:54:23 Xen3 SM: [13277] ['ifconfig', 'xapi12', '169.254.0.1',
'netmask', '255.255.0.0']
Jun 9 09:54:23 Xen3 SM: [13277] pread SUCCESS
Jun 9 09:54:23 Xen3 SM: [13277] ['ip', 'route', 'add',
'169.254.0.0/16', 'dev', 'xapi12', 'src', '169.254.0.1']
Jun 9 09:54:23 Xen3 SM: [13277] pread SUCCESS
Jeremy
-----Original Message-----
From: Jeremy Peterson [mailto:[email protected]]
Sent: Friday, June 9, 2017 9:53 AM
To: [email protected]
Subject: RE: Recreating SystemVM's
I am checking SMlog now on all hosts.
Jeremy
-----Original Message-----
From: Rajani Karuturi [mailto:[email protected]]
Sent: Friday, June 9, 2017 9:00 AM
To: Users <[email protected]>
Subject: Re: Recreating SystemVM's
on xenserver log, did you check what is causing "
HOST_CANNOT_ATTACH_NETWORK"?
~Rajani
http://cloudplatform.accelerite.com/
On Fri, Jun 9, 2017 at 7:00 PM, Jeremy Peterson <[email protected]>
wrote:
> 08:28:43 select * from vm_instance where name like 's-%' limit
> 10000 7481 row(s) returned 0.000 sec / 0.032 sec
>
> All vm's 'state' returned Destoryed outside of the current vm 7873
> which is in a Stopped state but that goes Destroyed and a new get
created.
>
> Any other suggestions?
>
> Jeremy
>
>
> -----Original Message-----
> From: Jeremy Peterson [mailto:[email protected]]
> Sent: Thursday, June 8, 2017 12:47 AM
> To: [email protected]
> Subject: Re: Recreating SystemVM's
>
> I'll make that change in the am.
>
> Today I put a host in maintence and rebooted because proxy and
> secstore vm were constantly being created on that host and still no
change.
>
> Let you know tomorrow.
>
> Jeremy
>
>
> Sent from my Verizon, Samsung Galaxy smartphone
>
>
> -------- Original message --------
> From: Rajani Karuturi <[email protected]>
> Date: 6/8/17 12:07 AM (GMT-06:00)
> To: Users <[email protected]>
> Subject: Re: Recreating SystemVM's
>
> Did you check SMLog on xenserver?
> unable to destroy task(com.xensource.xenapi.Task@256829a8) on
> host(b34f086e-fabf-471e-9feb-8f54362d7d0f) due to You gave an invalid
> object reference. The object may have recently been deleted. The
> class parameter gives the type of reference given, and the handle
> parameter echoes the bad value given.
>
> Looks like Destroy of SSVM failed. What state is SSVM in? mark it as
> Destroyed in cloud DB and wait for cloudstack to create a new SSVM.
>
> ~Rajani
> http://cloudplatform.accelerite.com/
>
> On Thu, Jun 8, 2017 at 1:11 AM, Jeremy Peterson
> <[email protected]>
> wrote:
>
> > Probably agreed.
> >
> > But I ran toolstack restart on all hypervisors and v-3193 just
tried
> > to create and fail along with s-5398.
> >
> > The PIF error went away. But VM's are still recreating
> >
> > https://pastebin.com/4n4xBgMT
> >
> > New log from this afternoon.
> >
> > My catalina.out is over 4GB
> >
> > Jeremy
> >
> >
> > -----Original Message-----
> > From: Makrand [mailto:[email protected]]
> > Sent: Wednesday, June 7, 2017 12:52 AM
> > To: [email protected]
> > Subject: Re: Recreating SystemVM's
> >
> > Hi there,
> >
> > Looks more like hypervisor issue.
> >
> > Just run *xe-toolstack-restart* on hosts where these VMs are trying
> > to start or if you don't have too many hosts, better run on all
> > members including master. most of i/o related issues squared off by
> > toolstack bounce.
> >
> > --
> > Makrand
> >
> >
> > On Wed, Jun 7, 2017 at 3:01 AM, Jeremy Peterson
> > <[email protected]>
> > wrote:
> >
> > > Ok so I pulled this from Sunday morning.
> > >
> > > https://pastebin.com/nCETw1sC
> > >
> > >
> > > errorInfo: [HOST_CANNOT_ATTACH_NETWORK,
> > > OpaqueRef:65d0c844-bd70-81e9-4518-8809e1dc0ee7,
> > > OpaqueRef:0093ac3f-9f3a-37e1-9cdb-581398d27ba2]
> > >
> > > XenServer error.
> > >
> > > Now this still gets me because all of the other VM's launched just
> fine.
> > >
> > > Going into XenCenter I see an error at the bottom This PIF is a
> > > bond slave and cannot be plugged.
> > >
> > > ???
> > >
> > > If I go to networking on the hosts I see the storage vlans and
> > > bonds are all there.
> > >
> > > I see my GUEST-PUB bond is there and LACP is setup correct.
> > >
> > > Any suggestions ?
> > >
> > >
> > > Jeremy
> > >
> > >
> > > -----Original Message-----
> > > From: Jeremy Peterson [mailto:[email protected]]
> > > Sent: Tuesday, June 6, 2017 9:23 AM
> > > To: [email protected]
> > > Subject: RE: Recreating SystemVM's
> > >
> > > Thank you all for those responses.
> > >
> > > I'll comb through my management-server.log and post a pastebin if
> > > I'm scratching my head.
> > >
> > > Jeremy
> > >
> > > -----Original Message-----
> > > From: Rajani Karuturi [mailto:[email protected]]
> > > Sent: Tuesday, June 6, 2017 6:53 AM
> > > To: [email protected]
> > > Subject: Re: Recreating SystemVM's
> > >
> > > If the zone is enabled, cloudstack should recreate them
automatically.
> > >
> > > ~ Rajani
> > >
> > > http://cloudplatform.accelerite.com/
> > >
> > > On June 6, 2017 at 11:37 AM, Erik Weber ([email protected])
> > > wrote:
> > >
> > > CloudStack should recreate automatically, check the mgmt server
> > > logs for hints of why it doesn't happen.
> > >
> > > --
> > > Erik
> > >
> > > tir. 6. jun. 2017 kl. 04.29 skrev Jeremy Peterson
> > > <[email protected]>:
> > >
> > > I had an issue Sunday morning with cloudstack 4.9.0 and xenserver
> 6.5.0.
> > > My hosts stop sending LACP PDU's and caused a network drop to
> > > iSCSI primary storage.
> > >
> > > So all my instances recovered via HA enabled.
> > >
> > > But my console proxy and secondary storage system VM's got stuck
> > > in a boot state that would not power on.
> > >
> > > At this time they are expunged and gone.
> > >
> > > How do I tell cloudstack-management to recreate system VM's?
> > >
> > > I'm drawing a blank since deploying CS two years ago and just
> > > keeping things running and adding hosts and more storage
> > > everything has been so stable.
> > >
> > > Jeremy
> > >
> >
>
- proIO GmbH -
Geschäftsführer: Swen Brüseke
Sitz der Gesellschaft: Frankfurt am Main
USt-IdNr. DE 267 075 918
Registergericht: Frankfurt am Main - HRB 86239
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte
Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich
erhalten haben,
informieren Sie bitte sofort den Absender und vernichten Sie diese
Mail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail sind
nicht gestattet.
This e-mail may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this e-mail in
error) please notify
the sender immediately and destroy this e-mail.
Any unauthorized copying, disclosure or distribution of the material in
this e-mail is strictly forbidden.
[email protected]
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
[email protected]
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
