I downgraded to 4.2.1, restored the database backup I made during the initial upgrade to 4.3, copied the rpmsave files over /etc/cloudstack/agent/agent.properties and /etc/cloudstack/managment/db.properties, and started the agent and management services. The service_ip field in the mshost table changed to 127.0.0.1 again, so I changed it to the actual IP address of the server. Then I ran
cloud-install-sys-tmplt -m /var/storage/secondary -u http://download.cloud.com/templates/4.2/systemvmtemplate-2013-06-12-master-kvm.qcow2.bz2-h kvm -F which said it successfully installed the system VM template to /var/storage/secondary/template/tmpl/1/3/. Next, I restarted the management service. When I logged in, the management console said there was an SSVM, console proxy, and router present but they were all stopped. I tried starting the SSVM but it failed to start. Here is an excerpt from management.log: http://pastebin.com/uW4NPZbC Is there some sort of general troubleshooting script that could identify major configuration problems? I feel like this system is mostly functional but one or two misconfigured things are causing it to break. I'd hate to have to wipe it clean and start over, losing all my VMs in the process. On Wed, Apr 30, 2014 at 6:00 PM, Ian Young <[email protected]> wrote: > I read this article about upgrading the system VMs: > > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+4.2+(KVM)+System+Vm+Upgrade > > However, there's just an empty set in the template_host_ref table. Is > this no longer used in 4.3? > > > On Wed, Apr 30, 2014 at 5:41 PM, Ian Young <[email protected]> wrote: > >> I've tried upgrading to 4.3 again. After poking around some more in the >> database, I've discovered that the KVM system VM template was only 27% >> downloaded. I think this is why the virtual router was unable to >> start--the template was incomplete. Is there a way to force it to resume >> downloading? >> >> >> On Wed, Apr 30, 2014 at 3:03 PM, Ian Young <[email protected]>wrote: >> >>> The address in Infrastructure > Hosts > (management server) is set to >>> the correct IP address, not 127.0.0.1. Why are the logs referring to >>> 127.0.0.1? >>> >>> >>> On Wed, Apr 30, 2014 at 3:00 PM, Ian Young <[email protected]>wrote: >>> >>>> I notice my dashboard says "Management server node 127.0.0.1 is up." >>>> It used to have an actual address, not localhost. Could this be causing >>>> problems and if so, how can I set it back? >>>> >>>> >>>> On Wed, Apr 30, 2014 at 12:40 PM, Ian Young <[email protected]>wrote: >>>> >>>>> Yes, I replaced the new files with the rpmsave ones, which allowed the >>>>> agent to start. However, most of the functions in the management console >>>>> fail. >>>>> >>>>> >>>>> On Wed, Apr 30, 2014 at 12:34 PM, stevenliang >>>>> <[email protected]>wrote: >>>>> >>>>>> Do you have the file db.properties.rpmsave on management server and >>>>>> agent.properties.rpmsave on agents? If so, and the date is correct, you >>>>>> can >>>>>> use it rather than db.properties and agent.properties. >>>>>> And then restart management and agent services. >>>>>> >>>>>> >>>>>> On 30/04/14 03:26 PM, Ian Young wrote: >>>>>> >>>>>>> Yes, I restored the DB from the backup. When I try to start the >>>>>>> router it >>>>>>> says: >>>>>>> >>>>>>> Resource [Host:1] is unreachable: Host 1: Unable to start instance >>>>>>> due to >>>>>>> Unable to start VM[DomainRouter|r-63-VM] due to error in >>>>>>> finalizeStart, not >>>>>>> retrying >>>>>>> >>>>>>> The management server log says: >>>>>>> >>>>>>> 2014-04-30 12:20:52,485 ERROR [cloud.async.AsyncJobManagerImpl] >>>>>>> (Job-Executor-7:job-520 = [ 9d7c898c-b5d0-4bd0-a711-563a91d7acc9 ]) >>>>>>> Unexpected exception while executing >>>>>>> org.apache.cloudstack.api.command.admin.router.StartRouterCmd >>>>>>> com.cloud.exception.AgentUnavailableException: Resource [Host:1] is >>>>>>> unreachable: Host 1: Unable to start instance due to Unable to start >>>>>>> VM[DomainRouter|r-63-VM] due to error in finalizeStart, not retrying >>>>>>> >>>>>>> >>>>>>> On Wed, Apr 30, 2014 at 12:02 PM, stevenliang <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>> I think you had backed up database, when you upgraded. >>>>>>>> When you downgraded CS, you also need to restore DB. >>>>>>>> >>>>>>>> >>>>>>>> On 30/04/14 02:58 PM, Ian Young wrote: >>>>>>>> >>>>>>>> I think my problem stems from a partially downloaded system VM >>>>>>>>> template. >>>>>>>>> I >>>>>>>>> just noticed systemvm-kvm-4.3 is stuck at 27% downloaded. It must >>>>>>>>> have >>>>>>>>> been interrupted during the upgrade to 4.3. At the moment I've >>>>>>>>> rolled >>>>>>>>> back >>>>>>>>> to 4.2.1 with a somewhat usable management interface, although the >>>>>>>>> system >>>>>>>>> VMs won't start. I suspect there is something in the database >>>>>>>>> that is >>>>>>>>> causing it to try to use the 4.3 template. How can I delete the >>>>>>>>> template >>>>>>>>> and make sure the management server is using the older one? >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Apr 29, 2014 at 8:23 PM, Ian Young <[email protected] >>>>>>>>> > >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Ok, so I've figured out a way to identify volumes in the >>>>>>>>> filesystem. For >>>>>>>>> >>>>>>>>>> instance, /var/storage/primary/4d324e1a-e3a6-4da8-9c4d-44ad723482ad >>>>>>>>>> is >>>>>>>>>> the >>>>>>>>>> root volume for an instance I want to back up. Is this in qcow2 >>>>>>>>>> format >>>>>>>>>> or >>>>>>>>>> something else? I'm using KVM. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Apr 29, 2014 at 7:38 PM, Ian Young < >>>>>>>>>> [email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Now I can't start cloudstack-agent. The agent.log says: >>>>>>>>>> >>>>>>>>>>> Unable to start agent: Failed to get private nic name >>>>>>>>>>> >>>>>>>>>>> I know this is because the network bridge is no longer set up >>>>>>>>>>> correctly. >>>>>>>>>>> I used to have a cloud0 and a cloudbr0 interface. Now I only >>>>>>>>>>> have >>>>>>>>>>> cloudbr0. I haven't changed my network configuration. Somehow >>>>>>>>>>> it's >>>>>>>>>>> been >>>>>>>>>>> changed by CloudStack during the upgrade/downgrade. This is >>>>>>>>>>> getting >>>>>>>>>>> worse >>>>>>>>>>> and worse the more I try to recover my data. Is there any way >>>>>>>>>>> to back >>>>>>>>>>> up >>>>>>>>>>> the instances' volumes via the command line? I can't tell which >>>>>>>>>>> is >>>>>>>>>>> which >>>>>>>>>>> because the filenames are all hashes. I really need to get these >>>>>>>>>>> instances >>>>>>>>>>> up and running--there are several months worth of work at stake >>>>>>>>>>> here. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, Apr 29, 2014 at 6:13 PM, ma y <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> I got the same problem, and how to downgrade CS 4.3.0 to 4.2.1 >>>>>>>>>>> safely? >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 2014-04-30 8:45 GMT+08:00 Ian Young <[email protected]>: >>>>>>>>>>>> >>>>>>>>>>>> Ok, my Cloudstack installation is now so broken that I think >>>>>>>>>>>> it's >>>>>>>>>>>> probably >>>>>>>>>>>> >>>>>>>>>>>> best to backup all my instances and templates, wipe the >>>>>>>>>>>>> databases, and >>>>>>>>>>>>> start from scratch. However, I can't take snapshots or >>>>>>>>>>>>> download >>>>>>>>>>>>> >>>>>>>>>>>>> volumes >>>>>>>>>>>> >>>>>>>>>>>> anymore. What's causing these errors? >>>>>>>>>>>>> >>>>>>>>>>>>> 2014-04-29 17:40:51,264 DEBUG [o.a.c.s.m. >>>>>>>>>>>>> AncientDataMotionStrategy] >>>>>>>>>>>>> (Job-Executor-11:ctx-0a3ead79 ctx-315eda05) copy object failed: >>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to >>>>>>>>>>>>> send >>>>>>>>>>>>> >>>>>>>>>>>>> command, >>>>>>>>>>>> >>>>>>>>>>>> due to Agent:1, com.cloud.exception. >>>>>>>>>>>>> OperationTimedoutException: >>>>>>>>>>>>> >>>>>>>>>>>>> Commands >>>>>>>>>>>> >>>>>>>>>>>> 841744457 to Host 1 timed out after 21600 >>>>>>>>>>>>> 2014-04-29 17:40:51,265 DEBUG [o.a.c.s.m. >>>>>>>>>>>>> AncientDataMotionStrategy] >>>>>>>>>>>>> (Job-Executor-11:ctx-0a3ead79 ctx-315eda05) copy failed >>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: >>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to >>>>>>>>>>>>> send >>>>>>>>>>>>> >>>>>>>>>>>>> command, >>>>>>>>>>>> >>>>>>>>>>>> due to Agent:1, com.cloud.exception. >>>>>>>>>>>>> OperationTimedoutException: >>>>>>>>>>>>> >>>>>>>>>>>>> Commands >>>>>>>>>>>> >>>>>>>>>>>> 841744457 to Host 1 timed out after 21600 >>>>>>>>>>>>> 2014-04-29 17:40:51,269 WARN [o.a.c.s.d. >>>>>>>>>>>>> ObjectInDataStoreManagerImpl] >>>>>>>>>>>>> (Job-Executor-11:ctx-0a3ead79 ctx-315eda05) Unsupported data >>>>>>>>>>>>> object >>>>>>>>>>>>> (VOLUME, >>>>>>>>>>>>> org.apache.cloudstack.storage.datastore. >>>>>>>>>>>>> PrimaryDataStoreImpl@7bbbd901 >>>>>>>>>>>>> ), >>>>>>>>>>>>> >>>>>>>>>>>>> no >>>>>>>>>>>> >>>>>>>>>>>> need to delete from object in store ref table >>>>>>>>>>>>> 2014-04-29 17:40:51,280 ERROR [c.c.a.ApiAsyncJobDispatcher] >>>>>>>>>>>>> (Job-Executor-11:ctx-0a3ead79) Unexpected exception while >>>>>>>>>>>>> executing >>>>>>>>>>>>> org.apache.cloudstack.api.command.user.volume.ExtractVolumeCmd >>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to >>>>>>>>>>>>> copy the >>>>>>>>>>>>> >>>>>>>>>>>>> volume >>>>>>>>>>>> >>>>>>>>>>>> from the source primary storage pool to secondary storage. >>>>>>>>>>>>> 2014-04-29 17:40:51,282 DEBUG [o.a.c.f.j.i. >>>>>>>>>>>>> AsyncJobManagerImpl] >>>>>>>>>>>>> (Job-Executor-11:ctx-0a3ead79) Complete async job-501, >>>>>>>>>>>>> jobStatus: >>>>>>>>>>>>> >>>>>>>>>>>>> FAILED, >>>>>>>>>>>> >>>>>>>>>>>> resultCode: 530, result: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/ >>>>>>>>>>>>> >>>>>>>>>>>> null/{"uuidList":[],"errorcode":530,"errortext":"Failed >>>>>>>>>>>> >>>>>>>>>>>> to copy the volume from the source primary storage pool to >>>>>>>>>>>>> secondary >>>>>>>>>>>>> storage."} >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Apr 29, 2014 at 4:15 PM, Ian Young < >>>>>>>>>>>>> [email protected]> >>>>>>>>>>>>> >>>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> I downgraded to 4.2.1 again but cloudstack-management won't >>>>>>>>>>>>> start >>>>>>>>>>>>> because >>>>>>>>>>>>> the database is version 4.3. Is it safe to restore the database >>>>>>>>>>>>> backup I >>>>>>>>>>>>> made prior to this whole process? In the meantime I have >>>>>>>>>>>>> destroyed >>>>>>>>>>>>> and >>>>>>>>>>>>> created system VMs, so I'm not sure it's a good idea. >>>>>>>>>>>>> >>>>>>>>>>>>>> On Apr 29, 2014 3:09 PM, "Ian Young" <[email protected]> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> @stevenliang: I take it back--you can't set the VM size >>>>>>>>>>>>>> when you >>>>>>>>>>>>>> register >>>>>>>>>>>>>> the template. >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Tue, Apr 29, 2014 at 3:02 PM, motty cruz < >>>>>>>>>>>>>>> [email protected]> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> yes, you would have to shutdown the router, then click on >>>>>>>>>>>>>> "Change >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Service >>>>>>>>>>>>>>> >>>>>>>>>>>>>> Offering" >>>>>>>>>>>>>> >>>>>>>>>>>>>>> restart the VR. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> To Ian, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I suspect you forgot the last step: " >>>>>>>>>>>>>>>> cloudstack-setup-management" >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> that would fix your issue, I think, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> --- >>>>>>>>>>>>>>>> I downgraded to 4.2.1 and then upgraded to 4.3. Now the >>>>>>>>>>>>>>>> cloudstack-management service can't start because it can't >>>>>>>>>>>>>>>> connect >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>> >>>>>>>>>>>>>> the >>>>>>>>>>>>> >>>>>>>>>>>>> database. >>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2014-04-29 14:51:36,424 ERROR [c.c.u.d.Merovingian2] >>>>>>>>>>>>>>>> (main:null) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Unable >>>>>>>>>>>>>>> >>>>>>>>>>>>>> to >>>>>>>>>>>>> >>>>>>>>>>>>>> get a new db connection >>>>>>>>>>>>>>>> Caused by: java.sql.SQLException: Access denied for user >>>>>>>>>>>>>>>> 'cloud'@ >>>>>>>>>>>>>>>> 'localhost' >>>>>>>>>>>>>>>> (using password: YES) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Where are the credentials stored? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Tue, Apr 29, 2014 at 2:57 PM, stevenliang < >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [email protected]> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> oh, then change service offering for vr? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 29/04/14 05:53 PM, motty cruz wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> for my VR, I created a new >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "System Offering For Software Router" >>>>>>>>>>>>>>>>>> CPU in (MHz) 1.00GHz >>>>>>>>>>>>>>>>>> Memory (in MB) 1.00GB >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> this are my current offerings, I'm sure the more RAM and >>>>>>>>>>>>>>>>>> CPU >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> better >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> performance. >>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Tue, Apr 29, 2014 at 2:44 PM, stevenliang < >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Thank you again, motty. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I didn't notice this earlier. >>>>>>>>>>>>>>>>>>> BTW, how did you make your vr had 1GB CPU and 512MB RAM? >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On 29/04/14 05:33 PM, motty cruz wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Stevellang, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I not sure if you saw this in the forums earlier : >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> http://mail-archives.apache. >>>>>>>>>>>>>>>>>>>> org/mod_mbox/cloudstack-users/ >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> 201404.mbox/% >>>>>>>>>>>> >>>>>>>>>>>> 3CCALoOYy6A10bz1zOQQs1VyFb9epqLfhf7mu6hc= >>>>>>>>>>>>> >>>>>>>>>>>>>> [email protected]%3E >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I don't know if the bug was fixed yet, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I will try upgrade in the next couple of days on a testing >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> cluster, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> will >>>>>>>>>>>>>> >>>>>>>>>>>>>>> report back if the bug was fixed. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Tue, Apr 29, 2014 at 2:25 PM, stevenliang < >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thank you, motty. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I am also running kvm. Since that time I failed >>>>>>>>>>>>>>>>>>>> upgrade, I am >>>>>>>>>>>>>>>>>>>> still >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> using >>>>>>>>>>>>>> >>>>>>>>>>>>>>> 4.2.1. I'll try as your advice. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On 29/04/14 05:19 PM, motty cruz wrote: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Stevenllang, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> I had the similar issue with VR, I notice it was >>>>>>>>>>>>>>>>>>>>> because I >>>>>>>>>>>>>>>>>>>>> leave >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>> >>>>>>>>>>>>>> default system specs on the VR, for instance by default >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 500MHz >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> on >>>>>>>>>>>>> >>>>>>>>>>>>> CPU >>>>>>>>>>>>>> >>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 128MB on RAM, if you upgrade to at least 1GB on CPU and >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> 512MB of >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> RAM >>>>>>>>>>>>> >>>>>>>>>>>>>> your >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> VR will survive the upgrade from 4.2.1 to 4.3.1. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> I am running KVM, when I upgrade from 4.2.1 to 4.3 my >>>>>>>>>>>>>>>>>>>>>> VMs >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> were >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> not >>>>>>>>>>>>> >>>>>>>>>>>>> able >>>>>>>>>>>>>> >>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>> access outside world, even if I created a new router. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> wish you the best, >>>>>>>>>>>>>>>>>>>>>> -motty >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 29, 2014 at 2:13 PM, stevenliang < >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> [email protected]> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Yes, I had two zones(one is basic, another is >>>>>>>>>>>>>>>>>>>>>> advanced >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> mode). >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> After I upgraded from 4.2.1 to 4.3, the vrouter lost. >>>>>>>>>>>>> >>>>>>>>>>>>>> So I rolled back to 4.2.1, the vrouter came back. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> On 29/04/14 04:54 PM, Ian Young wrote: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Did rolling back to 4.2 fix the problem? >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 29, 2014 at 1:22 PM, stevenliang < >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I met your situation before. Finally I rolled >>>>>>>>>>>>>>>>>>>>>>>> back to >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> 4.2 >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On 29/04/14 04:18 PM, Ian Young wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> I destroyed the old virtual router and was able to >>>>>>>>>>>>>>>>>>>>>>>> create >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> a new >>>>>>>>>>>>> >>>>>>>>>>>>>> one >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> by >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> adding a new instance. However, this new >>>>>>>>>>>>>>>>>>>>>>>>> router also >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> failed >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>> >>>>>>>>>>>>>>> start, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> citing the same error. After that, the expungement >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> delay >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> elapsed >>>>>>>>>>>>> >>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>> virtual router was expunged, so now I have none. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Apr 28, 2014 at 8:52 PM, Ian Young < >>>>>>>>>>>>>>>>>>>>>>>>>> [email protected]> >>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> I upgraded from 4.2.1 to 4.3.0 tonight, >>>>>>>>>>>>>>>>>>>>>>>>>> following >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> instructions >>>>>>>>>>>>> >>>>>>>>>>>>>> here: >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> http://docs.cloudstack.apache. >>>>>>>>>>>>>>>>>>>>>>>>>> org/projects/cloudstack- >>>>>>>>>>>>>>>>>>>>>>>>>> release-notes/en/latest/ >>>>>>>>>>>>>>>>>>>>>>>>>> rnotes.html#upgrade-from-4-2- >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> x-to-4-3 >>>>>>>>>>>>> >>>>>>>>>>>>> At the last step, I tried to restart the system VMs. >>>>>>>>>>>>>> >>>>>>>>>>>>>>> The >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> virtual >>>>>>>>>>>>> >>>>>>>>>>>>>> router >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> failed to start. Here is the message that was >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> displayed in >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>> >>>>>>>>>>>>>> web >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> UI: >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Resource [Host:1] is unreachable: Host 1: Unable >>>>>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> start >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> instance >>>>>>>>>>>>> >>>>>>>>>>>>>> due >>>>>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>> Unable to start VM[DomainRouter|r-4-VM] due to >>>>>>>>>>>>>>>>>>>>>>>>>>> error in >>>>>>>>>>>>>>>>>>>>>>>>>>> finalizeStart, >>>>>>>>>>>>>>>>>>>>>>>>>>> not >>>>>>>>>>>>>>>>>>>>>>>>>>> retrying >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> I tried running the script to restart the VMs >>>>>>>>>>>>>>>>>>>>>>>>>>> but this >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> time >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> it >>>>>>>>>>>>> >>>>>>>>>>>>>> failed >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>> start the console proxy: >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> [root@virthost1 ~]$ cloudstack-sysvmadm -d >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> 192.168.100.6 >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> -u >>>>>>>>>>>>> >>>>>>>>>>>>> cloud >>>>>>>>>>>>>> >>>>>>>>>>>>>>> -p >>>>>>>>>>>>>>>>>>>>>>>>>>> -a >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Stopping and starting 1 secondary storage >>>>>>>>>>>>>>>>>>>>>>>>>>> vm(s)... >>>>>>>>>>>>>>>>>>>>>>>>>>> Done stopping and starting secondary storage >>>>>>>>>>>>>>>>>>>>>>>>>>> vm(s) >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Stopping and starting 1 console proxy vm(s)... >>>>>>>>>>>>>>>>>>>>>>>>>>> ERROR: Failed to start console proxy vm with id 2 >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Done stopping and starting console proxy vm(s) . >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Stopping and starting 0 running routing vm(s)... >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Is there a way to wipe the system VMs out and >>>>>>>>>>>>>>>>>>>>>>>>>>> start >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> over? >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>> >>>>> >>>> >>> >> >
