Yes, I restored the DB from the backup. When I try to start the router it says:
Resource [Host:1] is unreachable: Host 1: Unable to start instance due to Unable to start VM[DomainRouter|r-63-VM] due to error in finalizeStart, not retrying The management server log says: 2014-04-30 12:20:52,485 ERROR [cloud.async.AsyncJobManagerImpl] (Job-Executor-7:job-520 = [ 9d7c898c-b5d0-4bd0-a711-563a91d7acc9 ]) Unexpected exception while executing org.apache.cloudstack.api.command.admin.router.StartRouterCmd com.cloud.exception.AgentUnavailableException: Resource [Host:1] is unreachable: Host 1: Unable to start instance due to Unable to start VM[DomainRouter|r-63-VM] due to error in finalizeStart, not retrying On Wed, Apr 30, 2014 at 12:02 PM, stevenliang <[email protected]> wrote: > I think you had backed up database, when you upgraded. > When you downgraded CS, you also need to restore DB. > > > On 30/04/14 02:58 PM, Ian Young wrote: > >> I think my problem stems from a partially downloaded system VM template. >> I >> just noticed systemvm-kvm-4.3 is stuck at 27% downloaded. It must have >> been interrupted during the upgrade to 4.3. At the moment I've rolled >> back >> to 4.2.1 with a somewhat usable management interface, although the system >> VMs won't start. I suspect there is something in the database that is >> causing it to try to use the 4.3 template. How can I delete the template >> and make sure the management server is using the older one? >> >> >> On Tue, Apr 29, 2014 at 8:23 PM, Ian Young <[email protected]> >> wrote: >> >> Ok, so I've figured out a way to identify volumes in the filesystem. For >>> instance, /var/storage/primary/4d324e1a-e3a6-4da8-9c4d-44ad723482ad is >>> the >>> root volume for an instance I want to back up. Is this in qcow2 format >>> or >>> something else? I'm using KVM. >>> >>> >>> On Tue, Apr 29, 2014 at 7:38 PM, Ian Young <[email protected]> >>> wrote: >>> >>> Now I can't start cloudstack-agent. The agent.log says: >>>> >>>> Unable to start agent: Failed to get private nic name >>>> >>>> I know this is because the network bridge is no longer set up correctly. >>>> I used to have a cloud0 and a cloudbr0 interface. Now I only have >>>> cloudbr0. I haven't changed my network configuration. Somehow it's >>>> been >>>> changed by CloudStack during the upgrade/downgrade. This is getting >>>> worse >>>> and worse the more I try to recover my data. Is there any way to back >>>> up >>>> the instances' volumes via the command line? I can't tell which is >>>> which >>>> because the filenames are all hashes. I really need to get these >>>> instances >>>> up and running--there are several months worth of work at stake here. >>>> >>>> >>>> On Tue, Apr 29, 2014 at 6:13 PM, ma y <[email protected]> wrote: >>>> >>>> I got the same problem, and how to downgrade CS 4.3.0 to 4.2.1 safely? >>>>> >>>>> >>>>> 2014-04-30 8:45 GMT+08:00 Ian Young <[email protected]>: >>>>> >>>>> Ok, my Cloudstack installation is now so broken that I think it's >>>>>> >>>>> probably >>>>> >>>>>> best to backup all my instances and templates, wipe the databases, and >>>>>> start from scratch. However, I can't take snapshots or download >>>>>> >>>>> volumes >>>>> >>>>>> anymore. What's causing these errors? >>>>>> >>>>>> 2014-04-29 17:40:51,264 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] >>>>>> (Job-Executor-11:ctx-0a3ead79 ctx-315eda05) copy object failed: >>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to send >>>>>> >>>>> command, >>>>> >>>>>> due to Agent:1, com.cloud.exception.OperationTimedoutException: >>>>>> >>>>> Commands >>>>> >>>>>> 841744457 to Host 1 timed out after 21600 >>>>>> 2014-04-29 17:40:51,265 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] >>>>>> (Job-Executor-11:ctx-0a3ead79 ctx-315eda05) copy failed >>>>>> com.cloud.utils.exception.CloudRuntimeException: >>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to send >>>>>> >>>>> command, >>>>> >>>>>> due to Agent:1, com.cloud.exception.OperationTimedoutException: >>>>>> >>>>> Commands >>>>> >>>>>> 841744457 to Host 1 timed out after 21600 >>>>>> 2014-04-29 17:40:51,269 WARN [o.a.c.s.d. >>>>>> ObjectInDataStoreManagerImpl] >>>>>> (Job-Executor-11:ctx-0a3ead79 ctx-315eda05) Unsupported data object >>>>>> (VOLUME, >>>>>> org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@7bbbd901 >>>>>> ), >>>>>> >>>>> no >>>>> >>>>>> need to delete from object in store ref table >>>>>> 2014-04-29 17:40:51,280 ERROR [c.c.a.ApiAsyncJobDispatcher] >>>>>> (Job-Executor-11:ctx-0a3ead79) Unexpected exception while executing >>>>>> org.apache.cloudstack.api.command.user.volume.ExtractVolumeCmd >>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to copy the >>>>>> >>>>> volume >>>>> >>>>>> from the source primary storage pool to secondary storage. >>>>>> 2014-04-29 17:40:51,282 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>>>>> (Job-Executor-11:ctx-0a3ead79) Complete async job-501, jobStatus: >>>>>> >>>>> FAILED, >>>>> >>>>>> resultCode: 530, result: >>>>>> >>>>>> >>>>>> org.apache.cloudstack.api.response.ExceptionResponse/ >>>>> null/{"uuidList":[],"errorcode":530,"errortext":"Failed >>>>> >>>>>> to copy the volume from the source primary storage pool to secondary >>>>>> storage."} >>>>>> >>>>>> >>>>>> On Tue, Apr 29, 2014 at 4:15 PM, Ian Young <[email protected]> >>>>>> >>>>> wrote: >>>>> >>>>>> I downgraded to 4.2.1 again but cloudstack-management won't start >>>>>>> >>>>>> because >>>>> >>>>>> the database is version 4.3. Is it safe to restore the database >>>>>>> >>>>>> backup I >>>>> >>>>>> made prior to this whole process? In the meantime I have destroyed >>>>>>> >>>>>> and >>>>> >>>>>> created system VMs, so I'm not sure it's a good idea. >>>>>>> On Apr 29, 2014 3:09 PM, "Ian Young" <[email protected]> wrote: >>>>>>> >>>>>>> @stevenliang: I take it back--you can't set the VM size when you >>>>>>>> >>>>>>> register >>>>>> >>>>>>> the template. >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Apr 29, 2014 at 3:02 PM, motty cruz <[email protected]> >>>>>>>> >>>>>>> wrote: >>>>>> >>>>>>> yes, you would have to shutdown the router, then click on "Change >>>>>>>>> >>>>>>>> Service >>>>>> >>>>>>> Offering" >>>>>>>>> restart the VR. >>>>>>>>> >>>>>>>>> To Ian, >>>>>>>>> >>>>>>>>> I suspect you forgot the last step: " cloudstack-setup-management" >>>>>>>>> >>>>>>>>> that would fix your issue, I think, >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> --- >>>>>>>>> I downgraded to 4.2.1 and then upgraded to 4.3. Now the >>>>>>>>> cloudstack-management service can't start because it can't connect >>>>>>>>> >>>>>>>> to >>>>> >>>>>> the >>>>>> >>>>>>> database. >>>>>>>>> >>>>>>>>> 2014-04-29 14:51:36,424 ERROR [c.c.u.d.Merovingian2] (main:null) >>>>>>>>> >>>>>>>> Unable >>>>> >>>>>> to >>>>>>>>> get a new db connection >>>>>>>>> Caused by: java.sql.SQLException: Access denied for user 'cloud'@ >>>>>>>>> 'localhost' >>>>>>>>> (using password: YES) >>>>>>>>> >>>>>>>>> Where are the credentials stored? >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Apr 29, 2014 at 2:57 PM, stevenliang < >>>>>>>>> >>>>>>>> [email protected]> >>>>> >>>>>> wrote: >>>>>>>>> >>>>>>>>> oh, then change service offering for vr? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 29/04/14 05:53 PM, motty cruz wrote: >>>>>>>>>> >>>>>>>>>> for my VR, I created a new >>>>>>>>>>> >>>>>>>>>>> "System Offering For Software Router" >>>>>>>>>>> CPU in (MHz) 1.00GHz >>>>>>>>>>> Memory (in MB) 1.00GB >>>>>>>>>>> >>>>>>>>>>> this are my current offerings, I'm sure the more RAM and CPU >>>>>>>>>>> >>>>>>>>>> better >>>>> >>>>>> performance. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, Apr 29, 2014 at 2:44 PM, stevenliang < >>>>>>>>>>> >>>>>>>>>> [email protected] >>>>> >>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Thank you again, motty. >>>>>>>>>>> >>>>>>>>>>>> I didn't notice this earlier. >>>>>>>>>>>> BTW, how did you make your vr had 1GB CPU and 512MB RAM? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 29/04/14 05:33 PM, motty cruz wrote: >>>>>>>>>>>> >>>>>>>>>>>> Stevellang, >>>>>>>>>>>> >>>>>>>>>>>>> I not sure if you saw this in the forums earlier : >>>>>>>>>>>>> >>>>>>>>>>>>> http://mail-archives.apache.org/mod_mbox/cloudstack-users/ >>>>> 201404.mbox/% >>>>> >>>>>> 3CCALoOYy6A10bz1zOQQs1VyFb9epqLfhf7mu6hc= >>>>>>>>>>>>> >>>>>>>>>>>> [email protected]%3E >>>>>>>>> >>>>>>>>>> I don't know if the bug was fixed yet, >>>>>>>>>>>>> >>>>>>>>>>>>> I will try upgrade in the next couple of days on a testing >>>>>>>>>>>>> >>>>>>>>>>>> cluster, >>>>>> >>>>>>> will >>>>>>>>> >>>>>>>>>> report back if the bug was fixed. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Apr 29, 2014 at 2:25 PM, stevenliang < >>>>>>>>>>>>> >>>>>>>>>>>> [email protected] >>>>>> >>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Thank you, motty. >>>>>>>>>>>>> >>>>>>>>>>>>> I am also running kvm. Since that time I failed upgrade, I am >>>>>>>>>>>>>> >>>>>>>>>>>>> still >>>>>> >>>>>>> using >>>>>>>>>>>>>> 4.2.1. I'll try as your advice. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 29/04/14 05:19 PM, motty cruz wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Stevenllang, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I had the similar issue with VR, I notice it was because I >>>>>>>>>>>>>>> >>>>>>>>>>>>>> leave >>>>> >>>>>> the >>>>>>>>> >>>>>>>>>> default system specs on the VR, for instance by default >>>>>>>>>>>>>>> >>>>>>>>>>>>>> 500MHz >>>>> >>>>>> on >>>>>> >>>>>>> CPU >>>>>>>>> >>>>>>>>>> and >>>>>>>>>>>>>>> 128MB on RAM, if you upgrade to at least 1GB on CPU and >>>>>>>>>>>>>>> >>>>>>>>>>>>>> 512MB of >>>>> >>>>>> RAM >>>>>>>>> >>>>>>>>>> your >>>>>>>>>>>>>>> VR will survive the upgrade from 4.2.1 to 4.3.1. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I am running KVM, when I upgrade from 4.2.1 to 4.3 my VMs >>>>>>>>>>>>>>> >>>>>>>>>>>>>> were >>>>> >>>>>> not >>>>>> >>>>>>> able >>>>>>>>>>>>>>> to >>>>>>>>>>>>>>> access outside world, even if I created a new router. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> wish you the best, >>>>>>>>>>>>>>> -motty >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Tue, Apr 29, 2014 at 2:13 PM, stevenliang < >>>>>>>>>>>>>>> >>>>>>>>>>>>>> [email protected]> >>>>>>>>> >>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Yes, I had two zones(one is basic, another is advanced >>>>>>>>>>>>>>> >>>>>>>>>>>>>> mode). >>>>> >>>>>> After I upgraded from 4.2.1 to 4.3, the vrouter lost. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> So I rolled back to 4.2.1, the vrouter came back. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 29/04/14 04:54 PM, Ian Young wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Did rolling back to 4.2 fix the problem? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Tue, Apr 29, 2014 at 1:22 PM, stevenliang < >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [email protected] >>>>>>>>> >>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I met your situation before. Finally I rolled back to >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 4.2 >>>>> >>>>>> On 29/04/14 04:18 PM, Ian Young wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I destroyed the old virtual router and was able to >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> create >>>>> >>>>>> a new >>>>>>>>> >>>>>>>>>> one >>>>>>>>>>>>>>>>>> by >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> adding a new instance. However, this new router also >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> failed >>>>>> >>>>>>> to >>>>>>>>> >>>>>>>>>> start, >>>>>>>>>>>>>>>>>>> citing the same error. After that, the expungement >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> delay >>>>> >>>>>> elapsed >>>>>>>>> >>>>>>>>>> and >>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>> virtual router was expunged, so now I have none. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Mon, Apr 28, 2014 at 8:52 PM, Ian Young < >>>>>>>>>>>>>>>>>>> [email protected]> >>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I upgraded from 4.2.1 to 4.3.0 tonight, following >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> the >>>>> >>>>>> instructions >>>>>>>>>>>>>>>>>>> here: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> http://docs.cloudstack.apache.org/projects/cloudstack- >>>>> >>>>>> >>>>>>>>>>>>>>>>>>> release-notes/en/latest/rnotes.html#upgrade-from-4-2- >>>>>> x-to-4-3 >>>>>> >>>>>>> At the last step, I tried to restart the system VMs. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> The >>>>> >>>>>> virtual >>>>>>>>> >>>>>>>>>> router >>>>>>>>>>>>>>>>>>>> failed to start. Here is the message that was >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> displayed in >>>>> >>>>>> the >>>>>>>>> >>>>>>>>>> web >>>>>>>>>>>>>>>>>>>> UI: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Resource [Host:1] is unreachable: Host 1: Unable to >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> start >>>>> >>>>>> instance >>>>>>>>>>>>>>>>>>>> due >>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>> Unable to start VM[DomainRouter|r-4-VM] due to error in >>>>>>>>>>>>>>>>>>>> finalizeStart, >>>>>>>>>>>>>>>>>>>> not >>>>>>>>>>>>>>>>>>>> retrying >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I tried running the script to restart the VMs but this >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> time >>>>> >>>>>> it >>>>>>>>> >>>>>>>>>> failed >>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>> start the console proxy: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> [root@virthost1 ~]$ cloudstack-sysvmadm -d >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> 192.168.100.6 >>>>> >>>>>> -u >>>>>> >>>>>>> cloud >>>>>>>>>>>>>>>>>>>> -p >>>>>>>>>>>>>>>>>>>> -a >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Stopping and starting 1 secondary storage vm(s)... >>>>>>>>>>>>>>>>>>>> Done stopping and starting secondary storage vm(s) >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Stopping and starting 1 console proxy vm(s)... >>>>>>>>>>>>>>>>>>>> ERROR: Failed to start console proxy vm with id 2 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Done stopping and starting console proxy vm(s) . >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Stopping and starting 0 running routing vm(s)... >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Is there a way to wipe the system VMs out and start >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> over? >>>>> >>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>> >>>> >
