I think my problem stems from a partially downloaded system VM template. I just noticed systemvm-kvm-4.3 is stuck at 27% downloaded. It must have been interrupted during the upgrade to 4.3. At the moment I've rolled back to 4.2.1 with a somewhat usable management interface, although the system VMs won't start. I suspect there is something in the database that is causing it to try to use the 4.3 template. How can I delete the template and make sure the management server is using the older one?
On Tue, Apr 29, 2014 at 8:23 PM, Ian Young <[email protected]> wrote: > Ok, so I've figured out a way to identify volumes in the filesystem. For > instance, /var/storage/primary/4d324e1a-e3a6-4da8-9c4d-44ad723482ad is the > root volume for an instance I want to back up. Is this in qcow2 format or > something else? I'm using KVM. > > > On Tue, Apr 29, 2014 at 7:38 PM, Ian Young <[email protected]> wrote: > >> Now I can't start cloudstack-agent. The agent.log says: >> >> Unable to start agent: Failed to get private nic name >> >> I know this is because the network bridge is no longer set up correctly. >> I used to have a cloud0 and a cloudbr0 interface. Now I only have >> cloudbr0. I haven't changed my network configuration. Somehow it's been >> changed by CloudStack during the upgrade/downgrade. This is getting worse >> and worse the more I try to recover my data. Is there any way to back up >> the instances' volumes via the command line? I can't tell which is which >> because the filenames are all hashes. I really need to get these instances >> up and running--there are several months worth of work at stake here. >> >> >> On Tue, Apr 29, 2014 at 6:13 PM, ma y <[email protected]> wrote: >> >>> I got the same problem, and how to downgrade CS 4.3.0 to 4.2.1 safely? >>> >>> >>> 2014-04-30 8:45 GMT+08:00 Ian Young <[email protected]>: >>> >>> > Ok, my Cloudstack installation is now so broken that I think it's >>> probably >>> > best to backup all my instances and templates, wipe the databases, and >>> > start from scratch. However, I can't take snapshots or download >>> volumes >>> > anymore. What's causing these errors? >>> > >>> > 2014-04-29 17:40:51,264 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] >>> > (Job-Executor-11:ctx-0a3ead79 ctx-315eda05) copy object failed: >>> > com.cloud.utils.exception.CloudRuntimeException: Failed to send >>> command, >>> > due to Agent:1, com.cloud.exception.OperationTimedoutException: >>> Commands >>> > 841744457 to Host 1 timed out after 21600 >>> > 2014-04-29 17:40:51,265 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] >>> > (Job-Executor-11:ctx-0a3ead79 ctx-315eda05) copy failed >>> > com.cloud.utils.exception.CloudRuntimeException: >>> > com.cloud.utils.exception.CloudRuntimeException: Failed to send >>> command, >>> > due to Agent:1, com.cloud.exception.OperationTimedoutException: >>> Commands >>> > 841744457 to Host 1 timed out after 21600 >>> > 2014-04-29 17:40:51,269 WARN [o.a.c.s.d.ObjectInDataStoreManagerImpl] >>> > (Job-Executor-11:ctx-0a3ead79 ctx-315eda05) Unsupported data object >>> > (VOLUME, >>> > org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@7bbbd901), >>> no >>> > need to delete from object in store ref table >>> > 2014-04-29 17:40:51,280 ERROR [c.c.a.ApiAsyncJobDispatcher] >>> > (Job-Executor-11:ctx-0a3ead79) Unexpected exception while executing >>> > org.apache.cloudstack.api.command.user.volume.ExtractVolumeCmd >>> > com.cloud.utils.exception.CloudRuntimeException: Failed to copy the >>> volume >>> > from the source primary storage pool to secondary storage. >>> > 2014-04-29 17:40:51,282 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>> > (Job-Executor-11:ctx-0a3ead79) Complete async job-501, jobStatus: >>> FAILED, >>> > resultCode: 530, result: >>> > >>> > >>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed >>> > to copy the volume from the source primary storage pool to secondary >>> > storage."} >>> > >>> > >>> > On Tue, Apr 29, 2014 at 4:15 PM, Ian Young <[email protected]> >>> wrote: >>> > >>> > > I downgraded to 4.2.1 again but cloudstack-management won't start >>> because >>> > > the database is version 4.3. Is it safe to restore the database >>> backup I >>> > > made prior to this whole process? In the meantime I have destroyed >>> and >>> > > created system VMs, so I'm not sure it's a good idea. >>> > > On Apr 29, 2014 3:09 PM, "Ian Young" <[email protected]> wrote: >>> > > >>> > >> @stevenliang: I take it back--you can't set the VM size when you >>> > register >>> > >> the template. >>> > >> >>> > >> >>> > >> On Tue, Apr 29, 2014 at 3:02 PM, motty cruz <[email protected]> >>> > wrote: >>> > >> >>> > >>> yes, you would have to shutdown the router, then click on "Change >>> > Service >>> > >>> Offering" >>> > >>> restart the VR. >>> > >>> >>> > >>> To Ian, >>> > >>> >>> > >>> I suspect you forgot the last step: " cloudstack-setup-management" >>> > >>> >>> > >>> that would fix your issue, I think, >>> > >>> >>> > >>> Thanks, >>> > >>> --- >>> > >>> I downgraded to 4.2.1 and then upgraded to 4.3. Now the >>> > >>> cloudstack-management service can't start because it can't connect >>> to >>> > the >>> > >>> database. >>> > >>> >>> > >>> 2014-04-29 14:51:36,424 ERROR [c.c.u.d.Merovingian2] (main:null) >>> Unable >>> > >>> to >>> > >>> get a new db connection >>> > >>> Caused by: java.sql.SQLException: Access denied for user 'cloud'@ >>> > >>> 'localhost' >>> > >>> (using password: YES) >>> > >>> >>> > >>> Where are the credentials stored? >>> > >>> >>> > >>> >>> > >>> On Tue, Apr 29, 2014 at 2:57 PM, stevenliang < >>> [email protected]> >>> > >>> wrote: >>> > >>> >>> > >>> > oh, then change service offering for vr? >>> > >>> > >>> > >>> > >>> > >>> > On 29/04/14 05:53 PM, motty cruz wrote: >>> > >>> > >>> > >>> >> for my VR, I created a new >>> > >>> >> >>> > >>> >> "System Offering For Software Router" >>> > >>> >> CPU in (MHz) 1.00GHz >>> > >>> >> Memory (in MB) 1.00GB >>> > >>> >> >>> > >>> >> this are my current offerings, I'm sure the more RAM and CPU >>> better >>> > >>> >> performance. >>> > >>> >> >>> > >>> >> Thanks, >>> > >>> >> >>> > >>> >> >>> > >>> >> >>> > >>> >> On Tue, Apr 29, 2014 at 2:44 PM, stevenliang < >>> [email protected] >>> > > >>> > >>> >> wrote: >>> > >>> >> >>> > >>> >> Thank you again, motty. >>> > >>> >>> I didn't notice this earlier. >>> > >>> >>> BTW, how did you make your vr had 1GB CPU and 512MB RAM? >>> > >>> >>> >>> > >>> >>> >>> > >>> >>> >>> > >>> >>> On 29/04/14 05:33 PM, motty cruz wrote: >>> > >>> >>> >>> > >>> >>> Stevellang, >>> > >>> >>>> I not sure if you saw this in the forums earlier : >>> > >>> >>>> >>> > >>> >>> > >>> http://mail-archives.apache.org/mod_mbox/cloudstack-users/201404.mbox/% >>> > >>> >>>> 3CCALoOYy6A10bz1zOQQs1VyFb9epqLfhf7mu6hc= >>> > >>> [email protected]%3E >>> > >>> >>>> >>> > >>> >>>> I don't know if the bug was fixed yet, >>> > >>> >>>> >>> > >>> >>>> I will try upgrade in the next couple of days on a testing >>> > cluster, >>> > >>> will >>> > >>> >>>> report back if the bug was fixed. >>> > >>> >>>> >>> > >>> >>>> Thanks, >>> > >>> >>>> >>> > >>> >>>> >>> > >>> >>>> On Tue, Apr 29, 2014 at 2:25 PM, stevenliang < >>> > [email protected] >>> > >>> > >>> > >>> >>>> wrote: >>> > >>> >>>> >>> > >>> >>>> Thank you, motty. >>> > >>> >>>> >>> > >>> >>>>> I am also running kvm. Since that time I failed upgrade, I am >>> > still >>> > >>> >>>>> using >>> > >>> >>>>> 4.2.1. I'll try as your advice. >>> > >>> >>>>> >>> > >>> >>>>> >>> > >>> >>>>> On 29/04/14 05:19 PM, motty cruz wrote: >>> > >>> >>>>> >>> > >>> >>>>> Stevenllang, >>> > >>> >>>>> >>> > >>> >>>>>> I had the similar issue with VR, I notice it was because I >>> leave >>> > >>> the >>> > >>> >>>>>> default system specs on the VR, for instance by default >>> 500MHz >>> > on >>> > >>> CPU >>> > >>> >>>>>> and >>> > >>> >>>>>> 128MB on RAM, if you upgrade to at least 1GB on CPU and >>> 512MB of >>> > >>> RAM >>> > >>> >>>>>> your >>> > >>> >>>>>> VR will survive the upgrade from 4.2.1 to 4.3.1. >>> > >>> >>>>>> >>> > >>> >>>>>> I am running KVM, when I upgrade from 4.2.1 to 4.3 my VMs >>> were >>> > not >>> > >>> >>>>>> able >>> > >>> >>>>>> to >>> > >>> >>>>>> access outside world, even if I created a new router. >>> > >>> >>>>>> >>> > >>> >>>>>> wish you the best, >>> > >>> >>>>>> -motty >>> > >>> >>>>>> >>> > >>> >>>>>> >>> > >>> >>>>>> On Tue, Apr 29, 2014 at 2:13 PM, stevenliang < >>> > >>> [email protected]> >>> > >>> >>>>>> wrote: >>> > >>> >>>>>> >>> > >>> >>>>>> Yes, I had two zones(one is basic, another is advanced >>> mode). >>> > >>> >>>>>> >>> > >>> >>>>>> After I upgraded from 4.2.1 to 4.3, the vrouter lost. >>> > >>> >>>>>>> So I rolled back to 4.2.1, the vrouter came back. >>> > >>> >>>>>>> >>> > >>> >>>>>>> >>> > >>> >>>>>>> On 29/04/14 04:54 PM, Ian Young wrote: >>> > >>> >>>>>>> >>> > >>> >>>>>>> Did rolling back to 4.2 fix the problem? >>> > >>> >>>>>>> >>> > >>> >>>>>>> On Tue, Apr 29, 2014 at 1:22 PM, stevenliang < >>> > >>> [email protected] >>> > >>> >>>>>>>> > >>> > >>> >>>>>>>> wrote: >>> > >>> >>>>>>>> >>> > >>> >>>>>>>> I met your situation before. Finally I rolled back to >>> 4.2 >>> > >>> >>>>>>>> >>> > >>> >>>>>>>> On 29/04/14 04:18 PM, Ian Young wrote: >>> > >>> >>>>>>>> >>> > >>> >>>>>>>>> I destroyed the old virtual router and was able to >>> create >>> > >>> a new >>> > >>> >>>>>>>>> one >>> > >>> >>>>>>>>> by >>> > >>> >>>>>>>>> >>> > >>> >>>>>>>>> adding a new instance. However, this new router also >>> > failed >>> > >>> to >>> > >>> >>>>>>>>> >>> > >>> >>>>>>>>>> start, >>> > >>> >>>>>>>>>> citing the same error. After that, the expungement >>> delay >>> > >>> elapsed >>> > >>> >>>>>>>>>> and >>> > >>> >>>>>>>>>> the >>> > >>> >>>>>>>>>> virtual router was expunged, so now I have none. >>> > >>> >>>>>>>>>> >>> > >>> >>>>>>>>>> >>> > >>> >>>>>>>>>> On Mon, Apr 28, 2014 at 8:52 PM, Ian Young < >>> > >>> >>>>>>>>>> [email protected]> >>> > >>> >>>>>>>>>> wrote: >>> > >>> >>>>>>>>>> >>> > >>> >>>>>>>>>> I upgraded from 4.2.1 to 4.3.0 tonight, following >>> the >>> > >>> >>>>>>>>>> instructions >>> > >>> >>>>>>>>>> here: >>> > >>> >>>>>>>>>> >>> > >>> >>>>>>>>>> >>> http://docs.cloudstack.apache.org/projects/cloudstack- >>> > >>> >>>>>>>>>> >>> > >>> >>>>>>>>>> >>> > release-notes/en/latest/rnotes.html#upgrade-from-4-2-x-to-4-3 >>> > >>> >>>>>>>>>>> >>> > >>> >>>>>>>>>>> At the last step, I tried to restart the system VMs. >>> The >>> > >>> virtual >>> > >>> >>>>>>>>>>> router >>> > >>> >>>>>>>>>>> failed to start. Here is the message that was >>> displayed in >>> > >>> the >>> > >>> >>>>>>>>>>> web >>> > >>> >>>>>>>>>>> UI: >>> > >>> >>>>>>>>>>> >>> > >>> >>>>>>>>>>> Resource [Host:1] is unreachable: Host 1: Unable to >>> start >>> > >>> >>>>>>>>>>> instance >>> > >>> >>>>>>>>>>> due >>> > >>> >>>>>>>>>>> to >>> > >>> >>>>>>>>>>> Unable to start VM[DomainRouter|r-4-VM] due to error in >>> > >>> >>>>>>>>>>> finalizeStart, >>> > >>> >>>>>>>>>>> not >>> > >>> >>>>>>>>>>> retrying >>> > >>> >>>>>>>>>>> >>> > >>> >>>>>>>>>>> I tried running the script to restart the VMs but this >>> time >>> > >>> it >>> > >>> >>>>>>>>>>> failed >>> > >>> >>>>>>>>>>> to >>> > >>> >>>>>>>>>>> start the console proxy: >>> > >>> >>>>>>>>>>> >>> > >>> >>>>>>>>>>> [root@virthost1 ~]$ cloudstack-sysvmadm -d >>> 192.168.100.6 >>> > -u >>> > >>> >>>>>>>>>>> cloud >>> > >>> >>>>>>>>>>> -p >>> > >>> >>>>>>>>>>> -a >>> > >>> >>>>>>>>>>> >>> > >>> >>>>>>>>>>> Stopping and starting 1 secondary storage vm(s)... >>> > >>> >>>>>>>>>>> Done stopping and starting secondary storage vm(s) >>> > >>> >>>>>>>>>>> >>> > >>> >>>>>>>>>>> Stopping and starting 1 console proxy vm(s)... >>> > >>> >>>>>>>>>>> ERROR: Failed to start console proxy vm with id 2 >>> > >>> >>>>>>>>>>> >>> > >>> >>>>>>>>>>> Done stopping and starting console proxy vm(s) . >>> > >>> >>>>>>>>>>> >>> > >>> >>>>>>>>>>> Stopping and starting 0 running routing vm(s)... >>> > >>> >>>>>>>>>>> >>> > >>> >>>>>>>>>>> Is there a way to wipe the system VMs out and start >>> over? >>> > >>> >>>>>>>>>>> >>> > >>> >>>>>>>>>>> >>> > >>> >>>>>>>>>>> >>> > >>> >>>>>>>>>>> >>> > >>> >>>>>>>>>>> >>> > >>> >>>>>>>>>>> >>> > >>> > >>> > >>> >>> > >> >>> > >> >>> > >>> >> >> >
