My cloud is down for 2 days. I would really appreciate if you can help me fix the issue.
On Thu, Oct 3, 2019 at 4:44 PM Fariborz Navidan <[email protected]> wrote: > Any idea? > > On Thu, Oct 3, 2019 at 2:09 PM Fariborz Navidan <[email protected]> > wrote: > >> I did the steps exactly but still same error. It first says cannot >> upgrade database with no further sql errors stated. Thn says cannot find >> System VM template for KVM. It look likes system vm template 4.11.3 is >> installed only for router! >> >> MariaDB [cloud]> select * from vm_template where name like 'SystemVM%' >> and removed is null; >> >> +----+-------------+-------------------------------+--------------------------------------+--------+----------+--------+-----+------+--------------------------------------------------------------------------------------+--------+---------------------+---------+------------+----------------------------------+-------------------------------+-----------------+---------------+-------------+----------+-------------+-------------+-------------+-----------------+--------------------+--------------+----------+-----------+--------+--------------+---------+----------------------+--------------------+-----------------+ >> | id | unique_name | name | uuid >> | public | featured | type | hvm | bits | url >> | >> format | created | removed | account_id | checksum >> | display_text | enable_password | >> enable_sshkey | guest_os_id | bootable | prepopulate | cross_zones | >> extractable | hypervisor_type | source_template_id | template_tag | >> sort_key | size | state | update_count | updated | >> dynamically_scalable | parent_template_id | direct_download | >> >> +----+-------------+-------------------------------+--------------------------------------+--------+----------+--------+-----+------+--------------------------------------------------------------------------------------+--------+---------------------+---------+------------+----------------------------------+-------------------------------+-----------------+---------------+-------------+----------+-------------+-------------+-------------+-----------------+--------------------+--------------+----------+-----------+--------+--------------+---------+----------------------+--------------------+-----------------+ >> | 1 | routing-1 | SystemVM Template (XenServer) | >> c1ea2f68-ca75-11e9-8212-d05099d38681 | 0 | 0 | SYSTEM | 0 | >> 64 | >> https://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.2-xen.vhd.bz2 >> | VHD | 2019-08-29 17:57:52 | NULL | 1 | >> 6e8b3ae84ca8145736d1d7d3f7546e65 | SystemVM Template (XenServer) | >> 0 | 0 | 184 | 1 | 0 | >> 1 | 0 | XenServer | NULL | NULL | >> 0 | NULL | Active | 0 | NULL | 0 | >> NULL | 0 | >> | 3 | routing-3 | SystemVM Template (KVM) | >> e4621347-39b8-427e-a55f-82cf51275b69 | 0 | 0 | SYSTEM | 0 | >> 64 | >> http://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.3-kvm.qcow2.bz2 >> | QCOW2 | 2019-08-29 17:57:52 | NULL | 1 | >> 6d12cc764cd7d64112d8c35d70923eb1 | SystemVM Template (KVM) | >> 0 | 0 | 15 | 1 | 0 | >> 1 | 0 | KVM | NULL | NULL | >> 0 | 385242624 | Active | 0 | NULL | 0 | >> NULL | 0 | >> | 8 | routing-8 | SystemVM Template (vSphere) | >> c1ea7e39-ca75-11e9-8212-d05099d38681 | 0 | 0 | SYSTEM | 0 | >> 64 | >> https://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.2-vmware.ova >> | OVA | 2019-08-29 17:57:52 | NULL | 1 | >> e981f8cb951688efd93481913198c9cc | SystemVM Template (vSphere) | >> 0 | 0 | 99 | 1 | 0 | >> 1 | 0 | VMware | NULL | NULL | >> 0 | NULL | Active | 0 | NULL | 1 | >> NULL | 0 | >> | 9 | routing-9 | SystemVM Template (HyperV) | >> c1eaa7b0-ca75-11e9-8212-d05099d38681 | 0 | 0 | SYSTEM | 0 | >> 64 | >> https://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.2-hyperv.vhd.zip >> | VHD | 2019-08-29 17:57:52 | NULL | 1 | >> e9032635ffba021371780307162551b9 | SystemVM Template (HyperV) | >> 0 | 0 | 15 | 1 | 0 | >> 1 | 0 | Hyperv | NULL | NULL | >> 0 | NULL | Active | 0 | NULL | 0 | >> NULL | 0 | >> | 10 | routing-10 | SystemVM Template (LXC) | >> 6706cad5-ca77-11e9-8212-d05099d38681 | 0 | 0 | SYSTEM | 0 | >> 64 | >> https://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.2-kvm.qcow2.bz2 >> | QCOW2 | 2019-08-29 18:09:38 | NULL | 1 | >> 6d12cc764cd7d64112d8c35d70923eb1 | SystemVM Template (LXC) | >> 0 | 0 | 15 | 1 | 0 | >> 1 | 0 | LXC | NULL | NULL | >> 0 | NULL | Active | 0 | NULL | 0 | >> NULL | 0 | >> | 12 | routing-12 | SystemVM Template (Ovm3) | >> 694d5de8-ca77-11e9-8212-d05099d38681 | 0 | 0 | SYSTEM | 0 | >> 64 | >> https://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.2-ovm.raw.bz2 >> | RAW | 2019-08-29 18:09:42 | NULL | 1 | >> c4a91f8e52e4531a1c2a9a17c530d5fe | SystemVM Template (Ovm3) | >> 0 | 0 | 183 | 1 | 0 | >> 1 | 0 | Ovm3 | NULL | NULL | >> 0 | NULL | Active | NULL | NULL | 0 | >> NULL | 0 | >> >> +----+-------------+-------------------------------+--------------------------------------+--------+----------+--------+-----+------+--------------------------------------------------------------------------------------+--------+---------------------+---------+------------+----------------------------------+-------------------------------+-----------------+---------------+-------------+----------+-------------+-------------+-------------+-----------------+--------------------+--------------+----------+-----------+--------+--------------+---------+----------------------+--------------------+-----------------+ >> 6 rows in set (0.00 sec) >> >> On Thu, Oct 3, 2019 at 1:52 AM Andrija Panic <[email protected]> >> wrote: >> >>> - ake sure mgmt is stopped (you can leave it on 4.13 if not >>> already downgraded), stop cloudstack-agents on KVM hosts. >>> - Rollback DB with your SQL statements, that you shared a few emails ago >>> (unfortunately after this 4.12 is NOT possible to work normally...) >>> -Download the new systemVM template - it WILL OVERWRITE your old one - >>> i.e.: >>> From mgmt, mount the Secondary Storage NFS on /mnt/secondary and >>> execute: >>> >>> >>> >>> /usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt >>> -m /mnt/secondary -u >>> >>> http://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.3-kvm.qcow2.bz2 >>> -h >>> <http://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.3-kvm.qcow2.bz2-h> >>> kvm -F >>> This will overwrite existing systemVM template on the >>> /mnt/secondary/templates/1/3/xxxxxx and will update the DB with new names >>> etc... >>> - Upgrade packages to 4.13 now (if not already having 4.13 installed) >>> - start mgmt - upgrade path will find the new systemVM template and mgmt >>> server should start fine. >>> - Destroy your SSVM and CPVM, new ones will be recreated from new >>> systemVM >>> template. >>> - Restart all VRs (or restart networks/VPCs with Clean Up option) - new >>> will be created from the new systemVM template. >>> >>> I've just tested all this in my lab - let me know if any issues. >>> >>> >>> >>> On Wed, 2 Oct 2019 at 20:37, Fariborz Navidan <[email protected]> >>> wrote: >>> >>> > I cannot rollback DB to 4.12. How can I download 4.11.3 and and >>> register in >>> > DB? >>> > >>> > On Wed, Oct 2, 2019 at 9:43 PM Andrija Panic <[email protected]> >>> > wrote: >>> > >>> > > There's a GitHub issue about this (perhaps you or some colleague?) >>> > > >>> > > Did you register 4.11.3 (not 4.13) systemVM template? Documentation >>> has >>> > > just been updated... Sorry for this. >>> > > >>> > > i..e. downgrade packages to 4.12, make sure your DB is rolled back, >>> start >>> > > 4.12 normally, register 4.11.3 systemVM template as per the doc and >>> then >>> > > try again to upgrade. >>> > > >>> > > let me know how it goes. >>> > > >>> > > Andrija >>> > > >>> > > On Wed, 2 Oct 2019, 17:27 Fariborz Navidan, <[email protected]> >>> > wrote: >>> > > >>> > > > Any idea? >>> > > > >>> > > > On Wed, Oct 2, 2019 at 3:59 PM Fariborz Navidan < >>> [email protected] >>> > > >>> > > > wrote: >>> > > > >>> > > > > Yes, It was clean 4.12 installation. >>> > > > > >>> > > > > On Wed, Oct 2, 2019 at 3:52 PM Andrija Panic < >>> > [email protected]> >>> > > > > wrote: >>> > > > > >>> > > > >> "was this a clean 4.12 install, then later upgrade - or was it >>> > > > previously >>> > > > >> 4.11.x then 4.12 etc?" >>> > > > >> >>> > > > >> On Wed, 2 Oct 2019 at 14:20, Fariborz Navidan < >>> > [email protected]> >>> > > > >> wrote: >>> > > > >> >>> > > > >> > Which question? >>> > > > >> > >>> > > > >> > I have sent pastebin link. I lost 4.12 backup accidentally. I >>> am >>> > > > trying >>> > > > >> to >>> > > > >> > make 4.13 upgrade successfully. I do run following commands >>> before >>> > > > >> upgrade >>> > > > >> > retry: >>> > > > >> > >>> > > > >> > [root@fr-kvm1 ~]# cat rollback.sql >>> > > > >> > delete from guest_os_hypervisor where guest_os_id >= 277; >>> > > > >> > delete from guest_os where id >= 277; >>> > > > >> > SET FOREIGN_KEY_CHECKS=0; >>> > > > >> > drop table direct_download_certificate; >>> > > > >> > drop table vpc_offering_details; >>> > > > >> > drop table direct_download_certificate_host_map; >>> > > > >> > drop table template_ovf_properties; >>> > > > >> > SET FOREIGN_KEY_CHECKS=1; >>> > > > >> > alter table data_center drop column sort_key; >>> > > > >> > alter table vpc_offerings drop column sort_key; >>> > > > >> > alter table disk_offering add column domain_id int(32) not >>> null; >>> > > > >> > alter table service_offering_details add constraint >>> > > > >> > uk_service_offering_id_name unique (id); >>> > > > >> > alter table network_offering_details DROP COLUMN `display`; >>> > > > >> > ALTER TABLE `cloud_usage`.`usage_vmsnapshot` DROP COLUMN >>> > > > >> `vm_snapshot_id`; >>> > > > >> > ALTER TABLE `cloud_usage`.`usage_snapshot_on_primary` DROP >>> COLUMN >>> > > > >> > `vm_snapshot_id`; >>> > > > >> > [root@fr-kvm1 ~]# >>> > > > >> > >>> > > > >> > However I don't understand the error I showed you on log in >>> > > pastebin. >>> > > > >> > Please check and let me know what does it mean. >>> > > > >> > >>> > > > >> > On Wed, Oct 2, 2019 at 3:27 PM Andrija Panic < >>> > > [email protected] >>> > > > > >>> > > > >> > wrote: >>> > > > >> > >>> > > > >> > > It was not designed to rollback on its own, you are >>> expected to >>> > > > >> rollback >>> > > > >> > DB >>> > > > >> > > yourself (DB restore) and try again if needed. >>> > > > >> > > >>> > > > >> > > But you have not answered my question. >>> > > > >> > > >>> > > > >> > > On Wed, 2 Oct 2019, 13:50 Fariborz Navidan, < >>> > > [email protected]> >>> > > > >> > wrote: >>> > > > >> > > >>> > > > >> > > > In my experience, if DB upgrade fails, it does not fully >>> rolls >>> > > > back >>> > > > >> > > > remaining 4.13's schema upgraded partially and upon next >>> > restart >>> > > > it >>> > > > >> > does >>> > > > >> > > > not check existence of 4.13's table or columns causing SQL >>> > > errors >>> > > > >> > > > complaining about tables or columns already exists. >>> > > > >> > > > >>> > > > >> > > > On Wed, Oct 2, 2019 at 3:14 PM Andrija Panic < >>> > > > >> [email protected]> >>> > > > >> > > > wrote: >>> > > > >> > > > >>> > > > >> > > > > was this a clean 4.12 install, then later upgrade - or >>> was >>> > it >>> > > > >> > > previously >>> > > > >> > > > > 4.11.x then 4.12 etc? >>> > > > >> > > > > >>> > > > >> > > > > On Wed, 2 Oct 2019 at 12:21, Fariborz Navidan < >>> > > > >> [email protected] >>> > > > >> > > >>> > > > >> > > > > wrote: >>> > > > >> > > > > >>> > > > >> > > > > > https://pastebin.com/2EFm0jaW >>> > > > >> > > > > > >>> > > > >> > > > > > On Wed, Oct 2, 2019 at 1:19 PM Andrija Panic < >>> > > > >> > > [email protected]> >>> > > > >> > > > > > wrote: >>> > > > >> > > > > > >>> > > > >> > > > > > > (backup, backup, backup...) >>> > > > >> > > > > > > >>> > > > >> > > > > > > Can you send the full logs (passtebin or similar >>> please, >>> > > not >>> > > > >> in >>> > > > >> > > > email) >>> > > > >> > > > > ? >>> > > > >> > > > > > > >>> > > > >> > > > > > > Andrija >>> > > > >> > > > > > > >>> > > > >> > > > > > >>> > > > >> > > > > >>> > > > >> > > > > >>> > > > >> > > > > -- >>> > > > >> > > > > >>> > > > >> > > > > Andrija Panić >>> > > > >> > > > > >>> > > > >> > > > >>> > > > >> > > >>> > > > >> > >>> > > > >> >>> > > > >> >>> > > > >> -- >>> > > > >> >>> > > > >> Andrija Panić >>> > > > >> >>> > > > > >>> > > > >>> > > >>> > >>> >>> >>> -- >>> >>> Andrija Panić >>> >>
