The ID for SystemVM Template (KVM) is 3 and there is two references in
template_store_ref:
MariaDB [cloud]> select * from template_store_ref where template_id=3;
+-----+----------+-------------+---------------------+---------------------+--------------------------------------+--------------+-----------+------------+---------------+----------------+-----------+----------------------------------------------------------------------------------------------------+--------------------------------------------------------------+------------------------------------------------------------------------------------+-------+-----------+---------+--------------+---------+---------------------+----------------------+--------------+
| id | store_id | template_id | created | last_updated
| job_id | download_pct | size |
store_role | physical_size | download_state | error_str | local_path
| install_path | url
| state | destroyed | is_copy | update_count | ref_cnt | updated
| download_url_created | download_url |
+-----+----------+-------------+---------------------+---------------------+--------------------------------------+--------------+-----------+------------+---------------+----------------+-----------+----------------------------------------------------------------------------------------------------+--------------------------------------------------------------+------------------------------------------------------------------------------------+-------+-----------+---------+--------------+---------+---------------------+----------------------+--------------+
| 5 | 1 | 3 | 2019-08-29 16:55:07 | 2019-10-01 09:24:55
| NULL | 100 | 385242624 | Image
| 385242624 | DOWNLOADED | NULL | NULL
|
template/tmpl/1/3/4b1fdeba-f3f5-45c8-b5ba-851e8bfd7f2b.qcow2 |
http://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.2-kvm.qcow2.bz2
| Ready | 0 | 0 | 0 | 0 | NULL
| NULL | NULL |
| 233 | 2 | 3 | 2019-09-15 13:01:12 | 2019-10-01 09:24:56
| 755ab14c-c4df-4dd2-ae39-2a9ff7bb0177 | 100 | 385242624 | Image
| 385242624 | DOWNLOADED | |
/mnt/SecStorage/aa312459-a38e-3d25-a531-791cf3c1bd35/template/tmpl/1/3/dnld6265824134994923385tmp_
| template/tmpl/1/3/4b1fdeba-f3f5-45c8-b5ba-851e8bfd7f2b.qcow2 | NULL
|
Ready | 0 | 0 | 2 | 0 | 2019-09-15 13:01:25
| NULL | NULL |
+-----+----------+-------------+---------------------+---------------------+--------------------------------------+--------------+-----------+------------+---------------+----------------+-----------+----------------------------------------------------------------------------------------------------+--------------------------------------------------------------+------------------------------------------------------------------------------------+-------+-----------+---------+--------------+---------+---------------------+----------------------+--------------+
2 rows in set (0.00 sec)
MariaDB [cloud]>
On Thu, Oct 3, 2019 at 7:56 PM Andrija Panic <[email protected]>
wrote:
> Can you share logs during this last attempt?
>
> I'll try to craft some SQL additions for your vm_template and
> template_store_ref tables while also manually downloading and unpacking the
> file to specific folder on Secondary Storage...
>
> Andrija
>
> On Thu, 3 Oct 2019 at 16:55, Fariborz Navidan <[email protected]>
> wrote:
>
> > My cloud is down for 2 days. I would really appreciate if you can help me
> > fix the issue.
> >
> > On Thu, Oct 3, 2019 at 4:44 PM Fariborz Navidan <[email protected]>
> > wrote:
> >
> > > Any idea?
> > >
> > > On Thu, Oct 3, 2019 at 2:09 PM Fariborz Navidan <[email protected]
> >
> > > wrote:
> > >
> > >> I did the steps exactly but still same error. It first says cannot
> > >> upgrade database with no further sql errors stated. Thn says cannot
> find
> > >> System VM template for KVM. It look likes system vm template 4.11.3 is
> > >> installed only for router!
> > >>
> > >> MariaDB [cloud]> select * from vm_template where name like 'SystemVM%'
> > >> and removed is null;
> > >>
> > >>
> >
> +----+-------------+-------------------------------+--------------------------------------+--------+----------+--------+-----+------+--------------------------------------------------------------------------------------+--------+---------------------+---------+------------+----------------------------------+-------------------------------+-----------------+---------------+-------------+----------+-------------+-------------+-------------+-----------------+--------------------+--------------+----------+-----------+--------+--------------+---------+----------------------+--------------------+-----------------+
> > >> | id | unique_name | name | uuid
> > >> | public | featured | type | hvm | bits | url
> > >> |
> > >> format | created | removed | account_id | checksum
> > >> | display_text | enable_password |
> > >> enable_sshkey | guest_os_id | bootable | prepopulate | cross_zones |
> > >> extractable | hypervisor_type | source_template_id | template_tag |
> > >> sort_key | size | state | update_count | updated |
> > >> dynamically_scalable | parent_template_id | direct_download |
> > >>
> > >>
> >
> +----+-------------+-------------------------------+--------------------------------------+--------+----------+--------+-----+------+--------------------------------------------------------------------------------------+--------+---------------------+---------+------------+----------------------------------+-------------------------------+-----------------+---------------+-------------+----------+-------------+-------------+-------------+-----------------+--------------------+--------------+----------+-----------+--------+--------------+---------+----------------------+--------------------+-----------------+
> > >> | 1 | routing-1 | SystemVM Template (XenServer) |
> > >> c1ea2f68-ca75-11e9-8212-d05099d38681 | 0 | 0 | SYSTEM |
> 0
> > |
> > >> 64 |
> > >>
> >
> https://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.2-xen.vhd.bz2
> > >> | VHD | 2019-08-29 17:57:52 | NULL | 1 |
> > >> 6e8b3ae84ca8145736d1d7d3f7546e65 | SystemVM Template (XenServer) |
> > >> 0 | 0 | 184 | 1 | 0 |
> > >> 1 | 0 | XenServer | NULL | NULL
> |
> > >> 0 | NULL | Active | 0 | NULL |
> > 0 |
> > >> NULL | 0 |
> > >> | 3 | routing-3 | SystemVM Template (KVM) |
> > >> e4621347-39b8-427e-a55f-82cf51275b69 | 0 | 0 | SYSTEM |
> 0
> > |
> > >> 64 |
> > >>
> >
> http://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.3-kvm.qcow2.bz2
> > >> | QCOW2 | 2019-08-29 17:57:52 | NULL | 1 |
> > >> 6d12cc764cd7d64112d8c35d70923eb1 | SystemVM Template (KVM) |
> > >> 0 | 0 | 15 | 1 | 0 |
> > >> 1 | 0 | KVM | NULL | NULL
> |
> > >> 0 | 385242624 | Active | 0 | NULL |
> > 0 |
> > >> NULL | 0 |
> > >> | 8 | routing-8 | SystemVM Template (vSphere) |
> > >> c1ea7e39-ca75-11e9-8212-d05099d38681 | 0 | 0 | SYSTEM |
> 0
> > |
> > >> 64 |
> > >>
> >
> https://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.2-vmware.ova
> > >> | OVA | 2019-08-29 17:57:52 | NULL | 1 |
> > >> e981f8cb951688efd93481913198c9cc | SystemVM Template (vSphere) |
> > >> 0 | 0 | 99 | 1 | 0 |
> > >> 1 | 0 | VMware | NULL | NULL
> |
> > >> 0 | NULL | Active | 0 | NULL |
> > 1 |
> > >> NULL | 0 |
> > >> | 9 | routing-9 | SystemVM Template (HyperV) |
> > >> c1eaa7b0-ca75-11e9-8212-d05099d38681 | 0 | 0 | SYSTEM |
> 0
> > |
> > >> 64 |
> > >>
> >
> https://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.2-hyperv.vhd.zip
> > >> | VHD | 2019-08-29 17:57:52 | NULL | 1 |
> > >> e9032635ffba021371780307162551b9 | SystemVM Template (HyperV) |
> > >> 0 | 0 | 15 | 1 | 0 |
> > >> 1 | 0 | Hyperv | NULL | NULL
> |
> > >> 0 | NULL | Active | 0 | NULL |
> > 0 |
> > >> NULL | 0 |
> > >> | 10 | routing-10 | SystemVM Template (LXC) |
> > >> 6706cad5-ca77-11e9-8212-d05099d38681 | 0 | 0 | SYSTEM |
> 0
> > |
> > >> 64 |
> > >>
> >
> https://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.2-kvm.qcow2.bz2
> > >> | QCOW2 | 2019-08-29 18:09:38 | NULL | 1 |
> > >> 6d12cc764cd7d64112d8c35d70923eb1 | SystemVM Template (LXC) |
> > >> 0 | 0 | 15 | 1 | 0 |
> > >> 1 | 0 | LXC | NULL | NULL
> |
> > >> 0 | NULL | Active | 0 | NULL |
> > 0 |
> > >> NULL | 0 |
> > >> | 12 | routing-12 | SystemVM Template (Ovm3) |
> > >> 694d5de8-ca77-11e9-8212-d05099d38681 | 0 | 0 | SYSTEM |
> 0
> > |
> > >> 64 |
> > >>
> >
> https://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.2-ovm.raw.bz2
> > >> | RAW | 2019-08-29 18:09:42 | NULL | 1 |
> > >> c4a91f8e52e4531a1c2a9a17c530d5fe | SystemVM Template (Ovm3) |
> > >> 0 | 0 | 183 | 1 | 0 |
> > >> 1 | 0 | Ovm3 | NULL | NULL
> |
> > >> 0 | NULL | Active | NULL | NULL |
> > 0 |
> > >> NULL | 0 |
> > >>
> > >>
> >
> +----+-------------+-------------------------------+--------------------------------------+--------+----------+--------+-----+------+--------------------------------------------------------------------------------------+--------+---------------------+---------+------------+----------------------------------+-------------------------------+-----------------+---------------+-------------+----------+-------------+-------------+-------------+-----------------+--------------------+--------------+----------+-----------+--------+--------------+---------+----------------------+--------------------+-----------------+
> > >> 6 rows in set (0.00 sec)
> > >>
> > >> On Thu, Oct 3, 2019 at 1:52 AM Andrija Panic <[email protected]
> >
> > >> wrote:
> > >>
> > >>> - ake sure mgmt is stopped (you can leave it on 4.13 if not
> > >>> already downgraded), stop cloudstack-agents on KVM hosts.
> > >>> - Rollback DB with your SQL statements, that you shared a few emails
> > ago
> > >>> (unfortunately after this 4.12 is NOT possible to work normally...)
> > >>> -Download the new systemVM template - it WILL OVERWRITE your old
> one -
> > >>> i.e.:
> > >>> From mgmt, mount the Secondary Storage NFS on /mnt/secondary
> > and
> > >>> execute:
> > >>>
> > >>>
> > >>>
> >
> /usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt
> > >>> -m /mnt/secondary -u
> > >>>
> > >>>
> >
> http://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.3-kvm.qcow2.bz2
> > >>> -h
> > >>> <
> >
> http://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.3-kvm.qcow2.bz2-h
> > >
> > >>> kvm -F
> > >>> This will overwrite existing systemVM template on the
> > >>> /mnt/secondary/templates/1/3/xxxxxx and will update the DB with new
> > names
> > >>> etc...
> > >>> - Upgrade packages to 4.13 now (if not already having 4.13 installed)
> > >>> - start mgmt - upgrade path will find the new systemVM template and
> > mgmt
> > >>> server should start fine.
> > >>> - Destroy your SSVM and CPVM, new ones will be recreated from new
> > >>> systemVM
> > >>> template.
> > >>> - Restart all VRs (or restart networks/VPCs with Clean Up option) -
> new
> > >>> will be created from the new systemVM template.
> > >>>
> > >>> I've just tested all this in my lab - let me know if any issues.
> > >>>
> > >>>
> > >>>
> > >>> On Wed, 2 Oct 2019 at 20:37, Fariborz Navidan <[email protected]
> >
> > >>> wrote:
> > >>>
> > >>> > I cannot rollback DB to 4.12. How can I download 4.11.3 and and
> > >>> register in
> > >>> > DB?
> > >>> >
> > >>> > On Wed, Oct 2, 2019 at 9:43 PM Andrija Panic <
> > [email protected]>
> > >>> > wrote:
> > >>> >
> > >>> > > There's a GitHub issue about this (perhaps you or some
> colleague?)
> > >>> > >
> > >>> > > Did you register 4.11.3 (not 4.13) systemVM template?
> Documentation
> > >>> has
> > >>> > > just been updated... Sorry for this.
> > >>> > >
> > >>> > > i..e. downgrade packages to 4.12, make sure your DB is rolled
> back,
> > >>> start
> > >>> > > 4.12 normally, register 4.11.3 systemVM template as per the doc
> and
> > >>> then
> > >>> > > try again to upgrade.
> > >>> > >
> > >>> > > let me know how it goes.
> > >>> > >
> > >>> > > Andrija
> > >>> > >
> > >>> > > On Wed, 2 Oct 2019, 17:27 Fariborz Navidan, <
> [email protected]
> > >
> > >>> > wrote:
> > >>> > >
> > >>> > > > Any idea?
> > >>> > > >
> > >>> > > > On Wed, Oct 2, 2019 at 3:59 PM Fariborz Navidan <
> > >>> [email protected]
> > >>> > >
> > >>> > > > wrote:
> > >>> > > >
> > >>> > > > > Yes, It was clean 4.12 installation.
> > >>> > > > >
> > >>> > > > > On Wed, Oct 2, 2019 at 3:52 PM Andrija Panic <
> > >>> > [email protected]>
> > >>> > > > > wrote:
> > >>> > > > >
> > >>> > > > >> "was this a clean 4.12 install, then later upgrade - or was
> it
> > >>> > > > previously
> > >>> > > > >> 4.11.x then 4.12 etc?"
> > >>> > > > >>
> > >>> > > > >> On Wed, 2 Oct 2019 at 14:20, Fariborz Navidan <
> > >>> > [email protected]>
> > >>> > > > >> wrote:
> > >>> > > > >>
> > >>> > > > >> > Which question?
> > >>> > > > >> >
> > >>> > > > >> > I have sent pastebin link. I lost 4.12 backup
> accidentally.
> > I
> > >>> am
> > >>> > > > trying
> > >>> > > > >> to
> > >>> > > > >> > make 4.13 upgrade successfully. I do run following
> commands
> > >>> before
> > >>> > > > >> upgrade
> > >>> > > > >> > retry:
> > >>> > > > >> >
> > >>> > > > >> > [root@fr-kvm1 ~]# cat rollback.sql
> > >>> > > > >> > delete from guest_os_hypervisor where guest_os_id >= 277;
> > >>> > > > >> > delete from guest_os where id >= 277;
> > >>> > > > >> > SET FOREIGN_KEY_CHECKS=0;
> > >>> > > > >> > drop table direct_download_certificate;
> > >>> > > > >> > drop table vpc_offering_details;
> > >>> > > > >> > drop table direct_download_certificate_host_map;
> > >>> > > > >> > drop table template_ovf_properties;
> > >>> > > > >> > SET FOREIGN_KEY_CHECKS=1;
> > >>> > > > >> > alter table data_center drop column sort_key;
> > >>> > > > >> > alter table vpc_offerings drop column sort_key;
> > >>> > > > >> > alter table disk_offering add column domain_id int(32) not
> > >>> null;
> > >>> > > > >> > alter table service_offering_details add constraint
> > >>> > > > >> > uk_service_offering_id_name unique (id);
> > >>> > > > >> > alter table network_offering_details DROP COLUMN
> `display`;
> > >>> > > > >> > ALTER TABLE `cloud_usage`.`usage_vmsnapshot` DROP COLUMN
> > >>> > > > >> `vm_snapshot_id`;
> > >>> > > > >> > ALTER TABLE `cloud_usage`.`usage_snapshot_on_primary` DROP
> > >>> COLUMN
> > >>> > > > >> > `vm_snapshot_id`;
> > >>> > > > >> > [root@fr-kvm1 ~]#
> > >>> > > > >> >
> > >>> > > > >> > However I don't understand the error I showed you on log
> in
> > >>> > > pastebin.
> > >>> > > > >> > Please check and let me know what does it mean.
> > >>> > > > >> >
> > >>> > > > >> > On Wed, Oct 2, 2019 at 3:27 PM Andrija Panic <
> > >>> > > [email protected]
> > >>> > > > >
> > >>> > > > >> > wrote:
> > >>> > > > >> >
> > >>> > > > >> > > It was not designed to rollback on its own, you are
> > >>> expected to
> > >>> > > > >> rollback
> > >>> > > > >> > DB
> > >>> > > > >> > > yourself (DB restore) and try again if needed.
> > >>> > > > >> > >
> > >>> > > > >> > > But you have not answered my question.
> > >>> > > > >> > >
> > >>> > > > >> > > On Wed, 2 Oct 2019, 13:50 Fariborz Navidan, <
> > >>> > > [email protected]>
> > >>> > > > >> > wrote:
> > >>> > > > >> > >
> > >>> > > > >> > > > In my experience, if DB upgrade fails, it does not
> fully
> > >>> rolls
> > >>> > > > back
> > >>> > > > >> > > > remaining 4.13's schema upgraded partially and upon
> next
> > >>> > restart
> > >>> > > > it
> > >>> > > > >> > does
> > >>> > > > >> > > > not check existence of 4.13's table or columns causing
> > SQL
> > >>> > > errors
> > >>> > > > >> > > > complaining about tables or columns already exists.
> > >>> > > > >> > > >
> > >>> > > > >> > > > On Wed, Oct 2, 2019 at 3:14 PM Andrija Panic <
> > >>> > > > >> [email protected]>
> > >>> > > > >> > > > wrote:
> > >>> > > > >> > > >
> > >>> > > > >> > > > > was this a clean 4.12 install, then later upgrade -
> or
> > >>> was
> > >>> > it
> > >>> > > > >> > > previously
> > >>> > > > >> > > > > 4.11.x then 4.12 etc?
> > >>> > > > >> > > > >
> > >>> > > > >> > > > > On Wed, 2 Oct 2019 at 12:21, Fariborz Navidan <
> > >>> > > > >> [email protected]
> > >>> > > > >> > >
> > >>> > > > >> > > > > wrote:
> > >>> > > > >> > > > >
> > >>> > > > >> > > > > > https://pastebin.com/2EFm0jaW
> > >>> > > > >> > > > > >
> > >>> > > > >> > > > > > On Wed, Oct 2, 2019 at 1:19 PM Andrija Panic <
> > >>> > > > >> > > [email protected]>
> > >>> > > > >> > > > > > wrote:
> > >>> > > > >> > > > > >
> > >>> > > > >> > > > > > > (backup, backup, backup...)
> > >>> > > > >> > > > > > >
> > >>> > > > >> > > > > > > Can you send the full logs (passtebin or similar
> > >>> please,
> > >>> > > not
> > >>> > > > >> in
> > >>> > > > >> > > > email)
> > >>> > > > >> > > > > ?
> > >>> > > > >> > > > > > >
> > >>> > > > >> > > > > > > Andrija
> > >>> > > > >> > > > > > >
> > >>> > > > >> > > > > >
> > >>> > > > >> > > > >
> > >>> > > > >> > > > >
> > >>> > > > >> > > > > --
> > >>> > > > >> > > > >
> > >>> > > > >> > > > > Andrija Panić
> > >>> > > > >> > > > >
> > >>> > > > >> > > >
> > >>> > > > >> > >
> > >>> > > > >> >
> > >>> > > > >>
> > >>> > > > >>
> > >>> > > > >> --
> > >>> > > > >>
> > >>> > > > >> Andrija Panić
> > >>> > > > >>
> > >>> > > > >
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> > >>>
> > >>> --
> > >>>
> > >>> Andrija Panić
> > >>>
> > >>
> >
>
>
> --
>
> Andrija Panić
>
