Hi Nicolas, thanks for the work.
I started testing this RC and unfortunately I ran into some problems
related to building/upgrading ACS.
From the less to the more complex:
- When building ".deb", there is a problem with the Debian changelog:
invalid weekday 'lun';
- Regarding auto upgrading "systemvms-templates", I analyzed the code
and concluded that "systemvms-templates" will only "autoupgrade" if we
use the profile "noredist" when building the packages, which is
necessary only to VMWare. Some people, might not want the noredist JARs,
and therefore will not build with this profile. With that said, there
are some points regarding it that I think are important:
- Package "cloudstack-management" goes from ~100Mb to more than
1.5Gb, when using "noredist" while executing the 4.16.0.0 build. It
happens due to "cloudstack/engine/schema/pom.xml" automatically
downloading XenServer, KVM, and VMware "systemvms-templates". Every time
that we build the packages, it will download all these
systemvms-templates; we are unable to decide which one is necessary or
not. If we do not use XenServer, or other hypervisors, why download it?
- If we build the packages with "noredist", when we are first
running ACS after upgrading it to 4.16, it will try to autoupgrade the
systemvms-templates, however, some KVM/XenServer infrastructures don't
allow the management server to access the secondary storage directly
(except for the first systemvm-template seed). This will cause
"mount.nfs: access denied by server while mounting <path_to_secondary>".
After generating this error, ACS continues the upgrade and updates the
database; however, the new systemvm-template will not be in the
secondary storage. If we try to deploy a system (like virtual router),
it will fail (as expected).
- I rolled back my environment to 4.15, did a manual seed of
systemvmtemplate-4.16.0-kvm.qcow2.bz2, built the packages without
"noredist" and tried to upgrade the environment to 4.16. Then, ACS
automatically tried to upgrade the systemvm-templates; even though the
system VM template for 4.16 is already there, it throws some errors:
2021-10-19 23:08:02,684 DEBUG [c.c.u.d.Upgrade41520to41600] (main:null)
(logid:) Updating System Vm template IDs
2021-10-19 23:08:02,904 DEBUG [c.c.u.SystemVmTemplateRegistration]
(main:null) (logid:) Updating System Vm template IDs
2021-10-19 23:08:02,910 DEBUG [c.c.u.SystemVmTemplateRegistration]
(main:null) (logid:) Updating VMware System Vms
2021-10-19 23:08:02,915 WARN [c.c.u.SystemVmTemplateRegistration]
(main:null) (logid:) 4.16.0 VMware SystemVm template not found. Cannot
upgrade system Vms hypervisor is not used, so not failing upgrade
2021-10-19 23:08:02,962 DEBUG [c.c.u.SystemVmTemplateRegistration]
(main:null) (logid:) Updating KVM System Vms
2021-10-19 23:08:02,966 INFO [c.c.u.SystemVmTemplateRegistration]
(main:null) (logid:) Grabbing lock to register templates.
2021-10-19 23:08:02,976 ERROR [c.c.u.SystemVmTemplateRegistration]
(main:null) (logid:) Failed to calculate template checksum
java.nio.file.NoSuchFileException:
/usr/share/cloudstack-management/templates/systemvm/systemvmtemplate-4.16.0-kvm.qcow2.bz2
at
java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at
java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219)
at java.base/java.nio.file.Files.newByteChannel(Files.java:371)
at java.base/java.nio.file.Files.newByteChannel(Files.java:422)
at
java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420)
at java.base/java.nio.file.Files.newInputStream(Files.java:156)
at
com.cloud.upgrade.SystemVmTemplateRegistration.calculateChecksum(SystemVmTemplateRegistration.java:330)
at
com.cloud.upgrade.SystemVmTemplateRegistration.validateTemplates(SystemVmTemplateRegistration.java:690)
at
com.cloud.upgrade.SystemVmTemplateRegistration.registerTemplates(SystemVmTemplateRegistration.java:713)
at
com.cloud.upgrade.SystemVmTemplateRegistration$5.doInTransactionWithoutResult(SystemVmTemplateRegistration.java:824)
at
com.cloud.utils.db.TransactionCallbackNoReturn.doInTransaction(TransactionCallbackNoReturn.java:25)
at
com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:50)
at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
at com.cloud.utils.db.Transaction.execute(Transaction.java:47)
at
com.cloud.upgrade.SystemVmTemplateRegistration.updateSystemVmTemplates(SystemVmTemplateRegistration.java:802)
at
com.cloud.upgrade.dao.Upgrade41520to41600.updateSystemVmTemplates(Upgrade41520to41600.java:105)
at
com.cloud.upgrade.DatabaseUpgradeChecker.updateSystemVmTemplates(DatabaseUpgradeChecker.java:258)
at
com.cloud.upgrade.DatabaseUpgradeChecker.upgrade(DatabaseUpgradeChecker.java:347)
at
com.cloud.upgrade.DatabaseUpgradeChecker.check(DatabaseUpgradeChecker.java:384)
...
2021-10-19 23:08:02,994 ERROR [c.c.u.SystemVmTemplateRegistration]
(main:null) (logid:) updateSystemVmTemplates:Exception while getting ids
of templates
com.cloud.utils.exception.CloudRuntimeException: 4.16.0 KVM SystemVm
template not found. Cannot upgrade system Vms
at
com.cloud.upgrade.SystemVmTemplateRegistration$5.doInTransactionWithoutResult(SystemVmTemplateRegistration.java:827)
at
com.cloud.utils.db.TransactionCallbackNoReturn.doInTransaction(TransactionCallbackNoReturn.java:25)
at
com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:50)
at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
at com.cloud.utils.db.Transaction.execute(Transaction.java:47)
...
Due to these points, my vote for this RC is -1.
My proposals to work around these issues are the following:
- Separate the build/downloading of systemVM template with different
profiles to be activated, for instance, "systemVmTemplateXenServer" (for
people that want XenServer system VM template with the auto seed
process), and so on for the other hypervisors.
- Allow people to pre-seed the system VM template for the upgrade.
Therefore, one can generate ACS packages without the system VM
templates, and then, during the upgrade, if the system VM template is
already there, the auto seed code does not need to do anything.
I can work on both of these issues if you all agree to address this
situation with the proposals I described.
Best regards,
Daniel Salvador
On 18/10/2021 15:25, Nicolas Vazquez wrote:
Hi All,
I have created a 4.16.0.0 release (RC1) with the following artefacts up for
testing and a vote:
Git Branch and Commit SH:
https://github.com/apache/cloudstack/tree/4.16.0.0-RC20211018T1454
Commit: 506f8a8269551acb9b07b029da0f9e4fe858d150
Source release (checksums and signatures are available at the same location):
https://dist.apache.org/repos/dist/dev/cloudstack/4.16.0.0/
PGP release keys (signed using 656E1BCC8CB54F84):
https://dist.apache.org/repos/dist/release/cloudstack/KEYS
The vote will be open until 21st October 2021, 21.00 CET (72h).
For sanity in tallying the vote, can PMC members please be sure to indicate
"(binding)" with their vote?
[ ] +1 approve
[ ] +0 no opinion
[ ] -1 disapprove (and reason why)
4.16.0.0 System VM templates are available from here:
https://download.cloudstack.org/systemvm/4.16/
Regards,
Nicolas Vazquez
