Interesting. Thanks for that...
So it seems that if you run the build system on a EL7 the version check
approach won't even help us because they have changed the default behavior
of the installed qemu-img version to make it default to '1.1'.
In light of this and considering what a PITA it is to compare version
numbers in bash, I will solve the problem with my original code. I will do
some testing on a local system to work out the kinks and try to validate
that Rohit's concerns are addressed.
I will report back later today.
Thanks for the help on this Nux... :)
Cheers,
Will
*Will STEVENS*
Lead Developer
*CloudOps* *| *Cloud Solutions Experts
420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
w cloudops.com *|* tw @CloudOps_
On Tue, Dec 2, 2014 at 10:27 AM, Nux! <n...@li.nux.ro> wrote:
> Will, I might have gotten lost in the details, apologies if it's the case.
>
> What I wanted to point out is that the qemu-img in EL7 (v1.5.3 which is
> _lower_ than 1.7) will by default produce an image with compat = 1.1 which
> older qemu-kvm will not be able to use, such as the one in CentOS 6.
> So if you are running a systemvm build job on EL7 you should be aware of
> this. That's why I preferred your script, because it doesn't just rely on
> version numbers.
> I myself am not running any systemvm build jobs.
>
> qemu-img create -f raw test.raw 1G
> qemu-img convert -f raw test.raw -O qcow2 test.qcow2
> qemu-img info test.qcow2
> image: test.qcow2
> file format: qcow2
> virtual size: 1.0G (1073741824 bytes)
> disk size: 196K
> cluster_size: 65536
> Format specific information:
> compat: 1.1
> lazy refcounts: false
>
>
>
> --
> Sent from the Delta quadrant using Borg technology!
>
> Nux!
> www.nux.ro
>
> ----- Original Message -----
> > From: "Will Stevens" <wstev...@cloudops.com>
> > To: "Nux!" <n...@li.nux.ro>
> > Cc: dev@cloudstack.apache.org, "Edison Su" <edison...@citrix.com>,
> "Wilder Rodrigues" <wrodrig...@schubergphilis.com>,
> > "Leo Simons" <lsim...@schubergphilis.com>
> > Sent: Tuesday, 2 December, 2014 15:11:40
> > Subject: Re: 05bec59c - kvm, qcow, systemvm qemu-img -o compat
>
> > @Rohit: If the script is run with 'set -e' it should not be a problem
> > because I was planning to do an explicit 'set +e' before the call and
> then
> > a 'set -e' after the call. This should handle this case, no?
> >
> > @Nux: When you say "EL7 comes with qemu-img 1.5.3 and it requires the
> > compat option", what do you mean by that? Are you saying that if you run
> > the build system vms job on EL7, that you have to specify the compat
> option
> > even though it comes with qemu-img 1.5.3? That should not be the case
> > because the default compat flag should not have been changed till 1.7.
> If
> > you are talking about having a KVM host which runs EL7 and has a qemu-img
> > version of 1.5.3, then that is not really relevant for this bash script.
> > The version of qemu-img which this script checking is the version on the
> > build system, not on the KVM host. It is validating if the compat option
> > needs to be passed to the 'qemu-img convert' command in order for the
> image
> > to have the 'compat=0.10' option set. We are not debating that the
> > 'compat=0.10' option needs to be set, that is for sure. We are
> determining
> > when it needs to be explicitly set on the build systems.
> >
> > In short...
> > Jenkins kicks off a job to build the system vms. This build system could
> > have one of many different qemu-img versions. When this script runs on
> the
> > build system, we need to determine which command needs to be run in order
> > to output a 'compat=0.10' compatible image.
> >
> > I will do some testing with Edison's suggestion and see if I can come up
> > with a solution that way as well and then we can pick which we want to
> > implement.
> >
> > Cheers,
> >
> > Will
> >
> >
> > *Will STEVENS*
> > Lead Developer
> >
> > *CloudOps* *| *Cloud Solutions Experts
> > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > w cloudops.com *|* tw @CloudOps_
> >
> > On Tue, Dec 2, 2014 at 3:32 AM, Nux! <n...@li.nux.ro> wrote:
> >
> >> As I said above EL7 comes with qemu-img 1.5.3 and it requires the compat
> >> option, so it's not only 1.7+.
> >> Personally I'm in favour of Will's approach.
> >>
> >> Lucian
> >>
> >> --
> >> Sent from the Delta quadrant using Borg technology!
> >>
> >> Nux!
> >> www.nux.ro
> >>
> >> ----- Original Message -----
> >> > From: "Rohit Yadav" <bhais...@apache.org>
> >> > To: "Will Stevens" <wstev...@cloudops.com>
> >> > Cc: "Edison Su" <edison...@citrix.com>, "Wilder Rodrigues" <
> >> wrodrig...@schubergphilis.com>, dev@cloudstack.apache.org,
> >> > "Leo Simons" <lsim...@schubergphilis.com>
> >> > Sent: Tuesday, 2 December, 2014 07:36:21
> >> > Subject: Re: 05bec59c - kvm, qcow, systemvm qemu-img -o compat
> >>
> >> > Hi Will,
> >> >
> >> > While we can use your first solution that checks exit status ($?) but
> in
> >> > case some is running the script with a "set -e", the script will
> simply
> >> > break when $? is not zero. I think if we know above which qemu-img
> >> version
> >> > we should we the compat option (1.7?) than that's great we should we
> >> that.
> >> > Thanks for looking into this.
> >> >
> >> > Regards.
> >> >
> >> > On Tue, Dec 2, 2014 at 8:01 AM, Will Stevens <wstev...@cloudops.com>
> >> wrote:
> >> >
> >> >> Thanks for that. Does that mean that we prefer to do an if else
> based on
> >> >> version rather than basically doing a try catch?
> >> >>
> >> >> I have not been involved in previous bash scripting for CS, so I am
> >> >> willing to fall in line with the preferred method. What I proposed
> was
> >> >> based on how similar problems were solved in other code in the same
> >> file.
> >> >>
> >> >> If I check for version, do you agree that I should check for 1.7 as
> the
> >> >> version since that is the version where the compact default changed?
> >> >>
> >> >> I will try to get this resolved tomorrow.
> >> >>
> >> >> Cheers,
> >> >>
> >> >> Will
> >> >> On Dec 1, 2014 9:16 PM, "Edison Su" <edison...@citrix.com> wrote:
> >> >>
> >> >>> qemu-img |head -n 1|awk '{print $3}'
> >> >>>
> >> >>> should show the version of qemu-img
> >> >>>
> >> >>>
> >> >>>
> >> >>> *From:* williamstev...@gmail.com [mailto:williamstev...@gmail.com]
> *On
> >> >>> Behalf Of *Will Stevens
> >> >>> *Sent:* Monday, December 01, 2014 2:17 PM
> >> >>> *To:* dev@cloudstack.apache.org
> >> >>> *Cc:* Leo Simons; Wilder Rodrigues; Rohit Yadav; Edison Su
> >> >>> *Subject:* Re: 05bec59c - kvm, qcow, systemvm qemu-img -o compat
> >> >>>
> >> >>>
> >> >>>
> >> >>> Alright, so far I have found the following:
> >> >>>
> >>
> https://github.com/qemu/qemu/commit/9117b47717ad208b12786ce88eacb013f9b3dd1c
> >> >>>
> >> >>>
> >> >>>
> >> >>> Basically, if the qemu-img version is less than 1.7, we should run
> the
> >> >>> non 'compat' option version and if the version is 1.7 or greater, we
> >> should
> >> >>> run the new command with the 'compat' version.
> >> >>>
> >> >>>
> >> >>>
> >> >>> Unfortunately I am not able to find a way to get the qemu-img
> version
> >> >>> from the command line.
> >> >>>
> >> >>>
> >> >>>
> >> >>> I am looking to basically add a conditional to try running with the
> >> >>> compat option and if that fails, then run it without the compat
> option.
> >> >>>
> >> >>>
> >> >>>
> >> >>> Basically, I would be replacing this:
> >> >>>
> >> >>>
> >> >>>
> >> >>> qemu-img convert -o compat=0.10 -f raw -c -O qcow2 raw.img
> >> >>> "${appliance_build_name}-kvm.qcow2"
> >> >>>
> >> >>>
> >> >>>
> >> >>> With this:
> >> >>>
> >> >>>
> >> >>>
> >> >>> set +e
> >> >>>
> >> >>> qemu-img convert -o compat=0.10 -f raw -c -O qcow2 raw.img
> >> >>> "${appliance_build_name}-kvm.qcow2"
> >> >>>
> >> >>> local qemuresult=$?
> >> >>>
> >> >>> set -e
> >> >>>
> >> >>> if [ ${qemuresult} != 0 ]; then
> >> >>>
> >> >>> log INFO "qemu-img convert failed, trying without compat option"
> >> >>>
> >> >>> qemu-img convert -f raw -c -O qcow2 raw.img
> >> >>> "${appliance_build_name}-kvm.qcow2"
> >> >>>
> >> >>> fi
> >> >>>
> >> >>>
> >> >>>
> >> >>> What do you guys think? Is this a good enough solution? If you
> guys
> >> >>> agree I will implement it in master and make sure it works, then we
> can
> >> >>> merge the change back to 4.5 to fix that branch as well.
> >> >>>
> >> >>>
> >> >>>
> >> >>> Let me know if you have issues with this approach...
> >> >>>
> >> >>>
> >> >>>
> >> >>> Cheers,
> >> >>>
> >> >>>
> >> >>>
> >> >>> Will
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>> *Will STEVENS*
> >> >>>
> >> >>> Lead Developer
> >> >>>
> >> >>>
> >> >>>
> >> >>> *CloudOps | *Cloud Solutions Experts
> >> >>>
> >> >>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> >> >>>
> >> >>> w cloudops.com *|* tw @CloudOps_
> >> >>>
> >> >>>
> >> >>>
> >> >>> On Mon, Dec 1, 2014 at 4:06 PM, Will Stevens <wstev...@cloudops.com
> >
> >> >>> wrote:
> >> >>>
> >> >>> Edison,
> >> >>>
> >> >>> According to this page the default compat option is 0.10:
> >> >>> http://manpages.ubuntu.com/manpages/trusty/man1/qemu-img.1.html
> >> >>>
> >> >>>
> >> >>>
> >> >>> Did you find that to not be the case and is that why you had to add
> the
> >> >>> compat option?
> >> >>>
> >> >>>
> >> >>>
> >> >>> BTW, in an attempt to get the master system vms building again, I
> >> >>> committed a change to master to remove the compat option. We now
> have
> >> >>> master system vms building correctly, but Rohit rightly pointed out
> >> that I
> >> >>> had basically reverted your change.
> >> >>>
> >> >>>
> >> >>>
> >> >>> I created a Jira ticket for this issue:
> >> >>> https://issues.apache.org/jira/browse/CLOUDSTACK-7959
> >> >>>
> >> >>>
> >> >>>
> >> >>> I have not reverted my change in master at this point, so it is
> >> building
> >> >>> right now, but I also did not make the change to 4.5, so that
> branch is
> >> >>> currently failing to build system vms.
> >> >>>
> >> >>>
> >> >>>
> >> >>> I will see if I can track down in which version of qemu the compat
> >> option
> >> >>> was added so we can add some intelligent logic around this command.
> >> >>>
> >> >>>
> >> >>>
> >> >>> Will
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>> *Will STEVENS*
> >> >>>
> >> >>> Lead Developer
> >> >>>
> >> >>>
> >> >>>
> >> >>> *CloudOps | *Cloud Solutions Experts
> >> >>>
> >> >>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> >> >>>
> >> >>> w cloudops.com *|* tw @CloudOps_
> >> >>>
> >> >>>
> >> >>>
> >> >>> On Thu, Nov 20, 2014 at 1:07 PM, edison su <sudi...@gmail.com>
> wrote:
> >> >>>
> >> >>> Hi Leo,
> >> >>> Our new internal build system are using ubuntu 14.04 or something
> like
> >> >>> that, which has qemu 1.x installed by default, that's why I added
> the
> >> >>> "compat" option in the build script, otherwise, the image build by
> >> >>> qemu 1.x, won't work on RHEL 6.x.
> >> >>> The fix would be, in the build script, check the version of
> qemu-img,
> >> >>> if it's 0.x, then don't add "compat" option. There are a lot of
> people
> >> >>> still using RHEL 6.x as KVM hypervisor, we have to make sure the
> image
> >> >>> we build can still work on these machines.
> >> >>>
> >> >>>
> >> >>> On Thu, Nov 20, 2014 at 6:49 AM, Leo Simons <
> >> lsim...@schubergphilis.com>
> >> >>> wrote:
> >> >>> > Hey Edison, all,
> >> >>> >
> >> >>> >
> >> >>>
> >>
> https://github.com/apache/cloudstack/commit/05bec59c1498dbcfb8a1089c86855fd3b433ea58
> >> >>> >
> >> >>> > breaks our internal build with
> >> >>> >
> >> >>> > + qemu-img convert -o compat=0.10 -f raw -c -O qcow2 raw.img \
> >> >>> >
> systemvmtemplate-systemvm-persistent-config-4.5.0.124-kvm.qcow2
> >> >>> > Unknown option 'compat'
> >> >>> > qemu-img: Invalid options for file format 'qcow2'.
> >> >>> >
> >> >>> > This is on CentOS release 6.6, which has
> >> >>> qemu-img-0.15.0-1.el6.rfx.x86_64.
> >> >>> >
> >> >>> > Based on
> >> >>> >
> >> >>>
> >>
> http://secure-web.cisco.com/1F5zlT8tpWX5PfbJbuzUoso6OWYSnPdIEm5Ue6-gXqJ1a69oLY46nrS2BKoFjZFPlr0DmQwfZHdm2XjfkVhbuMFrET2ilAqQyrI76EYyuxNVnTHaRHH_4bMCtgrtgasDwW1zTJWKDQp8d-HuwyPYokzHAiPFGp7w1CyCabrBpkEw/http%3A%2F%2Fwiki.qemu.org%2FOlderNews
> >> >>> > that seems like it is a really old qemu (august 2011).
> >> >>> >
> >> >>> > I'm guessing you have a newer OS / newer qemu? Can you please let
> me
> >> >>> know what OS, OS version and qemu(-img) version you are using?
> >> >>> >
> >> >>> > Also, does anyone know if there some minimum version of qemu-img
> that
> >> >>> should be used / cloudstack assumes? Is 0.15 still ok to do an
> >> acceptable
> >> >>> image conversion with? (we currently don't have any kvm use
> ourselves,
> >> but,
> >> >>> I'd like for our build infra to produce useful kvm images
> nonetheless).
> >> >>> >
> >> >>> > According to
> >> >>> >
> >> >>>
> >>
> http://secure-web.cisco.com/1pFLbDg2BMTb54NUG5_lYr4mAwFcn9tHP-Weq28Sj5SFqgRKker_DkUKLf6RdwVXHI66uQQNVvD74D6rs6Pc0srpLX0Ejh6nW6o9mBZ-AhQzygMrB_OfUQNLtsB-myd-CA0rJowOhs8ZTsIVr27mgJM4v6ay93D_64JLEBebrlnM/http%3A%2F%2Fwiki.qemu.org%2FChangeLog%2F1.1
> >> >>> > the -o compat switch was introduced in 1.1.
> >> >>> >
> >> >>> > According to
> >> >>> >
> >> >>>
> >>
> https://github.com/qemu/qemu/commit/9117b47717ad208b12786ce88eacb013f9b3dd1c
> >> >>> > the default format was changed from 0.10 to 1.1 in qemu 1.7 and
> >> onwards.
> >> >>> >
> >> >>> > The libvirt people
> >> >>> > https://bugzilla.redhat.com/show_bug.cgi?id=997977
> >> >>> > say they pass -o compat when qemu supports it (so when v >= 1.1 I
> >> >>> imagine).
> >> >>> >
> >> >>> > I think we should do the same in the build script and I'll make a
> >> patch.
> >> >>> >
> >> >>> > But, should we publish newer-format images too? According to
> >> >>> >
> >> >>>
> >>
> http://secure-web.cisco.com/193Tci801hAyVHNqtfHy56-aPW58HHhzwlnmfvPZQnrbtHxURQC0m3kDHxk-PlYYLhCZX0f768ZYK2fV1WbMFpQPz5JfmFt0S8cuzCSrykjOEx9isfBwORZu6I4XdU3jo-WdJcDgtoH-KOwKcGRX5TGZrVva_BDOYDQzF-7QAg8w/http%3A%2F%2Fwiki.qemu.org%2FFeatures%2FQcow3
> >> >>> > the new format is much better so I imagine qcow/kvm users will
> really
> >> >>> appreciate the newer formats.
> >> >>> >
> >> >>> > Thoughts?
> >> >>> >
> >> >>> >
> >> >>> > cheers,
> >> >>> >
> >> >>> >
> >> >>> > Leo
> >> >>> >
> >> >>>
> >> >>>
> >> >>> --
> >> >>> Best Regards,
> >> >>> Edison
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>>
>
