> -----Original Message-----
> From: williamstev...@gmail.com [mailto:williamstev...@gmail.com] On
> Behalf Of Will Stevens
> Sent: Tuesday, December 02, 2014 8:06 AM
> To: dev@cloudstack.apache.org
> Subject: Re: 05bec59c - kvm, qcow, systemvm qemu-img -o compat
>
> I will do some testing and see what I can come up with.
>
> BTW, I would be doing it the reverse of what you said. I would try to specify
> the 'compat=0.10' option initially and if that is OK, then we are good. If
> it fails
> (it is currently failing on the 4.5 branch) this is most likely because the
> version
> of qemu-img does not support the 'compat' option being passed to the
> convert call (which is what is happening right now on our build systems). In
> that case, we will want to run the command without the compat option
> because if it is too old to support the 'compat' option then it will have the
> compat=0.10 default.
Seems the qemu version is a little bit of mess, I think above method is good
enough.
BTW, thanks to look into the issue...
>
> Basically, I will try it the new way that Edison introduced. If that fails,
> I will fall
> back to the way it was being done before Edison's change...
>
> Will
>
>
> *Will STEVENS*
> Lead Developer
>
> *CloudOps* *| *Cloud Solutions Experts
> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw
> @CloudOps_
>
> On Tue, Dec 2, 2014 at 10:53 AM, Rohit Yadav <rohit.ya...@shapeblue.com>
> wrote:
>
> > Hi Will,
> >
> > Reply in-line;
> >
> > On Tuesday 02 December 2014 08:41 PM, Will Stevens wrote:
> >
> >> @Rohit: If the script is run with 'set -e' it should not be a
> >> problem because I was planning to do an explicit 'set +e' before the
> >> call and then a 'set -e' after the call. This should handle this case, no?
> >>
> >
> > Yes, I think that will work as well. If we are not sure about qemu
> > version, it should be alright to try once without compat and check $?
> > and retry again with compat option. The only corner case I can think
> > of is when qemu-img genuinely exits with a non-zero exit code during
> > image conversion.
> >
> >
> >> @Nux: When you say "EL7 comes with qemu-img 1.5.3 and it requires the
> >> compat option", what do you mean by that? Are you saying that if you
> >> run the build system vms job on EL7, that you have to specify the
> >> compat option even though it comes with qemu-img 1.5.3? That should
> >> not be the case because the default compat flag should not have been
> >> changed till 1.7. If you are talking about having a KVM host which
> >> runs EL7 and has a qemu-img version of 1.5.3, then that is not really
> >> relevant for this bash script.
> >> The version of qemu-img which this script checking is the version on
> >> the build system, not on the KVM host. It is validating if the
> >> compat option needs to be passed to the 'qemu-img convert' command in
> >> order for the image to have the 'compat=0.10' option set. We are not
> >> debating that the 'compat=0.10' option needs to be set, that is for
> >> sure. We are determining when it needs to be explicitly set on the
> >> build systems.
> >>
> >> In short...
> >> Jenkins kicks off a job to build the system vms. This build system
> >> could have one of many different qemu-img versions. When this script
> >> runs on the build system, we need to determine which command needs
> to
> >> be run in order to output a 'compat=0.10' compatible image.
> >>
> >> I will do some testing with Edison's suggestion and see if I can come
> >> up with a solution that way as well and then we can pick which we
> >> want to implement.
> >>
> >> Cheers,
> >>
> >> Will
> >>
> >>
> >> *Will STEVENS*
> >> Lead Developer
> >>
> >> *CloudOps* *| *Cloud Solutions Experts
> >>
> >> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|*
> tw
> >> @CloudOps_
> >>
> >> On Tue, Dec 2, 2014 at 3:32 AM, Nux! <n...@li.nux.ro> wrote:
> >>
> >> As I said above EL7 comes with qemu-img 1.5.3 and it requires the
> >> compat
> >>> option, so it's not only 1.7+.
> >>> Personally I'm in favour of Will's approach.
> >>>
> >>> Lucian
> >>>
> >>> --
> >>> Sent from the Delta quadrant using Borg technology!
> >>>
> >>> Nux!
> >>> www.nux.ro
> >>>
> >>> ----- Original Message -----
> >>>
> >>>> From: "Rohit Yadav" <bhais...@apache.org>
> >>>> To: "Will Stevens" <wstev...@cloudops.com>
> >>>> Cc: "Edison Su" <edison...@citrix.com>, "Wilder Rodrigues" <
> >>>>
> >>> wrodrig...@schubergphilis.com>, dev@cloudstack.apache.org,
> >>>
> >>>> "Leo Simons" <lsim...@schubergphilis.com>
> >>>> Sent: Tuesday, 2 December, 2014 07:36:21
> >>>> Subject: Re: 05bec59c - kvm, qcow, systemvm qemu-img -o compat
> >>>>
> >>>
> >>> Hi Will,
> >>>>
> >>>> While we can use your first solution that checks exit status ($?)
> >>>> but in case some is running the script with a "set -e", the script
> >>>> will simply break when $? is not zero. I think if we know above
> >>>> which qemu-img
> >>>>
> >>> version
> >>>
> >>>> we should we the compat option (1.7?) than that's great we should
> >>>> we
> >>>>
> >>> that.
> >>>
> >>>> Thanks for looking into this.
> >>>>
> >>>> Regards.
> >>>>
> >>>> On Tue, Dec 2, 2014 at 8:01 AM, Will Stevens
> >>>> <wstev...@cloudops.com>
> >>>>
> >>> wrote:
> >>>
> >>>>
> >>>> Thanks for that. Does that mean that we prefer to do an if else
> >>>> based
> >>>>> on
> >>>>> version rather than basically doing a try catch?
> >>>>>
> >>>>> I have not been involved in previous bash scripting for CS, so I
> >>>>> am willing to fall in line with the preferred method. What I
> >>>>> proposed was based on how similar problems were solved in other
> >>>>> code in the same
> >>>>>
> >>>> file.
> >>>
> >>>>
> >>>>> If I check for version, do you agree that I should check for 1.7
> >>>>> as the version since that is the version where the compact default
> changed?
> >>>>>
> >>>>> I will try to get this resolved tomorrow.
> >>>>>
> >>>>> Cheers,
> >>>>>
> >>>>> Will
> >>>>> On Dec 1, 2014 9:16 PM, "Edison Su" <edison...@citrix.com> wrote:
> >>>>>
> >>>>> qemu-img |head -n 1|awk '{print $3}'
> >>>>>>
> >>>>>> should show the version of qemu-img
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> *From:* williamstev...@gmail.com
> >>>>>> [mailto:williamstev...@gmail.com] *On Behalf Of *Will Stevens
> >>>>>> *Sent:* Monday, December 01, 2014 2:17 PM
> >>>>>> *To:* dev@cloudstack.apache.org
> >>>>>> *Cc:* Leo Simons; Wilder Rodrigues; Rohit Yadav; Edison Su
> >>>>>> *Subject:* Re: 05bec59c - kvm, qcow, systemvm qemu-img -o
> compat
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Alright, so far I have found the following:
> >>>>>>
> >>>>>>
> >>>>>>
> https://github.com/qemu/qemu/commit/9117b47717ad208b12786ce88eacb
> >>>>>> 0
> >>> 13f9b3dd1c
> >>>
> >>>>
> >>>>>>
> >>>>>>
> >>>>>> Basically, if the qemu-img version is less than 1.7, we should
> >>>>>> run the non 'compat' option version and if the version is 1.7 or
> >>>>>> greater, we
> >>>>>>
> >>>>> should
> >>>
> >>>> run the new command with the 'compat' version.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Unfortunately I am not able to find a way to get the qemu-img
> >>>>>> version from the command line.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> I am looking to basically add a conditional to try running with
> >>>>>> the compat option and if that fails, then run it without the
> >>>>>> compat option.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Basically, I would be replacing this:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> qemu-img convert -o compat=0.10 -f raw -c -O qcow2 raw.img
> >>>>>> "${appliance_build_name}-kvm.qcow2"
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> With this:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> set +e
> >>>>>>
> >>>>>> qemu-img convert -o compat=0.10 -f raw -c -O qcow2 raw.img
> >>>>>> "${appliance_build_name}-kvm.qcow2"
> >>>>>>
> >>>>>> local qemuresult=$?
> >>>>>>
> >>>>>> set -e
> >>>>>>
> >>>>>> if [ ${qemuresult} != 0 ]; then
> >>>>>>
> >>>>>> log INFO "qemu-img convert failed, trying without compat option"
> >>>>>>
> >>>>>> qemu-img convert -f raw -c -O qcow2 raw.img
> >>>>>> "${appliance_build_name}-kvm.qcow2"
> >>>>>>
> >>>>>> fi
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> What do you guys think? Is this a good enough solution? If you
> >>>>>> guys agree I will implement it in master and make sure it works,
> >>>>>> then we can merge the change back to 4.5 to fix that branch as
> >>>>>> well.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Let me know if you have issues with this approach...
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Cheers,
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Will
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> *Will STEVENS*
> >>>>>>
> >>>>>> Lead Developer
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> *CloudOps | *Cloud Solutions Experts
> >>>>>>
> >>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> >>>>>>
> >>>>>> w cloudops.com *|* tw @CloudOps_
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Mon, Dec 1, 2014 at 4:06 PM, Will Stevens
> >>>>>> <wstev...@cloudops.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>> Edison,
> >>>>>>
> >>>>>> According to this page the default compat option is 0.10:
> >>>>>> http://manpages.ubuntu.com/manpages/trusty/man1/qemu-
> img.1.html
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Did you find that to not be the case and is that why you had to
> >>>>>> add the compat option?
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> BTW, in an attempt to get the master system vms building again, I
> >>>>>> committed a change to master to remove the compat option. We
> now
> >>>>>> have master system vms building correctly, but Rohit rightly
> >>>>>> pointed out
> >>>>>>
> >>>>> that I
> >>>
> >>>> had basically reverted your change.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> I created a Jira ticket for this issue:
> >>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-7959
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> I have not reverted my change in master at this point, so it is
> >>>>>>
> >>>>> building
> >>>
> >>>> right now, but I also did not make the change to 4.5, so that
> >>>> branch is
> >>>>>> currently failing to build system vms.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> I will see if I can track down in which version of qemu the
> >>>>>> compat
> >>>>>>
> >>>>> option
> >>>
> >>>> was added so we can add some intelligent logic around this command.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Will
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> *Will STEVENS*
> >>>>>>
> >>>>>> Lead Developer
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> *CloudOps | *Cloud Solutions Experts
> >>>>>>
> >>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> >>>>>>
> >>>>>> w cloudops.com *|* tw @CloudOps_
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Thu, Nov 20, 2014 at 1:07 PM, edison su <sudi...@gmail.com>
> wrote:
> >>>>>>
> >>>>>> Hi Leo,
> >>>>>> Our new internal build system are using ubuntu 14.04 or something
> >>>>>> like that, which has qemu 1.x installed by default, that's why I
> >>>>>> added the "compat" option in the build script, otherwise, the
> >>>>>> image build by qemu 1.x, won't work on RHEL 6.x.
> >>>>>> The fix would be, in the build script, check the version of
> >>>>>> qemu-img, if it's 0.x, then don't add "compat" option. There are
> >>>>>> a lot of people still using RHEL 6.x as KVM hypervisor, we have
> >>>>>> to make sure the image we build can still work on these machines.
> >>>>>>
> >>>>>>
> >>>>>> On Thu, Nov 20, 2014 at 6:49 AM, Leo Simons <
> >>>>>>
> >>>>> lsim...@schubergphilis.com>
> >>>
> >>>> wrote:
> >>>>>>
> >>>>>>> Hey Edison, all,
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>> https://github.com/apache/cloudstack/commit/
> >>> 05bec59c1498dbcfb8a1089c86855fd3b433ea58
> >>>
> >>>>
> >>>>>>> breaks our internal build with
> >>>>>>>
> >>>>>>> + qemu-img convert -o compat=0.10 -f raw -c -O qcow2 raw.img
> \
> >>>>>>> systemvmtemplate-systemvm-persistent-config-4.5.0.124-
> >>>>>>> kvm.qcow2
> >>>>>>> Unknown option 'compat'
> >>>>>>> qemu-img: Invalid options for file format 'qcow2'.
> >>>>>>>
> >>>>>>> This is on CentOS release 6.6, which has
> >>>>>>>
> >>>>>> qemu-img-0.15.0-1.el6.rfx.x86_64.
> >>>>>>
> >>>>>>>
> >>>>>>> Based on
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>> http://secure-
> web.cisco.com/1F5zlT8tpWX5PfbJbuzUoso6OWYSnPdIEm5Ue
> >>>>>> 6-
> >>> gXqJ1a69oLY46nrS2BKoFjZFPlr0DmQwfZHdm2XjfkVhbuMFrET2ilAqQyrI
> >>> 76EYyuxNVnTHaRHH_4bMCtgrtgasDwW1zTJWKDQp8d-
> >>>
> HuwyPYokzHAiPFGp7w1CyCabrBpkEw/http%3A%2F%2Fwiki.qemu.org%2FOl
> derNew
> >>> s
> >>>
> >>>> that seems like it is a really old qemu (august 2011).
> >>>>>>>
> >>>>>>> I'm guessing you have a newer OS / newer qemu? Can you please
> >>>>>>> let me
> >>>>>>>
> >>>>>> know what OS, OS version and qemu(-img) version you are using?
> >>>>>>
> >>>>>>>
> >>>>>>> Also, does anyone know if there some minimum version of qemu-
> img
> >>>>>>> that
> >>>>>>>
> >>>>>> should be used / cloudstack assumes? Is 0.15 still ok to do an
> >>>>>>
> >>>>> acceptable
> >>>
> >>>> image conversion with? (we currently don't have any kvm use
> >>>> ourselves,
> >>>>>>
> >>>>> but,
> >>>
> >>>> I'd like for our build infra to produce useful kvm images nonetheless).
> >>>>>>
> >>>>>>>
> >>>>>>> According to
> >>>>>>>
> >>>>>>>
> >>>>>> http://secure-
> web.cisco.com/1pFLbDg2BMTb54NUG5_lYr4mAwFcn9tHP-
> >>>
> Weq28Sj5SFqgRKker_DkUKLf6RdwVXHI66uQQNVvD74D6rs6Pc0srpLX0Ejh6n
> W6o9mB
> >>> Z-
> >>> AhQzygMrB_OfUQNLtsB-myd-CA0rJowOhs8ZTsIVr27mgJM4v6ay93
> >>> D_64JLEBebrlnM/http%3A%2F%2Fwiki.qemu.org%2FChangeLog%2F1.1
> >>>
> >>>> the -o compat switch was introduced in 1.1.
> >>>>>>>
> >>>>>>> According to
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> https://github.com/qemu/qemu/commit/9117b47717ad208b12786ce88eacb
> >>>>>> 0
> >>> 13f9b3dd1c
> >>>
> >>>> the default format was changed from 0.10 to 1.1 in qemu 1.7 and
> >>>>>>>
> >>>>>> onwards.
> >>>
> >>>>
> >>>>>>> The libvirt people
> >>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=997977
> >>>>>>> say they pass -o compat when qemu supports it (so when v >= 1.1
> >>>>>>> I
> >>>>>>>
> >>>>>> imagine).
> >>>>>>
> >>>>>>>
> >>>>>>> I think we should do the same in the build script and I'll make
> >>>>>>> a
> >>>>>>>
> >>>>>> patch.
> >>>
> >>>>
> >>>>>>> But, should we publish newer-format images too? According to
> >>>>>>>
> >>>>>>>
> >>>>>> http://secure-web.cisco.com/193Tci801hAyVHNqtfHy56-
> >>> aPW58HHhzwlnmfvPZQnrbtHxURQC0m3kDHxk-
> PlYYLhCZX0f768ZYK2fV1WbMFpQPz5
> >>> JfmFt0S8cuzCSrykjOEx9isfBwORZu6I4XdU3jo-WdJcDgtoH-
> >>> KOwKcGRX5TGZrVva_BDOYDQzF-7QAg8w/http%3A%2F%2Fwiki.qemu.
> >>> org%2FFeatures%2FQcow3
> >>>
> >>>> the new format is much better so I imagine qcow/kvm users will
> >>>> really
> >>>>>>>
> >>>>>> appreciate the newer formats.
> >>>>>>
> >>>>>>>
> >>>>>>> Thoughts?
> >>>>>>>
> >>>>>>>
> >>>>>>> cheers,
> >>>>>>>
> >>>>>>>
> >>>>>>> Leo
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Best Regards,
> >>>>>> Edison
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>
> >>
> > --
> > Regards,
> > Rohit Yadav
> > Software Architect, ShapeBlue
> > M. +41 779015219 | rohit.ya...@shapeblue.com
> >
> > Blog: bhaisaab.org | Twitter: @_bhaisaab Find out more about ShapeBlue
> > and our range of CloudStack related services
> >
> > IaaS Cloud Design & Build<http://shapeblue.com/
> > iaas-cloud-design-and-build//> CSForge – rapid IaaS deployment
> > framework<http://shapeblue.com/csforge/>
> > CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
> > CloudStack Software
> > Engineering<http://shapeblue.com/cloudstack-software-
> > engineering/>
> > CloudStack Infrastructure Support<http://shapeblue.com/
> > cloudstack-infrastructure-support/>
> > CloudStack Bootcamp Training Courses<http://shapeblue.com/
> > cloudstack-training/>
> >
> > This email and any attachments to it may be confidential and are
> > intended solely for the use of the individual to whom it is addressed.
> > Any views or opinions expressed are solely those of the author and do
> > not necessarily represent those of Shape Blue Ltd or related
> > companies. If you are not the intended recipient of this email, you
> > must neither take any action based upon its contents, nor copy or show
> > it to anyone. Please contact the sender if you believe you have
> > received this email in error. Shape Blue Ltd is a company incorporated
> > in England & Wales. ShapeBlue Services India LLP is a company
> > incorporated in India and is operated under license from Shape Blue
> > Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in
> > Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA
> > Pty Ltd is a company registered by The Republic of South Africa and is
> traded under license from Shape Blue Ltd. ShapeBlue is a registered
> trademark.
> >
