I will do some testing and see what I can come up with.
BTW, I would be doing it the reverse of what you said. I would try to
specify the 'compat=0.10' option initially and if that is OK, then we are
good. If it fails (it is currently failing on the 4.5 branch) this is most
likely because the version of qemu-img does not support the 'compat' option
being passed to the convert call (which is what is happening right now on
our build systems). In that case, we will want to run the command without
the compat option because if it is too old to support the 'compat' option
then it will have the compat=0.10 default.
Basically, I will try it the new way that Edison introduced. If that
fails, I will fall back to the way it was being done before Edison's
change...
Will
*Will STEVENS*
Lead Developer
*CloudOps* *| *Cloud Solutions Experts
420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
w cloudops.com *|* tw @CloudOps_
On Tue, Dec 2, 2014 at 10:53 AM, Rohit Yadav <rohit.ya...@shapeblue.com>
wrote:
> Hi Will,
>
> Reply in-line;
>
> On Tuesday 02 December 2014 08:41 PM, Will Stevens wrote:
>
>> @Rohit: If the script is run with 'set -e' it should not be a problem
>> because I was planning to do an explicit 'set +e' before the call and then
>> a 'set -e' after the call. This should handle this case, no?
>>
>
> Yes, I think that will work as well. If we are not sure about qemu
> version, it should be alright to try once without compat and check $?
> and retry again with compat option. The only corner case I can think of
> is when qemu-img genuinely exits with a non-zero exit code during image
> conversion.
>
>
>> @Nux: When you say "EL7 comes with qemu-img 1.5.3 and it requires the
>> compat option", what do you mean by that? Are you saying that if you run
>> the build system vms job on EL7, that you have to specify the compat
>> option
>> even though it comes with qemu-img 1.5.3? That should not be the case
>> because the default compat flag should not have been changed till 1.7. If
>> you are talking about having a KVM host which runs EL7 and has a qemu-img
>> version of 1.5.3, then that is not really relevant for this bash script.
>> The version of qemu-img which this script checking is the version on the
>> build system, not on the KVM host. It is validating if the compat option
>> needs to be passed to the 'qemu-img convert' command in order for the
>> image
>> to have the 'compat=0.10' option set. We are not debating that the
>> 'compat=0.10' option needs to be set, that is for sure. We are
>> determining
>> when it needs to be explicitly set on the build systems.
>>
>> In short...
>> Jenkins kicks off a job to build the system vms. This build system could
>> have one of many different qemu-img versions. When this script runs on
>> the
>> build system, we need to determine which command needs to be run in order
>> to output a 'compat=0.10' compatible image.
>>
>> I will do some testing with Edison's suggestion and see if I can come up
>> with a solution that way as well and then we can pick which we want to
>> implement.
>>
>> Cheers,
>>
>> Will
>>
>>
>> *Will STEVENS*
>> Lead Developer
>>
>> *CloudOps* *| *Cloud Solutions Experts
>>
>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
>> w cloudops.com *|* tw @CloudOps_
>>
>> On Tue, Dec 2, 2014 at 3:32 AM, Nux! <n...@li.nux.ro> wrote:
>>
>> As I said above EL7 comes with qemu-img 1.5.3 and it requires the compat
>>> option, so it's not only 1.7+.
>>> Personally I'm in favour of Will's approach.
>>>
>>> Lucian
>>>
>>> --
>>> Sent from the Delta quadrant using Borg technology!
>>>
>>> Nux!
>>> www.nux.ro
>>>
>>> ----- Original Message -----
>>>
>>>> From: "Rohit Yadav" <bhais...@apache.org>
>>>> To: "Will Stevens" <wstev...@cloudops.com>
>>>> Cc: "Edison Su" <edison...@citrix.com>, "Wilder Rodrigues" <
>>>>
>>> wrodrig...@schubergphilis.com>, dev@cloudstack.apache.org,
>>>
>>>> "Leo Simons" <lsim...@schubergphilis.com>
>>>> Sent: Tuesday, 2 December, 2014 07:36:21
>>>> Subject: Re: 05bec59c - kvm, qcow, systemvm qemu-img -o compat
>>>>
>>>
>>> Hi Will,
>>>>
>>>> While we can use your first solution that checks exit status ($?) but in
>>>> case some is running the script with a "set -e", the script will simply
>>>> break when $? is not zero. I think if we know above which qemu-img
>>>>
>>> version
>>>
>>>> we should we the compat option (1.7?) than that's great we should we
>>>>
>>> that.
>>>
>>>> Thanks for looking into this.
>>>>
>>>> Regards.
>>>>
>>>> On Tue, Dec 2, 2014 at 8:01 AM, Will Stevens <wstev...@cloudops.com>
>>>>
>>> wrote:
>>>
>>>>
>>>> Thanks for that. Does that mean that we prefer to do an if else based
>>>>> on
>>>>> version rather than basically doing a try catch?
>>>>>
>>>>> I have not been involved in previous bash scripting for CS, so I am
>>>>> willing to fall in line with the preferred method. What I proposed was
>>>>> based on how similar problems were solved in other code in the same
>>>>>
>>>> file.
>>>
>>>>
>>>>> If I check for version, do you agree that I should check for 1.7 as the
>>>>> version since that is the version where the compact default changed?
>>>>>
>>>>> I will try to get this resolved tomorrow.
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Will
>>>>> On Dec 1, 2014 9:16 PM, "Edison Su" <edison...@citrix.com> wrote:
>>>>>
>>>>> qemu-img |head -n 1|awk '{print $3}'
>>>>>>
>>>>>> should show the version of qemu-img
>>>>>>
>>>>>>
>>>>>>
>>>>>> *From:* williamstev...@gmail.com [mailto:williamstev...@gmail.com]
>>>>>> *On
>>>>>> Behalf Of *Will Stevens
>>>>>> *Sent:* Monday, December 01, 2014 2:17 PM
>>>>>> *To:* dev@cloudstack.apache.org
>>>>>> *Cc:* Leo Simons; Wilder Rodrigues; Rohit Yadav; Edison Su
>>>>>> *Subject:* Re: 05bec59c - kvm, qcow, systemvm qemu-img -o compat
>>>>>>
>>>>>>
>>>>>>
>>>>>> Alright, so far I have found the following:
>>>>>>
>>>>>> https://github.com/qemu/qemu/commit/9117b47717ad208b12786ce88eacb0
>>> 13f9b3dd1c
>>>
>>>>
>>>>>>
>>>>>>
>>>>>> Basically, if the qemu-img version is less than 1.7, we should run the
>>>>>> non 'compat' option version and if the version is 1.7 or greater, we
>>>>>>
>>>>> should
>>>
>>>> run the new command with the 'compat' version.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Unfortunately I am not able to find a way to get the qemu-img version
>>>>>> from the command line.
>>>>>>
>>>>>>
>>>>>>
>>>>>> I am looking to basically add a conditional to try running with the
>>>>>> compat option and if that fails, then run it without the compat
>>>>>> option.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Basically, I would be replacing this:
>>>>>>
>>>>>>
>>>>>>
>>>>>> qemu-img convert -o compat=0.10 -f raw -c -O qcow2 raw.img
>>>>>> "${appliance_build_name}-kvm.qcow2"
>>>>>>
>>>>>>
>>>>>>
>>>>>> With this:
>>>>>>
>>>>>>
>>>>>>
>>>>>> set +e
>>>>>>
>>>>>> qemu-img convert -o compat=0.10 -f raw -c -O qcow2 raw.img
>>>>>> "${appliance_build_name}-kvm.qcow2"
>>>>>>
>>>>>> local qemuresult=$?
>>>>>>
>>>>>> set -e
>>>>>>
>>>>>> if [ ${qemuresult} != 0 ]; then
>>>>>>
>>>>>> log INFO "qemu-img convert failed, trying without compat option"
>>>>>>
>>>>>> qemu-img convert -f raw -c -O qcow2 raw.img
>>>>>> "${appliance_build_name}-kvm.qcow2"
>>>>>>
>>>>>> fi
>>>>>>
>>>>>>
>>>>>>
>>>>>> What do you guys think? Is this a good enough solution? If you guys
>>>>>> agree I will implement it in master and make sure it works, then we
>>>>>> can
>>>>>> merge the change back to 4.5 to fix that branch as well.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Let me know if you have issues with this approach...
>>>>>>
>>>>>>
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>>
>>>>>>
>>>>>> Will
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Will STEVENS*
>>>>>>
>>>>>> Lead Developer
>>>>>>
>>>>>>
>>>>>>
>>>>>> *CloudOps | *Cloud Solutions Experts
>>>>>>
>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
>>>>>>
>>>>>> w cloudops.com *|* tw @CloudOps_
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Dec 1, 2014 at 4:06 PM, Will Stevens <wstev...@cloudops.com>
>>>>>> wrote:
>>>>>>
>>>>>> Edison,
>>>>>>
>>>>>> According to this page the default compat option is 0.10:
>>>>>> http://manpages.ubuntu.com/manpages/trusty/man1/qemu-img.1.html
>>>>>>
>>>>>>
>>>>>>
>>>>>> Did you find that to not be the case and is that why you had to add
>>>>>> the
>>>>>> compat option?
>>>>>>
>>>>>>
>>>>>>
>>>>>> BTW, in an attempt to get the master system vms building again, I
>>>>>> committed a change to master to remove the compat option. We now have
>>>>>> master system vms building correctly, but Rohit rightly pointed out
>>>>>>
>>>>> that I
>>>
>>>> had basically reverted your change.
>>>>>>
>>>>>>
>>>>>>
>>>>>> I created a Jira ticket for this issue:
>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-7959
>>>>>>
>>>>>>
>>>>>>
>>>>>> I have not reverted my change in master at this point, so it is
>>>>>>
>>>>> building
>>>
>>>> right now, but I also did not make the change to 4.5, so that branch is
>>>>>> currently failing to build system vms.
>>>>>>
>>>>>>
>>>>>>
>>>>>> I will see if I can track down in which version of qemu the compat
>>>>>>
>>>>> option
>>>
>>>> was added so we can add some intelligent logic around this command.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Will
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Will STEVENS*
>>>>>>
>>>>>> Lead Developer
>>>>>>
>>>>>>
>>>>>>
>>>>>> *CloudOps | *Cloud Solutions Experts
>>>>>>
>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
>>>>>>
>>>>>> w cloudops.com *|* tw @CloudOps_
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Nov 20, 2014 at 1:07 PM, edison su <sudi...@gmail.com> wrote:
>>>>>>
>>>>>> Hi Leo,
>>>>>> Our new internal build system are using ubuntu 14.04 or something like
>>>>>> that, which has qemu 1.x installed by default, that's why I added the
>>>>>> "compat" option in the build script, otherwise, the image build by
>>>>>> qemu 1.x, won't work on RHEL 6.x.
>>>>>> The fix would be, in the build script, check the version of qemu-img,
>>>>>> if it's 0.x, then don't add "compat" option. There are a lot of people
>>>>>> still using RHEL 6.x as KVM hypervisor, we have to make sure the image
>>>>>> we build can still work on these machines.
>>>>>>
>>>>>>
>>>>>> On Thu, Nov 20, 2014 at 6:49 AM, Leo Simons <
>>>>>>
>>>>> lsim...@schubergphilis.com>
>>>
>>>> wrote:
>>>>>>
>>>>>>> Hey Edison, all,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> https://github.com/apache/cloudstack/commit/
>>> 05bec59c1498dbcfb8a1089c86855fd3b433ea58
>>>
>>>>
>>>>>>> breaks our internal build with
>>>>>>>
>>>>>>> + qemu-img convert -o compat=0.10 -f raw -c -O qcow2 raw.img \
>>>>>>> systemvmtemplate-systemvm-persistent-config-4.5.0.124-
>>>>>>> kvm.qcow2
>>>>>>> Unknown option 'compat'
>>>>>>> qemu-img: Invalid options for file format 'qcow2'.
>>>>>>>
>>>>>>> This is on CentOS release 6.6, which has
>>>>>>>
>>>>>> qemu-img-0.15.0-1.el6.rfx.x86_64.
>>>>>>
>>>>>>>
>>>>>>> Based on
>>>>>>>
>>>>>>>
>>>>>> http://secure-web.cisco.com/1F5zlT8tpWX5PfbJbuzUoso6OWYSnPdIEm5Ue6-
>>> gXqJ1a69oLY46nrS2BKoFjZFPlr0DmQwfZHdm2XjfkVhbuMFrET2ilAqQyrI
>>> 76EYyuxNVnTHaRHH_4bMCtgrtgasDwW1zTJWKDQp8d-
>>> HuwyPYokzHAiPFGp7w1CyCabrBpkEw/http%3A%2F%2Fwiki.qemu.org%2FOlderNews
>>>
>>>> that seems like it is a really old qemu (august 2011).
>>>>>>>
>>>>>>> I'm guessing you have a newer OS / newer qemu? Can you please let me
>>>>>>>
>>>>>> know what OS, OS version and qemu(-img) version you are using?
>>>>>>
>>>>>>>
>>>>>>> Also, does anyone know if there some minimum version of qemu-img that
>>>>>>>
>>>>>> should be used / cloudstack assumes? Is 0.15 still ok to do an
>>>>>>
>>>>> acceptable
>>>
>>>> image conversion with? (we currently don't have any kvm use ourselves,
>>>>>>
>>>>> but,
>>>
>>>> I'd like for our build infra to produce useful kvm images nonetheless).
>>>>>>
>>>>>>>
>>>>>>> According to
>>>>>>>
>>>>>>>
>>>>>> http://secure-web.cisco.com/1pFLbDg2BMTb54NUG5_lYr4mAwFcn9tHP-
>>> Weq28Sj5SFqgRKker_DkUKLf6RdwVXHI66uQQNVvD74D6rs6Pc0srpLX0Ejh6nW6o9mBZ-
>>> AhQzygMrB_OfUQNLtsB-myd-CA0rJowOhs8ZTsIVr27mgJM4v6ay93
>>> D_64JLEBebrlnM/http%3A%2F%2Fwiki.qemu.org%2FChangeLog%2F1.1
>>>
>>>> the -o compat switch was introduced in 1.1.
>>>>>>>
>>>>>>> According to
>>>>>>>
>>>>>>>
>>>>>> https://github.com/qemu/qemu/commit/9117b47717ad208b12786ce88eacb0
>>> 13f9b3dd1c
>>>
>>>> the default format was changed from 0.10 to 1.1 in qemu 1.7 and
>>>>>>>
>>>>>> onwards.
>>>
>>>>
>>>>>>> The libvirt people
>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=997977
>>>>>>> say they pass -o compat when qemu supports it (so when v >= 1.1 I
>>>>>>>
>>>>>> imagine).
>>>>>>
>>>>>>>
>>>>>>> I think we should do the same in the build script and I'll make a
>>>>>>>
>>>>>> patch.
>>>
>>>>
>>>>>>> But, should we publish newer-format images too? According to
>>>>>>>
>>>>>>>
>>>>>> http://secure-web.cisco.com/193Tci801hAyVHNqtfHy56-
>>> aPW58HHhzwlnmfvPZQnrbtHxURQC0m3kDHxk-PlYYLhCZX0f768ZYK2fV1WbMFpQPz5
>>> JfmFt0S8cuzCSrykjOEx9isfBwORZu6I4XdU3jo-WdJcDgtoH-
>>> KOwKcGRX5TGZrVva_BDOYDQzF-7QAg8w/http%3A%2F%2Fwiki.qemu.
>>> org%2FFeatures%2FQcow3
>>>
>>>> the new format is much better so I imagine qcow/kvm users will really
>>>>>>>
>>>>>> appreciate the newer formats.
>>>>>>
>>>>>>>
>>>>>>> Thoughts?
>>>>>>>
>>>>>>>
>>>>>>> cheers,
>>>>>>>
>>>>>>>
>>>>>>> Leo
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best Regards,
>>>>>> Edison
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>
>>
> --
> Regards,
> Rohit Yadav
> Software Architect, ShapeBlue
> M. +41 779015219 | rohit.ya...@shapeblue.com
>
> Blog: bhaisaab.org | Twitter: @_bhaisaab
> Find out more about ShapeBlue and our range of CloudStack related services
>
> IaaS Cloud Design & Build<http://shapeblue.com/
> iaas-cloud-design-and-build//>
> CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
> CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-
> engineering/>
> CloudStack Infrastructure Support<http://shapeblue.com/
> cloudstack-infrastructure-support/>
> CloudStack Bootcamp Training Courses<http://shapeblue.com/
> cloudstack-training/>
>
> This email and any attachments to it may be confidential and are intended
> solely for the use of the individual to whom it is addressed. Any views or
> opinions expressed are solely those of the author and do not necessarily
> represent those of Shape Blue Ltd or related companies. If you are not the
> intended recipient of this email, you must neither take any action based
> upon its contents, nor copy or show it to anyone. Please contact the sender
> if you believe you have received this email in error. Shape Blue Ltd is a
> company incorporated in England & Wales. ShapeBlue Services India LLP is a
> company incorporated in India and is operated under license from Shape Blue
> Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil
> and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is
> a company registered by The Republic of South Africa and is traded under
> license from Shape Blue Ltd. ShapeBlue is a registered trademark.
>
