I have gotten a bit side tracked with other work today. I have pushed this change to master and I have verified that everything is working correctly.
I will be pushing a MERGE REQUEST for the same (ish) change to the 4.5 branch. Once it is committed I will validate that everything works as expected since I have to kick off a build of the system vms to be positive that everything works as expected. Cheers, *Will STEVENS* Lead Developer *CloudOps* *| *Cloud Solutions Experts 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw @CloudOps_ On Tue, Dec 2, 2014 at 11:34 AM, Edison Su <edison...@citrix.com> wrote: > > > > -----Original Message----- > > From: williamstev...@gmail.com [mailto:williamstev...@gmail.com] On > > Behalf Of Will Stevens > > Sent: Tuesday, December 02, 2014 8:06 AM > > To: dev@cloudstack.apache.org > > Subject: Re: 05bec59c - kvm, qcow, systemvm qemu-img -o compat > > > > I will do some testing and see what I can come up with. > > > > BTW, I would be doing it the reverse of what you said. I would try to > specify > > the 'compat=0.10' option initially and if that is OK, then we are good. > If it fails > > (it is currently failing on the 4.5 branch) this is most likely because > the version > > of qemu-img does not support the 'compat' option being passed to the > > convert call (which is what is happening right now on our build > systems). In > > that case, we will want to run the command without the compat option > > because if it is too old to support the 'compat' option then it will > have the > > compat=0.10 default. > Seems the qemu version is a little bit of mess, I think above method is > good enough. > BTW, thanks to look into the issue... > > > > > Basically, I will try it the new way that Edison introduced. If that > fails, I will fall > > back to the way it was being done before Edison's change... > > > > Will > > > > > > *Will STEVENS* > > Lead Developer > > > > *CloudOps* *| *Cloud Solutions Experts > > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw > > @CloudOps_ > > > > On Tue, Dec 2, 2014 at 10:53 AM, Rohit Yadav <rohit.ya...@shapeblue.com> > > wrote: > > > > > Hi Will, > > > > > > Reply in-line; > > > > > > On Tuesday 02 December 2014 08:41 PM, Will Stevens wrote: > > > > > >> @Rohit: If the script is run with 'set -e' it should not be a > > >> problem because I was planning to do an explicit 'set +e' before the > > >> call and then a 'set -e' after the call. This should handle this > case, no? > > >> > > > > > > Yes, I think that will work as well. If we are not sure about qemu > > > version, it should be alright to try once without compat and check $? > > > and retry again with compat option. The only corner case I can think > > > of is when qemu-img genuinely exits with a non-zero exit code during > > > image conversion. > > > > > > > > >> @Nux: When you say "EL7 comes with qemu-img 1.5.3 and it requires the > > >> compat option", what do you mean by that? Are you saying that if you > > >> run the build system vms job on EL7, that you have to specify the > > >> compat option even though it comes with qemu-img 1.5.3? That should > > >> not be the case because the default compat flag should not have been > > >> changed till 1.7. If you are talking about having a KVM host which > > >> runs EL7 and has a qemu-img version of 1.5.3, then that is not really > > >> relevant for this bash script. > > >> The version of qemu-img which this script checking is the version on > > >> the build system, not on the KVM host. It is validating if the > > >> compat option needs to be passed to the 'qemu-img convert' command in > > >> order for the image to have the 'compat=0.10' option set. We are not > > >> debating that the 'compat=0.10' option needs to be set, that is for > > >> sure. We are determining when it needs to be explicitly set on the > > >> build systems. > > >> > > >> In short... > > >> Jenkins kicks off a job to build the system vms. This build system > > >> could have one of many different qemu-img versions. When this script > > >> runs on the build system, we need to determine which command needs > > to > > >> be run in order to output a 'compat=0.10' compatible image. > > >> > > >> I will do some testing with Edison's suggestion and see if I can come > > >> up with a solution that way as well and then we can pick which we > > >> want to implement. > > >> > > >> Cheers, > > >> > > >> Will > > >> > > >> > > >> *Will STEVENS* > > >> Lead Developer > > >> > > >> *CloudOps* *| *Cloud Solutions Experts > > >> > > >> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* > > tw > > >> @CloudOps_ > > >> > > >> On Tue, Dec 2, 2014 at 3:32 AM, Nux! <n...@li.nux.ro> wrote: > > >> > > >> As I said above EL7 comes with qemu-img 1.5.3 and it requires the > > >> compat > > >>> option, so it's not only 1.7+. > > >>> Personally I'm in favour of Will's approach. > > >>> > > >>> Lucian > > >>> > > >>> -- > > >>> Sent from the Delta quadrant using Borg technology! > > >>> > > >>> Nux! > > >>> www.nux.ro > > >>> > > >>> ----- Original Message ----- > > >>> > > >>>> From: "Rohit Yadav" <bhais...@apache.org> > > >>>> To: "Will Stevens" <wstev...@cloudops.com> > > >>>> Cc: "Edison Su" <edison...@citrix.com>, "Wilder Rodrigues" < > > >>>> > > >>> wrodrig...@schubergphilis.com>, dev@cloudstack.apache.org, > > >>> > > >>>> "Leo Simons" <lsim...@schubergphilis.com> > > >>>> Sent: Tuesday, 2 December, 2014 07:36:21 > > >>>> Subject: Re: 05bec59c - kvm, qcow, systemvm qemu-img -o compat > > >>>> > > >>> > > >>> Hi Will, > > >>>> > > >>>> While we can use your first solution that checks exit status ($?) > > >>>> but in case some is running the script with a "set -e", the script > > >>>> will simply break when $? is not zero. I think if we know above > > >>>> which qemu-img > > >>>> > > >>> version > > >>> > > >>>> we should we the compat option (1.7?) than that's great we should > > >>>> we > > >>>> > > >>> that. > > >>> > > >>>> Thanks for looking into this. > > >>>> > > >>>> Regards. > > >>>> > > >>>> On Tue, Dec 2, 2014 at 8:01 AM, Will Stevens > > >>>> <wstev...@cloudops.com> > > >>>> > > >>> wrote: > > >>> > > >>>> > > >>>> Thanks for that. Does that mean that we prefer to do an if else > > >>>> based > > >>>>> on > > >>>>> version rather than basically doing a try catch? > > >>>>> > > >>>>> I have not been involved in previous bash scripting for CS, so I > > >>>>> am willing to fall in line with the preferred method. What I > > >>>>> proposed was based on how similar problems were solved in other > > >>>>> code in the same > > >>>>> > > >>>> file. > > >>> > > >>>> > > >>>>> If I check for version, do you agree that I should check for 1.7 > > >>>>> as the version since that is the version where the compact default > > changed? > > >>>>> > > >>>>> I will try to get this resolved tomorrow. > > >>>>> > > >>>>> Cheers, > > >>>>> > > >>>>> Will > > >>>>> On Dec 1, 2014 9:16 PM, "Edison Su" <edison...@citrix.com> wrote: > > >>>>> > > >>>>> qemu-img |head -n 1|awk '{print $3}' > > >>>>>> > > >>>>>> should show the version of qemu-img > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> *From:* williamstev...@gmail.com > > >>>>>> [mailto:williamstev...@gmail.com] *On Behalf Of *Will Stevens > > >>>>>> *Sent:* Monday, December 01, 2014 2:17 PM > > >>>>>> *To:* dev@cloudstack.apache.org > > >>>>>> *Cc:* Leo Simons; Wilder Rodrigues; Rohit Yadav; Edison Su > > >>>>>> *Subject:* Re: 05bec59c - kvm, qcow, systemvm qemu-img -o > > compat > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> Alright, so far I have found the following: > > >>>>>> > > >>>>>> > > >>>>>> > > https://github.com/qemu/qemu/commit/9117b47717ad208b12786ce88eacb > > >>>>>> 0 > > >>> 13f9b3dd1c > > >>> > > >>>> > > >>>>>> > > >>>>>> > > >>>>>> Basically, if the qemu-img version is less than 1.7, we should > > >>>>>> run the non 'compat' option version and if the version is 1.7 or > > >>>>>> greater, we > > >>>>>> > > >>>>> should > > >>> > > >>>> run the new command with the 'compat' version. > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> Unfortunately I am not able to find a way to get the qemu-img > > >>>>>> version from the command line. > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> I am looking to basically add a conditional to try running with > > >>>>>> the compat option and if that fails, then run it without the > > >>>>>> compat option. > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> Basically, I would be replacing this: > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> qemu-img convert -o compat=0.10 -f raw -c -O qcow2 raw.img > > >>>>>> "${appliance_build_name}-kvm.qcow2" > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> With this: > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> set +e > > >>>>>> > > >>>>>> qemu-img convert -o compat=0.10 -f raw -c -O qcow2 raw.img > > >>>>>> "${appliance_build_name}-kvm.qcow2" > > >>>>>> > > >>>>>> local qemuresult=$? > > >>>>>> > > >>>>>> set -e > > >>>>>> > > >>>>>> if [ ${qemuresult} != 0 ]; then > > >>>>>> > > >>>>>> log INFO "qemu-img convert failed, trying without compat > option" > > >>>>>> > > >>>>>> qemu-img convert -f raw -c -O qcow2 raw.img > > >>>>>> "${appliance_build_name}-kvm.qcow2" > > >>>>>> > > >>>>>> fi > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> What do you guys think? Is this a good enough solution? If you > > >>>>>> guys agree I will implement it in master and make sure it works, > > >>>>>> then we can merge the change back to 4.5 to fix that branch as > > >>>>>> well. > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> Let me know if you have issues with this approach... > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> Cheers, > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> Will > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> *Will STEVENS* > > >>>>>> > > >>>>>> Lead Developer > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> *CloudOps | *Cloud Solutions Experts > > >>>>>> > > >>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 > > >>>>>> > > >>>>>> w cloudops.com *|* tw @CloudOps_ > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> On Mon, Dec 1, 2014 at 4:06 PM, Will Stevens > > >>>>>> <wstev...@cloudops.com> > > >>>>>> wrote: > > >>>>>> > > >>>>>> Edison, > > >>>>>> > > >>>>>> According to this page the default compat option is 0.10: > > >>>>>> http://manpages.ubuntu.com/manpages/trusty/man1/qemu- > > img.1.html > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> Did you find that to not be the case and is that why you had to > > >>>>>> add the compat option? > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> BTW, in an attempt to get the master system vms building again, I > > >>>>>> committed a change to master to remove the compat option. We > > now > > >>>>>> have master system vms building correctly, but Rohit rightly > > >>>>>> pointed out > > >>>>>> > > >>>>> that I > > >>> > > >>>> had basically reverted your change. > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> I created a Jira ticket for this issue: > > >>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-7959 > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> I have not reverted my change in master at this point, so it is > > >>>>>> > > >>>>> building > > >>> > > >>>> right now, but I also did not make the change to 4.5, so that > > >>>> branch is > > >>>>>> currently failing to build system vms. > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> I will see if I can track down in which version of qemu the > > >>>>>> compat > > >>>>>> > > >>>>> option > > >>> > > >>>> was added so we can add some intelligent logic around this command. > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> Will > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> *Will STEVENS* > > >>>>>> > > >>>>>> Lead Developer > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> *CloudOps | *Cloud Solutions Experts > > >>>>>> > > >>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 > > >>>>>> > > >>>>>> w cloudops.com *|* tw @CloudOps_ > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> On Thu, Nov 20, 2014 at 1:07 PM, edison su <sudi...@gmail.com> > > wrote: > > >>>>>> > > >>>>>> Hi Leo, > > >>>>>> Our new internal build system are using ubuntu 14.04 or something > > >>>>>> like that, which has qemu 1.x installed by default, that's why I > > >>>>>> added the "compat" option in the build script, otherwise, the > > >>>>>> image build by qemu 1.x, won't work on RHEL 6.x. > > >>>>>> The fix would be, in the build script, check the version of > > >>>>>> qemu-img, if it's 0.x, then don't add "compat" option. There are > > >>>>>> a lot of people still using RHEL 6.x as KVM hypervisor, we have > > >>>>>> to make sure the image we build can still work on these machines. > > >>>>>> > > >>>>>> > > >>>>>> On Thu, Nov 20, 2014 at 6:49 AM, Leo Simons < > > >>>>>> > > >>>>> lsim...@schubergphilis.com> > > >>> > > >>>> wrote: > > >>>>>> > > >>>>>>> Hey Edison, all, > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>> https://github.com/apache/cloudstack/commit/ > > >>> 05bec59c1498dbcfb8a1089c86855fd3b433ea58 > > >>> > > >>>> > > >>>>>>> breaks our internal build with > > >>>>>>> > > >>>>>>> + qemu-img convert -o compat=0.10 -f raw -c -O qcow2 raw.img > > \ > > >>>>>>> systemvmtemplate-systemvm-persistent-config-4.5.0.124- > > >>>>>>> kvm.qcow2 > > >>>>>>> Unknown option 'compat' > > >>>>>>> qemu-img: Invalid options for file format 'qcow2'. > > >>>>>>> > > >>>>>>> This is on CentOS release 6.6, which has > > >>>>>>> > > >>>>>> qemu-img-0.15.0-1.el6.rfx.x86_64. > > >>>>>> > > >>>>>>> > > >>>>>>> Based on > > >>>>>>> > > >>>>>>> > > >>>>>> > > >>>>>> http://secure- > > web.cisco.com/1F5zlT8tpWX5PfbJbuzUoso6OWYSnPdIEm5Ue > > >>>>>> 6- > > >>> gXqJ1a69oLY46nrS2BKoFjZFPlr0DmQwfZHdm2XjfkVhbuMFrET2ilAqQyrI > > >>> 76EYyuxNVnTHaRHH_4bMCtgrtgasDwW1zTJWKDQp8d- > > >>> > > HuwyPYokzHAiPFGp7w1CyCabrBpkEw/http%3A%2F%2Fwiki.qemu.org%2FOl > > derNew > > >>> s > > >>> > > >>>> that seems like it is a really old qemu (august 2011). > > >>>>>>> > > >>>>>>> I'm guessing you have a newer OS / newer qemu? Can you please > > >>>>>>> let me > > >>>>>>> > > >>>>>> know what OS, OS version and qemu(-img) version you are using? > > >>>>>> > > >>>>>>> > > >>>>>>> Also, does anyone know if there some minimum version of qemu- > > img > > >>>>>>> that > > >>>>>>> > > >>>>>> should be used / cloudstack assumes? Is 0.15 still ok to do an > > >>>>>> > > >>>>> acceptable > > >>> > > >>>> image conversion with? (we currently don't have any kvm use > > >>>> ourselves, > > >>>>>> > > >>>>> but, > > >>> > > >>>> I'd like for our build infra to produce useful kvm images > nonetheless). > > >>>>>> > > >>>>>>> > > >>>>>>> According to > > >>>>>>> > > >>>>>>> > > >>>>>> http://secure- > > web.cisco.com/1pFLbDg2BMTb54NUG5_lYr4mAwFcn9tHP- > > >>> > > Weq28Sj5SFqgRKker_DkUKLf6RdwVXHI66uQQNVvD74D6rs6Pc0srpLX0Ejh6n > > W6o9mB > > >>> Z- > > >>> AhQzygMrB_OfUQNLtsB-myd-CA0rJowOhs8ZTsIVr27mgJM4v6ay93 > > >>> D_64JLEBebrlnM/http%3A%2F%2Fwiki.qemu.org%2FChangeLog%2F1.1 > > >>> > > >>>> the -o compat switch was introduced in 1.1. > > >>>>>>> > > >>>>>>> According to > > >>>>>>> > > >>>>>>> > > >>>>>> > > >>>>>> > > https://github.com/qemu/qemu/commit/9117b47717ad208b12786ce88eacb > > >>>>>> 0 > > >>> 13f9b3dd1c > > >>> > > >>>> the default format was changed from 0.10 to 1.1 in qemu 1.7 and > > >>>>>>> > > >>>>>> onwards. > > >>> > > >>>> > > >>>>>>> The libvirt people > > >>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=997977 > > >>>>>>> say they pass -o compat when qemu supports it (so when v >= 1.1 > > >>>>>>> I > > >>>>>>> > > >>>>>> imagine). > > >>>>>> > > >>>>>>> > > >>>>>>> I think we should do the same in the build script and I'll make > > >>>>>>> a > > >>>>>>> > > >>>>>> patch. > > >>> > > >>>> > > >>>>>>> But, should we publish newer-format images too? According to > > >>>>>>> > > >>>>>>> > > >>>>>> http://secure-web.cisco.com/193Tci801hAyVHNqtfHy56- > > >>> aPW58HHhzwlnmfvPZQnrbtHxURQC0m3kDHxk- > > PlYYLhCZX0f768ZYK2fV1WbMFpQPz5 > > >>> JfmFt0S8cuzCSrykjOEx9isfBwORZu6I4XdU3jo-WdJcDgtoH- > > >>> KOwKcGRX5TGZrVva_BDOYDQzF-7QAg8w/http%3A%2F%2Fwiki.qemu. > > >>> org%2FFeatures%2FQcow3 > > >>> > > >>>> the new format is much better so I imagine qcow/kvm users will > > >>>> really > > >>>>>>> > > >>>>>> appreciate the newer formats. > > >>>>>> > > >>>>>>> > > >>>>>>> Thoughts? > > >>>>>>> > > >>>>>>> > > >>>>>>> cheers, > > >>>>>>> > > >>>>>>> > > >>>>>>> Leo > > >>>>>>> > > >>>>>>> > > >>>>>> > > >>>>>> -- > > >>>>>> Best Regards, > > >>>>>> Edison > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>> > > >> > > > -- > > > Regards, > > > Rohit Yadav > > > Software Architect, ShapeBlue > > > M. +41 779015219 | rohit.ya...@shapeblue.com > > > > > > Blog: bhaisaab.org | Twitter: @_bhaisaab Find out more about ShapeBlue > > > and our range of CloudStack related services > > > > > > IaaS Cloud Design & Build<http://shapeblue.com/ > > > iaas-cloud-design-and-build//> CSForge – rapid IaaS deployment > > > framework<http://shapeblue.com/csforge/> > > > CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> > > > CloudStack Software > > > Engineering<http://shapeblue.com/cloudstack-software- > > > engineering/> > > > CloudStack Infrastructure Support<http://shapeblue.com/ > > > cloudstack-infrastructure-support/> > > > CloudStack Bootcamp Training Courses<http://shapeblue.com/ > > > cloudstack-training/> > > > > > > This email and any attachments to it may be confidential and are > > > intended solely for the use of the individual to whom it is addressed. > > > Any views or opinions expressed are solely those of the author and do > > > not necessarily represent those of Shape Blue Ltd or related > > > companies. If you are not the intended recipient of this email, you > > > must neither take any action based upon its contents, nor copy or show > > > it to anyone. Please contact the sender if you believe you have > > > received this email in error. Shape Blue Ltd is a company incorporated > > > in England & Wales. ShapeBlue Services India LLP is a company > > > incorporated in India and is operated under license from Shape Blue > > > Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in > > > Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA > > > Pty Ltd is a company registered by The Republic of South Africa and is > > traded under license from Shape Blue Ltd. ShapeBlue is a registered > > trademark. > > > >
