Re: Comments on live-build, vmdebootstrap, bootstrap-vz, and live-wrapper
On Wed, Aug 17, 2016 at 11:19:42AM +0200, Thomas Goirand wrote: > I'm however surprised that in your review, you didn't take into account: > - openstack-debian-images (currently the only tool which is officially > building cloud image at http://cdimage.debian.org/cdimage/). > - diskimage-builder openstack-debian-images looks quite nice. I continue to use and maintain debian-image-builder (over at https://github.com/sitepoint/debian-image-builder) which is somewhat similar in many respects. Also written in Bash (and uses debootstrap), but easy enough to follow and extend. It primarily targets AWS, but theoretically supports GCE too (which I haven't used in ages but nobody reported bugs against) and should be straightforward to support other targets. All generic build steps are split out into small files in the 'tasks' directory, and platform-specific tasks are split out into sub-directories specific to a Debian release and target environment such as EC2. It's trivial to delete any existing tasks, or insert custom tasks in the desired order, and there are number of plugins that show of how easy it is to do, such as a plugins to remove systemd or adjust EC2 launch permissions. signature.asc Description: Digital signature
Re: Comments on live-build, vmdebootstrap, bootstrap-vz, and live-wrapper
On 22/08/16 18:49, Thomas Goirand wrote: > On 08/19/2016 03:02 AM, Adam Bolte wrote: >> I continue to use and maintain debian-image-builder (over at >> https://github.com/sitepoint/debian-image-builder) which is somewhat >> similar in many respects. Also written in Bash (and uses debootstrap), > > openstack-debian-images uses ISO shell (ie: it's compatible with dash), > and doesn't include bashism. That's fair enough. I've never had that as a goal, and do deliberately make use of some of the Bashisms. >> but easy enough to follow and extend. It primarily targets AWS, but >> theoretically supports GCE too (which I haven't used in ages but >> nobody reported bugs against) and should be straightforward to support >> other targets. > > Oh, that's nice. I'd love to add AWS & GCE support to > openstack-debian-images, so that we have all of our targets covered. > Instead of a huge (pointless) debate about features and such, I'd very > much prefer if you helped me to do that. Do you have time for it? I might look over what is required to add AWS support in the near future (I don't have a lot of free time right now, but AWS support is what I would have the most use for). I do really like that openstack-debian-images is packaged for Debian and maintained by a Debian developer already. :) > I just had a quick look. Your tool looks very cleanly written. > > It is harder to read the script in once since it has lots of small > files. I prefer a single flat file with conditionals. No worries. I feel that the multiple file approach is easier to look at at a glance, and get a quick feel for how the application would behave. Having said that, I am also really impressed with how you have managed to fit so much into such a concise script yet keep it very readable. > Also, apt-get > install during the script is a no-go, and should be replaced by runtime > Depends: in a package, but that's easily fixed. I fully agree with this, particularly since your project is already packaged. > Would you like to help me push this into openstack-debian-images? Cool. I'll be a bit busy over the next few weeks, but I should have more free time after that. Although I've just skimmed the openstack-debian-images script again and it might not take as long as I had suspected. Then again, the long part is usually all the testing. If I come up with something that looks good enough to consider (for AWS support), I'll let you know. But don't wait up! Cheers, Adam signature.asc Description: OpenPGP digital signature
Re: Comments on live-build, vmdebootstrap, bootstrap-vz, and live-wrapper
On Tue, Aug 23, 2016 at 08:31:28AM +0100, Neil Williams wrote: > On Fri, 19 Aug 2016 09:02:30 + > Riku Voipio wrote: > > On Tue, Aug 16, 2016 at 11:56:12AM -0400, Sam Hartman wrote: > > > We'd probably have to give up some of the tweaks we have, and add > > > support either for plugins for some of the more basic tweaks > > > directly into vmdebootstrap. As an example, vmdebootstrap would > > > almost certainly need to support raw images without a partition > > > table. > > I don't see what benefit that provides. Paravirtualization when running Xen, EC2, etc. eg. $ ls -l /dev/xvd* brw-rw---T 1 root disk 202, 1 Jul 10 23:48 /dev/xvda1 brw-rw---T 1 root disk 202, 2 Jul 10 23:48 /dev/xvda2 $ I'm not sure if it would be possible to use pygrub if your assigned volume is partitioned, which may be a problem if you don't control the Dom0. > This is why I'm unsure about the whole plugin request - if the build > tool needs special knowledge to handle your special snowflake device, > it is *your device* which is broken. It's not your device if you don't own it. I understand where you're coming from, but do you really want to forego support for various IaaS environments (and probably a significant number of other devices which might have legitimate reasons for behaving differently which we cannot anticipate)? I do agree that snowflake changes should be packaged wherever possible, but I don't agree that we shouldn't support something just because constraints of it dictate uncommon requirements. -Adam signature.asc Description: Digital signature
Re: Comments on live-build, vmdebootstrap, bootstrap-vz, and live-wrapper
On Tue, Aug 30, 2016 at 11:07:33AM +0200, Thomas Goirand wrote: > On 08/22/2016 01:29 PM, Adam Bolte wrote: > I believe I could write it myself if: > 1/ I had access to an AWS account. Can anyone help in this regard? Amazon offers a free-of-charge trial, with no-cost access to one of the micro instance types as I recall. That should suffice, and is probably your best bet to get started, although I think you will need a credit card. > 2/ Someone just told me what's missing in the current image. Adam, could > you give a quick summary of what's needed? A good starting point would be the ec2 directory of debian-image-builder, since that splits out the EC2-specific steps reasonably well I think. Those files can be seen here: https://github.com/sitepoint/debian-image-builder/tree/master/tasks/ec2 Some of it is specific to EC2 image types, such as paravirtual Vs HVM images, and EC2-backed Vs instance-backed images. I'm not sure if the plan for openstack-debian-images is to support all possible EC2 instance types, or just the latest ones AWS recommends (albeit perhaps not compatible with all existing environments). debian-image-builder does no partitioning when building HVM images since only paravirtual images existed when it was written. I would aim to make sure that partitioning is properly taken care of for HVM images when adding new code to openstack-debian-images. As part of the creation process, I attach an EBS volume and install GRUB on that to avoid some issues I had previously with grub-install. Maybe that's no longer an issue, so should be re-tested with the grub-install approach openstack-debian-images uses before adding that code. My suspicion is that it won't be an issue when partitioning for HVM images, and Jessie has the grub-xen package which you can use to create a menu.lst / grub.conf file without needing to run grub-install for paravirtual images. Actually, the paravirtual update-grub approach taken in debian-image-builder could use some improvement. For paravirtual images, installing grub-xen, removing execute permission on everything in /etc/default/grub/ except 10_linux, and setting GRUB_DISABLE_SUBMENU="y" in /etc/default/grub should be enough to cleanly generate a grub.conf file that's compatible with pygrub, without the need for the 40_custom Grub script you can see I deploy. But honestly I don't know if many people use paravirtual images on AWS these days, and AWS was the main focus of my project since that's what I use it for. That's everything I can think of that's noteworthy. Let me know if you have any questions. > Since both OpenStack and AWS are using the EC2 metadata server, I'm > convince that it should be possible (and even desirable) for Debian to > ship a single image that would accommodate both. I'm willing to do > so. That sounds good. Although I personally use EC2 right now, I'd like to see OpenStack supported as well. Another tool to maybe take hints from is the xen-create-image script included in the xen-tools package. It basically creates a Debian paravirtual image. Written in Perl, but probably simple enough to understand without much Perl knowledge. It's officially packaged for Debian, and you can compare the image modifications it makes by default. Cheers, Adam signature.asc Description: Digital signature
