Re: Debian images on Microsoft Azure cloud
On Wed, Nov 11, 2015 at 5:02 PM Marcin Kulisz wrote: > n 2015-11-11 14:53:36, Steve McIntyre wrote: > > Hi, > > > My only concern is that I'd be happier if the builds were created and > > hosted on Debian project machines, like our existing official > > buildsi. > > This would be ideal. > > > I've been discussing that with other people for other types of > > build. How awkward/difficult would that be? > > From what I know it's not possible to build and then upload to Marketplace > AWS > images. > > There is a way of triggering build of this images on AWS hosts from Debian > infrastructure with bootstrap-vz though. > > I know it's not "ideal" but right now I don't know about any other option. > -- > > |_|0|_| | > |_|_|0| "Heghlu'Meh QaQ jajVam" | > |0|0|0| kuLa - | > > gpg --keyserver pgp.mit.edu --recv-keys 0x58C338B3 > 3DF1 A4DF C732 4688 38BC F121 6869 30DD 58C3 38B3 > > From what I know it's not possible to build and then upload to Marketplace AWS images. You got me thinking :-) It *should* actually be possible to bootstrap EBS backed instances locally: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/importing-your-volumes-into-amazon-ebs.html You can upload EBS snapshots. AWS supports VMDK, VHD and RAW formats! I didn't know that until just now, hence my previous insistence on runniong bootstrap-vz in a ec2 environment. This is definitely something we could support in bootstrap-vz. p.s.: Explicitly CC'ed Tiago and James to get their attention and input on this, though maybe it should be in another thread -- Anders Ingemann
Re: Debian images on Microsoft Azure cloud
On Fri, Nov 13, 2015 at 7:29 AM Richard Hartmann < richih.mailingl...@gmail.com> wrote: > On Thu, Nov 12, 2015 at 8:14 PM, Bastian Blank wrote: > > > While SHA2 is relatively cheep, it still takes a lot of time on the > > given image sizes of 30GiB, somewhat between four and six minutes. > > This does not have to be part of the _build_ process, it can be part > of the _publishing_ process. > > Out of interest: If you run the same build ten times, will you always > have the same binary output? > > > > Also I'm not really sure what you want to check with this checksums. > > The intention is to constrain images as much as possible to be able to > tell if they have been tampered with, intentionally or otherwise. If > we want to reproduce a certain scenario X time later for whatever > reason, checksums help. > > > > The image uploaded to the Azure infrastructure gets modified with an > > additional header, so you can't directly compare the checksum. > > Is it possible to remove the header for checksumming purposes? Does > said header enable any direct or indirect modifications? > > > Thanks, > RIchard > > > Out of interest: If you run the same build ten times, will you always have the same binary output? You got last modified timestamps on files etc., so no :-) -- Anders Ingemann
Re: Debian images on Microsoft Azure cloud
"official repos/mirrors", > > and if an image building team wanted to point to different repos, they > > would get the blessing of the team responsible for overseeing our mirror > > network. (DSA?). > > > > IE: Why can't these mirrors become "official mirrors", for use with a > > specific public cloud, if they follow Debian's rules, and don't have > > random arbitrary packages in them? > > > >>> 2) Require public review of images/plans (where? I think debian-cloud > >>> and debian-cd are the right places, but there may be others) > >> > >> I like the idea in general. Will we be able to support the review > process for > >> all different vendors? Will we be able to verify images / review images > for > >> cloud systems that are not that widely used as Azure, AWS, GCE or > Openstack? > > > > I don't think there is a formal process, but there has been countless > > discussions shaping decisions made for AWS and GCE on the debian-cloud > > mailing lists. I know the AWS images are announced to the list, and > > peer reviewed. > > > > I can also say with certainty that both AWS and GCE went through an > > initial public vetting on list. As for vetting images for unpopular > systems, > > I don't know the answer, but I think we can cross that bridge when we > > come to it. > > > > Cheers, > > Brian > > > >>> 4) Documentation? Is it enough to just put it in wiki.d.o, in the cloud > >>> section? > >> started on https://wiki.debian.org/MicrosoftAzure. > >> > >>> Other questions: > >>> > >>> 1) bootstrap-vz is used to build the AWS and GCE images. bootstrap-vz > has > >>> also had support for Azure for at least two years. Is there a reason > the > >>> same tool wasn't used? > >> > >> The answer to this is quite simple: At the time we started to create > images for > >> Azure, bootstrap-vz was not in shape for generating Azure images that > worked. > >> For the demonstration purpose during DebConf15 we needed an image and > Thomas > >> openstack-debian-images script generated an image that was more or less > out of > >> the box usable for Azure. So we continued to use that script. Long term > we plan > >> to support both scripts. > >> > >> Best regards, > >> > >> Martin > >> -- > >> Martin Zobel-Helas > >> Technischer Leiter Betrieb > >> Tel.: +49 (2161) 4643-0 > >> Fax:+49 (2161) 4643-100 > >> E-Mail: martin.zobel-he...@credativ.de > >> pgp fingerprint: 6B18 5642 8E41 EC89 3D5D BDBB 53B1 AC6D B11B 627B > >> http://www.credativ.de > >> > >> credativ GmbH, HRB Mönchengladbach 12080 > >> USt-ID-Nummer: DE204566209 > >> Hohenzollernstr. 133, 41061 Mönchengladbach > >> Geschäftsführung: Dr. Michael Meskes, Jörg Folz, Sascha Heuer > > > > Just my 5 cents: > "Official mirrors" shouldn't contain differences to debian.org repositories. Otherwise they should be named "Debian based" too. Isn't that what we have GPG package signatures for? In the end, the real showstopper would be the installation of public keys that are not controlled by Debian. As long as I know that the only keys software is verified with are official Debian ones I couldn't care less where I get my "data" from - or at least that's how I think it should be, I am not pretending to know the official stance on this. -- Anders Ingemann
Re: Reproducibility of image building (Re: Debian images on Microsoft Azure cloud)
On Mon, Nov 23, 2015 at 2:04 AM Charles Plessy wrote: > Hi Marcin and everybody, > > about reproducibility: > > Le Sat, Nov 21, 2015 at 03:17:22PM +, Marcin Kulisz a écrit : > > > > I'm not sure if it's possible to upload image and to build one to make > them bit > > for bit identical for reasons like ex. timestamps on files, etc.. I > think that > > at least some providers are adding some metadate which would change any > > checksums produced before upload. > > Indeed. > > In this discussion and before, I think that there is a strong consensus > that > there must be some reproducibility in image building, but we have a > difficulty > of translating this in a concrete requirement. > > Requiring that two images built at different times are bitwise identical > is not > realistic, not only because of time stamps, but also because some elements > of > configuration will differ, for instance the location of the package > sources. > > Having checksums of all the files on a given image would be nice, but let's > note that this is not a requirement currently. At the moment, I think > that we > should not request that the file checksums stay identical over rebuilds in > the > same environments: this would restrict design choices for the image > builders > (on timesamps, logs, etc), and therefore put pressure on the people writing > them. > > Of course, some of these goals can become standard practice later, but I > think > that this should evolve through consensus involving the people and teams > developing image builders. Doing the other way round would be hitting > those > who do the work with a trademark stick, which would be counter productive, > so > put it mildly. > > Altogether, for reproducibility, would the following be acceptable ? > (Wording, of course, can be improved) > > * When building an image twice in a row with the same package source >and parameters: >- the packages installed must be the same; >- the files created must be the same; >- the content of the files created may differ; > > * When releasing an image, a list of all the packages installed and a > list of >checksums of all the files must be provided. > > * For files which checksums vary, it would be good to provide their list >and an explanation on why they vary, although it is not a stict > requirement. > > Have a nice day, > > -- > Charles Plessy > Tsurumi, Kanagawa, Japan > > Sounds like a plan. > For files which checksums vary, it would be good to provide their list and an explanation on why they vary, although it is not a stict requirement. Remember logfiles, they have the same problem with timestamps. Though tbh we do our best with bootstrap-vz to not leave anything behind from the bootstrapping process, so it shouldn't be a problem. > - the content of the files created may differ; I think you mean "may *not* differ"...? -- Anders Ingemann
Re: Debian images on Oracle Compute Cloud Service
On Thu, Feb 4, 2016 at 7:00 PM Tiago Ilieve wrote: > On 31 January 2016 at 09:06, Tiago Ilieve wrote: > > This is expected to be resolved on the next version of Oracle Compute, > > but I don't know how much time it will take for it to be launched. > > That's why we're looking for an alternative. > > Good news, everyone. Oracle has enabled Xen HVM support on their > platform this week and a kernel compressed with GZIP is not required > anymore. The first image that is going to be published will use the > default one compressed with XZ. :-) > > -- > Tiago "Myhro" Ilieve > Blog: https://blog.myhro.info/ > GitHub: https://github.com/myhro > LinkedIn: https://br.linkedin.com/in/myhro > Montes Claros - MG, Brasil > > Niiice! I am sure that removes a lot of your headaches ;-) -- Anders Ingemann
Re: Timeline for official Debian cloud images for the Stretch release
On Wed, Mar 30, 2016 at 2:55 AM Steve McIntyre wrote:
> Thomas Goirand wrote:
> >On 03/25/2016 11:31 PM, Steve McIntyre wrote:
> >> * 3rd September: Requirements agreed for a test suite for cloud images
> >>
> >>To make sure that the images we're producing are sane with
> >>reasonable quality, we'll be running tests on them. We'll need to
> >>agree what should be tested (and how!).
> >
> >I have a full functional test suite in place for OpenStack, on my own
> >CI. Though having it ported to the Debian infrastructure will be a lot
> >of work.
>
> Cool!
>
> >I can invest time in such a work, though how do I get the needed
> >resources. It will require at least a VM with 8 GB of RAM, and it'd be
> >even best if it was on bare metal (so it'd go faster to install and run).
>
> IME it's much easier to automate using VMs if possible, but I'm open
> to being convinced otherwise.
>
> >For the CI which I'm using currently, I'm using Debian Live, booted over
> >PXE, so that just a reset is enough to "redeploy" a fresh Jessie. Would
> >the DSA provide that?
>
> We could do something like that quite readily, I hope. Neil Williams
> has already started setting up a Debian instance of LAVA [1] which is
> designed to do exactly this kind of thing! See lave.debian.net if
> you're interested.
>
> [1] https://wiki.linaro.org/LAVA
>
> --
> Steve McIntyre, Cambridge, UK.
> st...@einval.com
> "Further comment on how I feel about IBM will appear once I've worked out
> whether they're being malicious or incompetent. Capital letters are
> forecast."
> Matthew Garrett, http://www.livejournal.com/users/mjg59/30675.html
>
>
Hi there
> One thing *I* would love to see is a clear guide to the different
> cloud images that we could/should be providing
I can help with that :-)
bootstrap-vz is used for the official gce, ec2 and I think oracle (or will
be used, not sure) images.
There are quite a few packages added to get the gce images working: optional
ones from the manifest
<https://github.com/andsens/bootstrap-vz/blob/cc5435237cd4139efe85dfd2787e6b99005f9c5a/manifests/official/gce/jessie.yml>
(but
part of the official image) and mandatory ones
<https://github.com/andsens/bootstrap-vz/blob/cc5435237cd4139efe85dfd2787e6b99005f9c5a/bootstrapvz/providers/gce/tasks/packages.py#L17>
ec2 images are pretty basic: manifest here
<https://github.com/andsens/bootstrap-vz/blob/cc5435237cd4139efe85dfd2787e6b99005f9c5a/manifests/official/ec2/ebs-jessie-amd64-hvm.yml>
, packages here
<https://github.com/andsens/bootstrap-vz/blob/cc5435237cd4139efe85dfd2787e6b99005f9c5a/bootstrapvz/providers/ec2/tasks/packages.py#L6>
oracle images are the same: manifest here
<https://github.com/andsens/bootstrap-vz/blob/cc5435237cd4139efe85dfd2787e6b99005f9c5a/manifests/official/oracle/jessie.yml>,
packages here
<https://github.com/andsens/bootstrap-vz/blob/cc5435237cd4139efe85dfd2787e6b99005f9c5a/bootstrapvz/providers/oracle/tasks/packages.py#L7>
Keep in mind that those are only the packages that are explicitly added,
the cloud_init plugin adds the cloud-init packages of course (ntp plugin
adds ntp), and most of the images use grub for booting.
There are quite a few more places
<https://github.com/andsens/bootstrap-vz/search?utf8=%E2%9C%93&q=info.packages>
where some packages may be added: All in all the list should be: locales,
sudo, openssh-server and isc-dhcp-{client,common}
> I'm naively hoping that this kind of doc is already available and I
> just haven't found it, but more realistically I'm expecting not... :-)
Sorry, no :-/
But it would be a great idea to create a wikipage where all those
deviations from standard debootstrap are documented!
Luckily the architecture of bootstrap-vz works in a way where you have
small tasks that do one thing only (with descriptions to boot!).
When running `./bootstrap-vz --dry-run
manifests/official/ec2/ebs-jessie-amd64-hvm.yml --log - --debug`, you get
this <https://gist.github.com/andsens/30ba30882e7098fd8c9c20ce13d3161e>.
You can almost take that log and convert it into a document, it contains
the full list of changes to an image :-)
--
Anders Ingemann
Re: DebConf session proposed
Hey guys, whe would this discussion be? I'd love to join via skype maybe? On tir. 3. maj 2016 at 17.32, Steve McIntyre wrote: > On Tue, May 03, 2016 at 03:34:21PM +0100, Marcin Kulisz wrote: > >On 2016-05-03 15:25:40, Steve McIntyre wrote: > >> I know that some people won't be able to make this session, but it > >> would be good to get some discussion anyway. If you have any input, > >> please let us know. > > > >I'm not able to go to this year DC, so I'm hoping that this session can be > >streamed and irc channel available so we can discus this in real time. > > I'm hoping for that too; I've just added information to the proposal > to say so explicitly. > > -- > Steve McIntyre, Cambridge, UK. > st...@einval.com > "...In the UNIX world, people tend to interpret `non-technical user' > as meaning someone who's only ever written one device driver." -- Daniel > Pead > > -- Anders Ingemann
