Re: [rdo-dev] [infra][tripleoci] ppc64le container images in registry.rdoproject.org
I've been working with mjturek and baha on this a bit. I've responded inline below, but also want to clarify on the desired workflow. TL;DR: The desired workflow is to have ppc64le and x86_64 seamlessly integrated and uploaded. This can be done with docker manifest list images. The following link explains in greater detail: https://docs.docker.com/registry/spec/manifest-v2-2/ The process boils down to the following steps: 1) Upload an image of the first architecture (ex: image1:x86_64_01012019) 2) Upload an image of the second architecture (ex: image1:ppc64le_01012019) 3) Upload manifest list image of the image (ex: image1:01012019) Step 3 is essentially just pushing a JSON body that has descriptors and references to the other two images, such that when someone does a pull request of the manifest list image, it will gather the appropriate architecture for that image based on the host's architecture. -Trevor PS. If I've missed something important with the overall concerns here I apologize, but thought it necessary to spell out the goal as I understand it. > On Mar 21, 2019, at 12:28 PM, Javier Pena wrote: > > > - Original Message - >> Hi all, >> >> Over the last few weeks, mjturek and baha have been busy working on a set of >> periodic jobs to build TripleO images for the ppc64le arch [1]. >> >> The current missing step is publishing those images, and they are proposing >> to push those to the RDO Registry instance at registry.rdoproject.org, just >> like we do with our TripleO images. I have tried to understand the >> requirements, and would like to get input on the following topics: >> >> - Which namespace would these images use? Based on some logs [2] it looks >> like they use tripleomaster-ppc64le, will they also push the images to that >> namespace? I have no experience in namespaces inside of a registry or how that differentiates images from one another, but the images should be pushed (in my opinion) to the same location in which the x86 images reside. >> - Could this create any conflicts with our current promotion pipeline? This should not cause conflicts in current promotion pipeline, as the process should be an extension to existing functionality. >> - Is registry.rdo the right place for those images? I'm not familiar with the >> next steps for ppc64le images after that (will it then go through a >> promotion pipeline?), so that might affect. If the x86 images exist in registry.rdo, then the ppc64le (and any other architecture image) should exist there as well. I can't think of a reason to differentiate between architectures when the desired result is parity and support of more architectures. >> >> If we decide to upload the images to images.rdo, we'll need to do the > > Correction: this should read "registry.rdo" > >> following: >> >> - Create the tripleomaster-ppc64le namespace in registry.rdo, following a >> similar pattern to [3]. >> - Schedule a short registry downtime to increase its disk space, since it is >> currently near its limit. This is definitely necessary, given the capacity requirement will double, give or take, to support the additional architecture. >> - Update the job at ci.centos to include the REGISTRY_PASSWORD environment >> variable with the right token (see [4]). This is missing today, and causing >> the job failure. >> >> Once we get input from all interested parties, we will decide on the next >> steps. >> >> Thanks, >> Javier >> >> >> [1] - >> https://ci.centos.org/job/tripleo-upstream-containers-build-master-ppc64le/ >> [2] - >> https://centos.logs.rdoproject.org/tripleo-upstream-containers-build-master-ppc64le/422/logs/logs/000_FAILED_tripleoclient.log >> [3] - https://review.rdoproject.org/r/19063 >> [4] - >> https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/playbooks/tripleo-ci-periodic-base/containers-build.yaml#L12-L20 >> ___ >> dev mailing list >> dev@lists.rdoproject.org >> http://lists.rdoproject.org/mailman/listinfo/dev >> >> To unsubscribe: dev-unsubscr...@lists.rdoproject.org >> > ___ > dev mailing list > dev@lists.rdoproject.org > http://lists.rdoproject.org/mailman/listinfo/dev > > To unsubscribe: dev-unsubscr...@lists.rdoproject.org ___ dev mailing list dev@lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/dev To unsubscribe: dev-unsubscr...@lists.rdoproject.org
Re: [rdo-dev] [infra][tripleoci] ppc64le container images in registry.rdoproject.org
> On Mar 21, 2019, at 1:12 PM, Wesley Hayutin wrote: > > > > On Thu, Mar 21, 2019 at 12:03 PM Trevor Vardeman wrote: > I've been working with mjturek and baha on this a bit. I've responded inline > below, but also want to clarify on the desired workflow. > > TL;DR: The desired workflow is to have ppc64le and x86_64 seamlessly > integrated and uploaded. This can be done with docker manifest list images. > > The following link explains in greater detail: > https://docs.docker.com/registry/spec/manifest-v2-2/ > > The process boils down to the following steps: > > 1) Upload an image of the first architecture (ex: image1:x86_64_01012019) > 2) Upload an image of the second architecture (ex: image1:ppc64le_01012019) > 3) Upload manifest list image of the image (ex: image1:01012019) > > > I think this is implying we upload x86_64, and PPC containers to the rdo > registry together. We specifically do NOT want to do that. One set of > images should not be blocked by the other at this stage of the testing. That's not quite right, the images can be uploaded separately, but for the manifest list image to function properly you'd need both images uploaded. Also, as for the tagging, I wasn't suggesting a change to how images are tagged save for potentially modifying it such that the manifest list image is the default tag, versus the x86_64 image. > > I only just read through the manifest doc, so I may be misunderstanding > something. > Any container built w/ tripleo should be tagged w/ the dlrn_hash used while > building so if we needed some containers from x86_64 and some from PPC that > should be relatively simple. > > > Step 3 is essentially just pushing a JSON body that has descriptors and > references to the other two images, such that when someone does a pull > request of the manifest list image, it will gather the appropriate > architecture for that image based on the host's architecture. > > -Trevor > > PS. If I've missed something important with the overall concerns here I > apologize, but thought it necessary to spell out the goal as I understand it. > > > On Mar 21, 2019, at 12:28 PM, Javier Pena wrote: > > > > > > - Original Message - > >> Hi all, > >> > >> Over the last few weeks, mjturek and baha have been busy working on a set > >> of > >> periodic jobs to build TripleO images for the ppc64le arch [1]. > >> > >> The current missing step is publishing those images, and they are proposing > >> to push those to the RDO Registry instance at registry.rdoproject.org, just > >> like we do with our TripleO images. I have tried to understand the > >> requirements, and would like to get input on the following topics: > >> > >> - Which namespace would these images use? Based on some logs [2] it looks > >> like they use tripleomaster-ppc64le, will they also push the images to that > >> namespace? > > I have no experience in namespaces inside of a registry or how that > differentiates images from one another, but the images should be pushed (in > my opinion) to the same location in which the x86 images reside. > > >> - Could this create any conflicts with our current promotion pipeline? > > This should not cause conflicts in current promotion pipeline, as the process > should be an extension to existing functionality. > > >> - Is registry.rdo the right place for those images? I'm not familiar with > >> the > >> next steps for ppc64le images after that (will it then go through a > >> promotion pipeline?), so that might affect. > > If the x86 images exist in registry.rdo, then the ppc64le (and any other > architecture image) should exist there as well. I can't think of a reason to > differentiate between architectures when the desired result is parity and > support of more architectures. > > >> > >> If we decide to upload the images to images.rdo, we'll need to do the > > > > Correction: this should read "registry.rdo" > > > >> following: > >> > >> - Create the tripleomaster-ppc64le namespace in registry.rdo, following a > >> similar pattern to [3]. > >> - Schedule a short registry downtime to increase its disk space, since it > >> is > >> currently near its limit. > > This is definitely necessary, given the capacity requirement will double, > give or take, to support the additional architecture. > > >> - Update the job at ci.centos to include the REGISTRY_PASSWORD environment > >> variable wi
Re: [rdo-dev] [infra][tripleoci] ppc64le container images in registry.rdoproject.org
> On Mar 21, 2019, at 1:36 PM, Wesley Hayutin wrote: > > > > On Thu, Mar 21, 2019 at 12:20 PM Trevor Vardeman wrote: > > > > On Mar 21, 2019, at 1:12 PM, Wesley Hayutin wrote: > > > > > > > > On Thu, Mar 21, 2019 at 12:03 PM Trevor Vardeman > > wrote: > > I've been working with mjturek and baha on this a bit. I've responded > > inline below, but also want to clarify on the desired workflow. > > > > TL;DR: The desired workflow is to have ppc64le and x86_64 seamlessly > > integrated and uploaded. This can be done with docker manifest list images. > > > > The following link explains in greater detail: > > https://docs.docker.com/registry/spec/manifest-v2-2/ > > > > The process boils down to the following steps: > > > > 1) Upload an image of the first architecture (ex: image1:x86_64_01012019) > > 2) Upload an image of the second architecture (ex: image1:ppc64le_01012019) > > 3) Upload manifest list image of the image (ex: image1:01012019) > > > > > > I think this is implying we upload x86_64, and PPC containers to the rdo > > registry together. We specifically do NOT want to do that. One set of > > images should not be blocked by the other at this stage of the testing. > > That's not quite right, the images can be uploaded separately, but for the > manifest list image to function properly you'd need both images uploaded. > Also, as for the tagging, I wasn't suggesting a change to how images are > tagged save for potentially modifying it such that the manifest list image is > the default tag, versus the x86_64 image. > > OK.. thanks Trevor for the explanation. Since you are executing in ci.centos > and not via Zuul I can see how you are having to readdress a lot of the > infrastructure bits like uploading to the registry. > > If we can get you on the zuul platform you won't have to rework or redo at > least some of that work. In zuul we have some post playbooks that execute > after the build job [1]. I think you'll be able to find just about everything > we do w/ containers here [2] now. > > Perhaps you can join the tripleo community sync on Tuesdays directly after > the TripleO upstream community meeting on #tripleo so we can get you > introduced to some folks who may not have met you yet. I'll do exactly that, thanks for the information! Sorry if I was running in circles a bit, my exposure has been relatively minimal. I'll do my best to join in next week :) > > Thanks > > [1] > https://github.com/openstack-infra/tripleo-ci/blob/master/playbooks/tripleo-buildcontainers/tag.yaml > [2] > https://github.com/openstack-infra/tripleo-ci/tree/master/playbooks/tripleo-buildcontainers > > > > I only just read through the manifest doc, so I may be misunderstanding > > something. > > Any container built w/ tripleo should be tagged w/ the dlrn_hash used while > > building so if we needed some containers from x86_64 and some from PPC that > > should be relatively simple. > > > > > > Step 3 is essentially just pushing a JSON body that has descriptors and > > references to the other two images, such that when someone does a pull > > request of the manifest list image, it will gather the appropriate > > architecture for that image based on the host's architecture. > > > > -Trevor > > > > PS. If I've missed something important with the overall concerns here I > > apologize, but thought it necessary to spell out the goal as I understand > > it. > > > > > On Mar 21, 2019, at 12:28 PM, Javier Pena wrote: > > > > > > > > > - Original Message - > > >> Hi all, > > >> > > >> Over the last few weeks, mjturek and baha have been busy working on a > > >> set of > > >> periodic jobs to build TripleO images for the ppc64le arch [1]. > > >> > > >> The current missing step is publishing those images, and they are > > >> proposing > > >> to push those to the RDO Registry instance at registry.rdoproject.org, > > >> just > > >> like we do with our TripleO images. I have tried to understand the > > >> requirements, and would like to get input on the following topics: > > >> > > >> - Which namespace would these images use? Based on some logs [2] it looks > > >> like they use tripleomaster-ppc64le, will they also push the images to > > >> that > > >> namespace? > > > > I have no experience in
Re: [rdo-dev] [infra][tripleoci] ppc64le container images in registry.rdoproject.org
> On Mar 22, 2019, at 8:46 AM, Wesley Hayutin wrote: > > > > On Fri, Mar 22, 2019 at 3:36 AM Javier Pena wrote: > > > - Original Message - > > I've been working with mjturek and baha on this a bit. I've responded > > inline > > below, but also want to clarify on the desired workflow. > > > > TL;DR: The desired workflow is to have ppc64le and x86_64 seamlessly > > integrated and uploaded. This can be done with docker manifest list images. > > > > The following link explains in greater detail: > > https://docs.docker.com/registry/spec/manifest-v2-2/ > > > > The process boils down to the following steps: > > > > 1) Upload an image of the first architecture (ex: image1:x86_64_01012019) > > 2) Upload an image of the second architecture (ex: image1:ppc64le_01012019) > > 3) Upload manifest list image of the image (ex: image1:01012019) > > > > This is one of the details where I had my doubts. Currently, the images > uploaded to the registry use the following naming convention: > > tripleomaster/centos-binary-neutron-l3-agent:42a882962919b867c91a182b83acca6d8004096e_ee467b40 > > > Where: > > - tripleomaster is associated to the release (we have tripleomaster, > tripleostein, tripleorocky...) > - centos is associated to the OS (we have centos and fedora) > - 42a882962919b867c91a182b83acca6d8004096e_ee467b40 refers to the repository > in trunk.rdoproject.org used to build the image (commit hash and short distro > hash) > > If we want to go multi-arch, we need to change that tag to include the > architecture, is this correct? Otherwise, we could have conflicts between the > x86_64 and ppc64le pipelines trying to upload the same image. > > Regards, > Javier > > Right.. I'm agreeing w/ you.. > > I'm recommending the following change > tripleomaster -> tripleomaster_x86_64 > and add > tripleomaster_ppc64le I'm not sure if I'm getting the right information here, but from what I understand, a namespace is just a separate repo inside the same image registry, is that right? I'm not sure if manifest list images can be created between images in different namespaces, but if its possible, that's how things *should* be done. I'm not advocating for it to be done this way immediately (unless the changes turn out to be relatively trivial), but it should be the end goal of making this multiarch aware. To explain: the suggested change to putting specific architecture images in their own separate namespaces might not be a problem, assuming a manifest list image can be set up across namespaces. However, if that's not possible, then we should append (or prepend) the architecture to the tag, and create the manifest list image with the original name that has no architecture described. This would extend existing functionality within the registry itself, and allow for a seamless transition to the consumers of these images. The final list of images would be as follows: x86_64 specific image: - tripleomaster/centos-binary-neutron-l3-agent:42a882962919b867c91a182b83acca6d8004096e_ee467b40_x86_64 ppc64le specific image: - tripleomaster/centos-binary-neutron-l3-agent:42a882962919b867c91a182b83acca6d8004096e_ee467b40_ppc64le manifest list image: - tripleomaster/centos-binary-neutron-l3-agent:42a882962919b867c91a182b83acca6d8004096e_ee467b40 So the consumer would do "docker pull tripleomaster/centos-binary-neutron-l3-agent:42a882962919b867c91a182b83acca6d8004096e_ee467b40" and it would automatically retrieve the appropriate architecture image based on the host machine's architecture. *IF* manifest list images can span across namespaces, all you'd have to do is move those architecture tags to the namespace definition, and you could *still* use the same manifest list image definition above and it would work the same way. -Trevor > > > > > Step 3 is essentially just pushing a JSON body that has descriptors and > > references to the other two images, such that when someone does a pull > > request of the manifest list image, it will gather the appropriate > > architecture for that image based on the host's architecture. > > > > -Trevor > > > > PS. If I've missed something important with the overall concerns here I > > apologize, but thought it necessary to spell out the goal as I understand > > it. > > > > > On Mar 21, 2019, at 12:28 PM, Javier Pena wrote: > > > > > > > > > - Original Message - > > >> Hi all, > > >> > > >> Over the last few weeks, mjturek and baha have been busy working on a set > > >> of > > >> periodic jobs to build TripleO images for the ppc64le arch [1]. > > >> > > >> The current missing step is publishing those images, and they are > > >> proposing > > >> to push those to the RDO Registry instance at registry.rdoproject.org, > > >> just > > >> like we do with our TripleO images. I have tried to understand the > > >> requirements, and would like to get input on the following topics: > > >> > > >> - Which namespace would these images use
