Looking at some of the capabilities jinja2 has, it's hard to justify changing the method already in place. I think jinja2 can provide a clear and operational way for operators to customize the dockerfiles as needed. Kolla just hasn't applied them yet.
I'm extremely hesitant to agree on changing this because I think kolla can solve these issues without having the overhead that will come with this change. -Ryan ----- Original Message ----- From: "Michał Jastrzębski" <inc...@gmail.com> To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org> Sent: Wednesday, June 1, 2016 4:55:50 PM Subject: Re: [openstack-dev] [kolla] prototype of a DSL for generating Dockerfiles Aaaand correct link: https://review.openstack.org/#/c/323589/ sorry for pastefail. On 1 June 2016 at 15:55, Michał Jastrzębski <inc...@gmail.com> wrote: > So this is prototype of working template overrides: > https://review.openstack.org/#/c/323612/ > > Pass --template-overrides=path-to-file to build.py > in file override you can add any custom code/logic/dockerfile stuff to > any of hooks we provide in Dockerfiles, and we'll provide a lot of > them as it's free and non breaking operation. With enough block you'll > be able to do virtually anything with any of the containers. > > This one is already working. Only work needed is to provide more > hooks/continue with refactoring of dockerfiles. > > Cheers, > Michal > > On 31 May 2016 at 19:36, Steven Dake (stdake) <std...@cisco.com> wrote: >> >> >> On 5/31/16, 1:42 PM, "Michał Jastrzębski" <inc...@gmail.com> wrote: >> >>>I am opposed to this idea as I don't think we need this. We can solve >>>many problems by using jinja2 to greater extend. I'll publish demo of >>>few improvements soon, please bear with me before we make a >>>arch-changing call. >> >> Can you make a specification please as you have asked me to do? >> >>> >>>On 29 May 2016 at 14:41, Steven Dake (stdake) <std...@cisco.com> wrote: >>>> >>>>>On 5/27/16, 1:58 AM, "Steven Dake (stdake)" <std...@cisco.com> wrote: >>>>> >>>>>> >>>>>> >>>>>>On 5/26/16, 8:45 PM, "Swapnil Kulkarni (coolsvap)" <m...@coolsvap.net> >>>>>>wrote: >>>>>> >>>>>>>On Fri, May 27, 2016 at 8:35 AM, Steven Dake (stdake) >>>>>>><std...@cisco.com> >>>>>>>wrote: >>>>>>>> Hey folks, >>>>>>>> >>>>>>>> While Swapnil has been busy churning the dockerfile.j2 files to all >>>>>>>>match >>>>>>>> the same style, and we also had summit where we declared we would >>>>>>>>solve >>>>>>>>the >>>>>>>> plugin problem, I have decided to begin work on a DSL prototype. >>>>>>>> >>>>>>>> Here are the problems I want to solve in order of importance by this >>>>>>>>work: >>>>>>>> >>>>>>>> Build CentOS, Ubuntu, Oracle Linux, Debian, Fedora containers >>>>>>>> Provide a programmatic way to manage Dockerfile construction rather >>>>>>>>then a >>>>>>>> manual (with vi or emacs or the like) mechanism >>>>>>>> Allow complete overrides of every facet of Dockerfile construction, >>>>>>>>most >>>>>>>> especially repositories per container (rather than in the base >>>>>>>>container) to >>>>>>>> permit the use case of dependencies from one version with >>>>>>>>dependencies >>>>>>>>in >>>>>>>> another version of a different service >>>>>>>> Get out of the business of maintaining 100+ dockerfiles but instead >>>>>>>>maintain >>>>>>>> one master file which defines the data that needs to be used to >>>>>>>>construct >>>>>>>> Dockerfiles >>>>>>>> Permit different types of optimizations or Dockerfile building by >>>>>>>>changing >>>>>>>> around the parser implementation to allow layering of each >>>>>>>>operation, >>>>>>>>or >>>>>>>> alternatively to merge layers as we do today >>>>>>>> >>>>>>>> I don't believe we can proceed with both binary and source plugins >>>>>>>>given our >>>>>>>> current implementation of Dockerfiles in any sane way. >>>>>>>> >>>>>>>> I further don't believe it is possible to customize repositories & >>>>>>>>installed >>>>>>>> files per container, which I receive increasing requests for >>>>>>>>offline. >>>>>>>> >>>>>>>> To that end, I've created a very very rough prototype which builds >>>>>>>>the >>>>>>>>base >>>>>>>> container as well as a mariadb container. The mariadb container >>>>>>>>builds >>>>>>>>and >>>>>>>> I suspect would work. >>>>>>>> >>>>>>>> An example of the DSL usage is here: >>>>>>>> https://review.openstack.org/#/c/321468/4/dockerdsl/dsl.yml >>>>>>>> >>>>>>>> A very poorly written parser is here: >>>>>>>> https://review.openstack.org/#/c/321468/4/dockerdsl/load.py >>>>>>>> >>>>>>>> I played around with INI as a format, to take advantage of >>>>>>>>oslo.config >>>>>>>>and >>>>>>>> kolla-build.conf, but that didn't work out. YML is the way to go. >>>>>>>> >>>>>>>> I'd appreciate reviews on the YML implementation especially. >>>>>>>> >>>>>>>> How I see this work progressing is as follows: >>>>>>>> >>>>>>>> A yml file describing all docker containers for all distros is >>>>>>>>placed >>>>>>>>in >>>>>>>> kolla/docker >>>>>>>> The build tool adds an option ‹use-yml which uses the YML file >>>>>>>> A parser (such as load.py above) is integrated into build.py to lay >>>>>>>>down he >>>>>>>> Dockerfiles >>>>>>>> Wait 4-6 weeks for people to find bugs and complain >>>>>>>> Make the ‹use-yml the default for 4-6 weeks >>>>>>>> Once we feel confident in the yml implementation, remove all >>>>>>>>Dockerfile.j2 >>>>>>>> files >>>>>>>> Remove ‹use-yml option >>>>>>>> Remove all jinja2-isms from build.py >>>>>>>> >>>>>>>> This is similar to the work that took place to convert from raw >>>>>>>>Dockerfiles >>>>>>>> to Dockerfile.j2 files. We are just reusing that pattern. >>>>>>>>Hopefully >>>>>>>>this >>>>>>>> will be the last major refactor of the dockerfiles unless someone >>>>>>>>has >>>>>>>>some >>>>>>>> significant complaints about the approach. >>>>>>>> >>>>>>>> Regards >>>>>>>> -steve >> >> Hey folks, >> >> I have produced a specification for Kolla's DSL (which I call Elemental). >> The spec is ready for review here: >> https://review.openstack.org/#/c/323612/ >> >> >> Regards >> -steve >> >>>> >>>> >>>> On 5/27/16, 3:44 AM, "Britt Houser (bhouser)" <bhou...@cisco.com> wrote: >>>> >>>>>I admit I'm not as knowledgable about the Kolla codebase as I'd like to >>>>>be, so most of what you're saying is going over my head. I think mainly >>>>>I don't understand the problem statement. It looks like you're pulling >>>>>all the "hard coded" things out of the docker files, and making them >>>>>user >>>>>replaceable? So the dockerfiles just become a list of required steps, >>>>>and the user can change how each step is implemented? Would this also >>>>>unify the dockefiles so there wouldn't be a huge if statements between >>>>>Centos and Ubuntu? >>>>> >>>>>Thx, >>>>>Britt >>>>> >>>> >>>> What is being pulled out is all of the metadata used by the Dockerfiles >>>>or >>>> Kolla in general. This metadata, being structured either as a >>>>dictionary >>>> or ordered list, can be manipulated by simple python tools to do things >>>> like merge sections and override sections or optimize the built images. >>>> FWIW it looks without even trying the Dockerfiles produce a 50MB smaller >>>> image produced by the parser. The jinja2 templates we have today cannot >>>> be easily overridden. We have to provide a new key for each type of >>>> override, which is super onerous on the build.py tool. >>>> >>>> To your question of docker files being a list of required steps, with >>>>this >>>> method Dockerfile.j2 would go away permanently. On each bulid, as is >>>>done >>>> now to process a jinja2 file into a Dockerfile, the elemental.yml file >>>> would be processed into a Dockerfile for that particular set of build >>>> options. >>>>> >>>> >>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>_____________________________________________________________________ >>>>>>>>__ >>>>>>>>__ >>>>>>>>_ >>>>>>>> OpenStack Development Mailing List (not for usage questions) >>>>>>>> Unsubscribe: >>>>>>>>openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>>>> >>>>>>> >>>>>>>The DSL template to generate the Dockerfile seems way better than the >>>>>>>jinja templates in terms of extension which is currently a major >>>>>>>bottleneck in the plugin implementation. I am +2+W on this plan of >>>>>>>action to test it for next 4-6 weeks and see thereon. >>>>>>> >>>>>>>Swapnil >>>>>>> >>>>>> >>>>>>Agree. >>>>>> >>>>>>Customization and plugins are the trigger for the work. I was thinking >>>>>>of >>>>>>the following: >>>>>> >>>>>>Elemental.yml (ships with Kolla) >>>>>>Elemental-merge.yml (operator provides in /etc/kolla, this file is yaml >>>>>>merged with elemental.yml) >>>>>>Elemental-override.yml (operator provides in /etc/kolla, this file >>>>>>overrides any YAML sections defined) >>>>>> >>>>>>I think merging and overriding the yaml files should be pretty easy, >>>>>>compared to jinja2, where I don't even know where to begin in a way >>>>>>that >>>>>>the operator doesn't have to have deep knowledge of Kolla's internal >>>>>>implementation. >>>>>> >>>>>>Regards >>>>>>-steve >>>>>> >>>>>>>______________________________________________________________________ >>>>>>>__ >>>>>>>__ >>>>>>>OpenStack Development Mailing List (not for usage questions) >>>>>>>Unsubscribe: >>>>>>>openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>> >>>>>> >>>>>>_______________________________________________________________________ >>>>>>__ >>>>>>_ >>>>>>OpenStack Development Mailing List (not for usage questions) >>>>>>Unsubscribe: >>>>>>openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>________________________________________________________________________ >>>>>__ >>>>>OpenStack Development Mailing List (not for usage questions) >>>>>Unsubscribe: >>>>>openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>> >>>> >>>>_________________________________________________________________________ >>>>_ >>>> OpenStack Development Mailing List (not for usage questions) >>>> Unsubscribe: >>>>openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>>__________________________________________________________________________ >>>OpenStack Development Mailing List (not for usage questions) >>>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
