On 5/31/16, 1:42 PM, "Michał Jastrzębski" <inc...@gmail.com> wrote:
>I am opposed to this idea as I don't think we need this. We can solve >many problems by using jinja2 to greater extend. I'll publish demo of >few improvements soon, please bear with me before we make a >arch-changing call. Can you make a specification please as you have asked me to do? > >On 29 May 2016 at 14:41, Steven Dake (stdake) <std...@cisco.com> wrote: >> >>>On 5/27/16, 1:58 AM, "Steven Dake (stdake)" <std...@cisco.com> wrote: >>> >>>> >>>> >>>>On 5/26/16, 8:45 PM, "Swapnil Kulkarni (coolsvap)" <m...@coolsvap.net> >>>>wrote: >>>> >>>>>On Fri, May 27, 2016 at 8:35 AM, Steven Dake (stdake) >>>>><std...@cisco.com> >>>>>wrote: >>>>>> Hey folks, >>>>>> >>>>>> While Swapnil has been busy churning the dockerfile.j2 files to all >>>>>>match >>>>>> the same style, and we also had summit where we declared we would >>>>>>solve >>>>>>the >>>>>> plugin problem, I have decided to begin work on a DSL prototype. >>>>>> >>>>>> Here are the problems I want to solve in order of importance by this >>>>>>work: >>>>>> >>>>>> Build CentOS, Ubuntu, Oracle Linux, Debian, Fedora containers >>>>>> Provide a programmatic way to manage Dockerfile construction rather >>>>>>then a >>>>>> manual (with vi or emacs or the like) mechanism >>>>>> Allow complete overrides of every facet of Dockerfile construction, >>>>>>most >>>>>> especially repositories per container (rather than in the base >>>>>>container) to >>>>>> permit the use case of dependencies from one version with >>>>>>dependencies >>>>>>in >>>>>> another version of a different service >>>>>> Get out of the business of maintaining 100+ dockerfiles but instead >>>>>>maintain >>>>>> one master file which defines the data that needs to be used to >>>>>>construct >>>>>> Dockerfiles >>>>>> Permit different types of optimizations or Dockerfile building by >>>>>>changing >>>>>> around the parser implementation to allow layering of each >>>>>>operation, >>>>>>or >>>>>> alternatively to merge layers as we do today >>>>>> >>>>>> I don't believe we can proceed with both binary and source plugins >>>>>>given our >>>>>> current implementation of Dockerfiles in any sane way. >>>>>> >>>>>> I further don't believe it is possible to customize repositories & >>>>>>installed >>>>>> files per container, which I receive increasing requests for >>>>>>offline. >>>>>> >>>>>> To that end, I've created a very very rough prototype which builds >>>>>>the >>>>>>base >>>>>> container as well as a mariadb container. The mariadb container >>>>>>builds >>>>>>and >>>>>> I suspect would work. >>>>>> >>>>>> An example of the DSL usage is here: >>>>>> https://review.openstack.org/#/c/321468/4/dockerdsl/dsl.yml >>>>>> >>>>>> A very poorly written parser is here: >>>>>> https://review.openstack.org/#/c/321468/4/dockerdsl/load.py >>>>>> >>>>>> I played around with INI as a format, to take advantage of >>>>>>oslo.config >>>>>>and >>>>>> kolla-build.conf, but that didn't work out. YML is the way to go. >>>>>> >>>>>> I'd appreciate reviews on the YML implementation especially. >>>>>> >>>>>> How I see this work progressing is as follows: >>>>>> >>>>>> A yml file describing all docker containers for all distros is >>>>>>placed >>>>>>in >>>>>> kolla/docker >>>>>> The build tool adds an option ‹use-yml which uses the YML file >>>>>> A parser (such as load.py above) is integrated into build.py to lay >>>>>>down he >>>>>> Dockerfiles >>>>>> Wait 4-6 weeks for people to find bugs and complain >>>>>> Make the ‹use-yml the default for 4-6 weeks >>>>>> Once we feel confident in the yml implementation, remove all >>>>>>Dockerfile.j2 >>>>>> files >>>>>> Remove ‹use-yml option >>>>>> Remove all jinja2-isms from build.py >>>>>> >>>>>> This is similar to the work that took place to convert from raw >>>>>>Dockerfiles >>>>>> to Dockerfile.j2 files. We are just reusing that pattern. >>>>>>Hopefully >>>>>>this >>>>>> will be the last major refactor of the dockerfiles unless someone >>>>>>has >>>>>>some >>>>>> significant complaints about the approach. >>>>>> >>>>>> Regards >>>>>> -steve Hey folks, I have produced a specification for Kolla's DSL (which I call Elemental). The spec is ready for review here: https://review.openstack.org/#/c/323612/ Regards -steve >> >> >> On 5/27/16, 3:44 AM, "Britt Houser (bhouser)" <bhou...@cisco.com> wrote: >> >>>I admit I'm not as knowledgable about the Kolla codebase as I'd like to >>>be, so most of what you're saying is going over my head. I think mainly >>>I don't understand the problem statement. It looks like you're pulling >>>all the "hard coded" things out of the docker files, and making them >>>user >>>replaceable? So the dockerfiles just become a list of required steps, >>>and the user can change how each step is implemented? Would this also >>>unify the dockefiles so there wouldn't be a huge if statements between >>>Centos and Ubuntu? >>> >>>Thx, >>>Britt >>> >> >> What is being pulled out is all of the metadata used by the Dockerfiles >>or >> Kolla in general. This metadata, being structured either as a >>dictionary >> or ordered list, can be manipulated by simple python tools to do things >> like merge sections and override sections or optimize the built images. >> FWIW it looks without even trying the Dockerfiles produce a 50MB smaller >> image produced by the parser. The jinja2 templates we have today cannot >> be easily overridden. We have to provide a new key for each type of >> override, which is super onerous on the build.py tool. >> >> To your question of docker files being a list of required steps, with >>this >> method Dockerfile.j2 would go away permanently. On each bulid, as is >>done >> now to process a jinja2 file into a Dockerfile, the elemental.yml file >> would be processed into a Dockerfile for that particular set of build >> options. >>> >> >> >>>>>> >>>>>> >>>>>> >>>>>>_____________________________________________________________________ >>>>>>__ >>>>>>__ >>>>>>_ >>>>>> OpenStack Development Mailing List (not for usage questions) >>>>>> Unsubscribe: >>>>>>openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>> >>>>> >>>>>The DSL template to generate the Dockerfile seems way better than the >>>>>jinja templates in terms of extension which is currently a major >>>>>bottleneck in the plugin implementation. I am +2+W on this plan of >>>>>action to test it for next 4-6 weeks and see thereon. >>>>> >>>>>Swapnil >>>>> >>>> >>>>Agree. >>>> >>>>Customization and plugins are the trigger for the work. I was thinking >>>>of >>>>the following: >>>> >>>>Elemental.yml (ships with Kolla) >>>>Elemental-merge.yml (operator provides in /etc/kolla, this file is yaml >>>>merged with elemental.yml) >>>>Elemental-override.yml (operator provides in /etc/kolla, this file >>>>overrides any YAML sections defined) >>>> >>>>I think merging and overriding the yaml files should be pretty easy, >>>>compared to jinja2, where I don't even know where to begin in a way >>>>that >>>>the operator doesn't have to have deep knowledge of Kolla's internal >>>>implementation. >>>> >>>>Regards >>>>-steve >>>> >>>>>______________________________________________________________________ >>>>>__ >>>>>__ >>>>>OpenStack Development Mailing List (not for usage questions) >>>>>Unsubscribe: >>>>>openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>> >>>> >>>>_______________________________________________________________________ >>>>__ >>>>_ >>>>OpenStack Development Mailing List (not for usage questions) >>>>Unsubscribe: >>>>openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>________________________________________________________________________ >>>__ >>>OpenStack Development Mailing List (not for usage questions) >>>Unsubscribe: >>>openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >>_________________________________________________________________________ >>_ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >>openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >__________________________________________________________________________ >OpenStack Development Mailing List (not for usage questions) >Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
