I am opposed to this idea as I don't think we need this. We can solve many problems by using jinja2 to greater extend. I'll publish demo of few improvements soon, please bear with me before we make a arch-changing call.
On 29 May 2016 at 14:41, Steven Dake (stdake) <std...@cisco.com> wrote: > >>On 5/27/16, 1:58 AM, "Steven Dake (stdake)" <std...@cisco.com> wrote: >> >>> >>> >>>On 5/26/16, 8:45 PM, "Swapnil Kulkarni (coolsvap)" <m...@coolsvap.net> >>>wrote: >>> >>>>On Fri, May 27, 2016 at 8:35 AM, Steven Dake (stdake) <std...@cisco.com> >>>>wrote: >>>>> Hey folks, >>>>> >>>>> While Swapnil has been busy churning the dockerfile.j2 files to all >>>>>match >>>>> the same style, and we also had summit where we declared we would >>>>>solve >>>>>the >>>>> plugin problem, I have decided to begin work on a DSL prototype. >>>>> >>>>> Here are the problems I want to solve in order of importance by this >>>>>work: >>>>> >>>>> Build CentOS, Ubuntu, Oracle Linux, Debian, Fedora containers >>>>> Provide a programmatic way to manage Dockerfile construction rather >>>>>then a >>>>> manual (with vi or emacs or the like) mechanism >>>>> Allow complete overrides of every facet of Dockerfile construction, >>>>>most >>>>> especially repositories per container (rather than in the base >>>>>container) to >>>>> permit the use case of dependencies from one version with dependencies >>>>>in >>>>> another version of a different service >>>>> Get out of the business of maintaining 100+ dockerfiles but instead >>>>>maintain >>>>> one master file which defines the data that needs to be used to >>>>>construct >>>>> Dockerfiles >>>>> Permit different types of optimizations or Dockerfile building by >>>>>changing >>>>> around the parser implementation to allow layering of each >>>>>operation, >>>>>or >>>>> alternatively to merge layers as we do today >>>>> >>>>> I don't believe we can proceed with both binary and source plugins >>>>>given our >>>>> current implementation of Dockerfiles in any sane way. >>>>> >>>>> I further don't believe it is possible to customize repositories & >>>>>installed >>>>> files per container, which I receive increasing requests for offline. >>>>> >>>>> To that end, I've created a very very rough prototype which builds the >>>>>base >>>>> container as well as a mariadb container. The mariadb container >>>>>builds >>>>>and >>>>> I suspect would work. >>>>> >>>>> An example of the DSL usage is here: >>>>> https://review.openstack.org/#/c/321468/4/dockerdsl/dsl.yml >>>>> >>>>> A very poorly written parser is here: >>>>> https://review.openstack.org/#/c/321468/4/dockerdsl/load.py >>>>> >>>>> I played around with INI as a format, to take advantage of oslo.config >>>>>and >>>>> kolla-build.conf, but that didn't work out. YML is the way to go. >>>>> >>>>> I'd appreciate reviews on the YML implementation especially. >>>>> >>>>> How I see this work progressing is as follows: >>>>> >>>>> A yml file describing all docker containers for all distros is placed >>>>>in >>>>> kolla/docker >>>>> The build tool adds an option ‹use-yml which uses the YML file >>>>> A parser (such as load.py above) is integrated into build.py to lay >>>>>down he >>>>> Dockerfiles >>>>> Wait 4-6 weeks for people to find bugs and complain >>>>> Make the ‹use-yml the default for 4-6 weeks >>>>> Once we feel confident in the yml implementation, remove all >>>>>Dockerfile.j2 >>>>> files >>>>> Remove ‹use-yml option >>>>> Remove all jinja2-isms from build.py >>>>> >>>>> This is similar to the work that took place to convert from raw >>>>>Dockerfiles >>>>> to Dockerfile.j2 files. We are just reusing that pattern. Hopefully >>>>>this >>>>> will be the last major refactor of the dockerfiles unless someone has >>>>>some >>>>> significant complaints about the approach. >>>>> >>>>> Regards >>>>> -steve > > > On 5/27/16, 3:44 AM, "Britt Houser (bhouser)" <bhou...@cisco.com> wrote: > >>I admit I'm not as knowledgable about the Kolla codebase as I'd like to >>be, so most of what you're saying is going over my head. I think mainly >>I don't understand the problem statement. It looks like you're pulling >>all the "hard coded" things out of the docker files, and making them user >>replaceable? So the dockerfiles just become a list of required steps, >>and the user can change how each step is implemented? Would this also >>unify the dockefiles so there wouldn't be a huge if statements between >>Centos and Ubuntu? >> >>Thx, >>Britt >> > > What is being pulled out is all of the metadata used by the Dockerfiles or > Kolla in general. This metadata, being structured either as a dictionary > or ordered list, can be manipulated by simple python tools to do things > like merge sections and override sections or optimize the built images. > FWIW it looks without even trying the Dockerfiles produce a 50MB smaller > image produced by the parser. The jinja2 templates we have today cannot > be easily overridden. We have to provide a new key for each type of > override, which is super onerous on the build.py tool. > > To your question of docker files being a list of required steps, with this > method Dockerfile.j2 would go away permanently. On each bulid, as is done > now to process a jinja2 file into a Dockerfile, the elemental.yml file > would be processed into a Dockerfile for that particular set of build > options. >> > > >>>>> >>>>> >>>>> >>>>>_______________________________________________________________________ >>>>>__ >>>>>_ >>>>> OpenStack Development Mailing List (not for usage questions) >>>>> Unsubscribe: >>>>>openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>> >>>> >>>>The DSL template to generate the Dockerfile seems way better than the >>>>jinja templates in terms of extension which is currently a major >>>>bottleneck in the plugin implementation. I am +2+W on this plan of >>>>action to test it for next 4-6 weeks and see thereon. >>>> >>>>Swapnil >>>> >>> >>>Agree. >>> >>>Customization and plugins are the trigger for the work. I was thinking >>>of >>>the following: >>> >>>Elemental.yml (ships with Kolla) >>>Elemental-merge.yml (operator provides in /etc/kolla, this file is yaml >>>merged with elemental.yml) >>>Elemental-override.yml (operator provides in /etc/kolla, this file >>>overrides any YAML sections defined) >>> >>>I think merging and overriding the yaml files should be pretty easy, >>>compared to jinja2, where I don't even know where to begin in a way that >>>the operator doesn't have to have deep knowledge of Kolla's internal >>>implementation. >>> >>>Regards >>>-steve >>> >>>>________________________________________________________________________ >>>>__ >>>>OpenStack Development Mailing List (not for usage questions) >>>>Unsubscribe: >>>>openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >>>_________________________________________________________________________ >>>_ >>>OpenStack Development Mailing List (not for usage questions) >>>Unsubscribe: >>>openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>__________________________________________________________________________ >>OpenStack Development Mailing List (not for usage questions) >>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
