On 14.7.2017 11:17, Flavio Percoco wrote:
Greetings, As some of you know, I've been working on the second phase of TripleO's containerization effort. This phase if about migrating the docker based deployment onto Kubernetes. These phase requires work on several areas: Kubernetes deployment, OpenStack deployment on Kubernetes, configuration management, etc. While I've been diving into all of these areas, this email is about the second point, OpenStack deployment on Kubernetes. There are several tools we could use for this task. kolla-kubernetes, openstack-helm, ansible roles, among others. I've looked into these tools and I've come to the conclusion that TripleO would be better of by having ansible roles that would allow for deploying OpenStack services on Kubernetes. The existing solutions in the OpenStack community require using Helm. While I like Helm and both, kolla-kubernetes and openstack-helm OpenStack projects, I believe using any of them would add an extra layer of complexity to TripleO, which is something the team has been fighting for years years - especially now that the snowball is being chopped off. Adopting any of the existing projects in the OpenStack communty would require TripleO to also write the logic to manage those projects. For example, in the case of openstack-helm, the TripleO team would have to write either ansible roles or heat templates to manage - install, remove, upgrade - the charts (I'm happy to discuss this point further but I'm keepping it at a high-level on purpose for the sake of not writing a 10k-words-long email). James Slagle sent an email[0], a couple of days ago, to form TripleO plans around ansible. One take-away from this thread is that TripleO is adopting ansible more and more, which is great and it fits perfectly with the conclusion I reached. Now, what this work means is that we would have to write an ansible role for each service that will deploy the service on a Kubernetes cluster. Ideally these roles will also generate the configuration files (removing the need of puppet entirely) and they would manage the lifecycle. The roles would be isolated and this will reduce the need of TripleO Heat templates. Doing this would give TripleO full control on the deployment process too. In addition, we could also write Ansible Playbook Bundles to contain these roles and run them using the existing docker-cmd implementation that is coming out in Pike (you can find a PoC/example of this in this repo[1]). Now, I do realize the amount of work this implies and that this is my opinion/conclusion. I'm sending this email out to kick-off the discussion and gather thoughts and opinions from the rest of the community.
I agree this is a direction we should explore further. This would give us the option to tailor things exactly as we need -- good for keeping our balance in having interfaces as stable as possible, while still making enough development progress. And we'd keep our ability to make important changes (e.g. bugfixes) without delays.
We'll have to write more code ourselves, but it's possible that if we picked up an existing tool, we'd have to spend that time (if not more) elsewhere. Migrating existing non-kubernetized TripleO deployments to kubernetized is going to be pretty difficult even if we do what you suggested. I imagine that if we also had to fit into some pre-existing external deployment/management interfaces, while trying to keep ours stable or make just iterative changes, it might turn out to be a surreal effort. We will have to design things with migration from "legacy TripleO" in mind, or make later amendments here and there solely for this purpose. Such design and patches would probably not be a good fit for non-tripleo projects.
What i recall from our old PoC [2], defining the resources and init containers etc. will probably not be the most difficult task, and furthermore we can largely draw inspiration from our current containerized solution too. I think the more challenging things might be e.g. config generation with Ansible, and how major upgrades and rolling updates will be done (how all this ties into the APB way of provisioning/deprovisioning). And of course how to fulfill the expectations that TripleO has set around network isolation and HA :)
I'm eager to give the latest code a try myself :) Thanks for working on this, it looks like there's been great progress lately!
Jirka
Finally, what I really like about writing pure ansible roles is that ansible is a known, powerfull, tool that has been adopted by many operators already. It'll provide the flexibility needed and, if structured correctly, it'll allow for operators (and other teams) to just use the parts they need/want without depending on the full-stack. I like the idea of being able to separate concerns in the deployment workflow and the idea of making it simple for users of TripleO to do the same at runtime. Unfortunately, going down this road means that my hope of creating a field where we could collaborate even more with other deployment tools will be a bit limited but I'm confident the result would also be useful for others and that we all will benefit from it... My hopes might be a bit naive *shrugs* Flavio [0] http://lists.openstack.org/pipermail/openstack-dev/2017-July/119405.html [1] https://github.com/tripleo-apb/tripleo-apbs
[2] https://github.com/jistr/tripleo-kube-test
-- @flaper87 Flavio Percoco __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
