Hi Josh, sorry for the double reply, I seem to be having ML bounce issues.
comments in-line, On Thu, May 12, 2016 at 4:04 PM, Joshua Harlow <harlo...@fastmail.com> wrote: > > Hi there all-ye-operators, > > I am investigating how to help move godaddy from rpms to a container-like > solution (virtualenvs, lxc, or docker...) and a set of questions that comes > up is the following (and I would think that some folks on this mailing list > may have some useful insight into the answers): > > * Have you done the transition? Back in the Havana timeframe I attempted a package to source conversion and it was fugly. I was attempting to convert nodes in-place and found that the distro packages we're applying "value added" out of tree patches and those patches caused me no end of pain. > * How did the transition go? As mentioned, the transition was painful while cleaning up packages leaves all sorts of crufty bits on a host the biggest problem I ran into during that time period was related to the DB. Some of the distro packages pulled in patches that added DB migrations and those migrations made going to the OpenStack source hard. While the conversion was a great learning experience in the end I abandoned the effort. > * Was/is kolla used or looked into? or something custom? Full disclosure, I work for Rackspace on the OpenStack-Ansible project. The OSA project uses both containers (LXC) and python virtual-environments. We do both because we want user, file system, and process isolation which containers gives us and we want to isolate OpenStack from the operating system which has python dependencies. The idea is to allow the host operating system to do what it does best and keep OpenStack as far away from it as possible. This has some great advantages which we've seen in our ability to scale services independently of a given hosts while keeping it fully isolated. In the end our solution is a hybrid one as we run the on metal for Cinder-Volume (when using the reference LVM driver), Nova-Compute (when using Linux+KVM), Swift-.* (except proxies). > * How long did it take to do the transition from a package based solution > (with say puppet/chef being used to deploy these packages)? I cant remember specifically but I think I worked on the effort for ~2 weeks before I decided it wasn't worth continuing. If I were to try it all again I'd likely have a better time today that I did then but I still think it'd be a mess. My basic plan of attack today would be to add nodes to the environment using the new infrastructure and slowly decommission the old deployment. You'll likely need to identify the point in time your distro packages currently are and take stock of all of the patches they may have applied. Then with all of that fully understood you will need to deploy a version of OpenStack just ahead of your release which "should" pulled the environment in line with the community. > * Follow-up being how big was the team to do this? It was just me, I was working on a 15+ node lab. > * What was the roll-out strategy to achieve the final container solution? My team at Rackspace created the initial release of what is now known as the OpenStack-Ansible project. In our evaluation of container technologies found docker to be a cute tool that was incapable of doing what we needed while remaining stable/functional so we went with LXC. Having run LXC for a few years now, 2 of which has been with production workloads, I've been very happy with the results. This is our basic reference architecture: [ http://docs.openstack.org/developer/openstack-ansible/install-guide/overvie= w-hostlayout.html ]. Regardless of your choices in container technologies your going to have to deal with some limitations. Before going full "container all the things" I'd look into what your current deployment needs are and see if there are known limitations going to cause you major headaches (like AF_NET_LINK not being namespace aware making it impossible to mount an iscsi target within a container https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1226855 unless you drop the network namespace). > Any other feedback (and/or questions that I missed)? * Dependency problems are still problems in every container technology. Having a reliable build environment or a package mirror is still a must. * Docker files are not packages no matter how many docker people tell you they are. However, a docker file is a really nice way to express a container runtime and if you treat it like a runtime expression engine and stay within those lines it works rather well. * OverlayFS has had some problems with various under mounts (example: https://bugzilla.redhat.com/show_bug.cgi?id=3D1319507). We're using LVM + EXT4 for the container root devices which I've not seen reliability issues with. * BTRFS may not be a good solution either (see the gotchas for more on that https://btrfs.wiki.kernel.org/index.php/Gotchas) * ZFS on linux looks really promising and has a long history of being awesome but I'm reserving full judgment on that until I can have a good long look at it. * If you're using containers and dropping all of the namespaces to work around problems or to make your life easier just run a chroot and save yourself a lot of frustration. * Generally a containerized solution will require you to re-imagine your infrastructure. I know containers are all the hype and everyone says it's so simple but there's nothing simple about running services in production that people rely on especially when you consider the network topology. If your deployment and operator teams are already accustom to a chef or puppet ecosystem, I'm assuming you're a chef or puppet shop based on one of the questions, I'd personally recommend popping into those communities IRC channels (if you're not already) to see what other folks are doing. It's likely others are working on similar things and you may be able to get what you need while helping out other deployers all without reinventing many wheels. That's about all I can think of right now and I hope it helps. Best of luck to you and if you have other questions or just want to chat drop by the #openstack-ansible channel (even if you don't use OpenStack-Ansible) I'm always happy to help and there are lots of other deployers running lots of different things and they may too be able to give you insight or perspective. > Thanks, > > Josh > > _______________________________________________ > OpenStack-operators mailing list > OpenStack-operators@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators -- Kevin Carter IRC: Cloudnull _______________________________________________ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
