Hi. The thread is a month old, but I sent a shorter version of this to Daneyon before with some info on the things we dealt with to get Magnum deployed successfully. We wrapped it up in a post (there's a video linked there with some demos at the end):
http://openstack-in-production.blogspot.ch/2016/04/containers-and-cern-cloud.html Hopefully the pointers to the relevant blueprints for some of the issues we found will be useful for others. Cheers, Ricardo On Fri, Mar 18, 2016 at 3:42 PM, Ricardo Rocha <rocha.po...@gmail.com> wrote: > Hi. > > We're running a Magnum pilot service - which means it's being > maintained just like all other OpenStack services and running on the > production infrastructure, but only available to a subset of tenants > for a start. > > We're learning a lot in the process and will happily report on this in > the next couple weeks. > > The quick summary is that it's looking good and stable with a few > hicks in the setup, which are handled by patches already under review. > The one we need the most is the trustee user (USER_TOKEN in the bay > heat params is preventing scaling after the token expires), but with > the review in good shape we look forward to try it very soon. > > Regarding barbican we'll keep you posted, we're working on the missing > puppet bits. > > Ricardo > > On Fri, Mar 18, 2016 at 2:30 AM, Daneyon Hansen (danehans) > <daneh...@cisco.com> wrote: >> Adrian/Hongbin, >> >> Thanks for taking the time to provide your input on this matter. After >> reviewing your feedback, my takeaway is that Magnum is not ready for >> production without implementing Barbican or some other future feature such >> as the Keystone option Adrian provided. >> >> All, >> >> Is anyone using Magnum in production? If so, I would appreciate your input. >> >> -Daneyon Hansen >> >>> On Mar 17, 2016, at 6:16 PM, Adrian Otto <adrian.o...@rackspace.com> wrote: >>> >>> Hongbin, >>> >>> One alternative we could discuss as an option for operators that have a >>> good reason not to use Barbican, is to use Keystone. >>> >>> Keystone credentials store: >>> http://specs.openstack.org/openstack/keystone-specs/api/v3/identity-api-v3.html#credentials-v3-credentials >>> >>> The contents are stored in plain text in the Keystone DB, so we would want >>> to generate an encryption key per bay, encrypt the certificate and store it >>> in keystone. We would then use the same key to decrypt it upon reading the >>> key back. This might be an acceptable middle ground for clouds that will >>> not or can not run Barbican. This should work for any OpenStack cloud since >>> Grizzly. The total amount of code in Magnum would be small, as the API >>> already exists. We would need a library function to encrypt and decrypt the >>> data, and ideally a way to select different encryption algorithms in case >>> one is judged weak at some point in the future, justifying the use of an >>> alternate. >>> >>> Adrian >>> >>>> On Mar 17, 2016, at 4:55 PM, Adrian Otto <adrian.o...@rackspace.com> wrote: >>>> >>>> Hongbin, >>>> >>>>> On Mar 17, 2016, at 2:25 PM, Hongbin Lu <hongbin...@huawei.com> wrote: >>>>> >>>>> Adrian, >>>>> >>>>> I think we need a boarder set of inputs in this matter, so I moved the >>>>> discussion from whiteboard back to here. Please check my replies inline. >>>>> >>>>>> I would like to get a clear problem statement written for this. >>>>>> As I see it, the problem is that there is no safe place to put >>>>>> certificates in clouds that do not run Barbican. >>>>>> It seems the solution is to make it easy to add Barbican such that it's >>>>>> included in the setup for Magnum. >>>>> No, the solution is to explore an non-Barbican solution to store >>>>> certificates securely. >>>> >>>> I am seeking more clarity about why a non-Barbican solution is desired. >>>> Why is there resistance to adopting both Magnum and Barbican together? I >>>> think the answer is that people think they can make Magnum work with >>>> really old clouds that were set up before Barbican was introduced. That >>>> expectation is simply not reasonable. If there were a way to easily add >>>> Barbican to older clouds, perhaps this reluctance would melt away. >>>> >>>>>> Magnum should not be in the business of credential storage when there is >>>>>> an existing service focused on that need. >>>>>> >>>>>> Is there an issue with running Barbican on older clouds? >>>>>> Anyone can choose to use the builtin option with Magnum if hey don't >>>>>> have Barbican. >>>>>> A known limitation of that approach is that certificates are not >>>>>> replicated. >>>>> I guess the *builtin* option you referred is simply placing the >>>>> certificates to local file system. A few of us had concerns on this >>>>> approach (In particular, Tom Cammann has gave -2 on the review [1]) >>>>> because it cannot scale beyond a single conductor. Finally, we made a >>>>> compromise to land this option and use it for testing/debugging only. In >>>>> other words, this option is not for production. As a result, Barbican >>>>> becomes the only option for production which is the root of the problem. >>>>> It basically forces everyone to install Barbican in order to use Magnum. >>>>> >>>>> [1] https://review.openstack.org/#/c/212395/ >>>>> >>>>>> It's probably a bad idea to replicate them. >>>>>> That's what Barbican is for. --adrian_otto >>>>> Frankly, I am surprised that you disagreed here. Back to July 2015, we >>>>> all agreed to have two phases of implementation and the statement was >>>>> made by you [2]. >>>>> >>>>> ================================================================ >>>>> #agreed Magnum will use Barbican for an initial implementation for >>>>> certificate generation and secure storage/retrieval. We will commit to a >>>>> second phase of development to eliminating the hard requirement on >>>>> Barbican with an alternate implementation that implements the functional >>>>> equivalent implemented in Magnum, which may depend on libraries, but not >>>>> Barbican. >>>>> ================================================================ >>>>> >>>>> [2] >>>>> http://lists.openstack.org/pipermail/openstack-dev/2015-July/069130.html >>>> >>>> The context there is important. Barbican was considered for two purposes: >>>> (1) CA signing capability, and (2) certificate storage. My willingness to >>>> implement an alternative was based on our need to get a certificate >>>> generation and signing solution that actually worked, as Barbican did not >>>> work for that at the time. I have always viewed Barbican as a suitable >>>> solution for certificate storage, as that was what it was first designed >>>> for. Since then, we have implemented certificate generation and signing >>>> logic within a library that does not depend on Barbican, and we can use >>>> that safely in production use cases. What we don’t have built in is what >>>> Barbican is best at, secure storage for our certificates that will allow >>>> multi-conductor operation. >>>> >>>> I am opposed to the idea that Magnum should re-implement Barbican for >>>> certificate storage just because operators are reluctant to adopt it. If >>>> we need to ship a Barbican instance along with each Magnum control plane, >>>> so be it, but I don’t see the value in re-inventing the wheel. I promised >>>> the OpenStack community that we were out to integrate with and enhance >>>> OpenStack not to replace it. >>>> >>>> Now, with all that said, I do recognize that not all clouds are motivated >>>> to use all available security best practices. They may be operating in >>>> environments that they believe are already secure (because of a secure >>>> perimeter), and that it’s okay to run fundamentally insecure software >>>> within those environments. As misguided as this viewpoint may be, it’s >>>> common. My belief is that it’s best to offer the best practice by default, >>>> and only allow insecure operation when someone deliberately turns of >>>> fundamental security features. >>>> >>>> With all this said, I also care about Magnum adoption as much as all of >>>> us, so I’d like us to think creatively about how to strike the right >>>> balance between re-implementing existing technology, and making that >>>> technology easily accessible. >>>> >>>> Thanks, >>>> >>>> Adrian >>>> >>>>> >>>>> Best regards, >>>>> Hongbin >>>>> >>>>> -----Original Message----- >>>>> From: Adrian Otto [mailto:adrian.o...@rackspace.com] >>>>> Sent: March-17-16 4:32 PM >>>>> To: OpenStack Development Mailing List (not for usage questions) >>>>> Subject: Re: [openstack-dev] [magnum] High Availability >>>>> >>>>> I have trouble understanding that blueprint. I will put some remarks on >>>>> the whiteboard. Duplicating Barbican sounds like a mistake to me. >>>>> >>>>> -- >>>>> Adrian >>>>> >>>>>> On Mar 17, 2016, at 12:01 PM, Hongbin Lu <hongbin...@huawei.com> wrote: >>>>>> >>>>>> The problem of missing Barbican alternative implementation has been >>>>>> raised several times by different people. IMO, this is a very serious >>>>>> issue that will hurt Magnum adoption. I created a blueprint for that [1] >>>>>> and set the PTL as approver. It will be picked up by a contributor once >>>>>> it is approved. >>>>>> >>>>>> [1] >>>>>> https://blueprints.launchpad.net/magnum/+spec/barbican-alternative-sto >>>>>> re >>>>>> >>>>>> Best regards, >>>>>> Hongbin >>>>>> >>>>>> -----Original Message----- >>>>>> From: Ricardo Rocha [mailto:rocha.po...@gmail.com] >>>>>> Sent: March-17-16 2:39 PM >>>>>> To: OpenStack Development Mailing List (not for usage questions) >>>>>> Subject: Re: [openstack-dev] [magnum] High Availability >>>>>> >>>>>> Hi. >>>>>> >>>>>> We're on the way, the API is using haproxy load balancing in the same >>>>>> way all openstack services do here - this part seems to work fine. >>>>>> >>>>>> For the conductor we're stopped due to bay certificates - we don't >>>>>> currently have barbican so local was the only option. To get them >>>>>> accessible on all nodes we're considering two options: >>>>>> - store bay certs in a shared filesystem, meaning a new set of >>>>>> credentials in the boxes (and a process to renew fs tokens) >>>>>> - deploy barbican (some bits of puppet missing we're sorting out) >>>>>> >>>>>> More news next week. >>>>>> >>>>>> Cheers, >>>>>> Ricardo >>>>>> >>>>>>> On Thu, Mar 17, 2016 at 6:46 PM, Daneyon Hansen (danehans) >>>>>>> <daneh...@cisco.com> wrote: >>>>>>> All, >>>>>>> >>>>>>> Does anyone have experience deploying Magnum in a highly-available >>>>>>> fashion? >>>>>>> If so, I'm interested in learning from your experience. My biggest >>>>>>> unknown is the Conductor service. Any insight you can provide is >>>>>>> greatly appreciated. >>>>>>> >>>>>>> Regards, >>>>>>> Daneyon Hansen >>>>>>> >>>>>>> _____________________________________________________________________ >>>>>>> _ ____ OpenStack Development Mailing List (not for usage questions) >>>>>>> Unsubscribe: >>>>>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>> >>>>>> ______________________________________________________________________ >>>>>> ____ OpenStack Development Mailing List (not for usage questions) >>>>>> Unsubscribe: >>>>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>> ______________________________________________________________________ >>>>>> ____ OpenStack Development Mailing List (not for usage questions) >>>>>> Unsubscribe: >>>>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>> >>>>> __________________________________________________________________________ >>>>> OpenStack Development Mailing List (not for usage questions) >>>>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>> >>>>> __________________________________________________________________________ >>>>> OpenStack Development Mailing List (not for usage questions) >>>>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>> >>>> __________________________________________________________________________ >>>> OpenStack Development Mailing List (not for usage questions) >>>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> __________________________________________________________________________ >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
