agree. "Native HA solution" was already ruled out in several email threads by keystone cores already (if i remember right). This is a devops issue and should be handled as such was the feedback.
Thanks, -- dims On Mon, Aug 3, 2015 at 7:03 AM, Sergii Golovatiuk <[email protected]> wrote: > Hi, > > -- > Best regards, > Sergii Golovatiuk, > Skype #golserge > IRC #holser > > On Mon, Aug 3, 2015 at 12:44 PM, Adam Heczko <[email protected]> wrote: >> >> Hi folks, we are discussing operations on sensitive data. >> May I ask you what security controls Pacemaker provides? > > > Pacemaker doesn't exchange any security information. > >> >> How we could audit its operations and data it is accessing? > > > Just audit all OCF scripts as they may contain some bits for storing > security data on CIB. If they store any data, then this data is exchanged > across all pacemaker nodes. > >> >> The same question arises when discussing native Keystone solution. >> From the security perspective, reduction of attack surface would be >> beneficial. >> IMO Keystone native solution would be the best possible, unless even today >> Pacemaker is accessing Keystone sensitive data (not sure about it). >> Bogdan, could you clarify this a bit? > > > Native HA solution is very costy which may require a lot of engineering > resource to make keystone ready with HA patterns (consensus algorithms, > network issues, split brain) > >> >> >> Regards, >> >> Adam >> >> >> On Mon, Aug 3, 2015 at 12:02 PM, Sergii Golovatiuk >> <[email protected]> wrote: >>> >>> Hi, >>> >>> I agree with Bogdan that key rotation procedure should be part of HA >>> solution. If you make a simple script then this script will be a single >>> point of failure. It requires operator's attention so it may lead to human >>> errors also. Adding monitoring around it or expiration time is not a >>> solution either. >>> >>> There are couple of approaches how to make 'key rotation' HA ready. >>> >>> 1. Make it as part of pacemaker OCF script. In this case pacemaker will >>> select the node which will be Master. It will be responsible for key >>> generations. In this case OCF script should have logic how to distribute >>> keys. It may be puppet or some rsync wrappers like lsyncd or special >>> function in OCF script itself. In this case when master is dead, pacemaker >>> will elect a new master while old one is down. >>> >>> 2. Make keystone HA ready by itself. In this case, all logic of >>> distributed system should be covered in keystone. keystone should be able to >>> detect peers, it should have some consensus algorithms (PAXOS, RAFT, ZAB). >>> Using this algorithm master should be elected. Master should generate keys >>> and distribute them somehow to all other peers. Key distribution may be done >>> via rsync or using memcache/db as centralized storage for keys. Master may >>> send a event to all peers or peers may check memcache/db periodically. >>> >>> >>> >>> >>> >>> -- >>> Best regards, >>> Sergii Golovatiuk, >>> Skype #golserge >>> IRC #holser >>> >>> On Mon, Aug 3, 2015 at 2:37 AM, David Medberry <[email protected]> >>> wrote: >>>> >>>> Glad to see you weighed in on this. -d >>>> >>>> On Sat, Aug 1, 2015 at 3:50 PM, Matt Fischer <[email protected]> >>>> wrote: >>>>> >>>>> Agree that you guys are way over thinking this. You don't need to >>>>> rotate keys at exactly the same time, we do it in within a one or two >>>>> hours >>>>> typically based on how our regions are setup. We do it with puppet, puppet >>>>> runs on one keystone node at a time and drops the keys into place. The >>>>> actual rotation and generation we handle with a script that then proposes >>>>> the new key structure as a review which is then approved and deployed via >>>>> the normal process. For this process I always drop keys 0, 1, 2 into >>>>> place, >>>>> I'm not bumping the numbers like the normal rotations do. >>>>> >>>>> We had also considered ansible which would be perfect for this, but >>>>> that makes our ability to setup throw away environments with a single >>>>> click >>>>> a bit more complicated. If you don't have that requirement, a simple >>>>> ansible >>>>> script is what you should do. >>>>> >>>>> >>>>> On Sat, Aug 1, 2015 at 3:41 PM, Clint Byrum <[email protected]> wrote: >>>>>> >>>>>> Excerpts from Boris Bobrov's message of 2015-08-01 14:18:21 -0700: >>>>>> > On Saturday 01 August 2015 16:27:17 [email protected] wrote: >>>>>> > > I suggest to use pacemaker multistate clone resource to rotate and >>>>>> > rsync >>>>>> > > fernet tokens from local directories across cluster nodes. The >>>>>> > > resource >>>>>> > > prototype is described here >>>>>> > > https://etherpad.openstack.org/p/fernet_tokens_pacemaker> Pros: >>>>>> > Pacemaker >>>>>> > > will care about CAP/split-brain stuff for us, we just design >>>>>> > > rotate and >>>>>> > > rsync logic. Also no shared FS/DB involved but only Corosync CIB - >>>>>> > > to >>>>>> > store >>>>>> > > few internal resource state related params, not tokens. Cons: >>>>>> > > Keystone >>>>>> > > nodes hosting fernet tokens directories must be members of >>>>>> > > pacemaker >>>>>> > > cluster. Also custom OCF script should be created to implement >>>>>> > > this. __ >>>>>> > > Regards, >>>>>> > > Bogdan Dobrelya. >>>>>> > > IRC: bogdando >>>>>> > >>>>>> > Looks complex. >>>>>> > >>>>>> > I suggest this kind of bash or python script, running on Fuel master >>>>>> > node: >>>>>> > >>>>>> > 0. Check that all controllers are online; >>>>>> > 1. Go to one of the controllers, rotate keys there; >>>>>> > 2. Fetch key 0 from there; >>>>>> > 3. For each other controller rotate keys there and put the 0-key >>>>>> > instead of >>>>>> > their new 0-key. >>>>>> > 4. If any of the nodes fail to get new keys (because they went >>>>>> > offline or for >>>>>> > some other reason) revert the rotate (move the key with the biggest >>>>>> > index >>>>>> > back to 0). >>>>>> > >>>>>> > The script can be launched by cron or by button in Fuel. >>>>>> > >>>>>> > I don't see anything critically bad if one rotation/sync event >>>>>> > fails. >>>>>> > >>>>>> >>>>>> This too is overly complex and will cause failures. If you replace key >>>>>> 0, >>>>>> you will stop validating tokens that were encrypted with the old key >>>>>> 0. >>>>>> >>>>>> You simply need to run rotate on one, and then rsync that key >>>>>> repository >>>>>> to all of the others. You _must not_ run rotate again until you rsync >>>>>> to >>>>>> all of the others, since the key 0 from one rotation becomes the >>>>>> primary >>>>>> token encrypting key going forward, so you need it to get pushed out >>>>>> to >>>>>> all nodes as 0 first. >>>>>> >>>>>> Don't over think it. Just read http://lbragstad.com/?p=133 and it will >>>>>> remain simple. >>>>>> >>>>>> >>>>>> __________________________________________________________________________ >>>>>> OpenStack Development Mailing List (not for usage questions) >>>>>> Unsubscribe: >>>>>> [email protected]?subject:unsubscribe >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>> >>>>> >>>>> >>>>> >>>>> __________________________________________________________________________ >>>>> OpenStack Development Mailing List (not for usage questions) >>>>> Unsubscribe: >>>>> [email protected]?subject:unsubscribe >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>> >>>> >>>> >>>> >>>> __________________________________________________________________________ >>>> OpenStack Development Mailing List (not for usage questions) >>>> Unsubscribe: >>>> [email protected]?subject:unsubscribe >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>> >>> >>> >>> >>> __________________________________________________________________________ >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: >>> [email protected]?subject:unsubscribe >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >> >> >> >> -- >> Adam Heczko >> Security Engineer @ Mirantis Inc. >> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: [email protected]?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: [email protected]?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Davanum Srinivas :: https://twitter.com/dims __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
