On 03/15/2017 02:26 PM, Davanum Srinivas wrote: > On Wed, Mar 15, 2017 at 9:19 AM, Monty Taylor <mord...@inaugust.com> wrote: >> On 03/15/2017 11:37 AM, Davanum Srinivas wrote: >>> Monty, Team, >>> >>> Sorry for the top post: >>> >>> Support for etcd/tooz in devstack (with file driver as default) - >>> https://review.openstack.org/#/c/445432/ >>> >>> As of right now both zookeeper driver and etcd driver is working fine: >>> https://review.openstack.org/#/c/445630/ >>> https://review.openstack.org/#/c/445629/ >>> >>> The problem we have from before is that we do not have any CI jobs >>> that used zookeeper. >>> >>> I am leaning towards just throwing the etcd as default and if folks >>> are interested in zookeeper then they can add specific CI jobs with >>> DLM_BACKEND variable set. >> >> That doesn't bother me - zk as the default choice was because at the >> time zk worked and etcd did not. >> >> That said - etcd3 is a newer/better thing - so maybe instead of driving >> etcd home as a default before we add etcd3 support, we just change tooz >> to support etcd3, add the devstack jobs to use that, and start from a >> position that doesn't involve dealing with any legacy? > > Yep, jd__ and i confirmed that things work with 3.x
WOOT! > Thanks, > Dims > >> >>> On Tue, Mar 14, 2017 at 11:00 PM, Monty Taylor <mord...@inaugust.com> wrote: >>>> On 03/15/2017 03:13 AM, Jay Pipes wrote: >>>>> On 03/14/2017 05:01 PM, Clint Byrum wrote: >>>>>> Excerpts from Jay Pipes's message of 2017-03-14 15:30:32 -0400: >>>>>>> On 03/14/2017 02:50 PM, Julien Danjou wrote: >>>>>>>> On Tue, Mar 14 2017, Jay Pipes wrote: >>>>>>>> >>>>>>>>> Not tooz, because I'm not interested in a DLM nor leader election >>>>>>>>> library >>>>>>>>> (that's what the underlying etcd3 cluster handles for me), only a >>>>>>>>> fast service >>>>>>>>> liveness/healthcheck system, but it shows usage of etcd3 and Google >>>>>>>>> Protocol >>>>>>>>> Buffers implementing a simple API for liveness checking and host >>>>>>>>> maintenance >>>>>>>>> reporting. >>>>>>>> >>>>>>>> Cool cool. So that's the same feature that we implemented in tooz 3 >>>>>>>> years ago. It's called "group membership". You create a group, make >>>>>>>> nodes join it, and you know who's dead/alive and get notified when >>>>>>>> their >>>>>>>> status change. >>>>>>> >>>>>>> The point of os-lively is not to provide a thin API over ZooKeeper's >>>>>>> group membership interface. The point of os-lively is to remove the need >>>>>>> to have a database (RDBMS) record of a service in Nova. >>>>>> >>>>>> That's also the point of tooz's group membership API: >>>>>> >>>>>> https://docs.openstack.org/developer/tooz/compatibility.html#grouping >>>>> >>>>> Did you take a look at the code I wrote in os-lively? What part of the >>>>> tooz group membership API do you think I would have used? >>>>> >>>>> Again, this was a weekend project that I was moving fast on. I looked at >>>>> tooz and didn't see how I could use it for my purposes, which was to >>>>> store a versioned object in a consistent key/value store with support >>>>> for transactional semantics when storing index and data records at the >>>>> same time [1] >>>>> >>>>> https://github.com/jaypipes/os-lively/blob/master/os_lively/service.py#L468-L511 >>>>> >>>>> >>>>> etcd3 -- and specifically etcd3, not etcd2 -- supports the transactional >>>>> semantics in a consistent key/value store that I needed. >>>>> >>>>> tooz is cool, but it's not what I was looking for. It's solving a >>>>> different problem than I was trying to solve. >>>>> >>>>> This isn't a case of NIH, despite what Julien is trying to intimate in >>>>> his emails. >>>>> >>>>>>> tooz simply abstracts a group membership API across a number of drivers. >>>>>>> I don't need that. I need a way to maintain a service record (with >>>>>>> maintenance period information, region, and an evolvable data record >>>>>>> format) and query those service records in an RDBMS-like manner but >>>>>>> without the RDBMS being involved. >>>>>>> >>>>>>>>> servicegroup API with os-lively and eliminate Nova's use of an >>>>>>>>> RDBMS for >>>>>>>>> service liveness checking, which should dramatically reduce the >>>>>>>>> amount of both >>>>>>>>> DB traffic as well as conductor/MQ service update traffic. >>>>>>>> >>>>>>>> Interesting. Joshua and Vilob tried to push usage of tooz group >>>>>>>> membership a couple of years ago, but it got nowhere. Well, no, they >>>>>>>> got >>>>>>>> 2 specs written IIRC: >>>>>>>> >>>>>>>> >>>>>>>> https://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/service-group-using-tooz.html >>>>>>>> >>>>>>>> >>>>>>>> But then it died for whatever reasons on Nova side. >>>>>>> >>>>>>> It died because it didn't actually solve a problem. >>>>>>> >>>>>>> The problem is that even if we incorporate tooz, we would still need to >>>>>>> have a service table in the RDBMS and continue to query it over and over >>>>>>> again in the scheduler and API nodes. >>>>>> >>>>>> Most likely it was designed with hesitance to have a tooz requirement >>>>>> to be a source of truth. But it's certainly not a problem for most tooz >>>>>> backends to be a source of truth. Certainly not for etcd or ZK, which >>>>>> are both designed to be that. >>>>>> >>>>>>> I want all service information in the same place, and I don't want to >>>>>>> use an RDBMS for that information. etcd3 provides an ideal place to >>>>>>> store service record information. Google Protocol Buffers is an ideal >>>>>>> data format for evolvable versioned objects. os-lively presents an API >>>>>>> that solves the problem I want to solve in Nova. tooz didn't. >>>>>> >>>>>> Was there something inherent in tooz's design that prevented you from >>>>>> adding it to tooz's group API? Said API already includes liveness (watch >>>>>> the group that corresponds to the service you want). >>>>> >>>>> See above about transactional semantics. >>>>> >>>>> I'm actually happy to add an etcd3 group membership driver to tooz, >>>>> though. After the experience gained this weekend using etcd3, I'd like >>>>> to do that. >>>>> >>>>> Still doesn't mean that tooz would be the appropriate choice for what I >>>>> was trying to do with os-lively, though. >>>>> >>>>>> The only thing missing is being able to get groups and group members >>>>>> by secondary indexes. etcd3's built in indexes by field are pretty nice >>>>> >>>>> Not sure what you're talking about. etcd3 doesn't have any indexing by >>>>> field. I built the os-lively library primarily as a well-defined set of >>>>> index overlays (by uuid, by host, by service type, and by region) over >>>>> etcd3's key/value store. >>>>> >>>>>> for that, but ZK can likely also do it too by maintaining the index in >>>>>> the driver. >>>>> >>>>> Maybe, I'm not sure, I didn't spend much time this weekend looking at >>>>> ZooKeeper. >>>> >>>> a) awesome. when the rest of this dips momentarily into words that might >>>> sound negative, please hear it all wrapped in an "awesome" and know that >>>> my personal desire is to see the thing you're working on be successful >>>> without undue burden... >>>> >>>> b) In Tokyo, we had the big discussion about DLMs (where at least my >>>> intent going in to the room was to get us to pick one and only one). >>>> There were three camps in the room who were all vocal: >>>> >>>> 1) YES! Let's just pick one, I don't care which one >>>> 2) I hate Java I don't want to run Zookeeper, so we can't pick that >>>> 3) I hate go/don't trust coreos I don't want to run etcd so we can't >>>> pick that >>>> >>>> Because of 2 and 3 the group represented by 1 lost and we ended up with: >>>> "crap, we have to use an abstraction library" >>>> >>>> I'd argue that unless something has changed significantly, having Nova >>>> grow a direct depend on etcd when the DLM discussion brought us to "the >>>> operators in the room have expressed a need for a pluggable choice >>>> between at least zk and etcd" should be pretty much a non-starter. >>>> >>>> Now, being that I was personally in group 1, I'd be THRILLED if we >>>> could, as a community, decide to pick one and skip having an abstraction >>>> library. I still don't care which one - and you know I love gRPC/protobuf. >>>> >>>> But I do think that given the anti-etcd sentiment that was expressed was >>>> equally as vehement as the anti-zk sentiment, that we need to circle >>>> back and make a legit call on this topic. >>>> >>>> If we can pick one, I think having special-purpose libraries like >>>> os-lively for specific purposes would be neat. >>>> >>>> If we still can't pick one, then I think adding the liveness check you >>>> implemented for os-lively as a new feature in tooz and also implementing >>>> the same thing in the zk driver would be necessary. (of course, that'll >>>> probably depend on getting etcd3 support added to tooz and making sure >>>> there is a good functional test for etcd3. >>>> >>>> c) awesome >>>> >>>>>> I understand abstractions can seem pretty cumbersome when you're moving >>>>>> fast. It's not something I want to see stand in your way. But it would >>>>>> be nice to see where there's deficiency in tooz so we can be there for >>>>>> the next project that needs it and maybe eventually factor out direct >>>>>> etcd3 usage so users who have maybe chosen ZK as their tooz backend can >>>>>> also benefit from your work. >>>>> >>>>> It's not a deficiency in tooz. It's a different problem domain. Look at >>>>> the os-lively API and show me how you think I could have used tooz to >>>>> implement that API. >>>> >>>> I think I already said this above - but what I was reading/hearing is >>>> "why not add add the feature you need to tooz" ... not "tooz does that >>>> already" - however, as you said, you were doing quick weekend POC work, >>>> so it's possible that adding this to tooz is a next step. >>>> >>>> >>>> __________________________________________________________________________ >>>> OpenStack Development Mailing List (not for usage questions) >>>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >>> >> >> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > >
signature.asc
Description: OpenPGP digital signature
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
