On mer 6 set 2017, 18:25 Sijie Guo <guosi...@gmail.com> wrote: > On Sep 6, 2017 4:57 AM, "Enrico Olivelli" <eolive...@gmail.com> wrote: > > Thank you Sijie and Jia for your comments and explanations, > answers inline > > 2017-09-06 2:23 GMT+02:00 Jia Zhai <zhaiji...@gmail.com>: > > > Thanks a lot Enrico and Sijie for your comments and information on this. > > > > On Tue, Sep 5, 2017 at 9:31 PM, Enrico Olivelli <eolive...@gmail.com> > > wrote: > > > > > Great to see you working on this ! > > > I would be great to have such feature, as it is the first step to a > > > 'standalone' BookKeeper mode > > > > > > Some complementary ideas/first look questions: > > > - the document does not talk about security, IMHO we have at least to > > cover > > > authentication and TLS, it would be great to leverage existing > > AuthPlugins, > > > as they are based on exchanging byte[] (as SASL wants) > > > > > [Jia] It is a good idea. We left the security part for now for a few > > reasons. 1) Make this BP more focus on removing zookeeper dependencies > from > > client. 2) It is introduced as a separated implementation of existing > > interfaces. So it won’t impact existing security story. And for sure, > We > > will add the security part later after this. > > > > > I am fine, I am only afraid that we won't be able to support it in the > (near) future, > maybe you could just only cite the security story and add some reference to > how we would deal with it in future > > > The new ledger manager will be first marked as experimental, until it is > stable and have security feature. > > How does that sound? >
Ok > > > > > > > - do we have some kind of "bootstrap servers list" configuration option ? > > > the list should be complete or just a subset of bookies ? at connection > > the > > > client could discover the list of other bookies > > > > > [Jia] Yes, we will have a `clientBootstrapBookies` settings in the server > > set. It can be a list of bookies or just simple a DNS over the bookies. > > Will add this to the BP > > > > - will the client connect to only one bookie at a time ? how we will deal > > > with errors ? > > > > > [Jia] It will connect the the list of bootstrap servers. gPRC will load > > balance the requests and manage the connection errors. > > > > - should the bookie write on ZK metadata its gRPC endpoint info ? (this > > > will be useful for a bookie to tell about other bookies to the > connected > > > clients) > > > > > [Jia]No, it won’t. We don’t see a strong reason to add it. Especially > > eventually we may eliminate zookeeper completely. > > It can be a fixed port `3281`, or in a scheduler-based environment, it is > > very easy to have a load balancer sitting in front of those bookies. > > > > I think a fixed port is not a good way. > You will not be able to run more than one bookie on a single host. > > We should support: > - configurable port > - ephemeral port for tests > > > I think what Jia means is a configurable port, but it is a relatively fixed > port, which client doesn't discover this port from zookeeper. > Very good > > > Ideally I would like to have the local transport option, in order to have a > single JVM, but this is not a blocker problem, as we are running gRPC on > netty it should be feasible or we can create some kind of short-circut > between the client and the Bookie > > > GRPC supports inprocess channel. So you don't need to use the low level > netty settings. > Great So it sounds all good to me thanks Enrico > > I am OK for not writing this to the bookie metadata, leaving up to the > client have a configured list of bookies enabled to metadata operations > > > > > > > > - the bookie will be somehow a proxy for zookeeper, I think that the > > > 'watch' part is the more complex, we will have to deal with > > reconnections, > > > errors....maybe it is worth to write more detail about this > > > > > [Jia] The `watch` API is using the `streaming` rpc in gRPC. It is a > > straightforward proxy behavior, if a connection is broken, the client > will > > simply retry on watching again. > > > > > > > Minor issues: > > > - Maybe you can consider using ledgerId and not ledger_id, like in > > > LedgerMetadataFormat we are using lastEntryId > > > > > [Jia] Thanks, It is a protobuf style. The protobuf will convert > `ledger_id` > > to `ledgerId`. We don’t need to worry about this. > > > > got it, thanks > > > > > > > > > -In the "motivation" part you write that the fact the having more > clients > > > than the number of bookies would be a problem for zookeeper, actually > > > zookeeper is very good at dealing with a huge number of clients. > > Actually I > > > am always running clusters with 3-5 bookies and 10-100 writing clients > > and > > > this has never given troubles > > > > [Jia] :) Seems “10-100 writing clients” is not “a huge number of > clients”. > > > > OK, I agree with you an Sijie, I have no experience of larger clusters > > > > > > > > > > > > > > > > Future: > > > - as bookies will be proxies maybe we should take care not to overwhelm > a > > > bookie with too many clients > > > > > [Jia] First, gRPC is based on Netty, the protocol is http2, so the > > connection is multiplexed. We don’t need to worry about connection count. > > Second, all the bookies are treated equally for the metadata operations, > > gRPC will load balancing the requests across the bookies. We don’t need > to > > worry about some bookies are overwhelmed. > > > > gRPC sounds great > > > > > > > > > - iteration on ledgers, sometimes the clients enumerates ledgers but it > > is > > > not interested in having all of them, as we are using the bookie as > proxy > > > maybe some kind of "filter" (at least on custom metadata) would be > create > > > to limit the number of returned items. Other point I don't know gRPC > but > > it > > > does not seems to be very clear how to 'stop' the iteration > > > > > [Jia] Thanks, We can add it later. For now, we would like to focus on > > adding the features the ledger manager needs. > > > > Yup > > -- Enrico > > > > > > > > > > -- Enrico > > > > > > > > > 2017-09-05 15:10 GMT+02:00 Jia Zhai <zhaiji...@gmail.com>: > > > > > > > Hi all, > > > > > > > > I have just posted a proposal to remove zookeeper dependency from > > > > bookkeeper client, to make bookkeeper client a thin client: > > > > > > > > https://cwiki.apache.org/confluence/display/BOOKKEEPER/ > > > > BP-16%3A+remove+zookeeper+dependency+from+bookkeeper+client > > > > > > > > > > > > BookKeeper uses zookeeper for service discovery (discovering the > > > available > > > > bookies in the cluster), metadata management (storing all the > metadata > > > for > > > > ledgers). However it exposes the metadata storage directly to the > > > clients, > > > > making bookkeeper client a very thick client. It also exposes some > > > > problems. > > > > > > > > This BP explores the possibility of eliminating zookeeper completely > > from > > > > client side, to produce a thin bookkeeper client. > > > > > > > > I will send a patch as soon as we agree on the proposal. > > > > > > > > > > > > Thanks. > > > > > > > > -Jia > > > > > > > > > > -- -- Enrico Olivelli
