We have inMemoryMetaStore.java already to store metadata locally isn't it? there are some tests that make use of that too. So if you don't need ZK for consensus and you can use this inMemoryMetaStore for metadata store.
On Thu, Jul 27, 2017 at 12:42 AM, Sijie Guo <[email protected]> wrote: > it might be better to call it 'local model' or 'standalone mode', rather > than 'zookeeper-less'. > > On Wed, Jul 26, 2017 at 8:26 PM, Enrico Olivelli <[email protected]> > wrote: > > > Hi, > > I would like to share this idea with you before starting a real design > > effort for this new configuration. > > > > Sometimes I have to run Bookie + BookKeeper inside the same JVM, > > essentially because I want my applications to run both in "cluster mode" > > and in "single instance mode". > > I have some other use case of embedding the Bookie, but this is not the > > scope of this email. > > > > In order to run BookKeeper we need ZooKeeper and running ZooKeeper inside > > the same process is tricky and you know, it is not the suggested > > configuration, due to several aspects (out of the scope of this email > > too....) > > > > I would like to make it run without ZooKeeper at all. > > This idea will support my use case (Bookie + Client inside the same JVM) > > and the single Bookie use case, which is a feature that could be useful > in > > general. > > Some other projects, like HBase, have "single server" vs "multi server" > > setup, and so I think it will be a good idea for the project and for > > gathering more interest and use cases. > > > > We are using ZooKeeper for (at least) these purposes: > > - bookie discovery > > - ledger metadata management > > - auditor leader election > > - global configuration settings (like autorecovery enabled/disabled....) > > > > my case is to have only one Bookie, so things are really simpler, no > > auditor, no shared configuration, no underreplication/overreplication, > no > > need for 'discovery' > > > > In order to address the bookie discovery we can just have some > client-side > > configuration to set the address of the Bookie and have some special > > PlacementPolicy which only returns the same bookie. > > > > The big thing is to implement the ledger metadata management, because > both > > the Bookie and the client must deal with it. > > My idea is to add to the Bookie protocol all the functions to access > > LedgersMetadata (create/list/update with CAS.....), this RPCs will call > the > > underlying LedgerManager > > > > I think this is in general very good approach to take. It will make the > bookie client not depends on metadata store directly and delegating the > heavy work to the server (bookie side). Once the client is thin, it is easy > to implement clients in other languages. > > However, I do not think it is necessarily for your use case. You can make > this as a separate topic. For RPC, I would suggest exploring gPRC (it is > protobuf + netty), that would be much better for metadata rpcs rather than > implementing the logic in bookie protocol. > > > > > > On the client side we will not use ZK but execute RPCs on the Bookie > > endpoint (a special LedgerManager which do not use ZK but uses RPCs) > > > > On the Bookie side we will have a local ledger metadata management which > > persists data on disk, we could use existing Key-Value stores, like > RocksDB > > or MapDB, or what ever we prefer, we need something which supports CAS > and > > eventually let us put bounds on used memory. > > > > In your use case, you might be able to just implement a ledger manager > using an embedded k/v. > > > > > > Does this make sense to you ? > > > > -- Enrico > > > -- Jvrao --- First they ignore you, then they laugh at you, then they fight you, then you win. - Mahatma Gandhi
