Hi, I would like to share this idea with you before starting a real design effort for this new configuration.
Sometimes I have to run Bookie + BookKeeper inside the same JVM, essentially because I want my applications to run both in "cluster mode" and in "single instance mode". I have some other use case of embedding the Bookie, but this is not the scope of this email. In order to run BookKeeper we need ZooKeeper and running ZooKeeper inside the same process is tricky and you know, it is not the suggested configuration, due to several aspects (out of the scope of this email too....) I would like to make it run without ZooKeeper at all. This idea will support my use case (Bookie + Client inside the same JVM) and the single Bookie use case, which is a feature that could be useful in general. Some other projects, like HBase, have "single server" vs "multi server" setup, and so I think it will be a good idea for the project and for gathering more interest and use cases. We are using ZooKeeper for (at least) these purposes: - bookie discovery - ledger metadata management - auditor leader election - global configuration settings (like autorecovery enabled/disabled....) my case is to have only one Bookie, so things are really simpler, no auditor, no shared configuration, no underreplication/overreplication, no need for 'discovery' In order to address the bookie discovery we can just have some client-side configuration to set the address of the Bookie and have some special PlacementPolicy which only returns the same bookie. The big thing is to implement the ledger metadata management, because both the Bookie and the client must deal with it. My idea is to add to the Bookie protocol all the functions to access LedgersMetadata (create/list/update with CAS.....), this RPCs will call the underlying LedgerManager On the client side we will not use ZK but execute RPCs on the Bookie endpoint (a special LedgerManager which do not use ZK but uses RPCs) On the Bookie side we will have a local ledger metadata management which persists data on disk, we could use existing Key-Value stores, like RocksDB or MapDB, or what ever we prefer, we need something which supports CAS and eventually let us put bounds on used memory. Does this make sense to you ? -- Enrico
