2017-05-16 15:53 GMT+02:00 Sijie Guo <guosi...@gmail.com>: > My personal feeling is the use cases are so different across organizations. > It would be really hard to design a **standard** policy for such things. > What is your major concern of customizing your own placement policy?
The first step I would like to do is to have a standard way to add "labels" to bookies, and drop my custom out-of-band "metadata management" about bookies. The idea of having a "custom" placement policy is feasible (and I am doing it in the short term) but for the long term I need to have somewhere a "common" code base to share among all of the apps. My limit actually is that every project (most of them are indipendent from each other) will need to implement such a policy. My idea is to implement a common policy to be included in standard community BookKeeper packages and make it available for each client. I think that it could be a new policy, without changes to existing policy -- Enrico > > - Sijie > > On Tue, May 16, 2017 at 4:05 AM, Enrico Olivelli <eolive...@gmail.com> > wrote: > >> Hi bookkeepers, >> I'm using BookKeeper for serveral projects, every project has its own >> workload characteristics and I would like to be able to assign bookies >> depending of the client type. It is quite common to share a BookKeeper >> cluster between different applications. >> >> For instance I am using Bookies to store Database logs, Task Brokers >> logs and recently I have started to use BookKeeper as data storage. >> >> Within the cluster I would like to use specific Bookies for mid-term >> storage, some bookies for logs...and so on, but current placement >> policies are not able to "distinguish" bookies. >> >> Actually I can achieve my goal by using a custom policy + custom >> metadata + out of band bookie metadata. >> >> I would like to introduce a first step, following the work of on >> "Resource aware data placement" (1), and introduce a list of "labels" >> to be assigned to every bookie. >> >> For instance: bookies for long term storage will have label >> "long-term", bookies for transaction logs may have label "wals". >> >> Another use case is to be able to request BookKeeper to write ledger >> data on specific sets of bookies depending on the "customer" who is >> the owner of data (I have customers already grouped by labels/tags) >> >> I would like to have a simple "standard" policy which uses some >> "standard" metadata to select bookies. >> >> Thinks to add: >> - a set of "labels" configurable for bookies >> - Enrich the API (getBookieInfo) to query for labels and BookKeeper >> client to keep a local cache of label-to-bookie assignments >> - add a standard "custom metadata field" which is a list of labels to >> use to select bookies, a bookie would be used only of it currently >> "has" all of the labels requested >> >> >> [1] https://cwiki.apache.org/confluence/display/BOOKKEEPER/ >> BP-2+-+Resource+aware+data+placement >> >> All comments are welcome >> >> -- Enrico >>
