On Wed, Sep 13, 2017 at 1:38 AM, Enrico Olivelli <eolive...@gmail.com> wrote:
> > > 2017-09-13 10:31 GMT+02:00 Jia Zhai <zhaiji...@gmail.com>: > >> Thanks a lot for your time Yiming and Enrico. :) >> >> Regarding the security, we could do it in a separate BP, and make this BP >> more focus on filling up the useful endpoints. How about it? >> > > OK, but please consider that the more utility you add to the endpoint the > more users will want to enable it and so security will be a blocker issue > My take on this - we all know security is important for any communication to bookies. However, security is a big different scope of problems to address. we shouldn't put everything in one big BP. I'd suggest us focusing on the problems that a BP tends to address, defer other things to separate BPs. Otherwise we can't land any changes quickly. > > >> >> On Wed, Sep 13, 2017 at 3:30 PM, Enrico Olivelli <eolive...@gmail.com> >> wrote: >> >> > Hi Jia, >> > I am OK with the idea of having standard HTTP API, this will help >> > development of (non-Java) tools. >> > >> > It is not clear to me if we are going to add an http API useful for >> > "managing bookies" or a new HTTP REST-like Client API to BookKeeper. >> > I am referring to the fact that in the proposal there are API calls to >> > create ledgers and to read data. >> > I think we should separate this two aspects and maybe it is better to >> > address the 'bookie management' first, which is the work that Yiming >> > started. >> > >> [jia] It is targeting for the admin portal. the existence of the ledger >> api >> is just to simplify debugging or operations. we can eliminate the ‘create’ >> endpoint. >> > > OK so are you already thinking about specific tools to manage bookies > without the bookie shell > We would like to integrate this for schedulers (e.g. k8s). so we can deploy and operate bookkeeper easily in those environments. > > >> >> >> > >> > Another point very important to me is that if we are going to introduce >> > management operations via http we have to take into account security, at >> > least TLS and some kind of authentication. >> > About TLS it is very simple to achieve this, every HTTP server supports >> TLS >> > (https) >> > About authentication the problem is not so simple, Kerberos on HTTPs is >> > very complex, but we need to introduce some auth mechanism. >> > IMHO We can require the 'http server implementation' to implement >> security >> > but out of the box we have to supply basic support at least for one >> > provider bundled in the distribution package. >> > >> [jia] The BP focuses on filling up the useful endpoints. The security will >> be a separate BP. >> >> >> > >> > Some API can return very large result sets like 'list ledgers', actually >> > the HTTP Server subsystem exchanges strings in memory, we will need to >> > introduce some more smart way because it would be easy to bring down the >> > bookie just by calling that API multiple times concurrencly (it is just >> an >> > example) >> > >> [jia] We could add pagination into all the `list` api. >> > > Yes but it will be really tricky to implement, but please do it > > >> >> > >> > IMHO The 'create ledger' API is not useful, due to the design of BK you >> > have to create a ledger and then write immediately to it, I think that >> such >> > an API should allow the client to stream the contents of the ledger in >> the >> > HTTP body at least. But I think that a more stateful http API needs to >> be >> > designed to implement a pure Http client >> > >> [jia] Thanks, we are not planning to implement an http client. we will >> remove ‘create’ here. >> > > OK, thanks > >> >> >> > >> > Maybe we should add a more narrowed motivation and add the only useful >> APIs >> > to address those issues. >> > For instance in my company we would like to start creating a BookKeeper >> Web >> > UI, so we need some way to talk to bookies and exchange data, but in >> this >> > case I am interested in asking to bookies their status and the effective >> > contents of the bookie >> > >> [jia] The BP proposes adding a standard naming convention for adding admin >> endpoints. Feel free to propose the endpoints you would like to appear in >> the http admin portal. >> > > The ones you wrote are enough complete for me, I would like to have > read-only operation in order to have a global view on the status of the > system. > > -- Enrico > > > >> >> >> > >> > -- Enrico >> > >> > >> > 2017-09-13 6:09 GMT+02:00 Yiming Zang <yz...@twitter.com.invalid>: >> > >> > > Sure, I think the current HTTP endpoints in Twitter are only designed >> for >> > > Twitter specific, such as check quorum loss, check rack/region >> diversity. >> > > So the endpoints convention in Twitter are not the same as in the >> > proposal. >> > > I think it would be great to have an agreement on the API naming >> design, >> > so >> > > I like the API design in the proposal, I think the proposal looks >> good to >> > > me. >> > > >> > > Besides, we're currently only using GET functionalities in Twitter, >> but I >> > > notice there're a lot of POST and PUT APIs in the proposal which could >> > > change the bookie state or trigger some heavy workload. These APIs >> looks >> > a >> > > bit risky to me if we don't have any authentication enabled (in >> Twitter). >> > > >> > > On Tue, Sep 12, 2017 at 6:11 PM, Sijie Guo <guosi...@gmail.com> >> wrote: >> > > >> > > > + Yiming >> > > > >> > > > Yiming, if you have time, please take a look at this BP. let's see >> if >> > > > there are any conflicts with those you added for autorecovery. >> > > > >> > > > - Sijie >> > > > >> > > > On Tue, Sep 12, 2017 at 8:00 AM, Jia Zhai <zhaiji...@gmail.com> >> wrote: >> > > > >> > > >> Hi all, >> > > >> >> > > >> Based on Github #278 <https://github.com/apache/boo >> kkeeper/pull/278>, >> > > I >> > > >> have just posted a proposal regarding define BookKeeper public http >> > > >> endpoints: >> > > >> >> > > >> https://cwiki.apache.org/confluence/display/BOOKKEEPER/BP- >> > > >> 17%3A+Define+BookKeeper+public+http+endpoints >> > > >> >> > > >> >> > > >> Github #278 <https://github.com/apache/bookkeeper/pull/278> >> > introduces >> > > >> BookKeeper Http Endpoint module. However there are only two >> endpoints, >> > > >> which is “/heartbeat” and “/api/config/serverconfig”, defined in >> #278. >> > > In >> > > >> order to fully leverage the http modules, The goal of this BP is to >> > add >> > > >> more endpoints to this modules. >> > > >> >> > > >> Any comments are welcome and appreciated. >> > > >> >> > > >> >> > > >> Thanks. >> > > >> >> > > >> -Jia >> > > >> >> > > > >> > > > >> > > >> > >> > >
