Cool. I would expect this is a big change. It would be good if you can divide it into smaller tasks, so people can review them easier.
- Sijie On Tue, Sep 12, 2017 at 1:05 AM, Enrico Olivelli <eolive...@gmail.com> wrote: > Thank you all ! > > I will copy the content of the Final draft to the Wiki and mark the > document as "Accepted" > > I will send a PR soon but it will depend on BP-15 New CreateLeader API > > I hope we could make it for 4.6 > > > Enrico > > > 2017-09-11 18:58 GMT+02:00 Sijie Guo <guosi...@gmail.com>: > > > Enrico, > > > > Feel free to close the thread and mark this BP as accepted, if there is > no > > -1. > > > > - Sijie > > > > On Mon, Sep 11, 2017 at 2:26 AM, Enrico Olivelli <eolive...@gmail.com> > > wrote: > > > > > Ping > > > > > > 2017-09-07 9:32 GMT+02:00 Enrico Olivelli <eolive...@gmail.com>: > > > > > > > Hi all, > > > > > > > > > > > > You can find the revised proposal here > > > > https://cwiki.apache.org/confluence/display/BOOKKEEPER/ > > > > BP-14+Relax+durability > > > > > > > > The link to the document open for comments is this: > > > > https://docs.google.com/document/d/1yNi9t2_ > > > deOOMXDaGzrnmaHTQeB3B3Fnym82DU > > > > ERH7LM/edit?usp=sharing > > > > > > > > Please check it out > > > > We are going to review this Proposal at the meeting > > > > > > > > -- Enrico > > > > > > > > > > > > 2017-08-30 8:56 GMT+02:00 Enrico Olivelli <eolive...@gmail.com>: > > > > > > > >> Thank you Sijie for summarizing and thanks to the community for > > helping > > > >> in this important enhancement to BookKeeper > > > >> > > > >> I am convinced that as JV pointed out we need to declare at ledger > > > >> creation time that the ledger is going to perform no-sync writes. > > > >> > > > >> I think we need an explicit declaration currently to make things > > "clear" > > > >> to the developer which is using the LedgerHandle API even and ledger > > > >> creation tyime. > > > >> > > > >> The case is that we are going to forbid "striping" ledgers (ensemble > > > size > > > >> > quorum size) for no-sync writes in the first implementation: > > > >> - one option is to fail at the first no-sync addEntry, but this > will > > be > > > >> really uncomfortable because usually the ack/write/ensemble sizes > are > > > >> configured by the admin, and there will be configurations in which > > > errors > > > >> will come out only after starting the system. > > > >> - the second option is to make the developer explicitly enable > no-sync > > > >> writes at creation time and fail the creation of the ledger if the > > > >> requested combination of options if not possible > > > >> > > > >> I am not sure that the changes to the bookie internals are a > > Client-API > > > >> matter, maybe we can leverage custom metadata (as JV said) in order > to > > > make > > > >> the bookie handle ledgers in a different manner, this way will be > > always > > > >> open as custom metadata are already here. > > > >> > > > >> JV preferred the ledger-type approach, the dual solution is to > > introduce > > > >> a list of "capabilities" or "ledger options". > > > >> I think that this ability to perform no-syc writes is so important > > that > > > >> "custom metadata" is not the good place to declare it, same for > > "ledger > > > >> type" > > > >> > > > >> So I am proposing to add a boolean 'allowNoSyncWrites" at ledger > > > creation > > > >> time, without writing in to ledger metadata on ZK, > > > >> I think that if further improvements will need ledger metadata > changes > > > we > > > >> will do. > > > >> > > > >> I have updated the BP-14 document, I have added an "Open issues" > > footer > > > >> with the open points, > > > >> please add comments and I will correct the document as soon as > > possible. > > > >> > > > >> > > > >> Enrico > > > >> > > > >> > > > >> > > > >> > > > >> 2017-08-30 1:24 GMT+02:00 Sijie Guo <guosi...@gmail.com>: > > > >> > > > >>> Thank you, Enrico, JV. > > > >>> > > > >>> These are great discussions. > > > >>> > > > >>> After reading these two proposals, I have a few very high-level > > > comments, > > > >>> dividing into three categories. > > > >>> > > > >>> > > > >>> *API* > > > >>> > > > >>> - I think there are not fundamentally differences between these two > > > >>> proposals. > > > >>> They are trying to achieve similar goals by exposing durability > > levels > > > in > > > >>> different way. > > > >>> So this will be a discussion on what API/interface should look like > > > from > > > >>> user / admin perspective. > > > >>> I would suggest focusing what would be the API itself, putting the > > > >>> implementation design aside when talking about this. > > > >>> > > > >>> *Core* > > > >>> > > > >>> - Both proposals need to deal with a core function - what happen to > > LAC > > > >>> and > > > >>> what semantic that bookkeeper provides. > > > >>> JV did a good summary in his proposal. However I am not a fan of > > > >>> maintaining two different semantics. So I am looking for > > > >>> a solution that bookkeeper can only maintain one semantic. The > > semantic > > > >>> is > > > >>> basically: > > > >>> > > > >>> 1) LAC only advanced when entries before LAC are committed to the > > > >>> persistent storage > > > >>> 2) All the entries until LAC are successfully committed to the > > > >>> persistence > > > >>> storage > > > >>> 3) Entries until LAC: all the entries must be readable all the > time. > > > >>> > > > >>> If we maintain such semantic, there is no need to change the auto > > > >>> recovery > > > >>> protocol in bookkeeper. All what we guarantee are the entries > durably > > > >>> persistent. > > > >>> > > > >>> In order to maintain such semantic, I think both me and JV proposed > > > >>> similar > > > >>> solution in either proposal. I am trying to finalize one here: > > > >>> > > > >>> * bookie maintains a LAS (Last Add Synced) point for each entry. > > > >>> * LAS can be piggybacked on AddResponses > > > >>> * Client uses the LAS to advance LAC. > > > >>> > > > >>> If we can agree on the core semantic we are going to provide, the > > other > > > >>> things are just logistics. > > > >>> > > > >>> *Others* > > > >>> > > > >>> - Regarding separating journal or bypassing journal, there is no > > > >>> difference > > > >>> when we talking from the core semantic. They are all non-durably > > writes > > > >>> (acknowledging before fsyncing). > > > >>> We can start with same journal approach (but just acknowledge > before > > > >>> fsyncing), implement the core and add other options later on. > > > >>> > > > >>> > > > >>> From my point of view, I'd be more interesting in providing a > single > > > >>> consistent durable semantic that application can rely on for both > > > durable > > > >>> writes and non-durable writes. The other stuffs seem to be more > > > logistics > > > >>> things. > > > >>> > > > >>> > > > >>> - Sijie > > > >>> > > > >>> > > > >>> On Mon, Aug 28, 2017 at 11:27 PM, Enrico Olivelli < > > eolive...@gmail.com > > > > > > > >>> wrote: > > > >>> > > > >>> > 2017-08-29 8:01 GMT+02:00 Venkateswara Rao Jujjuri < > > > jujj...@gmail.com > > > >>> >: > > > >>> > > > > >>> > > I don't believe I fully followed your second case. But even in > > this > > > >>> case, > > > >>> > > your major concern is about the additional 'sync' RPC? > > > >>> > > > > > >>> > > > > >>> > yes apart from that I am fine with your proposal too, that is to > > > have a > > > >>> > LedgerType which drives durability > > > >>> > and I think we need to add per-entry durability options > > > >>> > > > > >>> > I think that at least for the 'simple' no-sync addEntry we do not > > > need > > > >>> to > > > >>> > change many things, I am drafting a prototype, I will share it as > > > soon > > > >>> as > > > >>> > we all agree on the roadmap > > > >>> > > > > >>> > The first implementation can cover the first cases (no-sync > > addEntry) > > > >>> and > > > >>> > change the way the writer advances the LAC in order to support > > > 'relaxed > > > >>> > durability writes'. > > > >>> > This change will be compatible with future improvements and it > will > > > >>> open > > > >>> > the door for big changes on the bookie side like bypassing the > > > journal > > > >>> or > > > >>> > leveraging multiple journals..... > > > >>> > > > > >>> > -- Enrico > > > >>> > > > > >>> > or something else that the LedgerType proposal won't work? > > > >>> > > > > > >>> > > > > >>> > > > > > >>> > > > > > >>> > > On Mon, Aug 28, 2017 at 7:35 AM, Enrico Olivelli < > > > >>> eolive...@gmail.com> > > > >>> > > wrote: > > > >>> > > > > > >>> > > > I think that having a set of options on the ledger metadata > > will > > > >>> be a > > > >>> > > good > > > >>> > > > enhancement and I am sure we will do it as soon as it will be > > > >>> needed, > > > >>> > > maybe > > > >>> > > > we do not need it now. > > > >>> > > > > > > >>> > > > Actually I think we will need to declare this > durability-level > > at > > > >>> entry > > > >>> > > > level to support some uses cases in BP-14 document, let me > > > explain > > > >>> two > > > >>> > of > > > >>> > > > my usecases for which I need it: > > > >>> > > > > > > >>> > > > At higher level we have to choices: > > > >>> > > > > > > >>> > > > A) per-ledger durability options (JV proposal) > > > >>> > > > all addEntry operations are durable or non-durable and there > is > > > an > > > >>> > > explicit > > > >>> > > > 'sync' API (+ forced sync at close) > > > >>> > > > > > > >>> > > > B) per-entry durability options (original BP-14 proposal) > > > >>> > > > every addEntry has an own durable/non-durable option > > > >>> (sync/no-sync), > > > >>> > with > > > >>> > > > the ability to call 'sync' without addEntry (+ forced sync at > > > >>> close) > > > >>> > > > > > > >>> > > > I am speaking about the the database WAL case, I am using the > > > >>> ledger as > > > >>> > > > segment for the WAL of a database and I am writing all data > > > >>> changes in > > > >>> > > the > > > >>> > > > scope of a 'transaction' with the relaxed-durability flag, > > then I > > > >>> am > > > >>> > > > writing the 'transaction committed' entry with "strict > > > durability" > > > >>> > > > requirement, this will in fact require that all previous > > entries > > > >>> are > > > >>> > > > persisted durably and so that the transaction will never be > > lost. > > > >>> > > > > > > >>> > > > In this scenario we would need an addEntry + sync API in > fact: > > > >>> > > > > > > >>> > > > using option A) the WAL will look like: > > > >>> > > > - open ledger no-sync = true > > > >>> > > > - addEntry (set foo=bar) (this will be no-sync) > > > >>> > > > - addEntry (set foo=bar2) (this will be no-sync) > > > >>> > > > - addEntry (commit) > > > >>> > > > - sync > > > >>> > > > > > > >>> > > > using option B) the WAL will look like > > > >>> > > > - open ledger > > > >>> > > > - addEntry (set foo=bar), no-sync > > > >>> > > > - addEntry (set foo=bar2), no-sync > > > >>> > > > - addEntry (commit), sync > > > >>> > > > > > > >>> > > > in case B) we are "saving" one RPC call to every bookie (the > > > 'sync' > > > >>> > one) > > > >>> > > > same for single data change entries, like updating a single > > > record > > > >>> on > > > >>> > the > > > >>> > > > database, this with BK 4.5 "costs" only a single RPC to every > > > >>> bookie > > > >>> > > > > > > >>> > > > Second case: > > > >>> > > > I am using BookKeeper to store binary objects, so I am > packing > > > more > > > >>> > > > 'objects' (named sequences of bytes) into a single ledger, > like > > > >>> you do > > > >>> > > when > > > >>> > > > you write many records to a file in a streaming fashion and > > keep > > > >>> track > > > >>> > of > > > >>> > > > offsets of the beginning of every record (LedgerHandeAdv is > > > >>> perfect for > > > >>> > > > this case). > > > >>> > > > I am not using a single ledger per 'file' because it kills > > > >>> zookeeper to > > > >>> > > > create many ledgers very fast, in my systems I have big busts > > of > > > >>> > writes, > > > >>> > > > which need to be really "fast", so I am writing multiple > > 'files' > > > to > > > >>> > every > > > >>> > > > single ledger. So the close-to-open consistency at ledger > level > > > is > > > >>> not > > > >>> > > > suitable for this case. > > > >>> > > > I have to write as fast as possible to this 'ledger-backed' > > > >>> stream, and > > > >>> > > as > > > >>> > > > with a 'traditional' filesystem I am writing parts of each > > file > > > >>> and > > > >>> > than > > > >>> > > > requiring 'sync' at the end of each file. > > > >>> > > > Using BookKeeper you need to split big 'files' into "little" > > > >>> parts, you > > > >>> > > > cannot transmit the contents as to "real" stream on network. > > > >>> > > > > > > >>> > > > I am not talking about bookie level implementation details I > > > would > > > >>> like > > > >>> > > to > > > >>> > > > define the high level API in order to support all the > relevant > > > >>> known > > > >>> > use > > > >>> > > > cases and keep space for the future, > > > >>> > > > at this moment adding a per-entry 'durability option' seems > to > > be > > > >>> very > > > >>> > > > flexible and simple to implement, it does not prevent us from > > > doing > > > >>> > > further > > > >>> > > > improvements, like namely skipping the journal. > > > >>> > > > > > > >>> > > > Enrico > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > 2017-08-26 19:55 GMT+02:00 Enrico Olivelli < > > eolive...@gmail.com > > > >: > > > >>> > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > On sab 26 ago 2017, 19:19 Venkateswara Rao Jujjuri < > > > >>> > jujj...@gmail.com> > > > >>> > > > > wrote: > > > >>> > > > > > > > >>> > > > >> Hi all, > > > >>> > > > >> > > > >>> > > > >> As promised during Thursday call, here is my proposal. > > > >>> > > > >> > > > >>> > > > >> *NOTE*: Major difference in this proposal compared to > > Enrico’s > > > >>> > > > >> <https://docs.google.com/document/d/ > 1JLYO3K3tZ5PJGmyS0YK_- > > > >>> > > > >> NW8VOUUgUWVBmswCUOG158/edit#heading=h.q2rewiqndr5v> > > > >>> > > > >> is > > > >>> > > > >> making the durability a property of the ledger(type) as > > > opposed > > > >>> to > > > >>> > > > >> addEntry(). Rest of the technical details have a lot of > > > >>> > similarities. > > > >>> > > > >> > > > >>> > > > > > > > >>> > > > > Thank you JV. I have just read quickly the doc and your > view > > is > > > >>> > > centantly > > > >>> > > > > broader. > > > >>> > > > > I will dig into the doc as soon as possible on Monday. > > > >>> > > > > For me it is ok to have a ledger wide configuration I think > > > that > > > >>> the > > > >>> > > most > > > >>> > > > > important decision is about the API we will provide as in > the > > > >>> future > > > >>> > it > > > >>> > > > > will be difficult to change it. > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > Cheers > > > >>> > > > > Enrico > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > >> https://docs.google.com/document/d/ > > 1g1eBcVVCZrTG8YZliZP0LVqv > > > >>> Wpq43 > > > >>> > > > >> 2ODEghrGVQ4d4Q/edit?usp=sharing > > > >>> > > > >> > > > >>> > > > >> On Thu, Aug 24, 2017 at 1:14 AM, Enrico Olivelli < > > > >>> > eolive...@gmail.com > > > >>> > > > > > > >>> > > > >> wrote: > > > >>> > > > >> > > > >>> > > > >> > Thank you all for the comments and for taking a look to > > the > > > >>> > document > > > >>> > > > so > > > >>> > > > >> > soon. > > > >>> > > > >> > I have updated the doc, we will discuss the document at > > the > > > >>> > meeting, > > > >>> > > > >> > > > > >>> > > > >> > > > > >>> > > > >> > Enrico > > > >>> > > > >> > > > > >>> > > > >> > 2017-08-24 2:27 GMT+02:00 Sijie Guo <guosi...@gmail.com > >: > > > >>> > > > >> > > > > >>> > > > >> > > Enrico, > > > >>> > > > >> > > > > > >>> > > > >> > > Thank you so much! It is a great effort for putting > this > > > up. > > > >>> > > Overall > > > >>> > > > >> > looks > > > >>> > > > >> > > good. I made some comments, we can discuss at > tomorrow's > > > >>> > community > > > >>> > > > >> > meeting. > > > >>> > > > >> > > > > > >>> > > > >> > > - Sijie > > > >>> > > > >> > > > > > >>> > > > >> > > On Wed, Aug 23, 2017 at 8:25 AM, Enrico Olivelli < > > > >>> > > > eolive...@gmail.com > > > >>> > > > >> > > > > >>> > > > >> > > wrote: > > > >>> > > > >> > > > > > >>> > > > >> > > > Hi all, > > > >>> > > > >> > > > I have drafted a first proposal for BP-14 - Relax > > > >>> Durability > > > >>> > > > >> > > > > > > >>> > > > >> > > > We are talking about limiting the number of fsync to > > the > > > >>> > journal > > > >>> > > > >> while > > > >>> > > > >> > > > preserving the correctness of the LAC protocol. > > > >>> > > > >> > > > > > > >>> > > > >> > > > This is the link to the wiki page, but as the issue > is > > > >>> huge we > > > >>> > > > >> prefer > > > >>> > > > >> > to > > > >>> > > > >> > > > use Google Documents for sharing comments > > > >>> > > > >> > > > https://cwiki.apache.org/ > > confluence/display/BOOKKEEPER/ > > > >>> > > > >> > > > BP+-+14+Relax+durability > > > >>> > > > >> > > > > > > >>> > > > >> > > > This is the document > > > >>> > > > >> > > > https://docs.google.com/document/d/ > > > 1JLYO3K3tZ5PJGmyS0YK_- > > > >>> > > > >> > > > NW8VOUUgUWVBmswCUOG158/edit?usp=sharing > > > >>> > > > >> > > > > > > >>> > > > >> > > > All comments are welcome > > > >>> > > > >> > > > > > > >>> > > > >> > > > I have added DL dev list in cc as the discussion is > > > >>> > interesting > > > >>> > > > for > > > >>> > > > >> > both > > > >>> > > > >> > > > groups > > > >>> > > > >> > > > > > > >>> > > > >> > > > Enrico Olivelli > > > >>> > > > >> > > > > > > >>> > > > >> > > > > > >>> > > > >> > > > > >>> > > > >> > > > >>> > > > >> > > > >>> > > > >> > > > >>> > > > >> -- > > > >>> > > > >> Jvrao > > > >>> > > > >> --- > > > >>> > > > >> First they ignore you, then they laugh at you, then they > > fight > > > >>> you, > > > >>> > > then > > > >>> > > > >> you win. - Mahatma Gandhi > > > >>> > > > >> > > > >>> > > > > -- > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > -- Enrico Olivelli > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > -- > > > >>> > > Jvrao > > > >>> > > --- > > > >>> > > First they ignore you, then they laugh at you, then they fight > > you, > > > >>> then > > > >>> > > you win. - Mahatma Gandhi > > > >>> > > > > > >>> > > > > >>> > > > >> > > > >> > > > > > > > > > >
