+1. I liked this summary. JV, is this related to what you were writing? or anyone else want to drive this?
- Sijie On Mon, Oct 9, 2017 at 9:32 AM, Venkateswara Rao Jujjuri <jujj...@gmail.com> wrote: > Can we have a doc to put all these things? Thread has grown enough to cause > confusion. > > Immediate things. > 1. Don't assume new bookie if journal dir is empty. > 2. Put cookies through bookie format, and bookie never boots on an empty > cookie or mismatched cookie. > 3. We can live with operations procedure to deal with incarnation issue. > Infact we run an automated bookie decomm script which runs through the > entire metadata and makes sure that the bookie is not part of any ledger. > > For next step: > 1. Establish incarnation support. > 2. Deal with bitrot. > > Makes sense? > > JV > > On Mon, Oct 9, 2017 at 8:55 AM, Sijie Guo <guosi...@gmail.com> wrote: > > > On Oct 9, 2017 1:54 AM, "Ivan Kelly" <iv...@apache.org> wrote: > > > > Hi folks, > > > > I was travelling over the weekend, so I didn't have a chance to reply > > to anything on this thread. First off, as Enrico said, there's a lot > > of different topics being discussed at once. Perhaps each should be > > broken into a github issue, and then we can continue each conversation > > there, as it's getting a but unwieldy for email. > > > > I've created a cookie monster project, which we can throw all the issues > > into. > > https://github.com/apache/bookkeeper/projects/1 > > > > There's a few individual opinions I'd like to give here though. > > > > > Needing the check the instance of the bookie when auditing > > > > The auditor, while it does check when bookies have disappeared, it > > also periodically checks all ledgers by reading the first and last > > entry of each segment. So even if a bookie has resurrected, the > > auditor will find that it is missing entries it is supposed to have. > > > > > UUID in ledger metadata > > > > At least for the write path, I'm not sure if this is needed, but > > consider the following. > > > > Only one writer can "vote" on the entries of the ledger. Other writers > > are fencing writers. A fencing writer has to hit a majority of bookies > > to proceed to closing the ledger. Unless a majority have been wiped, > > it will not proceed to close as an empty ledger. However, if a > > majority have been wiped, the correct behaviour would be for it not be > > possible to close the ledger, as we cannot know what the end of the > > ledger is. > > > > That said, not boot if any ledger refers to a bookie could solve this. > > > > > No ledgers referencing bookie? (Sijie's suggestion) > > > > I'm resistant this idea, because it assumes a central oracle where all > > ledgers can be queried. I know we currently have this, but I don't > > think it scales for each bookie to read the metadata of the whole > > system. > > > > In any case, why not instead of refusing to start if any ledgers > > reference the bookie, on boot the bookie checks which ledgers it is > > supposed to have, and if it doesn't have them, start pulling the data > > for them itself. While doing this replication it should avoid all new > > writes. > > > > > > Yes, that's another thing we need to improve for auto recovery. It is not > > only on boot, you need to do it periodically, in the garbage collection > > thread. The bookie need to scan what ledgers are missing and what entries > > are missing and replicate them. > > > > > > > > > Storing the list of files in the cookie? (Enrico's suggestion) > > > > I don't think this is needed. The purpose of the cookie is to protect > > against stuff like a mount not coming up, or a machine being > > completely wiped. We assume that on a journalled filesystem, files > > don't just disappear arbitrarily. There may be corruption in > > individual files, but see my first point. > > > > Anyhow, as I said earlier, we should decide the broad topics here and > > move into issues. I've made a first pass. > > > > Regards, > > Ivan > > > > > > -- > Jvrao > --- > First they ignore you, then they laugh at you, then they fight you, then > you win. - Mahatma Gandhi >
