Re: Cookies and empty disks

2017-11-09 Thread Sijie Guo
Hi all, I've sent out a pull request on fixing the cookie issue. please take a look. https://github.com/apache/bookkeeper/pull/712 On Mon, Oct 9, 2017 at 1:50 PM, Ivan Kelly wrote: > On Mon, Oct 9, 2017 at 6:32 PM, Venkateswara Rao Jujjuri > wrote: > > Can we have a doc to put all these thing

Re: Cookies and empty disks

2017-10-09 Thread Ivan Kelly
On Mon, Oct 9, 2017 at 6:32 PM, Venkateswara Rao Jujjuri wrote: > Can we have a doc to put all these things? Thread has grown enough to cause > confusion. I created a github project earlier today. We can manage the different streams of work from there. Each stream should have a doc though. https:

Re: Cookies and empty disks

2017-10-09 Thread Enrico Olivelli
I like this too. I have no time immediately for working on this sorry. Maybe the only blocker isse is about the boot with empty dirs which Sijie pointed Enrico Il lun 9 ott 2017, 19:08 Sijie Guo ha scritto: > +1. I liked this summary. > > JV, is this related to what you were writing? or anyone

Re: Cookies and empty disks

2017-10-09 Thread Sijie Guo
+1. I liked this summary. JV, is this related to what you were writing? or anyone else want to drive this? - Sijie On Mon, Oct 9, 2017 at 9:32 AM, Venkateswara Rao Jujjuri wrote: > Can we have a doc to put all these things? Thread has grown enough to cause > confusion. > > Immediate things. >

Re: Cookies and empty disks

2017-10-09 Thread Venkateswara Rao Jujjuri
Can we have a doc to put all these things? Thread has grown enough to cause confusion. Immediate things. 1. Don't assume new bookie if journal dir is empty. 2. Put cookies through bookie format, and bookie never boots on an empty cookie or mismatched cookie. 3. We can live with operations procedur

Re: Cookies and empty disks

2017-10-09 Thread Sijie Guo
On Oct 9, 2017 1:54 AM, "Ivan Kelly" wrote: Hi folks, I was travelling over the weekend, so I didn't have a chance to reply to anything on this thread. First off, as Enrico said, there's a lot of different topics being discussed at once. Perhaps each should be broken into a github issue, and the

Re: Cookies and empty disks

2017-10-09 Thread Sijie Guo
On Oct 9, 2017 1:13 AM, "Enrico Olivelli" wrote: 2017-10-09 9:21 GMT+02:00 Sijie Guo : > okay, but why do you want to track the list of files? I don't get your idea > here. > If you allow a bookie to start with a journal directory which contains the cookie file but without the other files the

Re: Cookies and empty disks

2017-10-09 Thread Ivan Kelly
>> In any case, why not instead of refusing to start if any ledgers >> reference the bookie, on boot the bookie checks which ledgers it is >> supposed to have, > > How can you do this without querying the big oracle? You can use the local > view as source of truth. Maybe I am missing one piece Sor

Re: Cookies and empty disks

2017-10-09 Thread Enrico Olivelli
Il lun 9 ott 2017, 10:54 Ivan Kelly ha scritto: > Hi folks, > > I was travelling over the weekend, so I didn't have a chance to reply > to anything on this thread. First off, as Enrico said, there's a lot > of different topics being discussed at once. Perhaps each should be > broken into a github

Re: Cookies and empty disks

2017-10-09 Thread Ivan Kelly
Hi folks, I was travelling over the weekend, so I didn't have a chance to reply to anything on this thread. First off, as Enrico said, there's a lot of different topics being discussed at once. Perhaps each should be broken into a github issue, and then we can continue each conversation there, as

Re: Cookies and empty disks

2017-10-09 Thread Enrico Olivelli
2017-10-09 9:21 GMT+02:00 Sijie Guo : > okay, but why do you want to track the list of files? I don't get your idea > here. > If you allow a bookie to start with a journal directory which contains the cookie file but without the other files the bookie thinks that have been persisted durably you

Re: Cookies and empty disks

2017-10-09 Thread Sijie Guo
okay, but why do you want to track the list of files? I don't get your idea here. - Sijie On Sun, Oct 8, 2017 at 11:45 PM, Enrico Olivelli wrote: > 2017-10-09 7:52 GMT+02:00 Sijie Guo : > > > On Sat, Oct 7, 2017 at 9:53 AM, Enrico Olivelli > > wrote: > > > > > Il sab 7 ott 2017, 00:27 Sijie Gu

Re: Cookies and empty disks

2017-10-08 Thread Enrico Olivelli
2017-10-09 7:52 GMT+02:00 Sijie Guo : > On Sat, Oct 7, 2017 at 9:53 AM, Enrico Olivelli > wrote: > > > Il sab 7 ott 2017, 00:27 Sijie Guo ha scritto: > > > > > Enrico, > > > > > > Let's try to come to a conclusion or an agreement what we should fix > and > > > improve, before talking who is goin

Re: Cookies and empty disks

2017-10-08 Thread Sijie Guo
On Sat, Oct 7, 2017 at 9:53 AM, Enrico Olivelli wrote: > Il sab 7 ott 2017, 00:27 Sijie Guo ha scritto: > > > Enrico, > > > > Let's try to come to a conclusion or an agreement what we should fix and > > improve, before talking who is going to drive this. > > > > Sure. > > This is my point of vie

Re: Cookies and empty disks

2017-10-07 Thread Enrico Olivelli
Il sab 7 ott 2017, 00:27 Sijie Guo ha scritto: > Enrico, > > Let's try to come to a conclusion or an agreement what we should fix and > improve, before talking who is going to drive this. > Sure. This is my point of view: View have separate issues: - missing checksums, to protect fence bits - h

Re: Cookies and empty disks

2017-10-06 Thread Sijie Guo
Enrico, Let's try to come to a conclusion or an agreement what we should fix and improve, before talking who is going to drive this. - Sijie On Fri, Oct 6, 2017 at 1:14 PM, Enrico Olivelli wrote: > +1 for fixing the problem of missing cookie in 4.6 > > Who drives the issue? > > Thank you all f

Re: Cookies and empty disks

2017-10-06 Thread Sijie Guo
On Fri, Oct 6, 2017 at 12:27 PM, Venkateswara Rao Jujjuri wrote: > Thanks for the writeup Sijie, comments below. > > On Fri, Oct 6, 2017 at 12:14 PM, Sijie Guo wrote: > > > I think the question is mainly around "how do we recognize the bookie" or > > "incarnations". And the purpose of a cookie i

Re: Cookies and empty disks

2017-10-06 Thread Enrico Olivelli
+1 for fixing the problem of missing cookie in 4.6 Who drives the issue? Thank you all for the interesting points Enrico Il ven 6 ott 2017, 21:27 Venkateswara Rao Jujjuri ha scritto: > Thanks for the writeup Sijie, comments below. > > On Fri, Oct 6, 2017 at 12:14 PM, Sijie Guo wrote: > > > I

Re: Cookies and empty disks

2017-10-06 Thread Venkateswara Rao Jujjuri
Thanks for the writeup Sijie, comments below. On Fri, Oct 6, 2017 at 12:14 PM, Sijie Guo wrote: > I think the question is mainly around "how do we recognize the bookie" or > "incarnations". And the purpose of a cookie is designed for addressing > "incarnations". > > I will try to cover following

Re: Cookies and empty disks

2017-10-06 Thread Sijie Guo
I think the question is mainly around "how do we recognize the bookie" or "incarnations". And the purpose of a cookie is designed for addressing "incarnations". I will try to cover following aspects, and will try to answer questions that Ivan and JV raised. - what is cookie? - how the behavior be

Re: Cookies and empty disks

2017-10-06 Thread Ivan Kelly
> Just that may not be sufficient. > 1. The UUID needs to be part of the ledger metadata so that the auditor > knows it is looking at different bookie. Agreed, we should include it in ensemble info. > 2. Bookie need to know if the writes and reads from the client are intended > for it or not. If n

Re: Cookies and empty disks

2017-10-06 Thread Venkateswara Rao Jujjuri
On Fri, Oct 6, 2017 at 10:01 AM, Ivan Kelly wrote: > On Fri, Oct 6, 2017 at 6:35 PM, Venkateswara Rao Jujjuri > wrote: > >> However, imagine that the fenced message is only in the journal on b2, > >> b2 crashes, something wipes the journal directory and then b2 comes > >> back up. > > > > In thi

Re: Cookies and empty disks

2017-10-06 Thread Ivan Kelly
On Fri, Oct 6, 2017 at 6:35 PM, Venkateswara Rao Jujjuri wrote: >> However, imagine that the fenced message is only in the journal on b2, >> b2 crashes, something wipes the journal directory and then b2 comes >> back up. > > In this case what happened? > 1. We have WQ = 1 > 2. We had data loss (cr

Re: Cookies and empty disks

2017-10-06 Thread Venkateswara Rao Jujjuri
> However, imagine that the fenced message is only in the journal on b2, > b2 crashes, something wipes the journal directory and then b2 comes > back up. In this case what happened? 1. We have WQ = 1 2. We had data loss (crash and comeup clean) But yeah, in addition to dataloss we have fencing vi

Re: Cookies and empty disks

2017-10-06 Thread Ivan Kelly
> The case you described here is "almost correct". But there is an key here: > B2 can't startup itself if journal disk is wiped out, because the cookie is > missed. This is what I expected to see, but isn't the case. List journalCookies = Lists.newArrayList(); // try to read cook

Re: Cookies and empty disks

2017-10-06 Thread Sijie Guo
On Oct 6, 2017 3:07 AM, "Ivan Kelly" wrote: Hi folks, Following up from the meeting yesterday, I said I would look into the code to verify the behaviour because there could be a correctness problem. I think there could be an issue. The code is convoluted, but my understanding of it is as follow

Cookies and empty disks

2017-10-06 Thread Ivan Kelly
Hi folks, Following up from the meeting yesterday, I said I would look into the code to verify the behaviour because there could be a correctness problem. I think there could be an issue. The code is convoluted, but my understanding of it is as follows. We check all ledger, journal and index dir