Re: Question regarding Checkpoint logic in SortedLedgerStorage

2017-10-27 Thread Sijie Guo
FYI. I created a test to reproduce the behavior. https://github.com/apache/bookkeeper/pull/677 - Sijie On Tue, Oct 24, 2017 at 3:41 PM, Sijie Guo wrote: > I filed an issue for the problem discussed here: https://github.com/ > apache/bookkeeper/issues/659 > > - Sijie > > On Tue, Oct 24, 2017 at

Re: Question regarding Checkpoint logic in SortedLedgerStorage

2017-10-25 Thread Sijie Guo
On Oct 25, 2017 12:53 AM, "Enrico Olivelli - Diennea" < enrico.olive...@diennea.com> wrote: Il giorno mar, 24/10/2017 alle 15.41 -0700, Sijie Guo ha scritto: I filed an issue for the problem discussed here: https://github.com/apache/bookkeeper/issues/659 Sijie, Charan, thank you for catching th

Re: Question regarding Checkpoint logic in SortedLedgerStorage

2017-10-25 Thread Enrico Olivelli - Diennea
Il giorno mar, 24/10/2017 alle 15.41 -0700, Sijie Guo ha scritto: I filed an issue for the problem discussed here: https://github.com/apache/bookkeeper/issues/659 Sijie, Charan, thank you for catching this IMHO we should take options 1 o 2 as Sijie proposed Sijie, Do you have an approximate ide

Re: Question regarding Checkpoint logic in SortedLedgerStorage

2017-10-24 Thread Sijie Guo
I filed an issue for the problem discussed here: https://github.com/apache/bookkeeper/issues/659 - Sijie On Tue, Oct 24, 2017 at 2:06 PM, Charan Reddy G wrote: > Sure Sijie and Ivan. It sounds appropriate. > > Thanks, > Charan > > On Tue, Oct 24, 2017 at 1:00 AM, Sijie Guo wrote: > >> Yes, we

Re: Question regarding Checkpoint logic in SortedLedgerStorage

2017-10-24 Thread Charan Reddy G
Sure Sijie and Ivan. It sounds appropriate. Thanks, Charan On Tue, Oct 24, 2017 at 1:00 AM, Sijie Guo wrote: > Yes, we should fix this ASAP. > > Charan, what is your opinion on the fix? > > - Sijie > > On Tue, Oct 24, 2017 at 12:12 AM, Ivan Kelly wrote: > >> > Use the approach that I used befo

Re: Question regarding Checkpoint logic in SortedLedgerStorage

2017-10-24 Thread Sijie Guo
Yes, we should fix this ASAP. Charan, what is your opinion on the fix? - Sijie On Tue, Oct 24, 2017 at 12:12 AM, Ivan Kelly wrote: > > Use the approach that I used before (as in the old commits at twitter’s > > branch). Ledger storage is responsible for instantiating the checkpoints. > I would

Re: Question regarding Checkpoint logic in SortedLedgerStorage

2017-10-24 Thread Ivan Kelly
> Use the approach that I used before (as in the old commits at twitter’s > branch). Ledger storage is responsible for instantiating the checkpoints. I would go with this approach. It was probably me that asked for the changes in the other direction before, but I can't remember why I asked for them

Re: Question regarding Checkpoint logic in SortedLedgerStorage

2017-10-23 Thread Sijie Guo
Hi all, Sorry for late response. I apologize for being late. I have spent my time over the weekend going through all the history about the checkpoint changes, because the implementation in current master doesn’t match what I had in my mind. Now I think I have all the details ready for explain. Th

Re: Question regarding Checkpoint logic in SortedLedgerStorage

2017-10-17 Thread Sijie Guo
On Tue, Oct 17, 2017 at 11:46 AM, Charan Reddy G wrote: > @Sijie..Did you get chance to go through the scenario/code path. > Just come back to US today. I will check and get back to you. > > @JV..will create a bug, once I get clarity. > > Thanks, > Charan > > On Fri, Oct 13, 2017 at 5:19 PM, V

Re: Question regarding Checkpoint logic in SortedLedgerStorage

2017-10-17 Thread Charan Reddy G
@Sijie..Did you get chance to go through the scenario/code path. @JV..will create a bug, once I get clarity. Thanks, Charan On Fri, Oct 13, 2017 at 5:19 PM, Venkateswara Rao Jujjuri wrote: > Charan this looks like an issue to me. Do we have a defect/issue opened? > > On Fri, Oct 13, 2017 at 4:

Re: Question regarding Checkpoint logic in SortedLedgerStorage

2017-10-13 Thread Venkateswara Rao Jujjuri
Charan this looks like an issue to me. Do we have a defect/issue opened? On Fri, Oct 13, 2017 at 4:18 PM, Sijie Guo wrote: > Charan, > > Didn't mean to say the logic is correct. I was just trying to point out > the points that I remembered for checkpoint. > > I am currently traveling, so I don't

Re: Question regarding Checkpoint logic in SortedLedgerStorage

2017-10-13 Thread Sijie Guo
Charan, Didn't mean to say the logic is correct. I was just trying to point out the points that I remembered for checkpoint. I am currently traveling, so I don't have code available to check the sequence. I can check the logic when I am close to the laptop. Sijie On Oct 14, 2017 6:11 AM, "Char

Re: Question regarding Checkpoint logic in SortedLedgerStorage

2017-10-13 Thread Charan Reddy G
Hey Sijie, I'm not questioning the semantics of checkpoint or the optimization which was added with Bookkeeper-564. But my concern is are we sure, checkpoint logic/code is correct and "marker is only updated when all the entries added before are persisted.", in the case of SortedLedgerStorage. Can

Re: Question regarding Checkpoint logic in SortedLedgerStorage

2017-10-13 Thread Sijie Guo
The core of the checkpoint is: - marker is only updated when all the entries added before are persisted. That means it doesn't affect correctness if entries added after are flushed. - the flush in entry log files is just writing data to filesystem. The real fsync happens after checkpoint. The sep