Re: Journal Corruption when disk is full

2018-02-17 Thread Ivan Kelly
On Fri, Feb 16, 2018 at 9:48 PM, Matteo Merli wrote: >> One thing that should circumvent this is that the bookie should go >> into readonly mode when it hits 95% full disk. > > I think this is only applying to the ledgers disk, but not for the journal. That doesn't seem like the right thing to do

Re: Journal Corruption when disk is full

2018-02-16 Thread Matteo Merli
> One thing that should circumvent this is that the bookie should go > into readonly mode when it hits 95% full disk. I think this is only applying to the ledgers disk, but not for the journal. And, to answer to Bobby, the switch to read-only mode feature was already present in 4.3 (again, just f

Re: Journal Corruption when disk is full

2018-02-15 Thread Ivan Kelly
On Thu, Feb 15, 2018 at 9:49 PM, Bobby Evans wrote: > I don't have the read only mode on disk full feature yet. I will look at > pulling it back to our fork, but I will also look at fixing the journaling > in general. Having spoken with the HDFS team here, they have seen a lot of > scary things

Re: Journal Corruption when disk is full

2018-02-15 Thread Bobby Evans
I don't have the read only mode on disk full feature yet. I will look at pulling it back to our fork, but I will also look at fixing the journaling in general. Having spoken with the HDFS team here, they have seen a lot of scary things that appear similar to this situation when a disk starts to g

Re: Journal Corruption when disk is full

2018-02-15 Thread Enrico Olivelli
Il gio 15 feb 2018, 18:45 Ivan Kelly ha scritto: > Hi Bobby, > > One thing that should circumvent this is that the bookie should go > into readonly mode when it hits 95% full disk. > > This change is in your branch[1]. Have you disabled it? > > Regards, > Ivan > > [1] > https://github.com/yahoo/b

Re: Journal Corruption when disk is full

2018-02-15 Thread Ivan Kelly
Hi Bobby, One thing that should circumvent this is that the bookie should go into readonly mode when it hits 95% full disk. This change is in your branch[1]. Have you disabled it? Regards, Ivan [1] https://github.com/yahoo/bookkeeper/blob/7fb556fa2dbc1d308b5d7ec3e2676b8b11704698/bookkeeper-ser

Journal Corruption when disk is full

2018-02-15 Thread Bobby Evans
We recently had a situation where the disk a journal is on filled up and we ended up with a partial edit in the journal. This ended up being a METAENTRY_ID_LEDGER_KEY so only a partial master key was output and this ended up making the bookie fail on startup. Because we are running a forked versi