Re: [PATCH 0/4] dm verity: add support for error correction

2015-12-03 Thread Sami Tolvanen
On Thu, Nov 12, 2015 at 11:30:00AM +0100, Milan Broz wrote: > Yes, please, patches do not to be production ready (I can integrate > it to veritysetup upstream myself) I'm working on this and should have a patch for veritysetup for you next week. Sami -- To unsubscribe from this list: send the lin

Re: [PATCH 0/4] dm verity: add support for error correction

2015-12-03 Thread Sami Tolvanen
On Thu, Nov 12, 2015 at 01:50:04PM -0500, Mikulas Patocka wrote: > What flash controller and chips do you use? Considering the number of different devices running Android, I don't have a good answer for this. I'm guessing most of them. > Is the silent data corruption permanent or transient? Most

Re: [PATCH 0/4] dm verity: add support for error correction

2015-12-03 Thread Sami Tolvanen
On Wed, Dec 02, 2015 at 03:22:03PM -0500, Mike Snitzer wrote: > Any progress on v2 that takes into account previous review feedback? Yes. Sorry for the delay, the changes have gone through a couple of rounds of internal reviews and testing first. > Note I've staged the first 2 patches in this ser

Re: [PATCH 0/4] dm verity: add support for error correction

2015-12-02 Thread Mike Snitzer
On Mon, Nov 09 2015 at 2:19P -0500, Sami Tolvanen wrote: > On Mon, Nov 09, 2015 at 11:37:35AM -0500, Mike Snitzer wrote: > > I'm left wondering: can the new error correction code be made an > > optional feature that is off by default? -- so as to preserve some > > isolation of this new code from

Re: [PATCH 0/4] dm verity: add support for error correction

2015-11-12 Thread Mikulas Patocka
On Mon, 9 Nov 2015, Sami Tolvanen wrote: > > Also, the 2 other big questions from Mikulas need answering: > > 1) why aren't you actually adjustng error codes, returning success, if > >dm-verity was able to trap/correct the corruption? > > We don't see actual I/O errors very often. Most corr

Re: [PATCH 0/4] dm verity: add support for error correction

2015-11-12 Thread Milan Broz
On 11/09/2015 08:19 PM, Sami Tolvanen wrote: ... > We don't see actual I/O errors very often. Most corruption we've seen > is caused by flaky hardware that doesn't return errors. However, I can > certainly change to code to attempt recovery in this case too. So if I understand it correctly, there

Re: [PATCH 0/4] dm verity: add support for error correction

2015-11-09 Thread Mike Snitzer
On Mon, Nov 09 2015 at 2:19pm -0500, Sami Tolvanen wrote: > On Mon, Nov 09, 2015 at 11:37:35AM -0500, Mike Snitzer wrote: > > I'm left wondering: can the new error correction code be made an > > optional feature that is off by default? -- so as to preserve some > > isolation of this new code fro

Re: [PATCH 0/4] dm verity: add support for error correction

2015-11-09 Thread Sami Tolvanen
On Mon, Nov 09, 2015 at 11:37:35AM -0500, Mike Snitzer wrote: > I'm left wondering: can the new error correction code be made an > optional feature that is off by default? -- so as to preserve some > isolation of this new code from the old dm-verity behaviour. It's optional in the sense that you m

Re: [PATCH 0/4] dm verity: add support for error correction

2015-11-09 Thread Mike Snitzer
On Thu, Nov 05 2015 at 12:33pm -0500, Sami Tolvanen wrote: > On Thu, Nov 05, 2015 at 08:34:04AM +0100, Milan Broz wrote: > > could you please elaborate why is all this needed? To extend support > > of some faulty flash chips? > > This makes dm-verity more robust against corruption caused by eith

Re: [PATCH 0/4] dm verity: add support for error correction

2015-11-09 Thread Austin S Hemmelgarn
On 2015-11-07 10:18, Mikulas Patocka wrote: On Fri, 6 Nov 2015, Sami Tolvanen wrote: On Fri, Nov 06, 2015 at 12:23:29PM -0500, Mikulas Patocka wrote: Do you have some real case where such error corrections increase longevity of some device? Yes, there have been several cases where read-onl

Re: [dm-devel] [PATCH 0/4] dm verity: add support for error correction

2015-11-07 Thread Mikulas Patocka
On Fri, 6 Nov 2015, Sami Tolvanen wrote: > On Fri, Nov 06, 2015 at 10:05:24PM +0100, Zdenek Kabelac wrote: > > So how do you want to correctly 'restore' missing full sectors > > with just 0.8% data overhead ?? > > We use interleaving. Each byte in a 4k block is part of a different > Reed-Solom

Re: [dm-devel] [PATCH 0/4] dm verity: add support for error correction

2015-11-07 Thread Mikulas Patocka
On Fri, 6 Nov 2015, Sami Tolvanen wrote: > On Fri, Nov 06, 2015 at 08:20:15PM +0100, Zdenek Kabelac wrote: > > i.e. you have 1G of space - you want to give 250MB as 'redundancy' - > > so create 4 partition > > We cannot afford to set aside 25% of read-only partition space for You can have

Re: [PATCH 0/4] dm verity: add support for error correction

2015-11-07 Thread Mikulas Patocka
On Fri, 6 Nov 2015, Sami Tolvanen wrote: > On Fri, Nov 06, 2015 at 12:23:29PM -0500, Mikulas Patocka wrote: > > I'm also wondering what is this patch useful for. Disks and flash > > controllers have their own error detection and correction > > I think I addressed this earlier. Some storage dev

Re: [dm-devel] [PATCH 0/4] dm verity: add support for error correction

2015-11-06 Thread Sami Tolvanen
On Fri, Nov 06, 2015 at 10:05:24PM +0100, Zdenek Kabelac wrote: > So how do you want to correctly 'restore' missing full sectors > with just 0.8% data overhead ?? We use interleaving. Each byte in a 4k block is part of a different Reed-Solomon block, which means an entire lost 4k data block looks

Re: [dm-devel] [PATCH 0/4] dm verity: add support for error correction

2015-11-06 Thread Zdenek Kabelac
Dne 6.11.2015 v 21:27 Sami Tolvanen napsal(a): On Fri, Nov 06, 2015 at 08:20:15PM +0100, Zdenek Kabelac wrote: i.e. you have 1G of space - you want to give 250MB as 'redundancy' - so create 4 partition well data safety has it's price - user should choose what he prefers - more games and v

Re: [dm-devel] [PATCH 0/4] dm verity: add support for error correction

2015-11-06 Thread Sami Tolvanen
On Fri, Nov 06, 2015 at 08:20:15PM +0100, Zdenek Kabelac wrote: > i.e. you have 1G of space - you want to give 250MB as 'redundancy' - > so create 4 partition We cannot afford to set aside 25% of read-only partition space for redundancy on mobile devices, and would rather not impact performanc

Re: [dm-devel] [PATCH 0/4] dm verity: add support for error correction

2015-11-06 Thread Zdenek Kabelac
Dne 6.11.2015 v 20:06 Sami Tolvanen napsal(a): On Fri, Nov 06, 2015 at 12:23:29PM -0500, Mikulas Patocka wrote: I'm also wondering what is this patch useful for. Disks and flash controllers have their own error detection and correction I think I addressed this earlier. Some storage devices are

Re: [PATCH 0/4] dm verity: add support for error correction

2015-11-06 Thread Sami Tolvanen
On Fri, Nov 06, 2015 at 12:23:29PM -0500, Mikulas Patocka wrote: > I'm also wondering what is this patch useful for. Disks and flash > controllers have their own error detection and correction I think I addressed this earlier. Some storage devices are able to correct bit flips, but don't have eno

Re: [PATCH 0/4] dm verity: add support for error correction

2015-11-06 Thread Mikulas Patocka
On Thu, 5 Nov 2015, Milan Broz wrote: > On 11/05/2015 03:02 AM, Sami Tolvanen wrote: > > This patch set adds error correction support to dm-verity, which > > makes it possible to recover from data corruption in exchange of > > increased space overhead. > > > > The feature is implemented as part

Re: [PATCH 0/4] dm verity: add support for error correction

2015-11-05 Thread Sami Tolvanen
On Thu, Nov 05, 2015 at 08:34:04AM +0100, Milan Broz wrote: > could you please elaborate why is all this needed? To extend support > of some faulty flash chips? This makes dm-verity more robust against corruption caused by either hardware or software bugs, both of which we have seen in the past on

Re: [PATCH 0/4] dm verity: add support for error correction

2015-11-04 Thread Milan Broz
On 11/05/2015 03:02 AM, Sami Tolvanen wrote: > This patch set adds error correction support to dm-verity, which > makes it possible to recover from data corruption in exchange of > increased space overhead. > > The feature is implemented as part of dm-verity to take advantage > of the existing has

[PATCH 0/4] dm verity: add support for error correction

2015-11-04 Thread Sami Tolvanen
This patch set adds error correction support to dm-verity, which makes it possible to recover from data corruption in exchange of increased space overhead. The feature is implemented as part of dm-verity to take advantage of the existing hash tree to improve performance and locate known erasures.