Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-26 Thread Jörn Engel
On Mon, 26 March 2007 13:49:06 +0300, Artem Bityutskiy wrote: > On Sun, 2007-03-25 at 22:08 +0200, Jörn Engel wrote: > > > > Logical volume management can just as easily move its management > > information into a table, instead of having it spread across all blocks. > > Blocks can keep their origi

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-26 Thread Artem Bityutskiy
On Sun, 2007-03-25 at 22:08 +0200, Jörn Engel wrote: > And there is no fundamental reason why UBI should export blocks with > non-power-of-two sizes. False. There is. > UBI currently consists of two parts that are > intimately intertwined in the current implementation, but have > relatively lit

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-26 Thread David Woodhouse
On Mon, 2007-03-26 at 11:51 +0200, Jörn Engel wrote: > Are you sure? Do you have any specs or similar that state this? > > So far I have only encountered this limitation by word of mouth. And > such a myth coming from ECC effects is nothing that would surprise > me. See pp 6 and 31 of http://d

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-26 Thread Thomas Gleixner
On Mon, 2007-03-26 at 10:45 +0100, David Woodhouse wrote: > On Mon, 2007-03-26 at 03:04 +0200, Jörn Engel wrote: > > That limitation stems from ECC and ECC is done in software. Currently > > everyone and his dog is doing ECC in chunks of 256 bytes on NAND. So > > your minimum write size is 256 by

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-26 Thread Jörn Engel
On Mon, 26 March 2007 10:45:57 +0100, David Woodhouse wrote: > > No, on NAND flash it's a limitation of the hardware. The number of write > cycles you can perform to a given page is limited. Exceed it and the > contents of that page become undefined due to leakage, until you next > erase it. Are

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-26 Thread David Woodhouse
On Mon, 2007-03-26 at 03:04 +0200, Jörn Engel wrote: > That limitation stems from ECC and ECC is done in software. Currently > everyone and his dog is doing ECC in chunks of 256 bytes on NAND. So > your minimum write size is 256 bytes _if you care about ECC_. If you > don't care, you can write s

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-25 Thread Jörn Engel
On Mon, 26 March 2007 01:21:25 +0100, David Woodhouse wrote: > On Mon, 2007-03-26 at 02:01 +0200, Jörn Engel wrote: > > You can on NAND. ECC is done in software. And for a data structure as > > simple as the 'tally', foregoing ECC is not a huge problem - most > > bitflips are easily detected and

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-25 Thread David Woodhouse
On Mon, 2007-03-26 at 02:01 +0200, Jörn Engel wrote: > You can on NAND. ECC is done in software. And for a data structure as > simple as the 'tally', foregoing ECC is not a huge problem - most > bitflips are easily detected and the remaining only cause off-by-a-few > on the erase count. You're

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-25 Thread Jörn Engel
On Mon, 26 March 2007 00:46:33 +0100, David Woodhouse wrote: > On Mon, 2007-03-26 at 00:55 +0200, Jörn Engel wrote: > > > although, since you can flip bits to 1 without requireing an erase you > > [ vice versa. you can flip bits to 0 without erasing. ] > > And on NAND flash you can't just do it

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-25 Thread David Woodhouse
On Mon, 2007-03-26 at 00:55 +0200, Jörn Engel wrote: > > although, since you can flip bits to 1 without requireing an erase you > [ vice versa. you can flip bits to 0 without erasing. ] And on NAND flash you can't just do it in multiple cycles one bit at a time. The 'tally' trick isn't viable th

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-25 Thread Jörn Engel
On Sun, 25 March 2007 13:49:58 -0800, David Lang wrote: > On Sun, 25 Mar 2007, Jörn Engel wrote: > > >Logical volume management can just as easily move its management > >information into a table, instead of having it spread across all blocks. > >Blocks can keep their original size. Since you have

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-25 Thread David Lang
On Sun, 25 Mar 2007, Jörn Engel wrote: On Wed, 21 March 2007 12:25:34 +0100, Thomas Gleixner wrote: On Wed, 2007-03-21 at 12:05 +0100, Jörn Engel wrote: Also LogFS currently requires erasesizes of 2^n. Last time I talked to you about that, you said it would be possible and fixable. Actual

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-25 Thread Jörn Engel
On Wed, 21 March 2007 12:25:34 +0100, Thomas Gleixner wrote: > On Wed, 2007-03-21 at 12:05 +0100, Jörn Engel wrote: > > > > Also LogFS currently requires erasesizes of 2^n. > > Last time I talked to you about that, you said it would be possible and > fixable. Actually, no. LogFS is not broken,

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-21 Thread David Lang
On Wed, 21 Mar 2007, Frank Haverkamp wrote: several of these things sound like they would be useful to other block devices as well UBI solves here: 1. possiblitly to boot a controller with limited resources using NAND 2. transparent bitflip correction on read only data e.g. boot-code, kern

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-21 Thread Frank Haverkamp
Hi Ted, On Wed, 2007-03-21 at 09:50 -0400, Theodore Tso wrote: > Keeping this thought in mind and trying to help them, where in some > cases perhaps they are lacking in the same knowledge and experience > and those who have been working on UBI and have spent many months > thinking about the proble

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-21 Thread Artem Bityutskiy
On Wed, 2007-03-21 at 09:50 -0400, Theodore Tso wrote: > And at this point, I don't doubt that you will at some point going to > heed my comments --- but note that doing so will involve a massive > refactorization of the code, which will tend to invalidate the reviews > done of this current (take 3

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-21 Thread Josh Boyer
On Wed, 2007-03-21 at 09:50 -0400, Theodore Tso wrote: > > And at this point, I don't doubt that you will at some point going to > heed my comments --- but note that doing so will involve a massive > refactorization of the code, which will tend to invalidate the reviews > done of this current (tak

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-21 Thread Theodore Tso
On Wed, Mar 21, 2007 at 10:44:35AM +0200, Artem Bityutskiy wrote: > > As I mentioned to you in IRC, in the future if there is pending > > changes in response to reviewer comments, it might be a good idea to > > mention that, so that reviewers know not make those comments again, or > > worry that th

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-21 Thread Jörn Engel
*sigh* I really did not want to become involved in this. So please be nice and leave the flamethrower in your weapon closet or I will disappear again before you can say "fire". On Tue, 20 March 2007 21:32:40 +, David Woodhouse wrote: > On Tue, 2007-03-20 at 10:58 -0800, David Lang wrote: > >

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-21 Thread Artem Bityutskiy
On Wed, 2007-03-21 at 13:31 +0100, Jörn Engel wrote: > Correct. It may make sense to use UBI for that, I don't know. What I > do know is that UBI cannot make wear leveling decisions as well as > LogFS. So lets discuss this in different thread when you post a request to include LogFS into mainlin

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-21 Thread Jörn Engel
On Wed, 21 March 2007 12:57:42 +0100, Thomas Gleixner wrote: > On Wed, 2007-03-21 at 12:35 +0100, Jörn Engel wrote: > > Even if such flashes still contain a bootloader and a kernel, that will > > occupy less than 1% of the device. Wear leveling across the device is > > fairly pointless here. This

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-21 Thread Thomas Gleixner
On Wed, 2007-03-21 at 12:35 +0100, Jörn Engel wrote: > Even if such flashes still contain a bootloader and a kernel, that will > occupy less than 1% of the device. Wear leveling across the device is > fairly pointless here. This is what I designed LogFS for. Still you need to have a solution for

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-21 Thread Artem Bityutskiy
On Wed, 2007-03-21 at 12:25 +0100, Thomas Gleixner wrote: > Last time I talked to you about that, you said it would be possible and > fixable. We talked about several mechanisms, which would allow a > filesystem or other users to hint such things to UBI. > > Even if the LogFS wear levelling is so

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-21 Thread Jörn Engel
On Wed, 21 March 2007 12:25:34 +0100, Thomas Gleixner wrote: > On Wed, 2007-03-21 at 12:05 +0100, Jörn Engel wrote: > > > > Ok, now we have reached the absurd. UBI quite fundamentally cannot do > > wear leveling as good as LogFS can. Simply because UBI has zero > > knowledge of the _contents_ of

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-21 Thread Thomas Gleixner
On Wed, 2007-03-21 at 12:05 +0100, Jörn Engel wrote: > On Tue, 20 March 2007 01:42:46 +0100, Thomas Gleixner wrote: > > On Mon, 2007-03-19 at 17:32 -0500, Matt Mackall wrote: > > > > > > > 4. JFFS2 has its own wear-leving scheme, as do several other > > > > >filesystems, so they probably want

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-21 Thread Jörn Engel
On Tue, 20 March 2007 01:42:46 +0100, Thomas Gleixner wrote: > On Mon, 2007-03-19 at 17:32 -0500, Matt Mackall wrote: > > > > > 4. JFFS2 has its own wear-leving scheme, as do several other > > > >filesystems, so they probably want to bypass this piece of the stack. > > > > > > JFFS2 on top of

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-21 Thread Artem Bityutskiy
On Tue, 2007-03-20 at 21:36 +, David Woodhouse wrote: > On Tue, 2007-03-20 at 22:05 +0200, Artem Bityutskiy wrote: > > Guess why we still do not have a decent FTL? Because it is > > _difficult_. > > No. We don't have a decent FTL because it's _pointless_. We've got basic > implementations of

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-21 Thread Artem Bityutskiy
On Tue, 2007-03-20 at 18:03 -0400, Theodore Tso wrote: > So this is probably a stupid question, but what drives the design > decision to store the metadata in-band instead of out-of-band (and you > don't have to answer me here; putting it in the overall system > architecture document is just as goo

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-20 Thread Theodore Tso
On Tue, Mar 20, 2007 at 10:59:55AM -0500, Josh Boyer wrote: > Sure. But the larger question is *should* it be extended to do so. Absolutely, and so let's focus on that. > Except that flash filesystems don't use block devices at all. They use > MTD interfaces. Yes, so that would be first issue.

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-20 Thread David Woodhouse
On Tue, 2007-03-20 at 22:05 +0200, Artem Bityutskiy wrote: > Guess why we still do not have a decent FTL? Because it is > _difficult_. No. We don't have a decent FTL because it's _pointless_. We've got basic implementations of FTL, NFTL, INFTL etc. for compatibility with PCMCIA stuff and DiskOnCh

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-20 Thread David Woodhouse
On Tue, 2007-03-20 at 10:58 -0800, David Lang wrote: > What Matt and Ted are looking at is the question 'are flash devices close > enough > to other block devices that it would make sense to change the existing linux > definition of a block device to handle the special requirements of flash' I'

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-20 Thread Artem Bityutskiy
On Tue, 2007-03-20 at 10:58 -0800, David Lang wrote: > the fact that you erase in large blocks and then write in smaller blocks is a > difference, and one that the current block layer doesn't understand. but this > is > a difference that the current block layer could be changed to understand. >

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-20 Thread David Lang
On Tue, 20 Mar 2007, Josh Boyer wrote: On Tue, 2007-03-20 at 09:52 -0400, Theodore Tso wrote: As a suggestion, let's stop right here and see if we can get both sides talking in a more constructive fashion. Maybe it's just me, but I see both sides talking past each other in a rather dramatic fa

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-20 Thread Josh Boyer
On Tue, 2007-03-20 at 09:52 -0400, Theodore Tso wrote: > As a suggestion, let's stop right here and see if we can get both > sides talking in a more constructive fashion. Maybe it's just me, but > I see both sides talking past each other in a rather dramatic fashion. Perhaps, yes. Though I've be

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-20 Thread Artem Bityutskiy
On Tue, 2007-03-20 at 09:52 -0400, Theodore Tso wrote: > It would also help people understand why there are so many "units" in > UBI, since hopefully the high-level documentation would explain why > they fit together, and perhaps why some of the units weren't folded > together. What value do they

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-20 Thread Theodore Tso
On Tue, Mar 20, 2007 at 02:25:49PM +0200, Artem Bityutskiy wrote: > You failed to clearly define what is block until now, then you blame me > that I do not understand you. So I see block = eraseblock, lets assume > for further conversation. > > OK. Suppose we have done what you say, although I _do

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-20 Thread Artem Bityutskiy
> > iSCSI/nbd(6) > | > filesystem {swap | ext3ext3 jffs2 > \ | || / >/ \ | dm-crypt->snapshot(5) / > device mapper -|\ \ | / >

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-20 Thread Josh Boyer
On Mon, 2007-03-19 at 14:54 -0500, Matt Mackall wrote: > > False starts that get mainlined delay or prevent things getting done > right. The question is and remains "is UBI the right way to do > things?" Not "is UBI the easiest way to do things?" or "is UBI > something people have already adopted?

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-19 Thread Thomas Gleixner
On Mon, 2007-03-19 at 20:05 -0500, Matt Mackall wrote: > On Tue, Mar 20, 2007 at 01:42:46AM +0100, Thomas Gleixner wrote: > > On Mon, 2007-03-19 at 17:32 -0500, Matt Mackall wrote: > > > This is exactly the same problem as booting on a desktop PC. But > > > somehow LILO manages. My first Linux box

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-19 Thread Matt Mackall
On Tue, Mar 20, 2007 at 01:42:46AM +0100, Thomas Gleixner wrote: > On Mon, 2007-03-19 at 17:32 -0500, Matt Mackall wrote: > > > > If a static volume is simply a non-dynamic volume, then device mapper > > > > can do that too. And countless other things. Which is not an aside. > > > > UBI growing to

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-19 Thread Thomas Gleixner
On Mon, 2007-03-19 at 16:36 -0500, Matt Mackall wrote: > On Mon, Mar 19, 2007 at 11:06:33PM +0200, Artem Bityutskiy wrote: > > On Mon, 2007-03-19 at 14:54 -0500, Matt Mackall wrote: > > > The issue is 14000 lines of patch to make a parallel subsystem. > > > > Parallel system exists since very long

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-19 Thread Thomas Gleixner
On Mon, 2007-03-19 at 17:32 -0500, Matt Mackall wrote: > > > If a static volume is simply a non-dynamic volume, then device mapper > > > can do that too. And countless other things. Which is not an aside. > > > UBI growing to do all the things that device mapper does is exactly > > > the thing we s

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-19 Thread Matt Mackall
On Mon, Mar 19, 2007 at 10:05:29PM +0100, Thomas Gleixner wrote: > On Mon, 2007-03-19 at 14:54 -0500, Matt Mackall wrote: > > > (UBI also has static volumes which LVM doesn't but that is an aside.) > > > > If a static volume is simply a non-dynamic volume, then device mapper > > can do that too. A

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-19 Thread Matt Mackall
On Mon, Mar 19, 2007 at 11:06:33PM +0200, Artem Bityutskiy wrote: > On Mon, 2007-03-19 at 14:54 -0500, Matt Mackall wrote: > > The issue is 14000 lines of patch to make a parallel subsystem. > > Parallel system exists since very long. One is > flash->SW_or_HW_FTL->all_blkdev_stuff. The other is MT

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-19 Thread Thomas Gleixner
On Mon, 2007-03-19 at 14:54 -0500, Matt Mackall wrote: > > (UBI also has static volumes which LVM doesn't but that is an aside.) > > If a static volume is simply a non-dynamic volume, then device mapper > can do that too. And countless other things. Which is not an aside. > UBI growing to do all t

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-19 Thread Thomas Gleixner
On Mon, 2007-03-19 at 15:12 -0500, Matt Mackall wrote: > > Should we export block devices with 16/32/64/128 KiB size ? > > Sure, why not? Simply because we want to have the ability to write fine grained in order to write data safe to FLASH. If we export those large sizes we lose this ability and

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-19 Thread Artem Bityutskiy
On Mon, 2007-03-19 at 14:54 -0500, Matt Mackall wrote: > The issue is 14000 lines of patch to make a parallel subsystem. Parallel system exists since very long. One is flash->SW_or_HW_FTL->all_blkdev_stuff. The other is MTD->JFFS2. Think about _why_ there are 2 of them. Hint - reliability, perform

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-19 Thread Matt Mackall
On Mon, Mar 19, 2007 at 08:03:30PM +0100, Thomas Gleixner wrote: > Matt, > > On Mon, 2007-03-19 at 12:08 -0500, Matt Mackall wrote: > > On Sun, Mar 18, 2007 at 03:31:50PM -0500, Josh Boyer wrote: > > > On Sun, Mar 18, 2007 at 02:18:12PM -0500, Matt Mackall wrote: > > > > > > > > I'm well aware of

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-19 Thread Artem Bityutskiy
On Mon, 2007-03-19 at 14:54 -0500, Matt Mackall wrote: > The issue is 14000 lines of patch to make a parallel subsystem. It'll be much smaller after I remove "itsy-bitsy" and most of the debugging stuff, in progress - wait for take 4. -- Best regards, Artem Bityutskiy (Битюцкий Артём) - To unsu

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-19 Thread Matt Mackall
On Mon, Mar 19, 2007 at 01:16:28PM -0500, Josh Boyer wrote: > On Mon, 2007-03-19 at 12:08 -0500, Matt Mackall wrote: > > > > > > If the end goal is to end up with something that looks like a block > > > > device (which seems to be implied by adding transparent wear leveling > > > > > > Nope, not

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-19 Thread Thomas Gleixner
Matt, On Mon, 2007-03-19 at 12:08 -0500, Matt Mackall wrote: > On Sun, Mar 18, 2007 at 03:31:50PM -0500, Josh Boyer wrote: > > On Sun, Mar 18, 2007 at 02:18:12PM -0500, Matt Mackall wrote: > > > > > > I'm well aware of all that. I wrote a NAND driver just last month. > > > Let's consider this tab

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-19 Thread Josh Boyer
On Mon, 2007-03-19 at 12:08 -0500, Matt Mackall wrote: > > > > If the end goal is to end up with something that looks like a block > > > device (which seems to be implied by adding transparent wear leveling > > > > Nope, not the end goal. It's more about wear-leveling across the entire > > flash

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-19 Thread Matt Mackall
On Sun, Mar 18, 2007 at 03:31:50PM -0500, Josh Boyer wrote: > On Sun, Mar 18, 2007 at 02:18:12PM -0500, Matt Mackall wrote: > > > > I'm well aware of all that. I wrote a NAND driver just last month. > > Let's consider this table: > > > > HARD drives MTD device > > Consist

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-18 Thread Josh Boyer
On Sun, Mar 18, 2007 at 02:18:12PM -0500, Matt Mackall wrote: > > I'm well aware of all that. I wrote a NAND driver just last month. > Let's consider this table: > > HARD drives MTD device > Consists of sectors Consists of eraseblocks > Sectors are small

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-18 Thread Matt Mackall
On Sun, Mar 18, 2007 at 06:49:39PM +0200, Artem Bityutskiy wrote: > On Sun, 2007-03-18 at 11:27 -0500, Matt Mackall wrote: > > Forgive my ignorance, but why did you not implement the two features > > above as device mapper layers instead? A device mapper can arbitrarily > > transform I/O addresses

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-18 Thread Artem Bityutskiy
On Sun, 2007-03-18 at 11:27 -0500, Matt Mackall wrote: > Forgive my ignorance, but why did you not implement the two features > above as device mapper layers instead? A device mapper can arbitrarily > transform I/O addresses and contents and has direct access to the > mapped device's ioctl interfac

Re: [PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-18 Thread Matt Mackall
On Wed, Mar 14, 2007 at 05:19:34PM +0200, Artem Bityutskiy wrote: > Hello, > > This patch-set contains UBI, which stands for Unsorted Block Images. This > is closely related to the memory technology devices Linux subsystem (MTD), > so this new piece of software is from drivers/mtd/ubi. > > In sho

[PATCH 00/22 take 3] UBI: Unsorted Block Images

2007-03-14 Thread Artem Bityutskiy
Hello, This patch-set contains UBI, which stands for Unsorted Block Images. This is closely related to the memory technology devices Linux subsystem (MTD), so this new piece of software is from drivers/mtd/ubi. In short, UBI is kind of LVM layer but for flash (MTD) devices. It makes it possible t