Re: [HACKERS] [GENERAL] Undetected corruption of table files

2007-08-29 Thread Florian Weimer
* Alban Hertroys: > If you have a "proper" production database server, your memory has > error checking, and your RAID controller has something of the kind > as well. To my knowledge, no readily available controller performs validation on reads (not even for RAID-1 or RAID-10, where it would be p

Re: [HACKERS] [GENERAL] Undetected corruption of table files

2007-08-28 Thread Lincoln Yeoh
At 11:48 PM 8/27/2007, Trevor Talbot wrote: On 8/27/07, Jonah H. Harris <[EMAIL PROTECTED]> wrote: > On 8/27/07, Tom Lane <[EMAIL PROTECTED]> wrote: > > that and the lack of evidence that they'd actually gain anything > > I find it somewhat ironic that PostgreSQL strives to be fairly > non-corrup

Re: [HACKERS] [GENERAL] Undetected corruption of table files

2007-08-27 Thread Decibel!
On Mon, Aug 27, 2007 at 12:08:17PM -0400, Jonah H. Harris wrote: > On 8/27/07, Tom Lane <[EMAIL PROTECTED]> wrote: > > Indeed. In fact, the most likely implementation of this (refuse to do > > anything with a page with a bad CRC) would be a net loss from that > > standpoint, because you couldn't g

Re: [HACKERS] [GENERAL] Undetected corruption of table files

2007-08-27 Thread Jonah H. Harris
On 8/27/07, Tom Lane <[EMAIL PROTECTED]> wrote: > Indeed. In fact, the most likely implementation of this (refuse to do > anything with a page with a bad CRC) would be a net loss from that > standpoint, because you couldn't get *any* data out of a page, even if > only part of it had been zapped.

Re: [HACKERS] [GENERAL] Undetected corruption of table files

2007-08-27 Thread Tom Lane
"Trevor Talbot" <[EMAIL PROTECTED]> writes: > On 8/27/07, Jonah H. Harris <[EMAIL PROTECTED]> wrote: >> I find it somewhat ironic that PostgreSQL strives to be fairly >> non-corruptable, yet has no way to detect a corrupted page. > But how does detecting a corrupted data page gain you any durabili

Re: [HACKERS] [GENERAL] Undetected corruption of table files

2007-08-27 Thread Alban Hertroys
Jonah H. Harris wrote: > On 8/27/07, Tom Lane <[EMAIL PROTECTED]> wrote: >> that and the lack of evidence that they'd actually gain anything > > I find it somewhat ironic that PostgreSQL strives to be fairly > non-corruptable, yet has no way to detect a corrupted page. The only > reason for not h

Re: [GENERAL] Undetected corruption of table files

2007-08-27 Thread Gregory Stark
"Tom Lane" <[EMAIL PROTECTED]> writes: > "Albe Laurenz" <[EMAIL PROTECTED]> writes: >> Tom Lane wrote: Would it be an option to have a checksum somewhere in each data block that is verified upon read? > >>> That's been proposed before and rejected before. See the archives ... > >> I sea

Re: [HACKERS] [GENERAL] Undetected corruption of table files

2007-08-27 Thread Trevor Talbot
On 8/27/07, Jonah H. Harris <[EMAIL PROTECTED]> wrote: > On 8/27/07, Tom Lane <[EMAIL PROTECTED]> wrote: > > that and the lack of evidence that they'd actually gain anything > > I find it somewhat ironic that PostgreSQL strives to be fairly > non-corruptable, yet has no way to detect a corrupted pa

Re: [HACKERS] [GENERAL] Undetected corruption of table files

2007-08-27 Thread Jonah H. Harris
On 8/27/07, Tom Lane <[EMAIL PROTECTED]> wrote: > that and the lack of evidence that they'd actually gain anything I find it somewhat ironic that PostgreSQL strives to be fairly non-corruptable, yet has no way to detect a corrupted page. The only reason for not having CRCs is because it will slow

Re: [GENERAL] Undetected corruption of table files

2007-08-27 Thread Tom Lane
"Albe Laurenz" <[EMAIL PROTECTED]> writes: > Tom Lane wrote: >>> Would it be an option to have a checksum somewhere in each >>> data block that is verified upon read? >> That's been proposed before and rejected before. See the archives ... > I searched for "checksum" and couldn't find it. Could

Re: [GENERAL] Undetected corruption of table files

2007-08-27 Thread Albe Laurenz
Tom Lane wrote: >> Would it be an option to have a checksum somewhere in each >> data block that is verified upon read? > > That's been proposed before and rejected before. See the archives ... I searched for "checksum" and couldn't find it. Could someone give me a pointer? I'm not talking about

Re: [GENERAL] Undetected corruption of table files

2007-08-24 Thread Tom Lane
"Albe Laurenz" <[EMAIL PROTECTED]> writes: > Would it be an option to have a checksum somewhere in each > data block that is verified upon read? That's been proposed before and rejected before. See the archives ... regards, tom lane ---(end of bro

Re: [GENERAL] Undetected corruption of table files

2007-08-24 Thread Albe Laurenz
Tom Lane wrote: >> - Shouldn't there be an error, some kind of 'missing magic >> number' or similar, when a table file consists of only >> zeros? > > The particular case of an all-zeroes page is specifically allowed, > and has to be because it's a valid transient state in various > scenarios.

Re: [GENERAL] Undetected corruption of table files

2007-08-23 Thread Tom Lane
"Albe Laurenz" <[EMAIL PROTECTED]> writes: > - Shouldn't there be an error, some kind of 'missing magic > number' or similar, when a table file consists of only > zeros? The particular case of an all-zeroes page is specifically allowed, and has to be because it's a valid transient state in var

[GENERAL] Undetected corruption of table files

2007-08-23 Thread Albe Laurenz
I am slightly worried that corruption of data files may remain undetected in PostgreSQL. As an experiment, I created a simple table with a primary key index and inserted a couple of rows. The corresponding data file is 1 page = 8K long. Now when I stop the server, zero out the data file with dd i