Re: data checksums

On Wed, Aug 7, 2024 at 3:41 AM Daniel Gustafsson wrote: > > On 6 Aug 2024, at 18:29, Christophe Pettus wrote: > >> On Aug 6, 2024, at 08:11, bruno vieira da silva > wrote: > > >> the pg doc > >> mentions a considerable performance penality, how considerable it is? > > > > That line is probably

Re: data checksums

> On 6 Aug 2024, at 18:29, Christophe Pettus wrote: >> On Aug 6, 2024, at 08:11, bruno vieira da silva wrote: >> the pg doc >> mentions a considerable performance penality, how considerable it is? > > That line is probably somewhat out of date at this point. We haven't seen a > significant s

Re: data checksums

> On Aug 6, 2024, at 19:45, Laurenz Albe wrote: > I am surprised by that. Would you say that most storage systems will happily > give you a > garbage block if there was a hardware problem somewhere? "Most" is hard for me to judge. HDDs can have uncorrected and undetected errors, definitely

Re: data checksums

On Tue, 2024-08-06 at 09:29 -0700, Christophe Pettus wrote: > > > On Aug 6, 2024, at 08:11, bruno vieira da silva > > wrote: > > > > so my question is why data checksums aren't enabled by default on pg? > > At this point, mostly historical reasons. They're also superfluous if your > underlyi

Re: data checksums

The penalty is not "considerable", but it is not completely trivial either. But it's more on the trivial side. Part of the problem is that it is hard to measure, as it is very workload dependent. As to why it is not the default, Postgres is very careful and conservative by default, and not everybod

Re: data checksums

> On Aug 6, 2024, at 08:11, bruno vieira da silva wrote: > > so my question is why data checksums aren't enabled by default on pg? At this point, mostly historical reasons. They're also superfluous if your underlying file system or storage hardware does storage-level corruption checks (whi

Re: data checksums

On Tue, Aug 6, 2024 at 11:12 AM bruno vieira da silva wrote: > Hello. > I've been dealing with some database corruption events, so i've been > wondering to enable data checksums on my deployments. > > so my question is why data checksums aren't enabled by default on pg? the > pg doc > mentions a

Re: Sv: Re: Sv: Re: Sv: Re: Sv: Re: data-checksums

On 2018-01-10 11:39:21 -0800, Andres Freund wrote: > On 2018-01-09 20:51:17 -0500, Stephen Frost wrote: > > * Andreas Joseph Krogh (andr...@visena.com) wrote: > > > Aha, so enabling CRC causes hint-bits to be written causing extra > > > WAL-logging, > > > which woudn't be the case without CRC ena

Sv: Re: Sv: Re: Sv: Re: Sv: Re: Sv: Re: data-checksums

På onsdag 10. januar 2018 kl. 22:36:42, skrev Jeff Janes mailto:jeff.ja...@gmail.com>>: On Wed, Jan 10, 2018 at 12:23 PM, Thomas Poty mailto:thomas.p...@gmail.com>> wrote: Hello, A question seems to be, according to me, important : How a corruption, detected thanks to data-checksums, is fixed?  

Re: Sv: Re: Sv: Re: Sv: Re: Sv: Re: data-checksums

On Wed, Jan 10, 2018 at 12:23 PM, Thomas Poty wrote: > Hello, > A question seems to be, according to me, important : > How a corruption, detected thanks to data-checksums, is fixed? > Take two full cold backups of the current mess you have, including the executables, and lock one of them away wh

Re: Sv: Re: Sv: Re: Sv: Re: Sv: Re: data-checksums

Hello, A question seems to be, according to me, important : How a corruption, detected thanks to data-checksums, is fixed? Thank you, Thomas Le 10 janv. 2018 20:39, "Andres Freund" a écrit : > On 2018-01-09 20:51:17 -0500, Stephen Frost wrote: > > Greetings, > > > > * Andreas Joseph Krogh (andr

Re: Sv: Re: Sv: Re: Sv: Re: Sv: Re: data-checksums

On 2018-01-09 20:51:17 -0500, Stephen Frost wrote: > Greetings, > > * Andreas Joseph Krogh (andr...@visena.com) wrote: > > Aha, so enabling CRC causes hint-bits to be written causing extra > > WAL-logging, > > which woudn't be the case without CRC enabled? > > Thanks for pointing that out. > >

Re: Sv: Re: Sv: Re: Sv: Re: Sv: Re: data-checksums

On 2018-01-10 01:31:58 +0100, Andreas Joseph Krogh wrote: > På onsdag 10. januar 2018 kl. 01:01:26, skrev Andres Freund > >: > On 2018-01-10 00:25:08 +0100, Andreas Joseph Krogh wrote: > > But SIMD-instructions are also HW-accellerated by modern CPUs IIUC? > > Sure.

Re: Sv: Re: Sv: Re: Sv: Re: Sv: Re: data-checksums

Greetings, * Andreas Joseph Krogh (andr...@visena.com) wrote: > Aha, so enabling CRC causes hint-bits to be written causing extra > WAL-logging, > which woudn't be the case without CRC enabled? > Thanks for pointing that out. Yes, having checksums enabled forces logging of hint bits. You can e

Sv: Re: Sv: Re: Sv: Re: Sv: Re: data-checksums

På onsdag 10. januar 2018 kl. 01:01:26, skrev Andres Freund mailto:and...@anarazel.de>>: On 2018-01-10 00:25:08 +0100, Andreas Joseph Krogh wrote: > På tirsdag 09. januar 2018 kl. 23:42:45, skrev Rob Sargent < > robjsarg...@gmail.com >: >   > >    On 01/09/2018 03

Re: Sv: Re: Sv: Re: Sv: Re: data-checksums

On 2018-01-10 00:25:08 +0100, Andreas Joseph Krogh wrote: > På tirsdag 09. januar 2018 kl. 23:42:45, skrev Rob Sargent < > robjsarg...@gmail.com >: >   > >   On 01/09/2018 03:30 PM, Andreas Joseph Krogh wrote: > På tirsdag 09. januar 2018 kl. 23:06:06, skrev Andres F

Sv: Re: Sv: Re: Sv: Re: data-checksums

På tirsdag 09. januar 2018 kl. 23:42:45, skrev Rob Sargent < robjsarg...@gmail.com >:     On 01/09/2018 03:30 PM, Andreas Joseph Krogh wrote: På tirsdag 09. januar 2018 kl. 23:06:06, skrev Andres Freund < and...@anarazel.de >: Hi, On 2018

Re: Sv: Re: Sv: Re: data-checksums

On 01/09/2018 03:30 PM, Andreas Joseph Krogh wrote: På tirsdag 09. januar 2018 kl. 23:06:06, skrev Andres Freund mailto:and...@anarazel.de>>: Hi, On 2018-01-09 21:47:17 +0100, Andreas Joseph Krogh wrote: > Does PG use HW-accellerated crc if CPU supports it[1]? Yes we do, for

Sv: Re: Sv: Re: data-checksums

På tirsdag 09. januar 2018 kl. 23:06:06, skrev Andres Freund mailto:and...@anarazel.de>>: Hi, On 2018-01-09 21:47:17 +0100, Andreas Joseph Krogh wrote: > Does PG use HW-accellerated crc if CPU supports it[1]? Yes we do, for WAL checksums. The page checksums are a different algorithm though, o

Re: Sv: Re: data-checksums

Hi, On 2018-01-09 21:47:17 +0100, Andreas Joseph Krogh wrote: > Does PG use HW-accellerated crc if CPU supports it[1]? Yes we do, for WAL checksums. The page checksums are a different algorithm though, one which has the advantage of being SIMD compatible. The checksum computations have some impa

Sv: Re: data-checksums

På tirsdag 09. januar 2018 kl. 21:37:16, skrev Joshua D. Drake < j...@commandprompt.com >: On 01/09/2018 12:22 PM, Andres Freund wrote: > On 2018-01-09 20:04:04 +0100, Rakesh Kumar wrote: I also would like to believe that the hit is small, but when PG off

Re: data-checksums

On 01/09/2018 12:22 PM, Andres Freund wrote: On 2018-01-09 20:04:04 +0100, Rakesh Kumar wrote: I also would like to believe that the hit is small, but when PG official document writes "noticeable performance penalty", it becomes difficult to convince management that the hit is small :-) Why bel

Re: data-checksums

On 2018-01-09 20:04:04 +0100, Rakesh Kumar wrote: > > > > I also would like to believe that the hit is small, but when PG > > > official document writes "noticeable performance penalty", it becomes > > > difficult to convince management that the hit is small :-) > > > > Why believe, when you can

Re: data-checksums

"On Tue, 9 Jan 2018 20:02:37 +0100, "Rakesh Kumar" wrote: >Hello Mr. Pedantic, > >> noticeable != huge. > >and noticeable != small/negligible either, at least from English >point of view. small != negligible. The word "noticable" does not imply any particular magnitude of event. It means only

Re: data-checksums

On Tue, Jan 9, 2018 at 12:58 PM, Rakesh Kumar wrote: > > > That said, imv anyway, the performance hit is small and having checksums > > is well worth it. > > I also would like to believe that the hit is small, but when PG official > document writes "noticeable performance penalty", it becomes dif

Re: data-checksums

> > I also would like to believe that the hit is small, but when PG > > official document writes "noticeable performance penalty", it becomes > > difficult to convince management that the hit is small :-) > > Why believe, when you can measure? yup doing that. But I still feel that PG documentat

Re: data-checksums

Hello Mr. Pedantic, > noticeable != huge. and noticeable != small/negligible either, at least from English point of view.

Re: data-checksums

Rakesh Kumar wrote: > > > That said, imv anyway, the performance hit is small and having > > checksums is well worth it. > > I also would like to believe that the hit is small, but when PG > official document writes "noticeable performance penalty", it becomes > difficult to convince management t

Re: data-checksums

On 2018-01-09 18:58:41 +0100, Rakesh Kumar wrote: > > > That said, imv anyway, the performance hit is small and having checksums > > is well worth it. > > I also would like to believe that the hit is small, but when PG > official document writes "noticeable performance penalty", it becomes > diff

Re: data-checksums

> That said, imv anyway, the performance hit is small and having checksums > is well worth it. I also would like to believe that the hit is small, but when PG official document writes "noticeable performance penalty", it becomes difficult to convince management that the hit is small :-)

Re: data-checksums

Greetings, * Rakesh Kumar (rakeshkumar...@mail.com) wrote: > --data-checksums > Use checksums on data pages to help detect corruption by the I/O system that > would otherwise be silent. Enabling checksums may incur a noticeable > performance penalty. This option can only be set during initializa