Re: [HACKERS] no universally correct setting for fsync

2010-05-31 Thread Bruce Momjian
Josh Berkus wrote: > All, > > Updated docs based on tracking this discussion. fsync through full page > writes recorded below. I have applied this doc update with the attached patch. I added the change from "every night" to "frequently", and reworded it slightly so it was clear it affects the e

Re: [HACKERS] no universally correct setting for fsync

2010-05-11 Thread Yeb Havinga
Kevin Grittner wrote: "Joshua D. Drake" wrote: The answer to this is: PostgreSQL.org recommends that this setting be left on at all times. Turning it off, may lead to data corruption. Anything else is circumstantial and based on knowledge and facts we don't have about environmental fact

Re: [HACKERS] no universally correct setting for fsync

2010-05-10 Thread Greg Smith
Josh Berkus wrote: The real question is how much of a speed-up fsync provides compared to the same workload with synchronous_commit disabled. The only case for fsync=off is one where that number is much faster. I can't say I've tested this. Most of my head-to-heads on fsync were before

Re: [HACKERS] no universally correct setting for fsync

2010-05-10 Thread Josh Berkus
On 5/10/10 2:21 PM, Ross J. Reedstrom wrote: > On Mon, May 10, 2010 at 01:35:32PM -0700, Josh Berkus wrote: >> deleted, >> or on a reporting read-only clone of your database which gets >> recreated very >> night and is not used for failover. High quality hardware alone > > s/very/

Re: [HACKERS] no universally correct setting for fsync

2010-05-10 Thread Josh Berkus
> The real question is how much of a speed-up fsync provides compared to > the same workload with synchronous_commit disabled. The only case for > fsync=off is one where that number is much faster. I can't say I've tested this. Most of my head-to-heads on fsync were before asych existed. --

Re: [HACKERS] no universally correct setting for fsync

2010-05-10 Thread Greg Smith
Josh Berkus wrote: Wow. In a situation where you save seven minutes (4%), it's hardly worth turning off. I've had it be much higher, especially for really large databases. Cedric's system had a non-volatile write cache in it. In that case, a few percentage points of improvement is

Re: [HACKERS] no universally correct setting for fsync

2010-05-10 Thread Ross J. Reedstrom
On Mon, May 10, 2010 at 01:35:32PM -0700, Josh Berkus wrote: > deleted, > or on a reporting read-only clone of your database which gets > recreated very > night and is not used for failover. High quality hardware alone s/very/every/ or s/very night/periodically/ Ross -- Ross Re

Re: [HACKERS] no universally correct setting for fsync

2010-05-10 Thread Josh Berkus
> Wow. In a situation where you save seven minutes (4%), it's hardly > worth turning off. I've had it be much higher, especially for really large databases. -- -- Josh Berkus PostgreSQL Experts Inc.

Re: [HACKERS] no universally correct setting for fsync

2010-05-10 Thread Kevin Grittner
Cédric Villemain wrote: > On a recent pg_restore -j 32, with perc 6i with BBU, RAID10 8 hd, > results were not so bas with fsync turn on. (XFS with nobarrier su > and sw) > -- deactivate fsync > time pg_restore -U postgres -d foodb -j 32 foo.psql > real170m0.527s > user43m12.914s > sys

Re: [HACKERS] no universally correct setting for fsync

2010-05-10 Thread Josh Berkus
All, Updated docs based on tracking this discussion. fsync through full page writes recorded below. fsync configuration parameter fsync (boolean) If this parameter is on, the PostgreSQL server will try to make sure th

Re: [HACKERS] no universally correct setting for fsync

2010-05-10 Thread Cédric Villemain
2010/5/8 Bernd Helmle : > > > --On 7. Mai 2010 09:48:53 -0500 Kevin Grittner > wrote: > >> I think it goes beyond "tweaking" -- I think we should have a bald >> statement like "don't turn this off unless you're OK with losing the >> entire contents of the database cluster."  A brief listing of som

Re: [HACKERS] no universally correct setting for fsync

2010-05-10 Thread Tom Lane
"Kevin Grittner" writes: > Perhaps Josh's language for fsync could be modified to work here > (we're now talking about full_page_writes, for anyone who's lost > track): > | it is only advisable to turn off fsync if you can easily recreate > | your entire database from external data. > That cove

Re: [HACKERS] no universally correct setting for fsync

2010-05-10 Thread Kevin Grittner
"Joshua D. Drake" wrote: > The answer to this is: > > PostgreSQL.org recommends that this setting be left on at all > times. Turning it off, may lead to data corruption. > > Anything else is circumstantial and based on knowledge and facts > we don't have about environmental factors. Perhap

Re: [HACKERS] no universally correct setting for fsync

2010-05-10 Thread Joshua D. Drake
On Mon, 2010-05-10 at 18:46 +0100, Greg Stark wrote: > On Mon, May 10, 2010 at 4:55 PM, Kevin Grittner > wrote: > > Robert Haas wrote: > > > >> "It might be safe" is a bit of a waffle. It would be nice if we > >> could provide some more clear guidance as to whether it is or is > >> not, or how s

Re: [HACKERS] no universally correct setting for fsync

2010-05-10 Thread Greg Stark
On Mon, May 10, 2010 at 4:55 PM, Kevin Grittner wrote: > Robert Haas wrote: > >> "It might be safe" is a bit of a waffle.  It would be nice if we >> could provide some more clear guidance as to whether it is or is >> not, or how someone could go about testing their hardware to find >> out. > > I

Re: [HACKERS] no universally correct setting for fsync

2010-05-10 Thread Kevin Grittner
Robert Haas wrote: > "It might be safe" is a bit of a waffle. It would be nice if we > could provide some more clear guidance as to whether it is or is > not, or how someone could go about testing their hardware to find > out. I think that the issue is that you could have corruption if some,

Re: [HACKERS] no universally correct setting for fsync

2010-05-10 Thread Bruce Momjian
Robert Haas wrote: > On Mon, May 10, 2010 at 11:12 AM, Bruce Momjian wrote: > > Michael Tharp wrote: > >> On 05/08/2010 04:07 AM, Craig Ringer wrote: > >> > It's probably worth mentioning that people who want to turn off fsync to > >> > gain a performance boost should instead look at a RAID contro

Re: [HACKERS] no universally correct setting for fsync

2010-05-10 Thread Robert Haas
On Mon, May 10, 2010 at 11:12 AM, Bruce Momjian wrote: > Michael Tharp wrote: >> On 05/08/2010 04:07 AM, Craig Ringer wrote: >> > It's probably worth mentioning that people who want to turn off fsync to >> > gain a performance boost should instead look at a RAID controller with a >> > BBU so they

Re: [HACKERS] no universally correct setting for fsync

2010-05-10 Thread Bruce Momjian
Michael Tharp wrote: > On 05/08/2010 04:07 AM, Craig Ringer wrote: > > It's probably worth mentioning that people who want to turn off fsync to > > gain a performance boost should instead look at a RAID controller with a > > BBU so they can safely enable write-back caching, getting most of the > >

Re: [HACKERS] no universally correct setting for fsync

2010-05-10 Thread Michael Tharp
On 05/08/2010 04:07 AM, Craig Ringer wrote: It's probably worth mentioning that people who want to turn off fsync to gain a performance boost should instead look at a RAID controller with a BBU so they can safely enable write-back caching, getting most of the benefits of fsync=off safely. Which

Re: [HACKERS] no universally correct setting for fsync

2010-05-08 Thread Craig Ringer
On 8/05/2010 1:56 AM, Josh Berkus wrote: I never meant to suggest any statement in that section is factually wrong; it's just all too rosy, leading people to believe it's no big deal to turn it off. Yeah, that section is overdue for an update. I'll write some new text and post it to pgsql-doc

Re: [HACKERS] no universally correct setting for fsync

2010-05-07 Thread Bernd Helmle
--On 7. Mai 2010 19:49:15 -0400 Tom Lane wrote: Bernd Helmle writes: I've recently even started to wonder if the performance gain with fsync=off is still that large on modern hardware. While testing large migration procedures to a new version some time ago (on an admitedly fast storage) i

Re: [HACKERS] no universally correct setting for fsync

2010-05-07 Thread Tom Lane
Bernd Helmle writes: > I've recently even started to wonder if the performance gain with fsync=off > is still that large on modern hardware. While testing large migration > procedures to a new version some time ago (on an admitedly fast storage) i > forgot here and then to turn it off, without

Re: [HACKERS] no universally correct setting for fsync

2010-05-07 Thread Bernd Helmle
--On 7. Mai 2010 09:48:53 -0500 Kevin Grittner wrote: I think it goes beyond "tweaking" -- I think we should have a bald statement like "don't turn this off unless you're OK with losing the entire contents of the database cluster." A brief listing of some cases where that is OK might be il

Re: [HACKERS] no universally correct setting for fsync

2010-05-07 Thread Josh Berkus
I never meant to suggest any statement in that section is factually wrong; it's just all too rosy, leading people to believe it's no big deal to turn it off. Yeah, that section is overdue for an update. I'll write some new text and post it to pgsql-docs. --

Re: [HACKERS] no universally correct setting for fsync

2010-05-07 Thread Kevin Grittner
Andrew Dunstan wrote: > I think the critical question is really whether you are prepared > to lose your database. Precisely; and the docs don't make that at all clear. They mention the possibility of database corruption, but downplay it: | When fsync is disabled, the operating system is all

Re: [HACKERS] no universally correct setting for fsync

2010-05-07 Thread Andrew Dunstan
Kevin Grittner wrote: There are dire-sounding statements interspersed with: | using fsync results in a performance penalty | Due to the risks involved, there is no universally correct setting | for fsync. | If you trust your operating system, your hardware, and your | utility company (

Re: [HACKERS] no universally correct setting for fsync

2010-05-07 Thread Tom Lane
"Kevin Grittner" writes: > | If you trust your operating system, your hardware, and your > | utility company (or your battery backup), you can consider > | disabling fsync. > Isn't this a little too rosy a picture to paint? I think that statement is true as far as it goes, but I agree with reji

Re: [HACKERS] no universally correct setting for fsync

2010-05-07 Thread Magnus Hagander
On Fri, May 7, 2010 at 16:00, Robert Haas wrote: > On Fri, May 7, 2010 at 9:47 AM, Kevin Grittner > wrote: >> Someone just posted to the -admin list with a database corrupted >> while running with fsync=off.  I was all set to refer him to the >> documentation explaining why he should stop doing t

Re: [HACKERS] no universally correct setting for fsync

2010-05-07 Thread Robert Haas
On Fri, May 7, 2010 at 9:47 AM, Kevin Grittner wrote: > Someone just posted to the -admin list with a database corrupted > while running with fsync=off.  I was all set to refer him to the > documentation explaining why he should stop doing this, but to my > surprise the documentation waffles on th

[HACKERS] no universally correct setting for fsync

2010-05-07 Thread Kevin Grittner
Someone just posted to the -admin list with a database corrupted while running with fsync=off. I was all set to refer him to the documentation explaining why he should stop doing this, but to my surprise the documentation waffles on the issue way past what I think is reasonable. http://www.postg