Re: [HACKERS] auto-sizing wal_buffers

2011-01-22 Thread Tom Lane
Robert Haas writes: > On Sat, Jan 22, 2011 at 9:42 PM, Tom Lane wrote: >> It does strike me that we could provide SetConfigOptionInt, >> SetConfigOptionBool, and SetConfigOptionReal for the benefit of callers >> who'd prefer to pass values in those formats.  They'd still do sprintf >> internally,

Re: [HACKERS] auto-sizing wal_buffers

2011-01-22 Thread Robert Haas
On Sat, Jan 22, 2011 at 9:42 PM, Tom Lane wrote: > Robert Haas writes: >>> Oh, you're complaining about SetConfigOption, not the assign hooks. > >> I was actually complaining about the latter, and then switched gears >> to the former.  I'm an equal-opportunity complainer today, I guess... > > It

Re: [HACKERS] auto-sizing wal_buffers

2011-01-22 Thread Tom Lane
Robert Haas writes: >> Oh, you're complaining about SetConfigOption, not the assign hooks. > I was actually complaining about the latter, and then switched gears > to the former. I'm an equal-opportunity complainer today, I guess... It does strike me that we could provide SetConfigOptionInt, Se

Re: [HACKERS] auto-sizing wal_buffers

2011-01-22 Thread Robert Haas
On Sat, Jan 22, 2011 at 9:24 PM, Tom Lane wrote: > Robert Haas writes: >> I'm going with hacks.  Any API that requires you to print to a string >> so you can turn around and immediately convert it back to an integer >> is not too swift. > > Oh, you're complaining about SetConfigOption, not the as

Re: [HACKERS] auto-sizing wal_buffers

2011-01-22 Thread Tom Lane
Robert Haas writes: > I'm going with hacks. Any API that requires you to print to a string > so you can turn around and immediately convert it back to an integer > is not too swift. Oh, you're complaining about SetConfigOption, not the assign hooks. Not sure if it's really worth refactoring tha

Re: [HACKERS] auto-sizing wal_buffers

2011-01-22 Thread Robert Haas
On Sat, Jan 22, 2011 at 9:08 PM, Tom Lane wrote: > Robert Haas writes: >> This is certainly shorter than I wrote, which is good, but it strikes >> me that the fundamental problem here is that the API for an assign >> hook is fundamentally different for strings than it is for other data >> types.

Re: [HACKERS] auto-sizing wal_buffers

2011-01-22 Thread Tom Lane
Robert Haas writes: > This is certainly shorter than I wrote, which is good, but it strikes > me that the fundamental problem here is that the API for an assign > hook is fundamentally different for strings than it is for other data > types. I agree that that's annoying, but given that strings ar

Re: [HACKERS] auto-sizing wal_buffers

2011-01-22 Thread Robert Haas
On Sat, Jan 22, 2011 at 8:45 PM, Robert Haas wrote: >> "Just set" means "use SetConfigOption". > > OK. *reads patch as committed* This is certainly shorter than I wrote, which is good, but it strikes me that the fundamental problem here is that the API for an assign hook is fundamentally differe

Re: [HACKERS] auto-sizing wal_buffers

2011-01-22 Thread Tom Lane
Robert Haas writes: >> As penance, I've committed the corrected patch. > Which is obviously derived from my version rather than Greg's, without credit. Apologies --- I thought I'd ended up reverting basically all your changes, so I just credited him. regards, tom lane -

Re: [HACKERS] auto-sizing wal_buffers

2011-01-22 Thread Robert Haas
On Sat, Jan 22, 2011 at 8:33 PM, Tom Lane wrote: > Um: that was probably too brief. Yep. > "Just set" means "use SetConfigOption". OK. > As penance, I've committed the corrected patch. Which is obviously derived from my version rather than Greg's, without credit. -- Robert Haas EnterpriseDB

Re: [HACKERS] auto-sizing wal_buffers

2011-01-22 Thread Tom Lane
I wrote: > Robert Haas writes: >> Barring screams of agony^W^W^Whelpful suggestions for how to code this >> more neatly, I'll go ahead and commit this. > The show_hook sucks, and is unnecessary, as is exposing the format > function. Just set the GUC variable to the correct value. Um: that was p

Re: [HACKERS] auto-sizing wal_buffers

2011-01-22 Thread Tom Lane
Robert Haas writes: > Barring screams of agony^W^W^Whelpful suggestions for how to code this > more neatly, I'll go ahead and commit this. The show_hook sucks, and is unnecessary, as is exposing the format function. Just set the GUC variable to the correct value. regards

Re: [HACKERS] auto-sizing wal_buffers

2011-01-22 Thread Robert Haas
On Sat, Jan 22, 2011 at 1:30 AM, Jaime Casanova wrote: > why is this better than using XLOG_BUFFER_MIN? (the same for the 8 > buffers assigned just above of it) > > +       else if (XLOGbuffers < 4) > +               XLOGbuffers = 4; Oh, good point. Woops. > also this > +       Assert(XLOGbuffe

Re: [HACKERS] auto-sizing wal_buffers

2011-01-21 Thread Jaime Casanova
On Sat, Jan 22, 2011 at 12:33 AM, Robert Haas wrote: > On Sat, Jan 15, 2011 at 11:52 AM, Greg Smith wrote: >> Where I was expecting that setting to be "4" instead for 32kB.  So there's >> probably some minor bug left in where I inserted this into the >> initialization sequence. > > So I exposed t

Re: [HACKERS] auto-sizing wal_buffers

2011-01-21 Thread Robert Haas
On Sat, Jan 15, 2011 at 11:52 AM, Greg Smith wrote: > Where I was expecting that setting to be "4" instead for 32kB.  So there's > probably some minor bug left in where I inserted this into the > initialization sequence. So I started taking a look at this patch tonight with an eye to committing i

Re: [HACKERS] auto-sizing wal_buffers

2011-01-18 Thread Fujii Masao
On Tue, Jan 18, 2011 at 8:50 PM, Greg Smith wrote: >> Why is the minimum value 64kB only when wal_buffers is set to >> -1? This seems confusing for users. >> > > That's because the current default on older versions is 64kB.  Since the > automatic selection is going to be the new default, I hope, I

Re: [HACKERS] auto-sizing wal_buffers

2011-01-18 Thread Greg Smith
Josh Berkus wrote: I think we can be more specific on that last sentence; is there even any *theoretical* benefit to settings above 16MB, the size of a WAL segment? Certainly there have been no test results to show any. There was the set Marti just reminded about. The old wording suggested

Re: [HACKERS] auto-sizing wal_buffers

2011-01-18 Thread Greg Smith
Fujii Masao wrote: +/* Minimum setting used for a lower bound on wal_buffers */ +#define XLOG_BUFFER_MIN4 Why didn't you use XLOG_BUFFER_MIN instead of XLOGbuffersMin? XLOG_BUFFER_MIN is not used anywhere for now. That's a typo; will fix. + if (XLOGbu

Re: [HACKERS] auto-sizing wal_buffers

2011-01-16 Thread Jeff Janes
On Sun, Jan 16, 2011 at 9:32 AM, Tom Lane wrote: > Josh Berkus writes: >> I think we can be more specific on that last sentence; is there even any >> *theoretical* benefit to settings above 16MB, the size of a WAL segment? > > IIRC there's a forced fsync at WAL segment switch, so no. However oth

Re: [HACKERS] auto-sizing wal_buffers

2011-01-16 Thread Jeff Janes
On Sat, Jan 15, 2011 at 2:34 PM, Josh Berkus wrote: > On 1/14/11 10:51 PM, Greg Smith wrote: >> >> !         Since the data is written out to disk at every transaction >> commit, >> !         the setting many only need to be be large enough to hold the >> amount >> !         of WAL data generated

Re: [HACKERS] auto-sizing wal_buffers

2011-01-16 Thread Fujii Masao
On Sun, Jan 16, 2011 at 7:34 AM, Josh Berkus wrote: > I think we can be more specific on that last sentence; is there even any > *theoretical* benefit to settings above 16MB, the size of a WAL segment? >  Certainly there have been no test results to show any. If the workload generates 16MB or mor

Re: [HACKERS] auto-sizing wal_buffers

2011-01-16 Thread Fujii Masao
On Sun, Jan 16, 2011 at 1:52 AM, Greg Smith wrote: > Fujii Masao wrote: >> >> +int                    XLOGbuffersMin = 8; >> >> XLOGbuffersMin is a fixed value. I think that defining it as a macro >> rather than a variable seems better. >> >> +               if (XLOGbuffers > 2048) >> +          

Re: [HACKERS] auto-sizing wal_buffers

2011-01-16 Thread Marti Raudsepp
On Sun, Jan 16, 2011 at 00:34, Josh Berkus wrote: > I think we can be more specific on that last sentence; is there even any > *theoretical* benefit to settings above 16MB, the size of a WAL segment? >  Certainly there have been no test results to show any. I don't know if it's applicable to real

Re: [HACKERS] auto-sizing wal_buffers

2011-01-16 Thread Tom Lane
Josh Berkus writes: > I think we can be more specific on that last sentence; is there even any > *theoretical* benefit to settings above 16MB, the size of a WAL segment? IIRC there's a forced fsync at WAL segment switch, so no. regards, tom lane -- Sent via pgsql-hacker

Re: [HACKERS] auto-sizing wal_buffers

2011-01-15 Thread Josh Berkus
On 1/14/11 10:51 PM, Greg Smith wrote: > > ! Since the data is written out to disk at every transaction > commit, > ! the setting many only need to be be large enough to hold the > amount > ! of WAL data generated by one typical transaction. Larger values, > ! typi

Re: [HACKERS] auto-sizing wal_buffers

2011-01-15 Thread Greg Smith
Fujii Masao wrote: +intXLOGbuffersMin = 8; XLOGbuffersMin is a fixed value. I think that defining it as a macro rather than a variable seems better. + if (XLOGbuffers > 2048) + XLOGbuffers = 2048; Using "XLOG_SEG_SIZE/XLOG_BLCKSZ" rather

Re: [HACKERS] auto-sizing wal_buffers

2011-01-15 Thread Fujii Masao
On Sat, Jan 15, 2011 at 3:51 PM, Greg Smith wrote: > Agreed on both fronts.  Attached patch does the magic.  Also available in > branch "walbuffers" from git://github.com/greg2ndQuadrant/postgres.git +intXLOGbuffersMin = 8; XLOGbuffersMin is a fixed value. I think that defini

Re: [HACKERS] auto-sizing wal_buffers

2011-01-14 Thread Greg Smith
Kevin Grittner wrote: I guess a manual override doesn't bother me too much, but I am a bit dubious of its value, and there is value in keeping the GUC count down... It's a risk-reward thing really. The reward for removing it is that a few lines of code and a small section of the documentatio

Re: [HACKERS] auto-sizing wal_buffers

2011-01-14 Thread Greg Smith
Tom Lane wrote: I think we need to keep the override capability until the autotune algorithm has proven itself in the field for a couple of years. I agree with Josh that a negative value should be used to select the autotune method. Agreed on both fronts. Attached patch does the magic. Al

Re: [HACKERS] auto-sizing wal_buffers

2011-01-13 Thread Tom Lane
Robert Haas writes: > On Thu, Jan 13, 2011 at 5:29 PM, Magnus Hagander wrote: >> +1, I like the idea. Would it still be there to override if necessary? > Depends what people want to do. We could make the default "0kB", and > define that to mean "auto-tune", or we could remove the parameter > al

Re: [HACKERS] auto-sizing wal_buffers

2011-01-13 Thread Kevin Grittner
Robert Haas wrote: > Would anyone like to argue vigorously for or against the above > proposal? Greg's numbers look reasonable to me, and there's nobody I'd trust more to come up with reasonable numbers for this. One less tunable is a good thing, especially since this designed to scale from s

Re: [HACKERS] auto-sizing wal_buffers

2011-01-13 Thread Josh Berkus
Robert, >> Unfortunately, we might still need a manual parameter for override >> because of the interaction between wal_buffers and >> synchronous_commit=off, since it sets the max size of the unflushed data >> buffer. Discuss? > > Do we have any evidence there's actually a problem in that case,

Re: [HACKERS] auto-sizing wal_buffers

2011-01-13 Thread Robert Haas
On Thu, Jan 13, 2011 at 6:02 PM, Josh Berkus wrote: > >> Depends what people want to do.  We could make the default "0kB", and >> define that to mean "auto-tune", or we could remove the parameter >> altogether.  I think I was envisioning the latter, but if people are >> hesitant to do that we coul

Re: [HACKERS] auto-sizing wal_buffers

2011-01-13 Thread Josh Berkus
> Depends what people want to do. We could make the default "0kB", and > define that to mean "auto-tune", or we could remove the parameter > altogether. I think I was envisioning the latter, but if people are > hesitant to do that we could do the former instead. Unfortunately, we might still ne

Re: [HACKERS] auto-sizing wal_buffers

2011-01-13 Thread Robert Haas
On Thu, Jan 13, 2011 at 5:29 PM, Magnus Hagander wrote: > +1, I like the idea. Would it still be there to override if necessary? Depends what people want to do. We could make the default "0kB", and define that to mean "auto-tune", or we could remove the parameter altogether. I think I was envis

Re: [HACKERS] auto-sizing wal_buffers

2011-01-13 Thread Magnus Hagander
On Thu, Jan 13, 2011 at 23:19, Robert Haas wrote: > On Thu, Jan 6, 2011 at 11:37 PM, Greg Smith wrote: >> If it defaulted to 3% of shared_buffers, min 64K & max 16MB for the auto >> setting, it would for the most part become an autotuned parameter.  That >> would make it 0.75 to 1MB at the standa

[HACKERS] auto-sizing wal_buffers

2011-01-13 Thread Robert Haas
On Thu, Jan 6, 2011 at 11:37 PM, Greg Smith wrote: > If it defaulted to 3% of shared_buffers, min 64K & max 16MB for the auto > setting, it would for the most part become an autotuned parameter.  That > would make it 0.75 to 1MB at the standard anemic Linux default kernel > parameters.  Maybe more