Re: [HACKERS] pgbench: new feature allowing to launch shell commands

2009-12-01 Thread Greg Smith
Here's a first round of review of the patch submitted at http://archives.postgresql.org/message-id/c64c5f8b0910252350w42f7ea13g52a6a88a86143...@mail.gmail.com This is moving along nicely, and is now working (with some hacking) for the one use case I wanted it for. High-level summary of what I

Re: [HACKERS] [PATCH] bugfix for int2vectorin

2009-12-01 Thread Tom Lane
Caleb Welton writes: > On 12/1/09 7:38 PM, "Tom Lane" wrote: >> Under what circumstances would users (or anyone at all) be putting data into >> an int2vector? > What exactly is your objection to having the int2arrayin parser handle its > input conversion reasonably? I'm trying to gauge what t

Re: [HACKERS] SE-PgSQL patch review

2009-12-01 Thread KaiGai Kohei
Tom Lane wrote: > Even if we were to accept the SEPostgres patches lock stock and barrel > tomorrow, I don't foresee that it will ever get to the point of being > useful except to an extremely small group of users who are driven by > extreme need. Nobody else is going to have the motivation needed

Re: [HACKERS] Application name patch - v4

2009-12-01 Thread Tom Lane
Dave Page writes: > On Tue, Dec 1, 2009 at 4:19 PM, Tom Lane wrote: >> I don't think that we need to bump the protocol version.  The real >> alternative here would be that libpq sends a startup packet that >> includes application_name, and if it gets an error back from that, >> it starts over wit

Re: [HACKERS] Page-level version upgrade (was: Block-level CRC checks)

2009-12-01 Thread David Fetter
On Tue, Dec 01, 2009 at 10:34:11PM -0500, Bruce Momjian wrote: > Robert Haas wrote: > > The key issue, as I think Heikki identified at the time, is to > > figure out how you're eventually going to get rid of the old > > pages. He proposed running a pre-upgrade utility on each page to > > reserve t

Re: [HACKERS] Page-level version upgrade (was: Block-level CRC checks)

2009-12-01 Thread Bruce Momjian
Robert Haas wrote: > >> The key issue, as I think Heikki identified at the time, is to figure > >> out how you're eventually going to get rid of the old pages. ?He > >> proposed running a pre-upgrade utility on each page to reserve the > >> right amount of free space. > >> > >> http://archives.post

Re: [HACKERS] Block-level CRC checks

2009-12-01 Thread Aidan Van Dyk
* Greg Stark [091201 20:14]: > I'm not sure we're on the same page. As I understand it there are > three proposals on the table now: > > 1) set aside a section of the page to contain only non-checksummed > hint bits. That section has to be relocatable so the crc check would > have to read the

Re: [HACKERS] operator exclusion constraints

2009-12-01 Thread Robert Haas
On Fri, Nov 27, 2009 at 10:18 PM, Jeff Davis wrote: > On Thu, 2009-11-26 at 01:33 -0800, Jeff Davis wrote: >> Remaining issues: >>  * represent operator IDs in catalog, rather than strategy numbers > > Done, attached. > >>  * if someone thinks it's an issue, support search strategies that >>    re

Re: [HACKERS] Page-level version upgrade (was: Block-level CRC checks)

2009-12-01 Thread Robert Haas
On Tue, Dec 1, 2009 at 10:45 PM, Greg Smith wrote: > Robert Haas wrote: >> >> If the pre-upgrade utility is something >> that has to be run while the database is off-line, then it defeats the >> point of an in-place upgrade.  If it can be run while the database is >> up, I fear it will need to be

Re: [HACKERS] Page-level version upgrade (was: Block-level CRC checks)

2009-12-01 Thread Robert Haas
On Tue, Dec 1, 2009 at 10:34 PM, Bruce Momjian wrote: > Robert Haas wrote: >> > Well, there were quite a number of open issues relating to page >> > conversion: >> > >> > ? ? ? ?o ?Do we write the old version or just convert on read? >> > ? ? ? ?o ?How do we write pages that get larger on conversi

Re: [HACKERS] PL/Python array support

2009-12-01 Thread Joshua Tolley
On Fri, Nov 20, 2009 at 12:00:24AM +0200, Peter Eisentraut wrote: > On fre, 2009-11-13 at 18:46 +0300, Teodor Sigaev wrote: > > CREATE OR REPLACE FUNCTION incr(stuff int[]) RETURNS int[] AS $$ > > for x in stuff: > > yield x+1 > > $$ > > LANGUAGE 'plpythonu'; > > > > # select incr(ARRAY[1,2,3

Re: [HACKERS] [PATCH] bugfix for int2vectorin

2009-12-01 Thread Robert Haas
On Tue, Dec 1, 2009 at 10:38 PM, Tom Lane wrote: > Caleb Welton writes: >> I believe the int2vectorin function should handle invalid input more cleanly. > > Why bother?  Under what circumstances would users (or anyone at all) > be putting data into an int2vector?  It's an internal type and is > c

Re: [HACKERS] [PATCH] bugfix for int2vectorin

2009-12-01 Thread Caleb Welton
Tom, Thanks for the comments. Caleb Welton writes: > I believe the int2vectorin function should handle invalid input more cleanly. On 12/1/09 7:38 PM, "Tom Lane" wrote: >> Why bother? Because correctness is good. Results that produce unexpected results from incorrect input formatting lead

Re: [HACKERS] Page-level version upgrade (was: Block-level CRC checks)

2009-12-01 Thread Greg Smith
Robert Haas wrote: If the pre-upgrade utility is something that has to be run while the database is off-line, then it defeats the point of an in-place upgrade. If it can be run while the database is up, I fear it will need to be deeply integrated into the server. And since we can't know the req

Re: [HACKERS] [PATCH] bugfix for int2vectorin

2009-12-01 Thread Tom Lane
Caleb Welton writes: > I believe the int2vectorin function should handle invalid input more cleanly. Why bother? Under what circumstances would users (or anyone at all) be putting data into an int2vector? It's an internal type and is certainly not meant for use in user tables.

Re: [HACKERS] Page-level version upgrade (was: Block-level CRC checks)

2009-12-01 Thread Bruce Momjian
Robert Haas wrote: > > Well, there were quite a number of open issues relating to page > > conversion: > > > > ? ? ? ?o ?Do we write the old version or just convert on read? > > ? ? ? ?o ?How do we write pages that get larger on conversion to the > > ? ? ? ? ? new format? > > > > As I rember the pa

Re: [HACKERS] SE-PgSQL patch review

2009-12-01 Thread Tom Lane
KaiGai Kohei writes: > Joshua D. Drake wrote: >> I just did a little research and it appears the other two big names in >> this world (Novel and Ubuntu) are using something called App Armor. > As far as I can see, SUSE, Ubuntu and Debian provide SELinux option. > But they are more conservative th

Re: [HACKERS] Page-level version upgrade (was: Block-level CRC checks)

2009-12-01 Thread Robert Haas
On Tue, Dec 1, 2009 at 9:31 PM, Bruce Momjian wrote: > Robert Haas wrote: >> On Tue, Dec 1, 2009 at 5:15 PM, Greg Stark wrote: >> > On Tue, Dec 1, 2009 at 9:58 PM, decibel wrote: >> >> What happened to the work that was being done to allow a page to be >> >> upgraded >> >> on the fly when it wa

Re: [HACKERS] SE-PgSQL patch review

2009-12-01 Thread Bruce Momjian
Greg Williamson wrote: > As far as I can see, SUSE, Ubuntu and Debian provide SELinux > option. But they are more conservative than RedHat/Fedora, > because it is not enabled in the default installation. > > I don't think it is unpreferable decision. Users can choose the > option by themself acco

Re: [HACKERS] SE-PgSQL patch review

2009-12-01 Thread Greg Williamson
KaiGai Kohei wrote: === Joshua D. Drake wrote: > On Tue, 2009-12-01 at 14:46 -0500, Tom Lane wrote: >> "Joshua D. Drake" writes: >>> On Mon, 2009-11-30 at 20:28 -0800, David Fetter wrote: This is totally separate from the really important question of whether SE-Linux has a future, and

[HACKERS] [PATCH] bugfix for int2vectorin

2009-12-01 Thread Caleb Welton
I believe the int2vectorin function should handle invalid input more cleanly. Current behavior: -- Correct input format: SELECT '1 2 3'::int2vector; int2vector 1 2 3 (1 row) -- Three variations on incorrect input format SELECT '1not 2really_an 3int2vector'::int2vector; int2vector

Re: [HACKERS] Page-level version upgrade (was: Block-level CRC checks)

2009-12-01 Thread Bruce Momjian
Robert Haas wrote: > On Tue, Dec 1, 2009 at 5:15 PM, Greg Stark wrote: > > On Tue, Dec 1, 2009 at 9:58 PM, decibel wrote: > >> What happened to the work that was being done to allow a page to be > >> upgraded > >> on the fly when it was read in from disk? > > > > There were no page level changes

Re: [HACKERS] Page-level version upgrade (was: Block-level CRC checks)

2009-12-01 Thread Robert Haas
On Tue, Dec 1, 2009 at 5:15 PM, Greg Stark wrote: > On Tue, Dec 1, 2009 at 9:58 PM, decibel wrote: >> What happened to the work that was being done to allow a page to be upgraded >> on the fly when it was read in from disk? > > There were no page level changes between 8.3 and 8.4. That's true, b

Re: [HACKERS] SE-PgSQL patch review

2009-12-01 Thread KaiGai Kohei
Joshua D. Drake wrote: > On Tue, 2009-12-01 at 14:46 -0500, Tom Lane wrote: >> "Joshua D. Drake" writes: >>> On Mon, 2009-11-30 at 20:28 -0800, David Fetter wrote: This is totally separate from the really important question of whether SE-Linux has a future, and another about whether, if

Re: [HACKERS] SE-PgSQL patch review

2009-12-01 Thread KaiGai Kohei
Tom Lane wrote: > "Joshua D. Drake" writes: >> On Mon, 2009-11-30 at 20:28 -0800, David Fetter wrote: >>> This is totally separate from the really important question of whether >>> SE-Linux has a future, and another about whether, if SE-Linux has a >>> future, PostgreSQL needs to go there. > >> W

Re: [HACKERS] SE-PgSQL patch review

2009-12-01 Thread KaiGai Kohei
Josh Berkus wrote: >> This is totally separate from the really important question of whether >> SE-Linux has a future, and another about whether, if SE-Linux has a >> future, PostgreSQL needs to go there. > > If the hooks are generic enough that the could potentially be adapted to > other security

Re: [HACKERS] Block-level CRC checks

2009-12-01 Thread Greg Stark
On Wed, Dec 2, 2009 at 12:03 AM, Tom Lane wrote: > Greg Stark writes: >> On Tue, Dec 1, 2009 at 10:47 PM, Tom Lane wrote: >>> I don't think "relatively cheap" is the right criterion here --- the >>> question to me is how many assumptions are you making in order to >>> compute the page's CRC.  Ea

Re: [HACKERS] YAML Was: CommitFest status/management

2009-12-01 Thread Josh Berkus
On 11/30/09 8:17 PM, Andrew Dunstan wrote: > Do we have consensus yet that we want YAML? It seemed, well, > yet another format without all that much advantage over what's > there. Well, what's the code count? What dependencies, if any, does it add? --Josh -- Sent via pgsql-hackers mailing lis

Re: [HACKERS] Block-level CRC checks

2009-12-01 Thread Tom Lane
Greg Stark writes: > On Tue, Dec 1, 2009 at 10:47 PM, Tom Lane wrote: >> I don't think "relatively cheap" is the right criterion here --- the >> question to me is how many assumptions are you making in order to >> compute the page's CRC.  Each assumption degrades the reliability >> of the check,

Re: [HACKERS] [CORE] EOL for 7.4?

2009-12-01 Thread Andrew Dunstan
Tom Lane wrote: The other side of the coin is that people running such old versions are in it for stability --- they don't *want* bugs fixed, unless they're bugs they've hit themselves. Major fixes that would possibly destabilize the code base would be exactly what's not wanted. Every time I

Re: [HACKERS] Block-level CRC checks

2009-12-01 Thread decibel
On Dec 1, 2009, at 4:13 PM, Greg Stark wrote: On Tue, Dec 1, 2009 at 9:57 PM, Richard Huxton wrote: Why are we writing out the hint bits to disk anyway? Is it really so slow to calculate them on read + cache them that it's worth all this trouble? Are they not also to blame for the "write my im

Re: [HACKERS] Block-level CRC checks

2009-12-01 Thread Greg Stark
On Tue, Dec 1, 2009 at 10:47 PM, Tom Lane wrote: > Bruce Momjian writes: >> Greg Stark wrote: >>> It should be relatively cheap to skip the hint bits in the line >>> pointers since they'll be the same bits of every 16-bit value for a >>> whole range. Alternatively we could just CRC the tuples and

Re: [HACKERS] [CORE] EOL for 7.4?

2009-12-01 Thread Tom Lane
Greg Stark writes: > I still insist it's unrealistic to consider any of these, even 8.2, as > anything but "best effort" at this point. Agreed, and we should not pretend otherwise. > Declaring 8.0 "end of life" > today is implying that we haven't already been skipping fixing bugs in > it that wo

Re: [HACKERS] [CORE] EOL for 7.4?

2009-12-01 Thread Greg Smith
Greg Stark wrote: I can't imagine supporting recommending 8.1 for anything but a toy deployment today. Today's toy and test deployments are tomorrow's high-volume ones. It's pretty clear that anyone who cares about performance wants something newer. But that's a secondary concern behind s

Re: [HACKERS] Block-level CRC checks

2009-12-01 Thread Tom Lane
Bruce Momjian writes: > Tom Lane wrote: >> I don't think "relatively cheap" is the right criterion here --- the >> question to me is how many assumptions are you making in order to >> compute the page's CRC. Each assumption degrades the reliability >> of the check, not to mention creating another

Re: [HACKERS] [CORE] EOL for 7.4?

2009-12-01 Thread Greg Stark
On Tue, Dec 1, 2009 at 10:40 PM, Greg Smith wrote: > Sure, at some point in 2010, we may reach a point where it would be ill > advised to build a new system using RHEL5/PG8.1.  I was suggesting more that > there are completely reasonable reasons to deploy 8.1 even right now in > 2009, and people

Re: [HACKERS] Block-level CRC checks

2009-12-01 Thread Tom Lane
Richard Huxton writes: > So what is the cost of calculating the hint-bits for a whole block of > tuples in one go vs reading that block from actual spinning disk? Potentially a couple of hundred times worse, if you're unlucky and each XID on the page requires visiting a different block of clog th

Re: [HACKERS] enable-thread-safety defaults?

2009-12-01 Thread Bruce Momjian
Tom Lane wrote: > Magnus Hagander writes: > > 2009/12/1 Bruce Momjian : > >> What are we going to do for build farm members who don't support > >> threading? ?Is someone going to manually update their configure flags? > > > Yeah, I think so. > > > Unless there's a whole lot of them, in which cas

Re: [HACKERS] Block-level CRC checks

2009-12-01 Thread Bruce Momjian
Tom Lane wrote: > Bruce Momjian writes: > > Greg Stark wrote: > >> It should be relatively cheap to skip the hint bits in the line > >> pointers since they'll be the same bits of every 16-bit value for a > >> whole range. Alternatively we could just CRC the tuples and assume a > >> corrupted line

Re: [HACKERS] Block-level CRC checks

2009-12-01 Thread Tom Lane
Bruce Momjian writes: > Greg Stark wrote: >> It should be relatively cheap to skip the hint bits in the line >> pointers since they'll be the same bits of every 16-bit value for a >> whole range. Alternatively we could just CRC the tuples and assume a >> corrupted line pointer will show itself qui

Re: [HACKERS] Block-level CRC checks

2009-12-01 Thread Richard Huxton
Greg Stark wrote: > On Tue, Dec 1, 2009 at 9:57 PM, Richard Huxton wrote: >> Why are we writing out the hint bits to disk anyway? Is it really so >> slow to calculate them on read + cache them that it's worth all this >> trouble? Are they not also to blame for the "write my import data twice" >> f

Re: [CORE] [HACKERS] EOL for 7.4?

2009-12-01 Thread Greg Smith
Tom Lane wrote: Greg Smith writes: What I was trying to suggest was that right now, there are situations where a new deployment on 8.1 is still completely reasonable and possible to justify in the Enterprise Linux space, whereas I don't know of any situation where 7.4/8.0 can be similarly

Re: [CORE] [HACKERS] EOL for 7.4?

2009-12-01 Thread Devrim GÜNDÜZ
On Tue, 2009-12-01 at 17:14 -0500, Tom Lane wrote: > However, if you are paying attention to what has shipped in > recent Fedora releases, it's not hard to figure out that it will have > PG >= 8.4. I thought RHEL 6 would ship with 8.3. It will be perfect if it skips 8.3. -- Devrim GÜNDÜZ, RHCE Co

Re: [HACKERS] Page-level version upgrade (was: Block-level CRC checks)

2009-12-01 Thread Bruce Momjian
Greg Stark wrote: > On Tue, Dec 1, 2009 at 9:58 PM, decibel wrote: > > What happened to the work that was being done to allow a page to be upgraded > > on the fly when it was read in from disk? > > There were no page level changes between 8.3 and 8.4. Yea, we have the idea of how to do it (in ca

Re: [HACKERS] Block-level CRC checks

2009-12-01 Thread Bruce Momjian
Greg Stark wrote: > On Tue, Dec 1, 2009 at 6:41 PM, Tom Lane wrote: > > Bruce Momjian writes: > >> OK, here is another idea, maybe crazy: > > > >> When we read in a page that has an invalid CRC, we check the page to see > >> which hint bits are _not_ set, and we try setting them to see if can get

Re: [HACKERS] Page-level version upgrade (was: Block-level CRC checks)

2009-12-01 Thread Greg Stark
On Tue, Dec 1, 2009 at 9:58 PM, decibel wrote: > What happened to the work that was being done to allow a page to be upgraded > on the fly when it was read in from disk? There were no page level changes between 8.3 and 8.4. -- greg -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgr

Re: [HACKERS] Block-level CRC checks

2009-12-01 Thread decibel
On Dec 1, 2009, at 1:39 PM, Kevin Grittner wrote: Josh Berkus wrote: And a lot of our biggest users are having issues; it seems pretty much guarenteed that if you have more than 20 postgres servers, at least one of them will have bad memory, bad RAID and/or a bad driver. Huh?!? We have abou

Re: [CORE] [HACKERS] EOL for 7.4?

2009-12-01 Thread Tom Lane
Greg Smith writes: > What I was trying to suggest was that right now, there are situations > where a new deployment on 8.1 is still completely reasonable and > possible to justify in the Enterprise Linux space, whereas I don't know > of any situation where 7.4/8.0 can be similarly defended as a

Re: [HACKERS] Block-level CRC checks

2009-12-01 Thread Greg Stark
On Tue, Dec 1, 2009 at 9:57 PM, Richard Huxton wrote: > Why are we writing out the hint bits to disk anyway? Is it really so > slow to calculate them on read + cache them that it's worth all this > trouble? Are they not also to blame for the "write my import data twice" > feature? It would be int

Re: [HACKERS] Block-level CRC checks

2009-12-01 Thread Bruce Momjian
Tom Lane wrote: > Bruce Momjian writes: > > OK, crazy idea #3. What if we had a per-page counter of the number of > > hint bits set --- that way, we would only consider a CRC check failure > > to be corruption if the count matched the hint bit count on the page. > > Seems like rather a large hol

Re: [CORE] [HACKERS] EOL for 7.4?

2009-12-01 Thread Greg Smith
Tom Lane wrote: Well, actually, if it's just "what will RH support", I just today got launch commit on this... What I was trying to suggest was that right now, there are situations where a new deployment on 8.1 is still completely reasonable and possible to justify in the Enterprise Linux sp

Re: [CORE] [HACKERS] EOL for 7.4?

2009-12-01 Thread Andrew Dunstan
Tom Lane wrote: Greg Smith writes: Some people consider the extended support and easy upgrades of the RHEL5 versions valuable enough that they have a strong preference to use the version of PostgreSQL that ships with it. Right now, when such people ask me about using 8.1 in that context

Page-level version upgrade (was: [HACKERS] Block-level CRC checks)

2009-12-01 Thread decibel
On Dec 1, 2009, at 12:58 PM, Tom Lane wrote: The bottom line here seems to be that the only practical way to do anything like this is to move the hint bits into their own area of the page, and then exclude them from the CRC. Are we prepared to once again blow off any hope of in-place update for

Re: [HACKERS] Block-level CRC checks

2009-12-01 Thread Richard Huxton
Bruce Momjian wrote: > Tom Lane wrote: >> >> The suggestions that were made upthread about moving the hint bits >> could resolve the second objection, but once you do that you might >> as well just exclude them from the CRC and eliminate the guessing. > > OK, crazy idea #3. What if we had a per-p

Re: [HACKERS] Block-level CRC checks

2009-12-01 Thread Tom Lane
Bruce Momjian writes: > OK, crazy idea #3. What if we had a per-page counter of the number of > hint bits set --- that way, we would only consider a CRC check failure > to be corruption if the count matched the hint bit count on the page. Seems like rather a large hole in the ability to detect c

Re: [HACKERS] A thought about regex versus multibyte character sets

2009-12-01 Thread Tom Lane
Alvaro Herrera writes: > Tom Lane wrote: >> I just spent a bit of time considering what we might do to fix this. >> The idea mentioned in the above thread was to switch over to using >> wchar_t in the regex code, but that seems to have a number of problems. >> One showstopper is that on some platf

Re: [HACKERS] Block-level CRC checks

2009-12-01 Thread Bruce Momjian
Tom Lane wrote: > Bruce Momjian writes: > > OK, here is another idea, maybe crazy: > > > When we read in a page that has an invalid CRC, we check the page to see > > which hint bits are _not_ set, and we try setting them to see if can get > > a matching CRC. If there no missing hint bits and the

Re: [HACKERS] A thought about regex versus multibyte character sets

2009-12-01 Thread Alvaro Herrera
Tom Lane wrote: > I just spent a bit of time considering what we might do to fix this. > The idea mentioned in the above thread was to switch over to using > wchar_t in the regex code, but that seems to have a number of problems. > One showstopper is that on some platforms wchar_t is only 16 bits

Re: [HACKERS] Application name patch - v4

2009-12-01 Thread Marko Kreen
On 12/1/09, Tom Lane wrote: > Marko Kreen writes: > > On 12/1/09, Tom Lane wrote: > > >> If you're happy with handling the existing connection parameters in a given > >> way, why would you not want application_name behaving that same way? > > > Well, in pgbouncer case, the parameters tracked

Re: [CORE] [HACKERS] EOL for 7.4?

2009-12-01 Thread Tom Lane
Greg Smith writes: > Some people consider the extended support and easy upgrades of the RHEL5 > versions valuable enough that they have a strong preference to use the > version of PostgreSQL that ships with it. Right now, when such people > ask me about using 8.1 in that context, I tell them w

Re: [HACKERS] Block-level CRC checks

2009-12-01 Thread Bruce Momjian
Andrew Dunstan wrote: > > > Tom Lane wrote: > > Robert Haas writes: > > > >> OK, fair enough. My implication that only page formats were at issue > >> was off-base. My underlying point was that I think we have to be > >> prepared to write code that can understand old binary formats (on the

Re: [HACKERS] SE-PgSQL patch review

2009-12-01 Thread Joshua D. Drake
On Tue, 2009-12-01 at 14:46 -0500, Tom Lane wrote: > "Joshua D. Drake" writes: > > On Mon, 2009-11-30 at 20:28 -0800, David Fetter wrote: > >> This is totally separate from the really important question of whether > >> SE-Linux has a future, and another about whether, if SE-Linux has a > >> future

Re: [CORE] [HACKERS] EOL for 7.4?

2009-12-01 Thread Greg Smith
Tom Lane wrote: Personally I'll still be on the hook for maintaining 8.1 in RHEL5 so I'd be just as happy to keep it alive a bit longer, but if the community doesn't want to deal with it that makes perfect sense. Some people consider the extended support and easy upgrades of the RHEL5 version

Re: [HACKERS] Application name patch - v4

2009-12-01 Thread Tom Lane
Marko Kreen writes: > On 12/1/09, Tom Lane wrote: >> If you're happy with handling the existing connection parameters in a given >> way, why would you not want application_name behaving that same way? > Well, in pgbouncer case, the parameters tracked via ParamStatus are > handled transparently.

Re: [HACKERS] Application name patch - v4

2009-12-01 Thread Marko Kreen
On 12/1/09, Tom Lane wrote: > Marko Kreen writes: > > > No, at least both pgbouncer and pgpool consider only (username, database) > > pair as pool identifier. Rest of the startup params are tuned on the fly. > > And I think that should stay that way. > > > If you're happy with handling the exi

Re: [HACKERS] Block-level CRC checks

2009-12-01 Thread Andrew Dunstan
Tom Lane wrote: Robert Haas writes: OK, fair enough. My implication that only page formats were at issue was off-base. My underlying point was that I think we have to be prepared to write code that can understand old binary formats (on the tuple, page, or relation level) if we want this

Re: [HACKERS] Block-level CRC checks

2009-12-01 Thread Tom Lane
Robert Haas writes: > OK, fair enough. My implication that only page formats were at issue > was off-base. My underlying point was that I think we have to be > prepared to write code that can understand old binary formats (on the > tuple, page, or relation level) if we want this to work and work

Re: [HACKERS] Block-level CRC checks

2009-12-01 Thread Robert Haas
On Tue, Dec 1, 2009 at 3:04 PM, Greg Stark wrote: > I find that hard to understand. I believe the consensus is that an > on-demand page-level migration statregy like Aidan described is > precisely the plan when it's necessary to handle page format changes. > There were no page format changes for 8

Re: [HACKERS] Application name patch - v4

2009-12-01 Thread Tom Lane
Marko Kreen writes: > No, at least both pgbouncer and pgpool consider only (username, database) > pair as pool identifier. Rest of the startup params are tuned on the fly. > And I think that should stay that way. If you're happy with handling the existing connection parameters in a given way, wh

Re: [HACKERS] Application name patch - v4

2009-12-01 Thread Marko Kreen
On 12/1/09, Tom Lane wrote: > Marko Kreen writes: > > If the pooler gets new connection with same username:database > > as some existing connection, but with different appname, > > what it is supposed to do? > > > Whatever it wants to. People seem to be imagining that the appname > isn't und

Re: [HACKERS] Block-level CRC checks

2009-12-01 Thread Greg Stark
On Tue, Dec 1, 2009 at 8:04 PM, Greg Stark wrote: > And there was no point writing it for previously releases because there > was **no** pg_migrator anyways. oops -- greg -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.pos

Re: [HACKERS] Block-level CRC checks

2009-12-01 Thread Greg Stark
On Tue, Dec 1, 2009 at 7:51 PM, Robert Haas wrote: > On Tue, Dec 1, 2009 at 2:06 PM, Aidan Van Dyk wrote: >> Well, *I* think if we're ever going to have really reliable "in-place >> upgrades" that we can expect to function release after release, we're >> going to need to be able to read in "old v

Re: [HACKERS] Block-level CRC checks

2009-12-01 Thread Robert Haas
On Tue, Dec 1, 2009 at 2:06 PM, Aidan Van Dyk wrote: > Well, *I* think if we're ever going to have really reliable "in-place > upgrades" that we can expect to function release after release, we're > going to need to be able to read in "old version" pages, and convert > them to current version page

Re: [HACKERS] SE-PgSQL patch review

2009-12-01 Thread Tom Lane
"Joshua D. Drake" writes: > On Mon, 2009-11-30 at 20:28 -0800, David Fetter wrote: >> This is totally separate from the really important question of whether >> SE-Linux has a future, and another about whether, if SE-Linux has a >> future, PostgreSQL needs to go there. > Why would we think that it

Re: [HACKERS] Block-level CRC checks

2009-12-01 Thread Greg Stark
On Tue, Dec 1, 2009 at 7:19 PM, Josh Berkus wrote: > However, that solution would not detect subtle corruption, like > single-bit-flipping issues caused by quantum errors. Well there is a solution for this, ECC RAM. There's *no* software solution for it. The corruption can just as easily happen t

Re: [HACKERS] Block-level CRC checks

2009-12-01 Thread Kevin Grittner
Josh Berkus wrote: > And a lot of our biggest users are having issues; it seems pretty > much guarenteed that if you have more than 20 postgres servers, at > least one of them will have bad memory, bad RAID and/or a bad > driver. Huh?!? We have about 200 clusters running on about 100 boxes, a

Re: [CORE] [HACKERS] EOL for 7.4?

2009-12-01 Thread Tom Lane
Andrew Dunstan writes: > The time between these periodic debates seems to be getting shorter and > shorter. No, this is just a continuation of the unresolved thread from a month or so ago. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgres

Re: [HACKERS] enable-thread-safety defaults?

2009-12-01 Thread Tom Lane
Magnus Hagander writes: > 2009/12/1 Bruce Momjian : >> What are we going to do for build farm members who don't support >> threading?  Is someone going to manually update their configure flags? > Yeah, I think so. > Unless there's a whole lot of them, in which case we revert the patch. It would

Re: [HACKERS] SE-PgSQL patch review

2009-12-01 Thread Joshua D. Drake
On Mon, 2009-11-30 at 20:28 -0800, David Fetter wrote: > This is totally separate from the really important question of whether > SE-Linux has a future, and another about whether, if SE-Linux has a > future, PostgreSQL needs to go there. Why would we think that it doesn't? Maybe I haven't been fo

Re: [CORE] [HACKERS] EOL for 7.4?

2009-12-01 Thread Andrew Dunstan
Scrappy wrote: is there a reason why we can't follow a similar 4+3 life cycle? packagers r produced for the first 4y after .0 release and only source updates for year 5 thru 7? if we could advertise such on the web, there would be no question as to when bug reports are accepted (n+4y) and

Re: [HACKERS] SE-PgSQL patch review

2009-12-01 Thread Josh Berkus
> This is totally separate from the really important question of whether > SE-Linux has a future, and another about whether, if SE-Linux has a > future, PostgreSQL needs to go there. If the hooks are generic enough that the could potentially be adapted to other security frameworks, yes. The need

Re: [HACKERS] enable-thread-safety defaults?

2009-12-01 Thread Magnus Hagander
2009/12/1 Bruce Momjian : > Peter Eisentraut wrote: >> On m?n, 2009-11-30 at 12:21 -0500, Bruce Momjian wrote: >> > ! for thread safety;  use --disable-thread-safety to disable >> > threading.]) >> >> --disable-thread-safety does not disable threading, it disables thread >> safety. > > Good point!

Re: [HACKERS] Block-level CRC checks

2009-12-01 Thread Josh Berkus
All, I feel strongly that we should be verifying pages on write, or at least providing the option to do so, because hardware is simply not reliable. And a lot of our biggest users are having issues; it seems pretty much guarenteed that if you have more than 20 postgres servers, at least one of th

Re: [CORE] [HACKERS] EOL for 7.4?

2009-12-01 Thread Scrappy
is there a reason why we can't follow a similar 4+3 life cycle? packagers r produced for the first 4y after .0 release and only source updates for year 5 thru 7? if we could advertise such on the web, there would be no question as to when bug reports are accepted (n+4y) and when only secu

Re: [HACKERS] Block-level CRC checks

2009-12-01 Thread Tom Lane
Greg Stark writes: > Another thought is that would could use the MSSQL-style torn page > detection of including a counter (or even a bit?) in every 512-byte > chunk which gets incremented every time the page is written. I think we can dismiss that idea, or any idea involving a per-page status val

Re: [HACKERS] Deleted WAL files held open by backends in Linux

2009-12-01 Thread Kevin Grittner
Tom Lane wrote: > Perhaps, if you didn't mind sometimes getting a wrong answer. Well, it would be a heuristic which would close the deleted file *almost* all the time. When it didn't, the next check would probably catch it. Assuming that you would never get an indication that it was differen

Re: [HACKERS] Block-level CRC checks

2009-12-01 Thread Greg Stark
On Tue, Dec 1, 2009 at 6:41 PM, Tom Lane wrote: > Bruce Momjian writes: >> OK, here is another idea, maybe crazy: > >> When we read in a page that has an invalid CRC, we check the page to see >> which hint bits are _not_ set, and we try setting them to see if can get >> a matching CRC. Unfortuna

Re: [HACKERS] Re: [COMMITTERS] pgsql: Rewrite GEQO`s gimme_tree function so that it always finds a

2009-12-01 Thread Tom Lane
Robert Haas writes: > On Tue, Dec 1, 2009 at 11:39 AM, Greg Sabino Mullane > wrote: >> What about 14? Could we at least raise it to 14? 1/2 :) > I doubt we can raise it at all without lying to ourselves about the > likely results of so doing. The GEQO planning times are in the low > double dig

Re: [HACKERS] Block-level CRC checks

2009-12-01 Thread Aidan Van Dyk
* Tom Lane [091201 13:58]: > Actually, the killer problem with *any* scheme involving "guessing" > is that each bit you guess translates directly to removing one bit > of confidence from the CRC value. If you try to guess at as many > as 32 bits, it is practically guaranteed that you will find a

Re: [HACKERS] Deleted WAL files held open by backends in Linux

2009-12-01 Thread Tom Lane
"Kevin Grittner" writes: > Tom Lane wrote: >> Determining whether it's still the current append target is not so >> cheap though; it would require examining shared-memory status >> which means taking a lock on that status (and it's a high-traffic >> lock already). > I haven't reviewed the inter

Re: [HACKERS] Block-level CRC checks

2009-12-01 Thread Tom Lane
Simon Riggs writes: > On Tue, 2009-12-01 at 13:05 -0500, Bruce Momjian wrote: >> When we read in a page that has an invalid CRC, we check the page to see >> which hint bits are _not_ set, and we try setting them to see if can get >> a matching CRC. > Perhaps we could store a sector-based parity b

Re: [HACKERS] Block-level CRC checks

2009-12-01 Thread Tom Lane
Bruce Momjian writes: > OK, here is another idea, maybe crazy: > When we read in a page that has an invalid CRC, we check the page to see > which hint bits are _not_ set, and we try setting them to see if can get > a matching CRC. If there no missing hint bits and the CRC doesn't > match, we kno

Re: [CORE] [HACKERS] EOL for 7.4?

2009-12-01 Thread Tom Lane
"Marc G. Fournier" writes: > What are RedHats "EOL" dates for the various releases? Dave already mentioned a public page for that: http://www.redhat.com/security/updates/errata/ Based on track record so far, Red Hat isn't going to care about anything but high-priority security issues towards the

Re: [HACKERS] Block-level CRC checks

2009-12-01 Thread Joshua D. Drake
On Tue, 2009-12-01 at 13:20 -0500, Robert Haas wrote: > On Tue, Dec 1, 2009 at 1:02 PM, Joshua D. Drake > wrote: > > The hard core reality is this. *IF* it is one of the goals of this > > project to insure that the software can be safely, effectively, and > > responsibly operated in a manner that

Re: [HACKERS] Hot Standby remaining issues

2009-12-01 Thread Heikki Linnakangas
Simon Riggs wrote: > commit 02c3eadb766201db084b668daa271db4a900adc9 > Author: Simon Riggs > Date: Sat Nov 28 06:23:33 2009 + > > Added wal_standby_info GUC to turn RM_STANDBY_ID messages on/off. > Various comments added also. > This patch makes it unsafe to start hot standby mode

Re: [HACKERS] Application name patch - v4

2009-12-01 Thread Tom Lane
Marko Kreen writes: > If the pooler gets new connection with same username:database > as some existing connection, but with different appname, > what it is supposed to do? Whatever it wants to. People seem to be imagining that the appname isn't under the control of the pooler. It's a connection

Re: [HACKERS] Block-level CRC checks

2009-12-01 Thread Simon Riggs
On Tue, 2009-12-01 at 13:05 -0500, Bruce Momjian wrote: > Simon Riggs wrote: > > Also, we might > > > > * Put all hint bits in the block header to allow them to be excluded > > more easily from CRC checking. If we used 3 more bits from > > ItemIdData.lp_len (limiting tuple length to 4096) then we

Re: [HACKERS] Block-level CRC checks

2009-12-01 Thread Robert Haas
On Tue, Dec 1, 2009 at 1:02 PM, Joshua D. Drake wrote: > The hard core reality is this. *IF* it is one of the goals of this > project to insure that the software can be safely, effectively, and > responsibly operated in a manner that is acceptable to C* level people > in a Fortune level company th

Re: [HACKERS] Re: [COMMITTERS] pgsql: Rewrite GEQO`s gimme_tree function so that it always finds a

2009-12-01 Thread Robert Haas
On Tue, Dec 1, 2009 at 11:39 AM, Greg Sabino Mullane wrote: >> Playing around with this a bit, I was easily able to get 2-second >> planing times on 15 table join, 6-second planning times on a 16 table >> join and 30-second planning times on a 17 table join.  This makes it >> hard to support raisi

Re: [HACKERS] Deleted WAL files held open by backends in Linux

2009-12-01 Thread Kevin Grittner
Tom Lane wrote: > Determining whether it's still the current append target is not so > cheap though; it would require examining shared-memory status > which means taking a lock on that status (and it's a high-traffic > lock already). I haven't reviewed the internal locking techniques, so this

  1   2   >