Re: [HACKERS] [PATCH] Add PQconninfoParseParams and PQconninfodefaultsMerge to libpq

2013-02-02 Thread Phil Sorber
On Sun, Feb 3, 2013 at 1:37 AM, Magnus Hagander wrote: > > On Feb 3, 2013 4:16 AM, "Phil Sorber" wrote: >> >> This patch came up from discussion about pg_isready. >> >> PQconninfoParseParams is similar to PQconninfoParse but takes two >> arrays like PQconnectdbParams. It essentially exposes >> co

Re: [HACKERS] Suggestion: Issue warning when calling SET TRANSACTION outside transaction block

2013-02-02 Thread Amit kapila
On Saturday, February 02, 2013 9:08 PM Robert Haas wrote: On Fri, Feb 1, 2013 at 12:04 AM, Amit Kapila wrote: >> I think user should be aware of effect before using SET commands, as these >> are used at various levels (TRANSACTION, SESSION, ...). > Ideally, sure. But these kinds of mistakes ar

Re: [HACKERS] [PATCH] Add PQconninfoParseParams and PQconninfodefaultsMerge to libpq

2013-02-02 Thread Magnus Hagander
On Feb 3, 2013 4:16 AM, "Phil Sorber" wrote: > > This patch came up from discussion about pg_isready. > > PQconninfoParseParams is similar to PQconninfoParse but takes two > arrays like PQconnectdbParams. It essentially exposes > conninfo_array_parse(). > > PQconninfodefaultsMerge essentially expo

Re: [HACKERS] Setting visibility map in VACUUM's second phase

2013-02-02 Thread Pavan Deolasee
On Sun, Feb 3, 2013 at 2:31 AM, Jeff Janes wrote: > Hi Pavan, > > I get this warning: > vacuumlazy.c:890: warning: passing argument 6 of 'lazy_vacuum_page' > makes pointer from integer without a cast > > and make check then fails. > > I've added '&' to that line, and it now passes make check with

Re: [HACKERS] PL/Python result object str handler

2013-02-02 Thread Peter Eisentraut
On Sat, 2013-02-02 at 15:43 -0500, Steve Singer wrote: > I've looked through the code and everything looks fine. > > The patch includes no documentation. Adding a few lines to the > "Utility Functions" section of the plpython documentation so people know > about this feature would be good. Ad

[HACKERS] [PATCH] Add PQconninfoParseParams and PQconninfodefaultsMerge to libpq

2013-02-02 Thread Phil Sorber
This patch came up from discussion about pg_isready. PQconninfoParseParams is similar to PQconninfoParse but takes two arrays like PQconnectdbParams. It essentially exposes conninfo_array_parse(). PQconninfodefaultsMerge essentially exposes conninfo_add_defaults(). It allows you to pass a PQconni

Re: [HACKERS] [PATCH] pg_isready (was: [WIP] pg_ping utility)

2013-02-02 Thread Phil Sorber
On Tue, Jan 29, 2013 at 11:43 AM, Fujii Masao wrote: > On Tue, Jan 29, 2013 at 3:12 AM, Alvaro Herrera > wrote: >> Phil Sorber escribió: >>> On Mon, Jan 28, 2013 at 11:20 AM, Fujii Masao wrote: >>> > Maybe. But I'm not inclined to add new libpq interface at this stage. >>> > Because we are in th

Re: [HACKERS] autovacuum not prioritising for-wraparound tables

2013-02-02 Thread Gavin Flower
On 03/02/13 15:08, Christopher Browne wrote: On Sat, Feb 2, 2013 at 2:54 PM, Robert Haas wrote: On Sat, Feb 2, 2013 at 1:49 PM, Andres Freund wrote: You're right, this doesn't work superbly well, especially for insert-only tables... But imo the place to fix it is not the priorization logic bu

Re: [HACKERS] autovacuum not prioritising for-wraparound tables

2013-02-02 Thread Christopher Browne
On Sat, Feb 2, 2013 at 2:54 PM, Robert Haas wrote: > On Sat, Feb 2, 2013 at 1:49 PM, Andres Freund wrote: >> You're right, this doesn't work superbly well, especially for >> insert-only tables... But imo the place to fix it is not the >> priorization logic but relation_needs_vacanalyze, since fix

Re: [HACKERS] autovacuum not prioritising for-wraparound tables

2013-02-02 Thread Andres Freund
On 2013-02-03 02:40:04 +0100, Andres Freund wrote: > On 2013-02-01 15:09:34 -0800, Jeff Janes wrote: > > As an aside, it does seem like log_autovacuum_min_duration=0 should > > log whether a scan_all was done, and if so what relfrozenxid got set > > to. But looking at where the log message is gene

Re: [HACKERS] autovacuum not prioritising for-wraparound tables

2013-02-02 Thread Andres Freund
On 2013-02-01 15:09:34 -0800, Jeff Janes wrote: > As an aside, it does seem like log_autovacuum_min_duration=0 should > log whether a scan_all was done, and if so what relfrozenxid got set > to. But looking at where the log message is generated, I don't know > where to retrieve that info. What ab

Re: PATCH: Split stats file per database WAS: [HACKERS] autovacuum stress-testing our system

2013-02-02 Thread Tomáš Vondra
Ok, thanks for the info. I'll look into that and I'll also post info from some of our production systems (we've deployed a 9.1-backpatched version about 2 weeks ago). T. Původní zpráva Od: Jeff Janes Datum: Komu: Tomas Vondra Kopie: pgsql-hackers@postgresql.org Předmět:

Re: PATCH: Split stats file per database WAS: [HACKERS] autovacuum stress-testing our system

2013-02-02 Thread Jeff Janes
On Sat, Jan 5, 2013 at 8:03 PM, Tomas Vondra wrote: > On 3.1.2013 20:33, Magnus Hagander wrote: >> >> Yeah, +1 for a separate directory not in global. > > OK, I moved the files from "global/stat" to "stat". This has a warning: pgstat.c:5132: warning: 'pgstat_write_statsfile_needed' was used with

Re: [HACKERS] proposal - assign result of query to psql variable

2013-02-02 Thread Tom Lane
Pavel Stehule writes: > 2013/2/2 Tom Lane : >> Shigeru Hanada writes: >>> +1 for a). If users want to determine whether the result was NULL, or >>> want to use substitute string for NULL result, they can use coalesce >>> in SELECT clause. Anyway the feature should be documented clearly. >> Yea

Re: [HACKERS] logical changeset generation v4 - Heikki's thoughts about the patch state

2013-02-02 Thread Andres Freund
On 2013-01-28 16:55:52 -0500, Steve Singer wrote: > If your using non-surragate /natural primary keys this tends to come up > occasionally due to data-entry errors or renames. I'm looking at this from > the point of view of what do I need to use this as a source for a production > replication sys

Re: [HACKERS] autovacuum not prioritising for-wraparound tables

2013-02-02 Thread Andres Freund
On 2013-02-02 14:54:10 -0500, Robert Haas wrote: > On Sat, Feb 2, 2013 at 1:49 PM, Andres Freund wrote: > > I think scheduling a table for a partial vacuum every min_freeze * 2 > > xids, even if its insert only, would go a long way of reducing the > > impact of full-table vacuums. Obviously that w

Re: [HACKERS] Setting visibility map in VACUUM's second phase

2013-02-02 Thread Jeff Janes
On Sat, Jan 26, 2013 at 11:25 PM, Pavan Deolasee wrote: > On Thu, Jan 24, 2013 at 9:31 PM, Jeff Janes wrote: >> On Thu, Jan 24, 2013 at 1:28 AM, Pavan Deolasee >> wrote: > >>> >>> Good idea. Even though the cost of pinning/unpinning may not be high >>> with respect to the vacuum cost itself, but

Re: [HACKERS] PL/Python result object str handler

2013-02-02 Thread Steve Singer
On 13-01-07 09:58 PM, Peter Eisentraut wrote: By implementing a "str" handler for the result object, it now prints something like Patch attached for review. Here is a review: This patch adds a function that pl/python functions can call to convert a query result hash into a string suitable

Re: [HACKERS] proposal - assign result of query to psql variable

2013-02-02 Thread Pavel Stehule
Hello 2013/2/2 Tom Lane : > Shigeru Hanada writes: >> On Sat, Feb 2, 2013 at 7:30 PM, Pavel Stehule >> wrote: >>> possible variants >>> >>> a) don't store NULL values - and remove existing variable when NULL >>> be assigned - it is probably best, but should be confusing for users >>> b) implem

Re: [HACKERS] GetOldestXmin going backwards is dangerous after all

2013-02-02 Thread Simon Riggs
On 2 February 2013 18:38, Andres Freund wrote: > On 2013-02-02 18:32:44 +, Simon Riggs wrote: >> On 2 February 2013 14:24, Andres Freund wrote: >> >> > b) We don't assign the xmin early enough, we only set it when the first >> > feedback message arrives, but we should set it when walsender st

Re: [HACKERS] autovacuum not prioritising for-wraparound tables

2013-02-02 Thread Robert Haas
On Sat, Feb 2, 2013 at 1:49 PM, Andres Freund wrote: > You're right, this doesn't work superbly well, especially for > insert-only tables... But imo the place to fix it is not the > priorization logic but relation_needs_vacanalyze, since fixing it in > priorization won't prevent the BAM just the t

Re: [HACKERS] COPY FREEZE has no warning

2013-02-02 Thread Noah Misch
On Sat, Feb 02, 2013 at 10:12:54AM -0500, Bruce Momjian wrote: > On Sat, Feb 2, 2013 at 09:51:13AM -0500, Noah Misch wrote: > > Let's touch on the exception in passing by using the phrase "last > > truncated", > > giving this wording for both the second and the third COPY FREEZE error > > sites:

Re: [HACKERS] autovacuum not prioritising for-wraparound tables

2013-02-02 Thread Andres Freund
On 2013-02-02 11:25:01 -0500, Robert Haas wrote: > On Sat, Feb 2, 2013 at 8:41 AM, Andres Freund wrote: > >> - It's probably important to have a formula where we can be sure that > >> the wrap-around term will eventually dominate the dead-tuple term, > >> with enough time to spare to make sure not

Re: [HACKERS] Cascading replication: should we detect/prevent cycles?

2013-02-02 Thread Robert Haas
On Thu, Jan 31, 2013 at 9:48 PM, Josh Berkus wrote: > On 02/01/2013 12:01 PM, Josh Berkus wrote: >>> If we're going to start installing safeguards against doing stupid >>> things, there's a long list of scenarios that happen far more >>> regularly than this ever will and cause far more damage. >>

Re: [HACKERS] GetOldestXmin going backwards is dangerous after all

2013-02-02 Thread Andres Freund
On 2013-02-02 18:32:44 +, Simon Riggs wrote: > On 2 February 2013 14:24, Andres Freund wrote: > > > b) We don't assign the xmin early enough, we only set it when the first > > feedback message arrives, but we should set it when walsender starts > > streaming. > > That's easy to fix. Not tri

Re: [HACKERS] GetOldestXmin going backwards is dangerous after all

2013-02-02 Thread Simon Riggs
On 2 February 2013 00:24, Tom Lane wrote: > * if allDbs is FALSE and there are no transactions running in the current > * database, GetOldestXmin() returns latestCompletedXid. If a transaction > * begins after that, its xmin will include in-progress transactions in other > * databases that st

Re: [HACKERS] GetOldestXmin going backwards is dangerous after all

2013-02-02 Thread Simon Riggs
On 2 February 2013 00:24, Tom Lane wrote: > * The return value is also adjusted with vacuum_defer_cleanup_age, so > * increasing that setting on the fly is another easy way to make > * GetOldestXmin() move backwards, with no consequences for data integrity. > > And as for that, it's been prett

Re: [HACKERS] GetOldestXmin going backwards is dangerous after all

2013-02-02 Thread Simon Riggs
On 2 February 2013 14:24, Andres Freund wrote: > b) We don't assign the xmin early enough, we only set it when the first > feedback message arrives, but we should set it when walsender starts > streaming. That's easy to fix. > c) After a disconnect the feedback message will rather likely ask fo

Re: [HACKERS] COPY FREEZE has no warning

2013-02-02 Thread Bruce Momjian
On Sat, Feb 2, 2013 at 12:09:05PM -0500, Tom Lane wrote: > Bruce Momjian writes: > > Well, so you are saying that there really isn't any use-visible logic > > for those messages to be different, > > No, and in fact the whole block of code is badly written because it > conflates two unrelated tes

Re: [HACKERS] proposal - assign result of query to psql variable

2013-02-02 Thread Tom Lane
Shigeru Hanada writes: > On Sat, Feb 2, 2013 at 7:30 PM, Pavel Stehule wrote: >> possible variants >> >> a) don't store NULL values - and remove existing variable when NULL >> be assigned - it is probably best, but should be confusing for users >> b) implement flag IS NULL - for variables >> c)

Re: [HACKERS] COPY FREEZE has no warning

2013-02-02 Thread Tom Lane
Bruce Momjian writes: > Well, so you are saying that there really isn't any use-visible logic > for those messages to be different, No, and in fact the whole block of code is badly written because it conflates two unrelated tests. I guess somebody was trying to save a couple of nanoseconds by no

Re: [HACKERS] autovacuum not prioritising for-wraparound tables

2013-02-02 Thread Robert Haas
On Sat, Feb 2, 2013 at 8:41 AM, Andres Freund wrote: >> - It's probably important to have a formula where we can be sure that >> the wrap-around term will eventually dominate the dead-tuple term, >> with enough time to spare to make sure nothing really bad happens; on >> the other hand, it's also

Re: [HACKERS] autovacuum not prioritising for-wraparound tables

2013-02-02 Thread Robert Haas
On Fri, Feb 1, 2013 at 6:09 PM, Jeff Janes wrote: > As an aside, it does seem like log_autovacuum_min_duration=0 should > log whether a scan_all was done, and if so what relfrozenxid got set > to. That would be nifty. > [1] I don't know why it is that a scan_all vacuum with a > freeze_min_age of

Re: [HACKERS] Suggestion: Issue warning when calling SET TRANSACTION outside transaction block

2013-02-02 Thread Robert Haas
On Fri, Feb 1, 2013 at 12:04 AM, Amit Kapila wrote: > I think user should be aware of effect before using SET commands, as these > are used at various levels (TRANSACTION, SESSION, ...). Ideally, sure. But these kinds of mistakes are easy to make. That's why LOCK and DECLARE CURSOR already emi

Re: [HACKERS] GetOldestXmin going backwards is dangerous after all

2013-02-02 Thread Robert Haas
On Fri, Feb 1, 2013 at 6:56 PM, Tom Lane wrote: >> That is admittedly kind of weird behavior, but I think one could >> equally blame this on CLUSTER. This is hardly the first time we've >> had to patch CLUSTER's handling of TOAST tables (cf commits >> 21b446dd0927f8f2a187d9461a0d3f11db836f77, >>

Re: [HACKERS] GetOldestXmin going backwards is dangerous after all

2013-02-02 Thread Robert Haas
On Fri, Feb 1, 2013 at 7:24 PM, Tom Lane wrote: > Robert Haas writes: >> Having said that, I agree that a fix in GetOldestXmin() would be nice >> if we could find one, but since the comment describes at least three >> different ways the value can move backwards, I'm not sure that there's >> reall

Re: [HACKERS] Turning auto-analyze off (was Re: [GENERAL] Unusually high IO for autovacuum worker)

2013-02-02 Thread Vlad Bailescu
On Fri, Feb 1, 2013 at 5:54 PM, Pavan Deolasee wrote: > There is another problem that I noticed while looking at this case. > The analyze took close to 500sec on a fairly good hardware (40GB RAM, > 10K rpm disks on RAID10) because many large child tables were scanned > at once. > Just a small cor

Re: [HACKERS] COPY FREEZE has no warning

2013-02-02 Thread Bruce Momjian
On Sat, Feb 2, 2013 at 09:51:13AM -0500, Noah Misch wrote: > > OK, so, should we change the error message: > > > > cannot perform FREEZE because of transaction activity after table > > creation or truncation > > > > to > > > > cannot perform FREEZE because the table was not created

Re: [HACKERS] COPY FREEZE has no warning

2013-02-02 Thread Noah Misch
On Fri, Feb 01, 2013 at 12:57:18PM -0500, Bruce Momjian wrote: > On Tue, Jan 29, 2013 at 08:34:24PM -0500, Noah Misch wrote: > > On Fri, Jan 25, 2013 at 11:28:58PM -0500, Bruce Momjian wrote: > > > BEGIN; > > > TRUNCATE vistest; > > > SAVEPOINT s1; > > > COPY vistest FROM stdin CSV FREEZE;

Re: [HACKERS] proposal - assign result of query to psql variable

2013-02-02 Thread Pavel Stehule
2013/2/2 Shigeru Hanada : > On Sat, Feb 2, 2013 at 7:30 PM, Pavel Stehule wrote: >> possible variants >> >> a) don't store NULL values - and remove existing variable when NULL >> be assigned - it is probably best, but should be confusing for users >> b) implement flag IS NULL - for variables >> c

Re: [HACKERS] GetOldestXmin going backwards is dangerous after all

2013-02-02 Thread Andres Freund
On 2013-02-01 19:24:02 -0500, Tom Lane wrote: > * There are also replication-related effects: a walsender > * process can set its xmin based on transactions that are no longer running > * in the master but are still being replayed on the standby, thus possibly > * making the GetOldestXmin readi

Re: [HACKERS] autovacuum not prioritising for-wraparound tables

2013-02-02 Thread Andres Freund
On 2013-02-01 16:59:52 -0500, Robert Haas wrote: > I don't think I really understand the origin of the formula, so > perhaps if someone would try to characterize why it seems to behave > reasonably that would be helpful (at least to me). > > > f(deadtuples, relpages, age) = > >deadtuples/relpa

Re: [HACKERS] autovacuum not prioritising for-wraparound tables

2013-02-02 Thread Andres Freund
On 2013-02-01 15:09:34 -0800, Jeff Janes wrote: > On Fri, Feb 1, 2013 at 2:34 PM, Andres Freund wrote: > > On 2013-02-01 14:05:46 -0800, Jeff Janes wrote: > > >> As far as I can tell this bug kicks in when your cluster gets to be > >> older than freeze_min_age, and then lasts forever after. Afte

Re: [HACKERS] proposal - assign result of query to psql variable

2013-02-02 Thread Shigeru Hanada
On Sat, Feb 2, 2013 at 7:30 PM, Pavel Stehule wrote: > possible variants > > a) don't store NULL values - and remove existing variable when NULL > be assigned - it is probably best, but should be confusing for users > b) implement flag IS NULL - for variables > c) use nullPrint > d) use empty Str

Re: [HACKERS] GetOldestXmin going backwards is dangerous after all

2013-02-02 Thread Simon Riggs
On 1 February 2013 23:56, Tom Lane wrote: > Robert Haas writes: >> On Fri, Feb 1, 2013 at 2:35 PM, Tom Lane wrote: >>> In any case, I no longer have much faith in the idea that letting >>> GetOldestXmin go backwards is really safe. > >> That is admittedly kind of weird behavior, but I think one

Re: [HACKERS] GetOldestXmin going backwards is dangerous after all

2013-02-02 Thread Simon Riggs
On 1 February 2013 23:56, Tom Lane wrote: > Well, if we were tracking the latest value in shared memory, we could > certainly clamp to that to ensure it didn't go backwards. The problem > is where to find storage for a per-DB value. Adding new data columns to catalogs in backbranches seems like

Re: [HACKERS] proposal - assign result of query to psql variable

2013-02-02 Thread Pavel Stehule
Hello 2013/2/1 Tom Lane : > Pavel Stehule writes: >> here is patch related to your proposal > > I looked at this a bit. It's getting there, though I still don't trust > the places where you've chosen to clear the prefix setting. (Looking at > it, I'm now not sure that I trust the implementation

[HACKERS] Using indexes for partial index builds

2013-02-02 Thread Paul Norman
Hello, After a discussion on IRC in #postgresql, I had a feature suggestion and it was suggested I write it up here. I have a large (200GB, 1.7b rows) table with a number of columns, but the two of interest here are a hstore column, tags and a postgis geometry column, geom. There is a GIN index

[HACKERS] proposal 9.4 plpgsql: allows access to call stack from GET DIAGNOSTICS statement

2013-02-02 Thread Pavel Stehule
Hello I propose enhancing GET DIAGNOSTICS statement about new field PG_CONTEXT. It is similar to GET STACKED DIAGNOSTICS' PG_EXCEPTION_CONTEXT. Motivation for this proposal is possibility to get call stack for debugging without raising exception. This code is based on cleaned code from Orafce,