Re: [HACKERS] security labels on databases are bad for dump & restore

2015-07-31 Thread Noah Misch
On Thu, Jul 30, 2015 at 10:37:33PM -0400, Adam Brightwell wrote: > On Thu, Jul 30, 2015 at 02:26:34PM -0400, Robert Haas wrote: > >> 1. "pg_dumpall -g" > >> 2. "pg_dump --create" per database > > > > Gah, OK, I see your point. But we better document this, because if > > you need a PhD in PostgreSQ

Re: [HACKERS] [DESIGN] ParallelAppend

2015-07-31 Thread Amit Kapila
On Tue, Jul 28, 2015 at 6:08 PM, Kouhei Kaigai wrote: > > > On Tue, Jul 28, 2015 at 7:59 AM, Kouhei Kaigai wrote: > > > > > > > -Original Message- > > > > From: pgsql-hackers-ow...@postgresql.org > > > > [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Kouhei Kaigai > > > > Sent:

Re: [HACKERS] Solaris testers wanted for strxfrm() behavior

2015-07-31 Thread Josh Berkus
Noah, All: Where are we with this? Do we feel confident that this bug is only on old versions of Solaris we don't care about? Or does it remain to be resolved? -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)

Re: [HACKERS] [sqlsmith] Failed assertion in joinrels.c

2015-07-31 Thread Tom Lane
Andreas Seltenreich writes: > sqlsmith triggered the following assertion in master (c188204). > TRAP: FailedAssertion("!(!bms_overlap(joinrelids, sjinfo->min_lefthand))", > File: "joinrels.c", Line: 500) Cool, I'll take a look. > As usual, the query is against the regression database. It is r

[HACKERS] [sqlsmith] Failed assertion in joinrels.c

2015-07-31 Thread Andreas Seltenreich
Hi, sqlsmith triggered the following assertion in master (c188204). TRAP: FailedAssertion("!(!bms_overlap(joinrelids, sjinfo->min_lefthand))", File: "joinrels.c", Line: 500) As usual, the query is against the regression database. It is rather unwieldy… I wonder if I should stop working on new

Re: [HACKERS] 64-bit XIDs again

2015-07-31 Thread Tom Lane
Josh Berkus writes: > On 07/31/2015 02:46 PM, Heikki Linnakangas wrote: >> Well, sure, if you don't want the clog to grow arbitrarily large, then >> you need to freeze. And most people would want to freeze regularly, to >> keep the clog size in check. The point is that you wouldn't *have* to do >>

Re: [HACKERS] 64-bit XIDs again

2015-07-31 Thread Josh Berkus
On 07/31/2015 02:46 PM, Heikki Linnakangas wrote: > On 07/31/2015 12:29 AM, Josh Berkus wrote: >> On 07/30/2015 07:24 AM, Heikki Linnakangas wrote: >>> >>> I think we should move to 64-bit XIDs in in-memory structs snapshots, >>> proc array etc. And expand clog to handle 64-bit XIDs. But keep the >

Re: [HACKERS] patch: prevent user from setting wal_buffers over 2GB bytes

2015-07-31 Thread Josh Berkus
On 07/31/2015 10:43 AM, Robert Haas wrote: > On Thu, Jul 30, 2015 at 9:17 PM, Josh Berkus wrote: >> In guc.c, the maximum for wal_buffers is INT_MAX. However, wal_buffers >> is actually measured in 8KB buffers, not in bytes. This means that >> users are able to set wal_buffers > 2GB. When the d

Re: [HACKERS] Use of PRId64 with PostgreSQL functions

2015-07-31 Thread Tom Lane
"Paragon Corporation" writes: >> Huh. Apparently, whichever Windows compiler you're using defines >> PRId64 as "d", which surely seems pretty broken. > I think it was only failing on the postgresql function calls. > And this: gcc -Wall -E testint.c | grep '\(printf.*x)\| int64_t;\)' > Returne

Re: [HACKERS] Use of PRId64 with PostgreSQL functions

2015-07-31 Thread Paragon Corporation
Alvaro and Tom, A big thank you. That seems to have done the trick. My compiler warnings went away and no more syntax errors in the regress tests. I still have some regress failures I got to hunt down, but those could be caused by places in the code I did not replace PRId64 calls or something e

Re: [HACKERS] 64-bit XIDs again

2015-07-31 Thread Heikki Linnakangas
On 07/31/2015 12:29 AM, Josh Berkus wrote: On 07/30/2015 07:24 AM, Heikki Linnakangas wrote: I think we should move to 64-bit XIDs in in-memory structs snapshots, proc array etc. And expand clog to handle 64-bit XIDs. But keep the xmin/xmax fields on heap pages at 32-bits, and add an epoch-like

Re: [HACKERS] Use of PRId64 with PostgreSQL functions

2015-07-31 Thread Tom Lane
"Paragon Corporation" writes: > I've been currently suffering an issue in mingw64 with changes made in > PostGIS topology code base. Sadly it only happens to me (compiling in > mingw64) > Issue is when code like this was introduced > appendStringInfo(str, "%s%" PRId64, sep, edge->start_node);

Re: [HACKERS] Use of PRId64 with PostgreSQL functions

2015-07-31 Thread Alvaro Herrera
Paragon Corporation wrote: > I've been currently suffering an issue in mingw64 with changes made in > PostGIS topology code base. Sadly it only happens to me (compiling in > mingw64) > > Issue is when code like this was introduced > > > appendStringInfo(str, "%s%" PRId64, sep, edge->start_nod

Re: [HACKERS] brin index vacuum versus transaction snapshots

2015-07-31 Thread Alvaro Herrera
Kevin Grittner wrote: > Alvaro Herrera wrote: > > >> I think the real solution to this problem is to avoid use of > >> GetTransactionSnapshot(), and instead use GetLatestSnapshot(). As far > >> as I can see, that should completely close the hole. This requires > >> patching IndexBuildHeapRangeS

Re: [HACKERS] brin index vacuum versus transaction snapshots

2015-07-31 Thread Kevin Grittner
Alvaro Herrera wrote: >> I think the real solution to this problem is to avoid use of >> GetTransactionSnapshot(), and instead use GetLatestSnapshot(). As far >> as I can see, that should completely close the hole. This requires >> patching IndexBuildHeapRangeScan() to allow for that. > > Actua

Re: [HACKERS] Doubt about AccessExclusiveLock in ALTER TABLE .. SET ( .. );

2015-07-31 Thread Alvaro Herrera
Fabrízio de Royes Mello wrote: > In this patch I didn't change all lockmode comparison places previous > pointed by you, but I can change it maybe adding other method called > LockModeIsValid(lockmode) to do the comparison "lockmode >= NoLock && > lockmode < MAX_LOCKMODES" used in many places. I

Re: [HACKERS] brin index vacuum versus transaction snapshots

2015-07-31 Thread Alvaro Herrera
> I think the real solution to this problem is to avoid use of > GetTransactionSnapshot(), and instead use GetLatestSnapshot(). As far > as I can see, that should completely close the hole. This requires > patching IndexBuildHeapRangeScan() to allow for that. Actually I think there's another pro

[HACKERS] Use of PRId64 with PostgreSQL functions

2015-07-31 Thread Paragon Corporation
I've been currently suffering an issue in mingw64 with changes made in PostGIS topology code base. Sadly it only happens to me (compiling in mingw64) Issue is when code like this was introduced appendStringInfo(str, "%s%" PRId64, sep, edge->start_node); Which uses a PostgreSQL function appe

Re: [HACKERS] [PROPOSAL] VACUUM Progress Checker.

2015-07-31 Thread Alvaro Herrera
Robert Haas wrote: > On Fri, Jul 31, 2015 at 2:21 PM, Alvaro Herrera > wrote: > > I think the only way to produce usable estimates is to report more than > > one number. And in the particular case of lazy vacuuming, ISTM the way > > to do it is to consider heap scanning as one phase, index cleanu

Re: [HACKERS] Doubt about AccessExclusiveLock in ALTER TABLE .. SET ( .. );

2015-07-31 Thread Fabrízio de Royes Mello
On Fri, Jul 31, 2015 at 10:01 AM, Michael Paquier wrote: > > On Fri, Jul 31, 2015 at 11:41 AM, Fabrízio de Royes Mello wrote: > >> We usually don't compare lock values that way, i.e. there's not > >> guaranteed to be a strict monotonicity between lock levels. I don't > >> really agree with that po

Re: [HACKERS] 64-bit XIDs again

2015-07-31 Thread Robert Haas
On Thu, Jul 30, 2015 at 5:23 PM, Arthur Silva wrote: > In the other hand PG tuple overhead is already the largest among the > alternatives. > Even if storage keeps getting faster and cheaper stuff you can't ignore the > overhead of adding yet another 8bytes to each tuple. +1, very much. -- Robe

Re: [HACKERS] Planner debug views

2015-07-31 Thread Qingqing Zhou
On Thu, Jul 30, 2015 at 2:42 PM, Jim Nasby wrote: > > I think a better option would be shoving it into a backend tuplestore and > just leaving it there (maybe with a command to clear it for the paranoid). > That gives a relation you can query against, insert into another table, etc. > -- This is

Re: [HACKERS] [PROPOSAL] VACUUM Progress Checker.

2015-07-31 Thread Robert Haas
On Fri, Jul 31, 2015 at 2:21 PM, Alvaro Herrera wrote: > I think the only way to produce usable estimates is to report more than > one number. And in the particular case of lazy vacuuming, ISTM the way > to do it is to consider heap scanning as one phase, index cleanup as > another phase; these t

Re: [HACKERS] Transactions involving multiple postgres foreign servers

2015-07-31 Thread Robert Haas
On Fri, Jul 31, 2015 at 6:33 AM, Ashutosh Bapat wrote:> >> I'm not hung up on the table-level attribute, but I think having a >> server-level attribute rather than a global GUC is a good idea. >> However, I welcome other thoughts on that. > > The patch supports server level attribute. Let me repea

Re: [HACKERS] [PROPOSAL] VACUUM Progress Checker.

2015-07-31 Thread Joshua D. Drake
On 07/31/2015 11:21 AM, Alvaro Herrera wrote: This looks pretty complicated to understand from the user POV, but anything other than this seems to me too simplistic to be of any use. I would agree and I don't think it is all that complicated. This is an RDBMS not a web browser downloading a

Re: [HACKERS] [PROPOSAL] VACUUM Progress Checker.

2015-07-31 Thread Alvaro Herrera
I think the only way to produce usable estimates is to report more than one number. And in the particular case of lazy vacuuming, ISTM the way to do it is to consider heap scanning as one phase, index cleanup as another phase; these two phases can be interleaved. And there's a final heap scan whi

Re: [HACKERS] RequestAddinLWLocks(int n)

2015-07-31 Thread Jesper Pedersen
On 07/31/2015 01:35 PM, Robert Haas wrote: On Fri, Jul 31, 2015 at 11:56 AM, Jesper Pedersen wrote: Currently Max(lock_addin_request, NUM_USER_DEFINED_LWLOCKS); LWLock's are added during startup for extensions. However, this presents a problem if an extension doesn't specify the correct n

Re: [HACKERS] brin index vacuum versus transaction snapshots

2015-07-31 Thread Alvaro Herrera
Let me explain a bit more what's the mechanism at play. When scanning a block range to summarize it, we use an MVCC snapshot. Any tuples that are inserted by transactions that appear as in-progress to that snapshot will not be seen by that range scan; therefore we need another mechanism to include

Re: [HACKERS] patch: prevent user from setting wal_buffers over 2GB bytes

2015-07-31 Thread Robert Haas
On Thu, Jul 30, 2015 at 9:17 PM, Josh Berkus wrote: > In guc.c, the maximum for wal_buffers is INT_MAX. However, wal_buffers > is actually measured in 8KB buffers, not in bytes. This means that > users are able to set wal_buffers > 2GB. When the database is started, > this can cause a core dump

Re: [HACKERS] Autonomous Transaction is back

2015-07-31 Thread Robert Haas
On Tue, Jul 28, 2015 at 6:01 AM, Craig Ringer wrote: > That should be practical to special-case by maintaining a list of > parent transaction (virtual?) transaction IDs. Attempts to wait on a > lock held by any of those should fail immediately. There's no point > waiting for the deadlock detector

Re: [HACKERS] RequestAddinLWLocks(int n)

2015-07-31 Thread Robert Haas
On Fri, Jul 31, 2015 at 11:56 AM, Jesper Pedersen wrote: > Currently > > Max(lock_addin_request, NUM_USER_DEFINED_LWLOCKS); > > LWLock's are added during startup for extensions. > > However, this presents a problem if an extension doesn't specify the correct > number of LWLock's needed, if the t

Re: [HACKERS] Re: Using quicksort and a merge step to significantly improve on tuplesort's single run "external sort"

2015-07-31 Thread Robert Haas
On Fri, Jul 31, 2015 at 7:21 AM, Jeremy Harris wrote: > Heapification is O(n) already, whether siftup (existing) or down. That's not my impression, or what Wikipedia says. Source? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-ha

Re: [HACKERS] LWLock deadlock and gdb advice

2015-07-31 Thread Jeff Janes
On Thu, Jul 30, 2015 at 8:22 PM, Andres Freund wrote: > On 2015-07-30 09:03:01 -0700, Jeff Janes wrote: > > On Wed, Jul 29, 2015 at 6:10 AM, Andres Freund > wrote: > > > What do you think about something roughly like the attached? > > > > > > > I've not evaluated the code, but applying it does s

[HACKERS] RequestAddinLWLocks(int n)

2015-07-31 Thread Jesper Pedersen
Hi, Currently Max(lock_addin_request, NUM_USER_DEFINED_LWLOCKS); LWLock's are added during startup for extensions. However, this presents a problem if an extension doesn't specify the correct number of LWLock's needed, if the total number is <= 4. The attached patch requires extensions to

Re: [HACKERS] brin index vacuum versus transaction snapshots

2015-07-31 Thread Simon Riggs
On 31 July 2015 at 15:48, Kevin Grittner wrote: > > Reread the thread. I stand corrected. > The error is being hit on a VACUUM command > (which is not, of course in any explicit transaction). > Seems like VACUUM only ever needs a read-committed snapshot. Whether it takes a read-committed or

Re: [HACKERS] brin index vacuum versus transaction snapshots

2015-07-31 Thread Kevin Grittner
Simon Riggs wrote: > On 30 July 2015 at 22:24, Tom Lane wrote: >> Alvaro Herrera writes: >>> Kevin Grittner wrote: If you run `make installcheck` against a cluster with default_transaction_isolation = 'repeatable read' you get one failure: >>> >>> I am tempted to say that we shou

Re: [HACKERS] Encoding of early PG messages

2015-07-31 Thread Shay Rojansky
Thanks for the suggestions Tom. As I'm developing a general-purpose driver I can't do anything in PostgreSQL config, but it's a good workaround suggestion for users who encounter this error. Sending lc_messages in the startup packet could work, but if I understand correctly that setting combines

Re: [HACKERS] brin index vacuum versus transaction snapshots

2015-07-31 Thread Kevin Grittner
Alvaro Herrera wrote: > Kevin Grittner wrote: >> If you run `make installcheck` against a cluster with >> default_transaction_isolation = 'repeatable read' you get one >> failure: > > I am tempted to say that we should just disallow to run vacuum on > a table containing a brin index in a transact

Re: [HACKERS] brin index vacuum versus transaction snapshots

2015-07-31 Thread Simon Riggs
On 30 July 2015 at 22:24, Tom Lane wrote: > Alvaro Herrera writes: > > Kevin Grittner wrote: > >> If you run `make installcheck` against a cluster with > >> default_transaction_isolation = 'repeatable read' you get one > >> failure: > > > I am tempted to say that we should just disallow to run v

Re: [HACKERS] 64-bit XIDs again

2015-07-31 Thread Simon Riggs
On 31 July 2015 at 11:00, Alexander Korotkov wrote: > > If use upgrade database cluster with pg_upgrade, he would stop old > postmaster, pg_upgrade, start new postmaster. That means we start from the > point when there is no running transactions. Thus, between tuples of old > format there are two

Re: [HACKERS] Encoding of early PG messages

2015-07-31 Thread Tom Lane
Shay Rojansky writes: > Developing Npgsql I've encountered the problem described in > http://www.postgresql.org/message-id/20081223212414.gd3...@merkur.hilbert.loc: > a German installation of PostgreSQL seems to respond to an incorrect > password with a non-UTF8 encoding of the error messages, eve

Re: [HACKERS] deparsing utility commands

2015-07-31 Thread Shulgin, Oleksandr
On Wed, Jul 29, 2015 at 12:44 PM, Shulgin, Oleksandr < oleksandr.shul...@zalando.de> wrote: > On Tue, May 12, 2015 at 12:25 AM, Alvaro Herrera > wrote: > >> David Steele wrote: >> >> > I have reviewed and tested this patch and everything looks good to me. >> > It also looks like you added better

Re: [HACKERS] Proposal: backend "niceness" / session_priority

2015-07-31 Thread Amit Kapila
On Fri, Jul 31, 2015 at 3:48 AM, Jim Nasby wrote: > On 7/30/15 10:54 AM, Tom Lane wrote: > >> =?ISO-8859-15?Q?Jos=E9_Luis_Tall=F3n?= >> writes: >> >>> Since PostgreSQL lacks the resource management capabilities of the >>> "Big Ones" ( Resource Groups - Red, WorkLoad Manager - Blue ) or the

Re: [HACKERS] Doubt about AccessExclusiveLock in ALTER TABLE .. SET ( .. );

2015-07-31 Thread Michael Paquier
On Fri, Jul 31, 2015 at 10:05 PM, Simon Riggs wrote: > On 31 July 2015 at 02:46, Michael Paquier wrote: > >> >> > Added. I really don't know if my isolation tests are completely correct, >> > is >> > my first time writing this kind of tests. >> >> This patch size has increased from 16k to 157k be

Re: [HACKERS] creating extension including dependencies

2015-07-31 Thread Petr Jelinek
On 2015-07-31 03:03, Michael Paquier wrote: On Thu, Jul 30, 2015 at 10:58 PM, Petr Jelinek wrote: On 2015-07-27 15:18, Michael Paquier wrote: Something also has not been discussed yet: what to do with new_version and old_version (the options of CreateExtensionStmt)? As of now if those options a

Re: [HACKERS] Doubt about AccessExclusiveLock in ALTER TABLE .. SET ( .. );

2015-07-31 Thread Simon Riggs
On 31 July 2015 at 02:46, Michael Paquier wrote: > > Added. I really don't know if my isolation tests are completely correct, > is > > my first time writing this kind of tests. > > This patch size has increased from 16k to 157k because of the output > of the isolation tests you just added. Tha

Re: [HACKERS] Doubt about AccessExclusiveLock in ALTER TABLE .. SET ( .. );

2015-07-31 Thread Michael Paquier
On Fri, Jul 31, 2015 at 12:21 PM, Fabrízio de Royes Mello wrote: > > On Thu, Jul 30, 2015 at 10:46 PM, Michael Paquier wrote: > > >> On further notice, I would recommend not to use the same string name >> for the session and the query identifiers. And I think that inserting >> only one tuple at in

[HACKERS] Encoding of early PG messages

2015-07-31 Thread Shay Rojansky
Hi hackers. Developing Npgsql I've encountered the problem described in http://www.postgresql.org/message-id/20081223212414.gd3...@merkur.hilbert.loc: a German installation of PostgreSQL seems to respond to an incorrect password with a non-UTF8 encoding of the error messages, even if the startup m

Re: [HACKERS] 64-bit XIDs again

2015-07-31 Thread Alexander Korotkov
On Fri, Jul 31, 2015 at 12:23 AM, Tom Lane wrote: > But the elephant in the room is on-disk compatibility. There is > absolutely no way that we can just change xmin/xmax to 64 bits without a > disk format break. That seems problematic. But I'm not yet convinced that there is absolutely no way

Re: [HACKERS] Doubt about AccessExclusiveLock in ALTER TABLE .. SET ( .. );

2015-07-31 Thread Michael Paquier
On Fri, Jul 31, 2015 at 11:41 AM, Fabrízio de Royes Mello wrote: >> We usually don't compare lock values that way, i.e. there's not >> guaranteed to be a strict monotonicity between lock levels. I don't >> really agree with that policy, but it's nonetheless there. > > And how is the better way to c

[HACKERS] Re: Using quicksort and a merge step to significantly improve on tuplesort's single run "external sort"

2015-07-31 Thread Jeremy Harris
On 30/07/15 02:05, Peter Geoghegan wrote: > Since heapification is now a big fraction of the total cost of a sort > sometimes, even where the heap invariant need not be maintained for > any length of time afterwards, it might be worth revisiting the patch > to make that an O(n) rather than a O(log

Re: [HACKERS] LWLock deadlock and gdb advice

2015-07-31 Thread Heikki Linnakangas
On 07/30/2015 09:14 PM, Andres Freund wrote: On 2015-07-30 17:36:52 +0300, Heikki Linnakangas wrote: In 9.4, LWLockAcquire holds the spinlock when it marks the lock as held, until it has updated the variable. And LWLockWaitForVar() holds the spinlock when it checks that the lock is held and that

Re: [HACKERS] Transactions involving multiple postgres foreign servers

2015-07-31 Thread Ashutosh Bapat
On Thu, Jul 30, 2015 at 1:52 AM, Robert Haas wrote: > On Wed, Jul 29, 2015 at 6:58 AM, Ashutosh Bapat > wrote: > > A user may set atomic_foreign_transaction to ON to guarantee atomicity, > IOW > > it throws error when atomicity can not be guaranteed. Thus if application > > accidentally does som

Re: [HACKERS] 64-bit XIDs again

2015-07-31 Thread Alexander Korotkov
On Fri, Jul 31, 2015 at 1:27 AM, Petr Jelinek wrote: > On 2015-07-30 23:23, Tom Lane wrote: > >> But the elephant in the room is on-disk compatibility. There is >> absolutely no way that we can just change xmin/xmax to 64 bits without a >> disk format break. However, if we do something like wha

Re: [HACKERS] Using quicksort and a merge step to significantly improve on tuplesort's single run "external sort"

2015-07-31 Thread Heikki Linnakangas
On 07/31/2015 02:01 AM, Peter Geoghegan wrote: What prevents the tuple at the top of the in-memory heap at the point of tuplesort_performsort() (say, one of the ones added to the heap as our glut of memory was*partially* consumed) being less than the last/greatest tuple on tape? If the answer is