Re: [HACKERS] Crash on promotion when recovery.conf is renamed

2016-12-19 Thread Michael Paquier
On Sat, Dec 17, 2016 at 9:23 PM, Magnus Hagander wrote: > On Fri, Dec 16, 2016 at 7:08 AM, Michael Paquier > wrote: >> Looking at PrescanPreparedTransactions(), I am thinking as well that it >> would >> be better to get a hard failure when bumping on a corrupted 2PC file. >> Future >> files are o

Re: [HACKERS] increasing the default WAL segment size

2016-12-19 Thread Beena Emerson
Hello all, On Mon, Dec 19, 2016 at 3:14 PM, Beena Emerson wrote: > Hello all, > > Please find attached a patch to make wal segment size initdb configurable. > > The attached patch removes --with-wal-segsize configure option and adds a > new initdb option --wal-segsize. The module initdb passes t

Re: [HACKERS] too low cost of Bitmap index scan

2016-12-19 Thread Pavel Stehule
2016-12-19 23:28 GMT+01:00 Robert Haas : > On Sat, Dec 17, 2016 at 3:30 AM, Pavel Stehule > wrote: > > -> Bitmap Heap Scan on "Zasilka" (cost=5097.39..5670.64 rows=1 > width=12) > > (actual time=62.253..62.400 rows=3 loops=231) > ... > > When I disable bitmap scan, then the query is 6x time fas

Re: [HACKERS] Logical Replication WIP

2016-12-19 Thread Erik Rijkers
On 2016-12-19 08:04, Erik Rijkers wrote: On 2016-12-18 11:12, Petr Jelinek wrote: (now using latest: patchset:) 0001-Add-PUBLICATION-catalogs-and-DDL-v14.patch 0002-Add-SUBSCRIPTION-catalog-and-DDL-v14.patch 0003-Define-logical-replication-protocol-and-output-plugi-v14.patch 0004-Add-logical-re

Re: [HACKERS] Logical decoding on standby

2016-12-19 Thread Petr Jelinek
On 07/12/16 07:05, Craig Ringer wrote: > On 21 November 2016 at 16:17, Craig Ringer wrote: >> Hi all >> >> I've prepared a working initial, somewhat raw implementation for >> logical decoding on physical standbys. > > Hi all > > I've attached a significantly revised patch, which now incorporates

Re: [HACKERS] invalid combination of options "-D - -F t -X stream" in pg_basebackup

2016-12-19 Thread Fujii Masao
On Tue, Dec 20, 2016 at 1:43 AM, Magnus Hagander wrote: > > > On Mon, Dec 19, 2016 at 5:39 PM, Fujii Masao wrote: >> >> Hi, >> >> Isn't it better to forbid the conbination of the options "-D -", "-F t" >> and >> "-X stream" in pg_basebackup? This is obviously invalid setting and the >> docs >> wa

Re: [HACKERS] Quorum commit for multiple synchronous replication.

2016-12-19 Thread Michael Paquier
On Tue, Dec 20, 2016 at 2:31 PM, Masahiko Sawada wrote: > Do we need to consider the sorting method and the selecting k-th > latest LSN method? Honestly, nah. Tests are showing that there are many more bottlenecks before that with just memory allocation and parsing. -- Michael -- Sent via pgs

Re: [HACKERS] pg_background contrib module proposal

2016-12-19 Thread amul sul
On Tue, Dec 20, 2016 at 12:21 AM, David Fetter wrote: > On Thu, Nov 24, 2016 at 09:16:53AM +0530, amul sul wrote: >> Hi All, >> >> I would like to take over pg_background patch and repost for >> discussion and review. > > This looks great. > > Sadly, it now breaks initdb when applied atop > dd7288

Re: [HACKERS] Quorum commit for multiple synchronous replication.

2016-12-19 Thread Masahiko Sawada
On Mon, Dec 19, 2016 at 9:49 PM, Fujii Masao wrote: > On Sun, Dec 18, 2016 at 9:36 PM, Michael Paquier > wrote: >> On Fri, Dec 16, 2016 at 10:42 PM, Fujii Masao wrote: >>> Attached is the modified version of the patch. Barring objections, I will >>> commit this version. >> >> There is a whitespa

Re: [HACKERS] Declarative partitioning - another take

2016-12-19 Thread Amit Langote
On 2016/12/20 12:59, Robert Haas wrote: > On Sun, Dec 18, 2016 at 10:00 PM, Amit Langote > wrote: >> Here are updated patches including the additional information. > > Thanks. Committed 0001. Will have to review the others when I'm less tired. Thanks! > BTW, elog() is only supposed to be used

Re: [HACKERS] Declarative partitioning - another take

2016-12-19 Thread Robert Haas
On Sun, Dec 18, 2016 at 10:00 PM, Amit Langote wrote: > Here are updated patches including the additional information. Thanks. Committed 0001. Will have to review the others when I'm less tired. BTW, elog() is only supposed to be used for can't happen error messages; when it is used, no transl

Re: [HACKERS] Measuring replay lag

2016-12-19 Thread Thomas Munro
On Mon, Dec 19, 2016 at 10:46 PM, Simon Riggs wrote: > On 26 October 2016 at 11:34, Thomas Munro > wrote: > >> It works by taking advantage of the { time, end-of-WAL } samples that >> sending servers already include in message headers to standbys. That >> seems to provide a pretty good proxy fo

Re: pg_authid.rolpassword format (was Re: [HACKERS] Password identifiers, protocol aging and SCRAM protocol)

2016-12-19 Thread Robert Haas
On Sat, Dec 17, 2016 at 5:48 PM, Michael Paquier wrote: > On Sun, Dec 18, 2016 at 3:59 AM, Robert Haas wrote: >> On Fri, Dec 16, 2016 at 5:30 PM, Michael Paquier >> wrote: >>> From the discussions of last year on -hackers, it was decided to *not* >>> have an additional column per complains from

Re: [HACKERS] Creating a DSA area to provide work space for parallel execution

2016-12-19 Thread Robert Haas
On Mon, Dec 19, 2016 at 6:35 PM, Thomas Munro wrote: > On Tue, Dec 20, 2016 at 11:12 AM, Robert Haas wrote: >> On Thu, Dec 1, 2016 at 6:35 AM, Thomas Munro >> wrote: >>> On Sat, Nov 26, 2016 at 1:55 AM, Thomas Munro >>> wrote: Here's a new version to apply on top of dsa-v7.patch. >>> >>> H

Re: [HACKERS] Logical Replication WIP

2016-12-19 Thread Petr Jelinek
On 19/12/16 15:39, Steve Singer wrote: > On 12/18/2016 09:04 PM, Petr Jelinek wrote: >> On 18/12/16 19:02, Steve Singer wrote: >> >>> pg_dump is also generating warnings >>> >>> pg_dump: [archiver] WARNING: don't know how to set owner for object type >>> SUBSCRIPTION >>> >>> I know that the plan is

Re: [HACKERS] Password identifiers, protocol aging and SCRAM protocol

2016-12-19 Thread Michael Paquier
On Thu, Dec 15, 2016 at 3:17 PM, Michael Paquier wrote: > In the case where the binaries are *not* built with libidn, I think > that we had better reject valid UTF-8 string directly and just allow > ASCII? SASLprep is a no-op on ASCII characters. > > Thoughts about this approach? And Heikki has m

Re: [HACKERS] Logical replication existing data copy

2016-12-19 Thread Petr Jelinek
On 19/12/16 18:25, Peter Eisentraut wrote: > On 12/19/16 4:30 AM, Petr Jelinek wrote: >> So existing table data can be copied once subscription is created, but >> also new tables can be added and their data copied. This is where the >> REFRESH PUBLICATION comes into play. Adding table to publicatio

Re: [HACKERS] Minor correction in alter_table.sgml

2016-12-19 Thread Amit Langote
On 2016/12/20 4:08, Peter Eisentraut wrote: > On 11/30/16 8:47 PM, Amit Langote wrote: >>> So maybe something like >>> >>> All the forms of ALTER TABLE that act on a single table, >>> except RENAME and SET SCHEMA, can be combined into a >>> list of multiple alterations to be applied tog

Re: [HACKERS] Logical replication existing data copy

2016-12-19 Thread Craig Ringer
On 20 Dec. 2016 1:27 am, "Peter Eisentraut" < peter.eisentr...@2ndquadrant.com> wrote: On 12/19/16 4:30 AM, Petr Jelinek wrote: > So existing table data can be copied once subscription is created, but > also new tables can be added and their data copied. This is where the > REFRESH PUBLICATION com

Re: [HACKERS] Creating a DSA area to provide work space for parallel execution

2016-12-19 Thread Thomas Munro
On Tue, Dec 20, 2016 at 11:12 AM, Robert Haas wrote: > On Thu, Dec 1, 2016 at 6:35 AM, Thomas Munro > wrote: >> On Sat, Nov 26, 2016 at 1:55 AM, Thomas Munro >> wrote: >>> Here's a new version to apply on top of dsa-v7.patch. >> >> Here's a version to go with dsa-v8.patch. > > All right, so I've

Re: [HACKERS] too low cost of Bitmap index scan

2016-12-19 Thread Robert Haas
On Sat, Dec 17, 2016 at 3:30 AM, Pavel Stehule wrote: > -> Bitmap Heap Scan on "Zasilka" (cost=5097.39..5670.64 rows=1 width=12) > (actual time=62.253..62.400 rows=3 loops=231) ... > When I disable bitmap scan, then the query is 6x time faster >-> Index Scan using "Zasilka_idx_Dopravce

Re: [HACKERS] Cache Hash Index meta page.

2016-12-19 Thread Robert Haas
On Fri, Dec 16, 2016 at 5:16 AM, Mithun Cy wrote: > Shouldn't _hash_doinsert() be using the cache, too >> > > Yes, we have an opportunity there, added same in code. But one difference > is at the end at-least once we need to read the meta page to split and/or > write. Performance improvement migh

Re: [HACKERS] Creating a DSA area to provide work space for parallel execution

2016-12-19 Thread Robert Haas
On Thu, Dec 1, 2016 at 6:35 AM, Thomas Munro wrote: > On Sat, Nov 26, 2016 at 1:55 AM, Thomas Munro > wrote: >> Here's a new version to apply on top of dsa-v7.patch. > > Here's a version to go with dsa-v8.patch. All right, so I've committed this, but not before (1) renaming some things that you

Re: [HACKERS] Proposal for changes to recovery.conf API

2016-12-19 Thread Simon Riggs
On 19 December 2016 at 21:29, Peter Eisentraut wrote: > On 12/16/16 8:52 PM, Robert Haas wrote: >> If the explanation is just a few sentences long, I see no reason not >> to include it in the release notes. > > As far as I can tell from the latest posted patch, the upgrade > instructions are > > -

Re: [HACKERS] Proposal: add error message in backend/catalog/index.c

2016-12-19 Thread Peter Eisentraut
On 12/13/16 1:15 AM, Ioseph Kim wrote: > I propose to append an error message when index name and table name are > same. > > > example: > > postgres@postgres=# create table t (a int not null, constraint t primary > key (a)); > ERROR: relation "t" already exists The code to detect that would pr

Re: [HACKERS] pg_basebackups and slots

2016-12-19 Thread Peter Eisentraut
On 12/17/16 9:55 AM, Magnus Hagander wrote: > That makes a lot of sense now that we have temporary replication slots, > I agree. And then max_replication_slots could be something like > persistent_replication_slots? I was thinking was more like that each walsender gets one fixed slot, which can be

Re: [HACKERS] Proposal for changes to recovery.conf API

2016-12-19 Thread Peter Eisentraut
On 12/16/16 8:52 PM, Robert Haas wrote: > If the explanation is just a few sentences long, I see no reason not > to include it in the release notes. As far as I can tell from the latest posted patch, the upgrade instructions are - To trigger recovery, create an empty file recovery.trigger instead

Re: [HACKERS] Time to drop old-style (V0) functions?

2016-12-19 Thread Robert Haas
On Mon, Dec 19, 2016 at 3:13 PM, Peter Eisentraut wrote: > On 12/9/16 7:52 AM, Robert Haas wrote: >> It's kind of ironic, at least IMHO, that the DirectionFunctionCall >> macros are anything but direct. Each one is a non-inlined function >> call that does a minimum of 8 variable assignments befor

Re: [HACKERS] Time to drop old-style (V0) functions?

2016-12-19 Thread Peter Eisentraut
On 12/9/16 7:52 AM, Robert Haas wrote: > It's kind of ironic, at least IMHO, that the DirectionFunctionCall > macros are anything but direct. Each one is a non-inlined function > call that does a minimum of 8 variable assignments before actually > calling the function. If this is a problem (it mi

Re: [HACKERS] Minor correction in alter_table.sgml

2016-12-19 Thread Peter Eisentraut
On 11/30/16 8:47 PM, Amit Langote wrote: >> So maybe something like >> >> All the forms of ALTER TABLE that act on a single table, >> except RENAME and SET SCHEMA, can be combined into a >> list of multiple alterations to be applied together. > > Updated patch attached. Could you p

Re: [HACKERS] pg_background contrib module proposal

2016-12-19 Thread David Fetter
On Thu, Nov 24, 2016 at 09:16:53AM +0530, amul sul wrote: > Hi All, > > I would like to take over pg_background patch and repost for > discussion and review. This looks great. Sadly, it now breaks initdb when applied atop dd728826c538f000220af98de025c00114ad0631 with: performing post-bootstrap

Re: [HACKERS] PSQL commands: \quit_if, \quit_unless

2016-12-19 Thread David G. Johnston
On Mon, Dec 19, 2016 at 11:23 AM, Alvaro Herrera wrote: > David G. Johnston wrote: > > > Being able to do more conditional work in psql would make setting up more > > robust scripts easier and without either losing transaction capabilities > or > > session pooling for improved performance when la

Re: [HACKERS] PSQL commands: \quit_if, \quit_unless

2016-12-19 Thread Alvaro Herrera
David G. Johnston wrote: > Being able to do more conditional work in psql would make setting up more > robust scripts easier and without either losing transaction capabilities or > session pooling for improved performance when large numbers of small > commands are run in between flow control in do

Re: [HACKERS] PSQL commands: \quit_if, \quit_unless

2016-12-19 Thread Corey Huinker
> > > > > -- this is how I fake an 'exit 0' now: > > \set work_needs_to_be_done t > > select > > case > > when :'work_needs_to_be_done'::boolean then '' > > else '\q' > > end as cmd > > \gset > > :cmd > > > > -- ridiculous example to illustrate complications in remem

Re: [HACKERS] PSQL commands: \quit_if, \quit_unless

2016-12-19 Thread David G. Johnston
On Mon, Dec 19, 2016 at 10:30 AM, Robert Haas wrote: > > I see your point. Just out of curiosity, why in the world don't you > use something other than psql for scripting? I mean, if you accessed > the data from Perl or Python or > $INSERT_YOUR_FAVORITE_SCRIPTING_LANGUAGE_HERE, you could do all

Re: [HACKERS] PSQL commands: \quit_if, \quit_unless

2016-12-19 Thread Pavel Stehule
2016-12-19 18:30 GMT+01:00 Robert Haas : > On Sat, Dec 17, 2016 at 3:39 PM, Corey Huinker > wrote: > >> To implement \while, we'd also need to remember previous commands so > >> that when we reach the end of the loop, we can rewind and put all of > >> those commands back on the stack to be execut

Re: [HACKERS] Hash Indexes

2016-12-19 Thread Robert Haas
On Sun, Dec 18, 2016 at 8:54 AM, Amit Kapila wrote: >> I committed remove-hash-wrtbuf and fix_dirty_marking_v1 but I've got >> some reservations about fix_lock_chaining_v1. ISTM that the natural >> fix here would be to change the API contract for _hash_freeovflpage so >> that it doesn't release t

Re: [HACKERS] PSQL commands: \quit_if, \quit_unless

2016-12-19 Thread Robert Haas
On Sat, Dec 17, 2016 at 3:39 PM, Corey Huinker wrote: >> To implement \while, we'd also need to remember previous commands so >> that when we reach the end of the loop, we can rewind and put all of >> those commands back on the stack to be executed again, or perhaps to >> be skipped if the \while

Re: [HACKERS] Logical replication existing data copy

2016-12-19 Thread Peter Eisentraut
On 12/19/16 4:30 AM, Petr Jelinek wrote: > So existing table data can be copied once subscription is created, but > also new tables can be added and their data copied. This is where the > REFRESH PUBLICATION comes into play. Adding table to publication does > not make it automatically replicated by

Re: [HACKERS] Creating a DSA area to provide work space for parallel execution

2016-12-19 Thread Robert Haas
On Sun, Dec 18, 2016 at 10:33 PM, Thomas Munro wrote: > On Sat, Dec 17, 2016 at 5:41 AM, Robert Haas wrote: >> On Wed, Dec 14, 2016 at 3:25 PM, Robert Haas wrote: >>> Thoughts? >> >> Hearing no objections, I've gone ahead and committed this. If that >> makes somebody really unhappy I can revert

Re: [HACKERS] Declarative partitioning vs. sql_inheritance

2016-12-19 Thread Robert Haas
On Mon, Dec 19, 2016 at 11:48 AM, Alvaro Herrera wrote: > Any particular reason not to change inhOpt to be a simple boolean, and > remove the enum? No, no particular reason. I thought about it, but I didn't really see any advantage in getting rid of the typedef. -- Robert Haas EnterpriseDB: ht

Re: [HACKERS] Declarative partitioning vs. sql_inheritance

2016-12-19 Thread Alvaro Herrera
Robert Haas wrote: > On Fri, Dec 16, 2016 at 7:39 PM, Tom Lane wrote: > > Peter Eisentraut writes: > >> On 12/16/16 11:05 AM, Robert Haas wrote: > >>> If we were going to do anything about this, > >>> my vote would be to remove sql_inheritance. > > > >> Go for it. > > > >> Let's also remove the t

Re: [HACKERS] Quorum commit for multiple synchronous replication.

2016-12-19 Thread Alvaro Herrera
Fujii Masao wrote: > Regarding this feature, there are some loose ends. We should work on > and complete them until the release. Please list these in https://wiki.postgresql.org/wiki/Open_Items so that we don't forget. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL De

Re: [HACKERS] invalid combination of options "-D - -F t -X stream" in pg_basebackup

2016-12-19 Thread Magnus Hagander
On Mon, Dec 19, 2016 at 5:39 PM, Fujii Masao wrote: > Hi, > > Isn't it better to forbid the conbination of the options "-D -", "-F t" and > "-X stream" in pg_basebackup? This is obviously invalid setting and the > docs > warns this as follows. But currently users can specify such setting and > pg

[HACKERS] invalid combination of options "-D - -F t -X stream" in pg_basebackup

2016-12-19 Thread Fujii Masao
Hi, Isn't it better to forbid the conbination of the options "-D -", "-F t" and "-X stream" in pg_basebackup? This is obviously invalid setting and the docs warns this as follows. But currently users can specify such setting and pg_basebackup can exit unexpectedly with an error. -

Re: [HACKERS] pg_background contrib module proposal

2016-12-19 Thread Andrew Borodin
2016-12-19 4:21 GMT+05:00 David Fetter : > Couldn't it sleep in increments smaller than a second? Like maybe > milliseconds? Also, it's probably cleaner (or at least more > comprehensible) to write something using format() and dollar quoting > than the line with the double 's. Right. Here's vers

Re: [HACKERS] Retire src/backend/port/dynloader/linux.c ?

2016-12-19 Thread Andres Freund
On 2016-12-18 22:19:36 -0500, Tom Lane wrote: > Andres Freund writes: > > Shouldn't we just remove that code? > > What for? I every now and then end up looking at it for a few minutes, and wonder what the hell dld is, just to see that it's old stuff. > It's maintenance-free ... hasn't been touc

Re: [HACKERS] postgres_fdw bug in 9.6

2016-12-19 Thread Tom Lane
Etsuro Fujita writes: > On 2016/12/17 1:13, Tom Lane wrote: >> So I think the rule could be >> "When first asked to produce a path for a given foreign joinrel, collect >> the cheapest paths for its left and right inputs, and make a nestloop path >> (or hashjoin path, if full join) from those, usi

Re: [HACKERS] Logical Replication WIP

2016-12-19 Thread Steve Singer
On 12/18/2016 09:04 PM, Petr Jelinek wrote: On 18/12/16 19:02, Steve Singer wrote: pg_dump is also generating warnings pg_dump: [archiver] WARNING: don't know how to set owner for object type SUBSCRIPTION I know that the plan is to add proper ACL's for publications and subscriptions later. I

Re: [HACKERS] Quorum commit for multiple synchronous replication.

2016-12-19 Thread Fujii Masao
On Sun, Dec 18, 2016 at 9:36 PM, Michael Paquier wrote: > On Fri, Dec 16, 2016 at 10:42 PM, Fujii Masao wrote: >> Attached is the modified version of the patch. Barring objections, I will >> commit this version. > > There is a whitespace: > $ git diff master --check > src/backend/replication/sync

Re: [HACKERS] postgres_fdw bug in 9.6

2016-12-19 Thread Etsuro Fujita
On 2016/12/19 13:59, Ashutosh Bapat wrote: On Fri, Dec 16, 2016 at 9:43 PM, Tom Lane wrote: Etsuro Fujita writes: On 2016/12/16 11:25, Etsuro Fujita wrote: As I said upthread, an alternative I am thinking is (1) to create an equivalent nestloop join path using inner/outer paths of a foreign

[HACKERS] Protect syscache from bloating with negative cache entries

2016-12-19 Thread Kyotaro HORIGUCHI
Hello, recently one of my customer stumbled over an immoderate catcache bloat. This is a known issue living on the Todo page in the PostgreSQL wiki. https://wiki.postgresql.org/wiki/Todo#Cache_Usage > Fix memory leak caused by negative catcache entries https://www.postgresql.org/message-id/51c0a

Re: [HACKERS] Measuring replay lag

2016-12-19 Thread Thomas Munro
On Mon, Dec 19, 2016 at 4:03 PM, Peter Eisentraut wrote: > On 11/22/16 4:27 AM, Thomas Munro wrote: >> Thanks very much for testing! New version attached. I will add this >> to the next CF. > > I don't see it there yet. Thanks for the reminder. Added here: https://commitfest.postgresql.org/12

Re: [HACKERS] Make pg_basebackup -x stream the default

2016-12-19 Thread Vladimir Rusinov
On Sat, Dec 17, 2016 at 2:37 PM, Magnus Hagander wrote: > Attached is an updated patch that does this. As a bonus it simplifies the > code a bit. I also fixed an error message that I missed updating in the > previous patch. looks good to me. Still applies cleanly at head with all tests pass. I

Re: [HACKERS] Measuring replay lag

2016-12-19 Thread Simon Riggs
On 26 October 2016 at 11:34, Thomas Munro wrote: > It works by taking advantage of the { time, end-of-WAL } samples that > sending servers already include in message headers to standbys. That > seems to provide a pretty good proxy for when the WAL was written, if > you ignore messages where the

Re: [HACKERS] increasing the default WAL segment size

2016-12-19 Thread Beena Emerson
Hello all, Please find attached a patch to make wal segment size initdb configurable. The attached patch removes --with-wal-segsize configure option and adds a new initdb option --wal-segsize. The module initdb passes the wal-segsize value into an environment variable which is used to overwrite t

Re: [HACKERS] Slow I/O can break throttling of base backup

2016-12-19 Thread Magnus Hagander
On Fri, Dec 16, 2016 at 11:24 AM, Antonin Houska wrote: > Antonin Houska wrote: > > > It seems to be my bug. I'll check tomorrow. > > I could reproduce the problem by adding sufficient sleep time to the > loop. > > > Magnus Hagander wrote: > >> I wonder if the else if (sleep > 0) at the bottom

Re: [HACKERS] Logical Replication WIP

2016-12-19 Thread Petr Jelinek
On 19/12/16 08:04, Erik Rijkers wrote: > On 2016-12-18 11:12, Petr Jelinek wrote: > > (now using latest: patchset:) > > 0001-Add-PUBLICATION-catalogs-and-DDL-v14.patch > 0002-Add-SUBSCRIPTION-catalog-and-DDL-v14.patch > 0003-Define-logical-replication-protocol-and-output-plugi-v14.patch > 0004-Ad