Re: [HACKERS] Logical Replication WIP

2017-01-11 Thread Petr Jelinek
On 10/01/17 14:52, Peter Eisentraut wrote: > On 1/2/17 8:32 AM, Petr Jelinek wrote: >> On 02/01/17 05:23, Steve Singer wrote: >>> but I can't drop the subscription either >>> >>> >>> test_b=# drop subscription mysub; >>> ERROR: could

Re: [HACKERS] Logical Replication WIP

2017-01-11 Thread Petr Jelinek
On 10/01/17 15:06, Peter Eisentraut wrote: > On 1/3/17 5:23 PM, Petr Jelinek wrote: >> I got this remark about IsCatalogClass() from Andres offline as well, >> but it's not true, it only checks for FirstNormalObjectId for objects in >> pg_catalog and toast schemas, not

Re: [HACKERS] Logical Replication WIP

2017-01-11 Thread Petr Jelinek
On 11/01/17 18:32, Peter Eisentraut wrote: > On 1/11/17 3:29 AM, Petr Jelinek wrote: >> Okay, looking into my notes, I originally did this because we did not >> allow adding tables without pkeys to publications which effectively >> prohibited FOR ALL TABLES publication fr

Re: [HACKERS] Logical Replication WIP

2017-01-11 Thread Petr Jelinek
On 11/01/17 18:27, Peter Eisentraut wrote: > On 1/11/17 3:11 AM, Petr Jelinek wrote: >> That will not help, issue is that we consider names for origins to be >> unique across cluster while subscription names are per database so if >> there is origin per subscription (which the

Re: [HACKERS] Logical Replication WIP

2017-01-15 Thread Petr Jelinek
On 11/01/17 22:30, Peter Eisentraut wrote: > On 1/11/17 3:35 PM, Petr Jelinek wrote: >> On 11/01/17 18:32, Peter Eisentraut wrote: >>> On 1/11/17 3:29 AM, Petr Jelinek wrote: >>>> Okay, looking into my notes, I originally did this because we did not >>>

Re: [HACKERS] Logical Replication WIP

2017-01-15 Thread Petr Jelinek
success of replication is not confirmed by responding to COMMIT but by reporting LSNs of various replication stages. Perhaps the logical_replication_synchronous_commit should only be boolean that would translate to 'off' and 'local' for the real synchronous_commit. -- Petr J

Re: [HACKERS] Logical Replication WIP

2017-01-15 Thread Petr Jelinek
On 15/01/17 20:20, Euler Taveira wrote: > On 15-01-2017 15:13, Petr Jelinek wrote: >> I am not quite sure I agree with this. Either it's system object and we >> don't replicate it (which I would have considered to be anything with >> Oid < FirstNormalObjectId) or

Re: [HACKERS] Logical Replication WIP

2017-01-15 Thread Petr Jelinek
On 15/01/17 23:57, Erik Rijkers wrote: > On 2017-01-15 23:20, Petr Jelinek wrote: > >> 0001-Add-PUBLICATION-catalogs-and-DDL-v18.patch >> 0002-Add-SUBSCRIPTION-catalog-and-DDL-v18.patch >> 0003-Define-logical-replication-protocol-and-output-plugi-v18.patch >> 0004-A

Re: [HACKERS] Logical Replication WIP

2017-01-17 Thread Petr Jelinek
if the subscription name was changed. >>> I don't know why that is a problem. >> >> Because we don't have ALTER SUBSCRIPTION RENAME currently. Maybe should >> be Assert? > > Is there anything stopping anyone from implementing it? > No, just didn't se

Re: [HACKERS] Logical Replication WIP

2017-01-17 Thread Petr Jelinek
On 17/01/17 17:11, Peter Eisentraut wrote: > On 1/15/17 1:48 PM, Petr Jelinek wrote: >> It's meant to decouple the synchronous commit setting for logical >> replication workers from the one set for normal clients. Now that we >> have owners for subscription and subsc

Re: [HACKERS] Logical replication existing data copy

2017-01-18 Thread Petr Jelinek
On 18/01/17 17:35, Erik Rijkers wrote: > On 2017-01-18 14:46, Petr Jelinek wrote: > >> 0001-Logical-replication-support-for-initial-data-copy-v2.patch > > Applies and builds fine on top of the previous 5 patches. > > Two problems: > > 1. alter_subscription.sg

Re: [HACKERS] Logical Replication WIP

2017-01-18 Thread Petr Jelinek
On 17/01/17 22:43, Robert Haas wrote: > On Tue, Jan 17, 2017 at 11:15 AM, Petr Jelinek > wrote: >>> Is there anything stopping anyone from implementing it? >> >> No, just didn't seem priority for the functionality right now. > > Why is it OK for this to n

Re: [HACKERS] [PATCH] ALTER DEFAULT PRIVILEGES with GRANT/REVOKE ON SCHEMAS

2017-01-18 Thread Petr Jelinek
s are not handled by patch author but by translation project so no need. > > Attached a rebased version and with the docs update pointed by Ashutosh > Sharma. > The patch looks good, the only thing I am missing is tab completion support for psql. -- Petr Jelinek http

Re: [HACKERS] Logical Replication WIP - FailedAssertion, File: "array_typanalyze.c", Line: 340

2017-01-19 Thread Petr Jelinek
h it works fine for me. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h index ccb8880..0ad7b0e 100644 --- a/src/include/catal

Re: [HACKERS] Logical Replication WIP

2017-01-20 Thread Petr Jelinek
On 20/01/17 15:08, Peter Eisentraut wrote: > On 1/19/17 5:01 PM, Petr Jelinek wrote: >> There were some conflicting changes committed today so I rebased the >> patch on top of them. >> >> Other than that nothing much has changed, I removed the separate sync >>

Re: [HACKERS] Logical Replication WIP

2017-01-20 Thread Petr Jelinek
On 20/01/17 17:05, Fujii Masao wrote: > On Fri, Jan 20, 2017 at 11:08 PM, Peter Eisentraut > wrote: >> On 1/19/17 5:01 PM, Petr Jelinek wrote: >>> There were some conflicting changes committed today so I rebased the >>> patch on top of them. >>> >>

Re: [HACKERS] Logical Replication WIP

2017-01-20 Thread Petr Jelinek
On 20/01/17 17:33, Jaime Casanova wrote: > On 20 January 2017 at 11:25, Petr Jelinek > wrote: >> On 20/01/17 17:05, Fujii Masao wrote: >>> On Fri, Jan 20, 2017 at 11:08 PM, Peter Eisentraut >>> wrote: >>>> On 1/19/17 5:01 PM, Petr Jelinek wrote:

Re: [HACKERS] Logical Replication WIP

2017-01-20 Thread Petr Jelinek
On 20/01/17 17:23, Petr Jelinek wrote: > On 20/01/17 15:08, Peter Eisentraut wrote: >> On 1/19/17 5:01 PM, Petr Jelinek wrote: >>> There were some conflicting changes committed today so I rebased the >>> patch on top of them. >>> >>> Other than that not

Re: [HACKERS] remote_apply for logical replication?

2017-01-21 Thread Petr Jelinek
ription is running with synchronous_commit = off where flush position is behind. The xinfo & XACT_COMPLETION_APPLY_FEEDBACK does not affect logical replication though as it does not have access to that information (subscriber does not receive raw WAL and logical decoding does not decode this). -- Pe

Re: [HACKERS] Checksums by default?

2017-01-21 Thread Petr Jelinek
On 21/01/17 11:39, Magnus Hagander wrote: > Is it time to enable checksums by default, and give initdb a switch to > turn it off instead? I'd like to see benchmark first, both in terms of CPU and in terms of produced WAL (=network traffic) given that it turns on logging of hint bits.

Re: [HACKERS] Checksums by default?

2017-01-21 Thread Petr Jelinek
On 21/01/17 16:40, Stephen Frost wrote: > Petr, > > * Petr Jelinek (petr.jeli...@2ndquadrant.com) wrote: >> On 21/01/17 11:39, Magnus Hagander wrote: >>> Is it time to enable checksums by default, and give initdb a switch to >>> turn it off instead? >> >

Re: [HACKERS] Checksums by default?

2017-01-21 Thread Petr Jelinek
On 21/01/17 17:31, Stephen Frost wrote: > Petr, > > * Petr Jelinek (petr.jeli...@2ndquadrant.com) wrote: >> On 21/01/17 16:40, Stephen Frost wrote: >>> * Petr Jelinek (petr.jeli...@2ndquadrant.com) wrote: >>>> On 21/01/17 11:39, Magnus Hagander wrote: >

Re: [HACKERS] Checksums by default?

2017-01-21 Thread Petr Jelinek
On 21/01/17 17:51, Stephen Frost wrote: > * Petr Jelinek (petr.jeli...@2ndquadrant.com) wrote: >> On 21/01/17 17:31, Stephen Frost wrote: >>> This is just changing the *default*, not requiring checksums to always >>> be enabled. We do not hold the same standards for ou

Re: [HACKERS] Checksums by default?

2017-01-21 Thread Petr Jelinek
On 21/01/17 18:15, Stephen Frost wrote: > * Petr Jelinek (petr.jeli...@2ndquadrant.com) wrote: >> On 21/01/17 17:51, Stephen Frost wrote: >>> I'm quite sure that the performance numbers for the CREATE TABLE + COPY >>> case with wal_level=minimal would have been *fa

Re: [HACKERS] Checksums by default?

2017-01-21 Thread Petr Jelinek
On 21/01/17 18:46, Stephen Frost wrote: > * Petr Jelinek (petr.jeli...@2ndquadrant.com) wrote: >> As we don't know the performance impact is (there was no benchmark done >> on reasonably current code base) I really don't understand how you can >> judge if it's

Re: [HACKERS] Logical replication failing when foreign key present

2017-01-22 Thread Petr Jelinek
scription > 16408 (PID 19201) exited with exit code 1 > > Hi, thanks for report. Looks like I missed AfterTriggerBeginQuery/AfterTriggerEndQuery when moving the executor stuff around. Attached should fix it. -- Petr Jelinek http://www.2ndQuadra

Re: [HACKERS] Logical Replication WIP

2017-01-22 Thread Petr Jelinek
On 20/01/17 22:30, Petr Jelinek wrote: > Since it's not exactly straight forward to find when these need to be > initialized based on commands, I decided to move the initialization code > to exec_replication_command() since that's always called before anything > so that ma

Re: [HACKERS] Logical Replication WIP

2017-01-23 Thread Petr Jelinek
On 23/01/17 17:19, Fujii Masao wrote: > On Sat, Jan 21, 2017 at 1:39 AM, Petr Jelinek > wrote: >> On 20/01/17 17:33, Jaime Casanova wrote: >>> On 20 January 2017 at 11:25, Petr Jelinek >>> wrote: >>>> On 20/01/17 17:05, Fujii Masao wrote: >>>&g

Re: [HACKERS] Logical replication launcher's bgworker enabled by default, and max_logical_replication_workers

2017-01-23 Thread Petr Jelinek
ApplyLauncherRegister() is a noop if wal_level < logical. > As discussed elsewhere, there is no need for wal_level = logical downstream and launcher is only needed for downstream. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Traini

Re: [HACKERS] Checksums by default?

2017-01-23 Thread Petr Jelinek
systems without throwing a false positive or surfacing a >> bug." > > I am reading that similarly to what Tom is seeing: enabling it has > proved Heroku that it did not catch problems in years, meaning that > the performance cost induced by enabling it has paid noth

Re: [HACKERS] simplify sequence test

2017-01-25 Thread Petr Jelinek
tached patch to restructure the test a bit > to avoid this, without loss of test coverage. > +1, looks good. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hack

Re: [HACKERS] Logical Replication WIP

2017-01-26 Thread Petr Jelinek
On 25/01/17 18:16, Peter Eisentraut wrote: > On 1/22/17 8:11 PM, Petr Jelinek wrote: >> 0001 - Changes the libpqrcv_connect to use async libpq api so that it >> won't get stuck forever in case of connect is stuck. This is preexisting >> bug that also affects walreceiver b

Re: [HACKERS] Enabling replication connections by default in pg_hba.conf

2017-02-02 Thread Petr Jelinek
n't materially improve security control. > +1 > It would also be useful to be able prevent users with REPLICATION > capability from connecting as normal users, if the are marked as > NOLOGIN. > +1 -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Dev

Re: [HACKERS] Cannot shutdown subscriber after DROP SUBSCRIPTION

2017-02-04 Thread Petr Jelinek
calrep_worker_stop(). > > Another problem that I found while reading the code is that the launcher can > start up the worker with the subscription that DROP SUBSCRIPTION just removed. > That is, DROP SUBSCRIPTION removes the target entry from pg_subscription, > but the launcher

Re: [HACKERS] Logical Replication and Character encoding

2017-02-04 Thread Petr Jelinek
I wonder if we should just make the subscriber send the client_encoding always (based on server encoding of the subscriber). That should solve the issue in combination with your patch no? -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training &

Re: [HACKERS] IF [NOT] EXISTS for replication slots

2017-02-06 Thread Petr Jelinek
for the replication commands a > dedicated keyword. > > Any thoughts? > My thought is, how would this handle the snapshot creation that logical slot does when it's created? -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, T

Re: [HACKERS] Cannot shutdown subscriber after DROP SUBSCRIPTION

2017-02-06 Thread Petr Jelinek
On 06/02/17 17:33, Fujii Masao wrote: > On Sun, Feb 5, 2017 at 5:11 AM, Petr Jelinek > wrote: >> On 03/02/17 19:38, Fujii Masao wrote: >>> On Sat, Feb 4, 2017 at 12:49 AM, Fujii Masao wrote: >>>> On Fri, Feb 3, 2017 at 10:56 AM, Kyotaro HORIGUCHI >>>&

Re: [HACKERS] IF [NOT] EXISTS for replication slots

2017-02-06 Thread Petr Jelinek
On 07/02/17 01:00, Michael Paquier wrote: > On Tue, Feb 7, 2017 at 2:07 AM, Petr Jelinek > wrote: >> On 06/02/17 05:15, Michael Paquier wrote: >>> Hi all, >>> >>> Lately I have bumped into a case where it would have been useful to >>> make th

Re: [HACKERS] DROP SUBSCRIPTION and ROLLBACK

2017-02-07 Thread Petr Jelinek
BSCRIPTION/COMMIT and is not started because the delete from catalog is now visible so the subscription is no longer there? -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-ha

Re: [HACKERS] DROP SUBSCRIPTION and ROLLBACK

2017-02-07 Thread Petr Jelinek
On 07/02/17 16:26, Petr Jelinek wrote: > On 07/02/17 13:10, Masahiko Sawada wrote: >> I think the logical replication should not stop and the corresponding >> replication slot and replication origin should not be removed until >> the transaction commits. >> >> The

Re: [HACKERS] DROP SUBSCRIPTION and ROLLBACK

2017-02-08 Thread Petr Jelinek
On 08/02/17 07:40, Masahiko Sawada wrote: > On Wed, Feb 8, 2017 at 9:01 AM, Michael Paquier > wrote: >> On Wed, Feb 8, 2017 at 1:30 AM, Fujii Masao wrote: >>> On Wed, Feb 8, 2017 at 12:26 AM, Petr Jelinek >>> wrote: >>>> For example wha

Re: [HACKERS] Logical replication existing data copy

2017-02-08 Thread Petr Jelinek
55eb4-6b2e-b35b-c184-509efd6eba10%402ndquadrant.com -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscript

Re: [HACKERS] DROP SUBSCRIPTION and ROLLBACK

2017-02-11 Thread Petr Jelinek
On 10/02/17 19:55, Masahiko Sawada wrote: > On Thu, Feb 9, 2017 at 12:44 AM, Petr Jelinek > wrote: >> On 08/02/17 07:40, Masahiko Sawada wrote: >>> On Wed, Feb 8, 2017 at 9:01 AM, Michael Paquier >>> wrote: >>>> On Wed, Feb 8, 2017 at 1:30 AM, Fujii Mas

Re: [HACKERS] DROP SUBSCRIPTION and ROLLBACK

2017-02-15 Thread Petr Jelinek
On 15/02/17 06:43, Masahiko Sawada wrote: > On Tue, Feb 14, 2017 at 1:13 AM, Fujii Masao wrote: >> On Mon, Feb 13, 2017 at 4:05 PM, Masahiko Sawada >> wrote: >>> On Sat, Feb 11, 2017 at 8:18 PM, Petr Jelinek >>> wrote: >>>> On 10/02/17 19:55, Masahi

Re: [HACKERS] Logical Replication and Character encoding

2017-02-15 Thread Petr Jelinek
On 14/02/17 03:23, Kyotaro HORIGUCHI wrote: > At Sat, 4 Feb 2017 21:27:32 +0100, Petr Jelinek > wrote in > >> Hmm I wonder if we should just make the subscriber send the >> client_encoding always (based on server encoding of the subscriber). >> That should solve th

Re: [HACKERS] Logical replication existing data copy

2017-02-15 Thread Petr Jelinek
On 13/02/17 14:51, Erik Rijkers wrote: > On 2017-02-11 11:16, Erik Rijkers wrote: >> On 2017-02-08 23:25, Petr Jelinek wrote: >> >>> 0001-Use-asynchronous-connect-API-in-libpqwalreceiver-v2.patch >>> 0002-Always-initialize-stringinfo-buffers-in-walsender-v2

Re: [HACKERS] PATCH: two slab-like memory allocators

2016-11-27 Thread Petr Jelinek
lock 0x227c3f0 does not match bitmap 34 step s0start: SELECT data FROM pg_logical_slot_get_changes('isolation_slot', NULL, NULL, 'include-xids', 'false'); data Also, let's just rename the Gen to Generation. -- Petr Jelinek http://ww

Re: [HACKERS] Logical Replication WIP

2016-11-27 Thread Petr Jelinek
On 22/11/16 18:42, Erik Rijkers wrote: > On 2016-11-20 19:02, Petr Jelinek wrote: > >> 0001-Add-support-for-TE...cation-slots-v8.patch.gz (~8 KB) >> 0002-Refactor-libpqwalreceiver-v8.patch.gz (~9 KB) >> 0003-Add-PUBLICATION-catalogs-and-DDL-v8.patch.gz (~30 KB) >>

Re: [HACKERS] Logical decoding on standby

2016-11-27 Thread Petr Jelinek
n more pointless. Not going into deeper detail as this is still very WIP. I go agree with the general design though. This also replaces the previous timeline following and decoding threads/CF entries so maybe those should be closed in CF? -- Petr Jelinek http://www.2ndQuadrant.co

Re: [HACKERS] PATCH: two slab-like memory allocators

2016-11-27 Thread Petr Jelinek
hether you need -> or . since you don't know whether > it's a pointer or the actual object. All to save a * in parameter and > variable declaration?... > FWIW I don't like that pattern either although it's used in many parts of our code-base. -- Petr Jelinek

Re: [HACKERS] Logical Replication WIP

2016-11-27 Thread Petr Jelinek
On 27/11/16 23:42, Erik Rijkers wrote: > On 2016-11-27 19:57, Petr Jelinek wrote: >> On 22/11/16 18:42, Erik Rijkers wrote: >>> On 2016-11-20 19:02, Petr Jelinek wrote: >>> >>>> 0001-Add-support-for-TE...cation-slots-v8.patch.gz (~8 KB) >>>>

Re: [HACKERS] Tackling JsonPath support

2016-11-28 Thread Petr Jelinek
between those except contrib is included in distribution. Extensions that provide just functions are easy to integrate into core (that's how some of the existing json functions were added in the past as well). -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Dev

Re: [HACKERS] Tackling JsonPath support

2016-11-29 Thread Petr Jelinek
op of that). I think that should give us some hint on what the base functionality should look like. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Tackling JsonPath support

2016-11-29 Thread Petr Jelinek
On 29/11/16 17:28, Nico Williams wrote: > On Tue, Nov 29, 2016 at 05:18:17PM +0100, Petr Jelinek wrote: >> Just to add to this, the SQL/JSON proposals I've seen so far, and what >> Oracle, MSSQL and Teradata chose to implement already is basically >> subset o

Re: [HACKERS] Logical Replication WIP

2016-11-30 Thread Petr Jelinek
good to me, do you plan to commit this soon or would you rather me to resubmit the patches rebased on top of this (and including this) first? -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hacke

Re: [HACKERS] Logical Replication WIP

2016-12-02 Thread Petr Jelinek
On 02/12/16 02:55, Thomas Munro wrote: > On Fri, Dec 2, 2016 at 2:32 PM, Peter Eisentraut > wrote: >> On 11/30/16 8:06 PM, Petr Jelinek wrote: >>> On 30/11/16 22:37, Peter Eisentraut wrote: >>>> I have taken the libpqwalreceiver refactoring patch and split it

Re: [HACKERS] Logical Replication WIP

2016-12-05 Thread Petr Jelinek
nSlotAcquire() that you did will result in making it impossible to reacquire temporary slot once you switched to different one in the session as the if (active_pid != 0) will always be true for temp slot. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24

Re: [HACKERS] Logical Replication WIP

2016-12-08 Thread Petr Jelinek
On 08/12/16 20:16, Peter Eisentraut wrote: > On 12/6/16 11:58 AM, Peter Eisentraut wrote: >> On 12/5/16 6:24 PM, Petr Jelinek wrote: >>> I think that the removal of changes to ReplicationSlotAcquire() that you >>> did will result in making it impossible to reacquire

Re: [HACKERS] Logical Replication WIP

2016-12-09 Thread Petr Jelinek
nchronous-commit-control-for-logical--v11.patch # PJ > > Could (one of) you give me a hint? > I just sent in a rebased patch that includes all of it. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Service

[HACKERS] snapbuild woes

2016-12-10 Thread Petr Jelinek
y-snapshot-builds.patch changes this behavior so that we don't make snapshots for transactions that we seen wholly and know that they didn't make catalog changes even if we didn't reach consistency yet. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Developmen

Re: [HACKERS] snapbuild woes

2016-12-10 Thread Petr Jelinek
On 10/12/16 23:10, Petr Jelinek wrote: > > The attached 0002-Skip-unnecessary-snapshot-builds.patch changes this > behavior so that we don't make snapshots for transactions that we seen > wholly and know that they didn't make catalog changes even if we didn't >

Re: [HACKERS] PATCH: two slab-like memory allocators

2016-12-11 Thread Petr Jelinek
On 01/12/16 03:26, Tomas Vondra wrote: > > Dne 11/27/2016 v 11:02 PM Andres Freund napsal(a): >> On 2016-11-27 22:21:49 +0100, Petr Jelinek wrote: >>> On 27/11/16 21:47, Andres Freund wrote: >>>> Hi, >>>> >>>>>> +typedef stru

Re: [HACKERS] snapbuild woes

2016-12-12 Thread Petr Jelinek
On 12/12/16 22:42, Andres Freund wrote: > Hi, > > On 2016-12-10 23:10:19 +0100, Petr Jelinek wrote: >> Hi, >> First one is outright bug, which has to do with how we track running >> transactions. What snapbuild basically does while doing initial snapshot >> is

Re: [HACKERS] snapbuild woes

2016-12-12 Thread Petr Jelinek
On 12/12/16 23:33, Andres Freund wrote: > On 2016-12-12 23:27:30 +0100, Petr Jelinek wrote: >> On 12/12/16 22:42, Andres Freund wrote: >>> Hi, >>> >>> On 2016-12-10 23:10:19 +0100, Petr Jelinek wrote: >>>> Hi, >>>> First on

Re: [HACKERS] Logical Replication WIP

2016-12-12 Thread Petr Jelinek
On 13/12/16 01:33, Andres Freund wrote: > HJi, > > On 2016-12-12 09:18:48 -0500, Peter Eisentraut wrote: >> On 12/8/16 4:10 PM, Petr Jelinek wrote: >>> On 08/12/16 20:16, Peter Eisentraut wrote: >>>> On 12/6/16 11:58 AM, Peter Eisentraut wrote: >>

Re: [HACKERS] PATCH: two slab-like memory allocators

2016-12-12 Thread Petr Jelinek
On 13/12/16 01:45, Tomas Vondra wrote: > On 12/12/2016 11:39 PM, Tomas Vondra wrote: >> On 12/12/2016 05:05 AM, Petr Jelinek wrote: >>> >>> I'd be happy with this patch now (as in committer ready) except that it >>> does have some merge conflicts after th

Re: [HACKERS] Logical Replication WIP

2016-12-12 Thread Petr Jelinek
On 13/12/16 02:41, Andres Freund wrote: > On 2016-12-10 08:48:55 +0100, Petr Jelinek wrote: > >> diff --git a/src/backend/catalog/pg_publication.c >> b/src/backend/catalog/pg_publication.c >> new file mode 100644 >> index 000..e3560b7 >> --- /d

Re: [HACKERS] background sessions

2016-12-12 Thread Petr Jelinek
whatever you need there. > You do if source of your data is already in COPY (or csv) format. I do have similar usecase in logical replication followup patch that I plan to submit to Jan CF, so maybe that will be interesting for this as well. -- Petr Jelinek http://www.2ndQua

Re: [HACKERS] Logical Replication WIP

2016-12-12 Thread Petr Jelinek
On 13/12/16 03:26, Petr Jelinek wrote: > On 13/12/16 02:41, Andres Freund wrote: >> On 2016-12-10 08:48:55 +0100, Petr Jelinek wrote: >> >>> +static List * >>> +OpenTableList(List *tables) >>> +{ >>> + List *relids = NIL; >

Re: [HACKERS] Logical Replication WIP

2016-12-13 Thread Petr Jelinek
s('regression_slot2', NULL, NULL); >> >> Can we actually output something? Right now this doesn't test that >> much... > > This test was added because an earlier version of the patch would crash > on this. > I did improve the test as part of the tests impro

Re: [HACKERS] Logical Replication WIP

2016-12-13 Thread Petr Jelinek
ill problem in terms of the write filtering so the publications still have to reset relcache too. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.or

Re: [HACKERS] Logical Replication WIP

2016-12-13 Thread Petr Jelinek
On 13/12/16 21:42, Peter Eisentraut wrote: > On 12/10/16 2:48 AM, Petr Jelinek wrote: >> Attached new version with your updates and rebased on top of the current >> HEAD (the partitioning patch produced quite a few conflicts). > > I have attached a few more "fixup&

Re: [HACKERS] Logical Replication WIP

2016-12-15 Thread Petr Jelinek
On 13/12/16 01:33, Andres Freund wrote: > > On 2016-12-12 09:18:48 -0500, Peter Eisentraut wrote: >> On 12/8/16 4:10 PM, Petr Jelinek wrote: >>> On 08/12/16 20:16, Peter Eisentraut wrote: >>>> On 12/6/16 11:58 AM, Peter Eisentraut wrote: >>>>>

Re: [HACKERS] Logical Replication WIP

2016-12-15 Thread Petr Jelinek
On 13/12/16 21:42, Peter Eisentraut wrote: > On 12/10/16 2:48 AM, Petr Jelinek wrote: >> Attached new version with your updates and rebased on top of the current >> HEAD (the partitioning patch produced quite a few conflicts). > > I have attached a few more "fixup&

Re: [HACKERS] Logical Replication WIP

2016-12-15 Thread Petr Jelinek
On 15/12/16 13:06, Craig Ringer wrote: > On 15 Dec. 2016 18:19, "Petr Jelinek" <mailto:petr.jeli...@2ndquadrant.com>> wrote: > > On 13/12/16 21:42, Peter Eisentraut wrote: > > On 12/10/16 2:48 AM, Petr Jelinek wrote: > >> Attached new vers

Re: [HACKERS] pg_basebackups and slots

2016-12-16 Thread Petr Jelinek
who benefit from an automatic slot. > I think the issue with slots taking over disk space should be fixed by making it possible to cut off slots when necessary (ie some GUC that would say how much behind slot can be at maximum, with something like 0 being unlimited like it's now)

Re: [HACKERS] Logical Replication WIP

2016-12-18 Thread Petr Jelinek
On 17/12/16 13:37, Erik Rijkers wrote: > On 2016-12-16 13:49, Petr Jelinek wrote: >> >> version 13 of the patch. >> >> 0001-Add-PUBLICATION-catalogs-and-DDL-v13.patch.gz (~32 KB) >> 0002-Add-SUBSCRIPTION-catalog-and-DDL-v13.patch.gz (~28 KB) >> 0003-Define

Re: [HACKERS] Logical Replication WIP

2016-12-18 Thread Petr Jelinek
On 17/12/16 18:34, Steve Singer wrote: > On 12/16/2016 07:49 AM, Petr Jelinek wrote: >> Hi, >> >> attached is version 13 of the patch. >> >> I merged in changes from PeterE. And did following changes: >> - fixed the ownership error messages for both provid

Re: [HACKERS] Logical Replication WIP

2016-12-19 Thread Petr Jelinek
On 19/12/16 08:04, Erik Rijkers wrote: > On 2016-12-18 11:12, Petr Jelinek wrote: > > (now using latest: patchset:) > > 0001-Add-PUBLICATION-catalogs-and-DDL-v14.patch > 0002-Add-SUBSCRIPTION-catalog-and-DDL-v14.patch > 0003-Define-logical-replication-protocol-and-output-p

Re: [HACKERS] Logical replication existing data copy

2016-12-19 Thread Petr Jelinek
On 19/12/16 18:25, Peter Eisentraut wrote: > On 12/19/16 4:30 AM, Petr Jelinek wrote: >> So existing table data can be copied once subscription is created, but >> also new tables can be added and their data copied. This is where the >> REFRESH PUBLICATION comes into

Re: [HACKERS] Logical Replication WIP

2016-12-19 Thread Petr Jelinek
On 19/12/16 15:39, Steve Singer wrote: > On 12/18/2016 09:04 PM, Petr Jelinek wrote: >> On 18/12/16 19:02, Steve Singer wrote: >> >>> pg_dump is also generating warnings >>> >>> pg_dump: [archiver] WARNING: don't know how to set owner for object typ

Re: [HACKERS] Logical decoding on standby

2016-12-19 Thread Petr Jelinek
007 is unfinished as you said in your mail (missing option to specify behavior). And the last one 0009 is the implementation discussed above, which I think needs rework. IMHO 0007 and 0009 should be ultimately merged. I think parts of this could be committed separately and are ready for c

Re: [HACKERS] Logical Replication WIP

2016-12-20 Thread Petr Jelinek
On 20/12/16 10:41, Erik Rijkers wrote: > On 2016-12-20 09:43, Petr Jelinek wrote: > >> Thanks, this was very useful. We had wrong attribute index arithmetics >> in the place where we verify that replica identities match well enough. > > Well, I spent a lot of time on the

Re: [HACKERS] Logical Replication WIP

2016-12-20 Thread Petr Jelinek
On 20/12/16 10:56, Erik Rijkers wrote: > On 2016-12-20 10:48, Petr Jelinek wrote: > > Here is another small thing: > > $ psql -d testdb -p 6972 > psql (10devel_logical_replication_20161220_1008_db80acfc9d50) > Type "help" for help. > > testdb=# dr

Re: [HACKERS] Logical decoding on standby

2016-12-20 Thread Petr Jelinek
On 21/12/16 04:31, Craig Ringer wrote: > On 20 December 2016 at 15:03, Petr Jelinek > wrote: > >> But in 0003 I don't understand following code: >>> + if (endpos != InvalidXLogRecPtr && !do_start_slot) >>> + { >>> +

Re: [HACKERS] Logical decoding on standby

2016-12-20 Thread Petr Jelinek
On 21/12/16 04:06, Craig Ringer wrote: > On 20 December 2016 at 15:03, Petr Jelinek > wrote: > >>> The biggest change in this patch, and the main intrusive part, is that >>> procArray->replication_slot_catalog_xmin is no longer directly used by >>> va

Re: [HACKERS] [PATCH] Fix minor race in commit_ts SLRU truncation vs lookups

2016-12-28 Thread Petr Jelinek
eno %d, xid %u", > + trunc.pageno, trunc.oldestXid); This should probably say oldestXid instead of xid in the text description. About back-patching, I wonder if standby could be modified to move oldestXid based on pageno reverse calculation, but it's going to be

Re: [HACKERS] multivariate statistics (v19)

2016-12-30 Thread Petr Jelinek
plan tree turned out to be way more invasive than I'm > comfortable with (but maybe that'd be OK). > In theory it seems like possibly reasonable approach to me, mainly because mv statistics are user defined objects. I guess we'd have to see at least some PoC to see h

Re: [HACKERS] multivariate statistics (v19)

2016-12-30 Thread Petr Jelinek
relation) > +{ ... > + > +void > +update_mv_stats(Oid mvoid, MVNDistinct ndistinct, > + int2vector *attrs, VacAttrStats **stats) ... > +static double > +ndistinct_for_combination(double totalrows, int numrows, HeapTuple *rows, > +int

Re: [HACKERS] Logical Replication WIP

2017-01-02 Thread Petr Jelinek
On 02/01/17 05:23, Steve Singer wrote: > On 12/30/2016 05:53 AM, Petr Jelinek wrote: >> Hi, >> >> I rebased this for the changes made to inheritance and merged in the >> fixes that I previously sent separately. >> >> >> > > > I'm

Re: [HACKERS] Logical Replication WIP

2017-01-03 Thread Petr Jelinek
things together a bit? > I am actually thinking this should correspond to MAXNUMMESSAGES (4096) as that's the limit on buffer size. I didn't find it the first time around when I was looking for good number. -- Petr Jelinek http://www.2ndQuadrant.com/

Re: [HACKERS] Logical Replication WIP

2017-01-04 Thread Petr Jelinek
ply fine on top of it. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services 0001-Add-PUBLICATION-catalogs-and-DDL-v17.patch.gz Description: application/gzip -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresq

Re: [HACKERS] Re: [COMMITTERS] pgsql: Enable logical slots to follow timeline switches

2016-03-31 Thread Petr Jelinek
gmentation > fault > > in > pgsql.build/src/test/recovery/tmp_check/log/006_logical_decoding_timelines_replica2.log > > Could you have a look at whether you have core dumps from these? If so, > backtraces would be very useful. > The function does following: TransactionId

Re: [HACKERS] Proposal: Generic WAL logical messages

2016-04-03 Thread Petr Jelinek
Hi, I rebased this patch on top of current master as the generic wal commit added some conflicting changes. Also fixed couple of typos in comments and added non ascii message to test. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support

Re: [HACKERS] Timeline following for logical slots

2016-04-04 Thread Petr Jelinek
ion as HA solution for logical decoding because currently there is no way to have reasonable continuation of service when the original master dies. And waiting until we have logical that's on par with physical for failover will take long time as it needs the above. -- Petr Jelinek

Re: [HACKERS] Timeline following for logical slots

2016-04-04 Thread Petr Jelinek
ir status up to the master which would then take them into account when calculating oldest needed catalog xmin and lsn (simple way of doing that would be to add this to feedback protocol and let physical slot to keep the xmin/lsn as well), but that does not mean timeline following isn

Re: [HACKERS] Sequence Access Method WIP

2016-04-05 Thread Petr Jelinek
On 04/04/16 15:53, Fabrízio de Royes Mello wrote: On Thu, Mar 31, 2016 at 9:19 AM, Petr Jelinek mailto:p...@2ndquadrant.com>> wrote: > > Hi, > > new version attached that should fix the issue. It was alignment - honestly don't know what I was thinking using fixed ali

Re: [HACKERS] Timeline following for logical slots

2016-04-05 Thread Petr Jelinek
uat about sequences, and rolls over and dies if a timeline switch happens, I would consider that a huge step forward and I think a lot of other people would, too. I agree with the exception of working HA. I would consider it very sad if we got logical replication in core without having an

Re: [HACKERS] [PATH] Jsonb, insert a new value into an array at arbitrary position

2016-04-05 Thread Petr Jelinek
patch (it will throw an error for an existing key). Is it better now? This seems like reasonable compromise to me. I wonder if the errcode should be ERRCODE_INVALID_PARAMETER_VALUE but don't feel too strongly about that. -- Petr Jelinek http://www.2ndQuadrant.com/

Re: [HACKERS] Proposal: Generic WAL logical messages

2016-04-06 Thread Petr Jelinek
osed above. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services diff --git a/contrib/test_decoding/expected/messages.out b/contrib/test_decoding/expected/messages.out index 70130e9..a5b13c5 100644 --- a/contrib/test_dec

<    4   5   6   7   8   9   10   11   12   >