Re: [HACKERS] Documentation improvements for partitioning

2017-02-27 Thread Petr Jelinek
On 27/02/17 07:59, Amit Langote wrote: > On 2017/02/27 3:18, Petr Jelinek wrote: >> On 24/02/17 07:15, Robert Haas wrote: >>> On Fri, Feb 24, 2017 at 9:53 AM, Simon Riggs wrote: >>>> The good news is that logical replication DOES work with partitioning, >&g

Re: [HACKERS] logical replication access control patches

2017-02-27 Thread Petr Jelinek
u would simply need a user who is considered an > owner for all of the objects which you want to replicate, that doesn't > have to be a *direct* owner or a superuser. > > The tables can have individual roles, with some admin role who is a > member of those other roles, or another

Re: [HACKERS] Restricting maximum keep segments by repslots

2017-02-27 Thread Petr Jelinek
nt >> DETAIL: Some replication slots lose required WAL segnents to continue. > However this is dangerous as logical replication slot does not consider it error when too old LSN is requested so we'd continue replication, hiding data loss. -- Petr Jelinek h

Re: [HACKERS] Logical replication existing data copy

2017-03-01 Thread Petr Jelinek
here is something suspicious in there: > Hmm that didn't really reveal much :( -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make

Re: [HACKERS] logical decoding of two-phase transactions

2017-03-01 Thread Petr Jelinek
OMMIT PREPARED but at that point there is no real difference between replicating it as 2pc or 1pc since the 2pc behavior is for all intents and purposes lost at that point. Works for me. I guess the hard part is knowing if COMMIT PREPARED happened at the time PREPARE is decoded, but I existence of th

Re: [HACKERS] logical decoding of two-phase transactions

2017-03-02 Thread Petr Jelinek
uggest adding new callbacks for 2pc prepare, commit and rollback, > and if the output plugin doesn't set them fall back to the existing > behaviour. Plugins that aren't interested in 2PC (think ETL) should > probably not have to deal with it, we might as well just send them > o

Re: [HACKERS] snapbuild woes

2017-03-02 Thread Petr Jelinek
On 26/02/17 01:43, Petr Jelinek wrote: > On 24/02/17 22:56, Petr Jelinek wrote: >> On 22/02/17 03:05, Petr Jelinek wrote: >>> On 13/12/16 00:38, Petr Jelinek wrote: >>>> On 12/12/16 23:33, Andres Freund wrote: >>>>> On 2016-12-12 23:27:30 +0100, Petr Je

Re: [HACKERS] snapbuild woes

2017-03-02 Thread Petr Jelinek
On 03/03/17 01:53, Erik Rijkers wrote: > On 2017-03-03 01:30, Petr Jelinek wrote: > > With these patches: > > 0001-Use-asynchronous-connect-API-in-libpqwalreceiver.patch > 0002-Fix-after-trigger-execution-in-logical-replication.patch > 0003-Add-RENAME-support

Re: [HACKERS] DROP SUBSCRIPTION and ROLLBACK

2017-03-03 Thread Petr Jelinek
PTION ... CREATE SLOT" as that's afaik how we do it for other commands (and same with DROP). -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.o

Re: [HACKERS] Logical Replication and Character encoding

2017-03-03 Thread Petr Jelinek
in future so it's definitely not an omission. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Logical replication existing data copy

2017-03-03 Thread Petr Jelinek
last couple of weeks into investigation and fixing of various existing bugs in the snapshot builder so it's taking longer than I expected). But the patch for this is trivial anyway. Agreement is the what we need first. -- Petr Jelinek http://www.2ndQuadrant.com/ Po

Re: [HACKERS] [COMMITTERS] pgsql: Use asynchronous connect API in libpqwalreceiver

2017-03-03 Thread Petr Jelinek
the case, the attached should fix it, but I have no way of testing it on windows, I can only say that it still works on my machine so at least it hopefully does not make things worse. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training &am

Re: [HACKERS] Logical Replication and Character encoding

2017-03-06 Thread Petr Jelinek
On 06/03/17 11:06, Kyotaro HORIGUCHI wrote: > At Fri, 3 Mar 2017 21:31:24 -0500, Peter Eisentraut > wrote in > <88397afa-a8ec-8d8a-1c94-94a4795a3...@2ndquadrant.com> >> On 3/3/17 14:51, Petr Jelinek wrote: >>> On 03/03/17 20:37, Peter Eisentraut wrote: >>&g

Re: [HACKERS] Logical replication existing data copy

2017-03-06 Thread Petr Jelinek
On 06/03/17 23:40, Erik Rijkers wrote: > On 2017-03-06 16:10, Erik Rijkers wrote: >> On 2017-03-06 11:27, Petr Jelinek wrote: >>> Hi, >>> >>> updated and rebased version of the patch attached. >>> >> >> I compiled with /only/ this one

[HACKERS] logical replication apply to run with sync commit off by default

2017-03-06 Thread Petr Jelinek
there are conflicts between the two and this helps a bit with testing of copy patch. [1] https://www.postgresql.org/message-id/CA+TgmoY7Lk2YKArcp4O=Qu=xoor8j71mad1oteojawmuje3...@mail.gmail.com -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support

Re: [HACKERS] Logical replication existing data copy

2017-03-08 Thread Petr Jelinek
On 07/03/17 23:30, Erik Rijkers wrote: > On 2017-03-06 11:27, Petr Jelinek wrote: > >> 0001-Reserve-global-xmin-for-create-slot-snasphot-export.patch + >> 0002-Don-t-use-on-disk-snapshots-for-snapshot-export-in-l.patch+ >> 0003-Prevent-snapshot-builder-xmin-from-going-ba

Re: [HACKERS] SQL/JSON in PostgreSQL

2017-03-09 Thread Petr Jelinek
eSQL community can fix. > The original complain was about JSON_VALUE extracting date but I don't understand why there is problem with that, the SQL/JSON defines that behavior. The RETURNING clause there is more or less just shorthand for casting with some advanced options. -- Petr

Re: [HACKERS] Logical replication existing data copy

2017-03-09 Thread Petr Jelinek
On 09/03/17 18:46, Peter Eisentraut wrote: > On 3/6/17 05:27, Petr Jelinek wrote: >> updated and rebased version of the patch attached. > > Some comments on this patch: > > Can we have a better explanation of different snapshot options in > CREATE_REPLICATION_SLOT. We u

Re: [HACKERS] Logical replication existing data copy

2017-03-09 Thread Petr Jelinek
On 09/03/17 18:48, Peter Eisentraut wrote: > On 3/6/17 05:27, Petr Jelinek wrote: >> And lastly I changed the automagic around exporting, not exporting and >> using the snapshot produced by CREATE_REPLICATION_SLOT into explicit >> parameter for the CREATE_REPLICATION_SLOT comm

[HACKERS] Logical replication origin tracking fix

2017-03-09 Thread Petr Jelinek
-- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services 0001-Fix-remote-position-tracking-in-logical-replication.patch Description: binary/octet-stream -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To

Re: [HACKERS] logical replication access control patches

2017-03-14 Thread Petr Jelinek
quot; of tableA and "ownerB" of tableB, then you want role "publishe"r to be able to publish those, so you simply grant it the "ownerA" and "ownerB" roles. Obviously that might is many situations mean that the "publisher" role potentially also gets sweeping privileges to other tables which may not be desirable. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] logical replication access control patches

2017-03-14 Thread Petr Jelinek
On 14/03/17 19:47, Robert Haas wrote: > On Tue, Mar 14, 2017 at 2:41 PM, Petr Jelinek > wrote: >> My understanding of what Shephen is proposing is, you have "ownerA" of >> tableA and "ownerB" of tableB, then you want role "publishe"r to be able

Re: [HACKERS] logical replication access control patches

2017-03-14 Thread Petr Jelinek
On 14/03/17 19:49, Petr Jelinek wrote: > On 14/03/17 19:47, Robert Haas wrote: >> On Tue, Mar 14, 2017 at 2:41 PM, Petr Jelinek >> wrote: >>> My understanding of what Shephen is proposing is, you have "ownerA" of >>> tableA and "ownerB" of tab

Re: [HACKERS] logical replication access control patches

2017-03-14 Thread Petr Jelinek
On 14/03/17 20:09, Robert Haas wrote: > On Tue, Mar 14, 2017 at 2:56 PM, Petr Jelinek > wrote: >> Note that I am not necessarily saying it's better though, just trying to >> explain. It definitely has drawbacks, as in order to grant publish on >> one table you might b

Re: [HACKERS] logical decoding of two-phase transactions

2017-03-15 Thread Petr Jelinek
On 02/03/17 17:34, Petr Jelinek wrote: > On 02/03/17 13:23, Craig Ringer wrote: >> On 2 March 2017 at 16:20, Stas Kelvich wrote: >>> >>>> On 2 Mar 2017, at 11:00, Craig Ringer wrote: >>>> >>>> We already have it, because we just decod

Re: [HACKERS] logical replication launcher crash on buildfarm

2017-03-16 Thread Petr Jelinek
an > issue: > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=culicidae&dt=2017-03-16%2003%3A30%3A02 > Hmm, I tried with EXEC_BACKEND (and with --disable-spinlocks) and it seems to work fine on my two machines. I don't see anything else different on culicidae though. Sadly the b

Re: [HACKERS] logical replication launcher crash on buildfarm

2017-03-16 Thread Petr Jelinek
On 16/03/17 09:44, Andres Freund wrote: > On 2017-03-16 09:40:48 +0100, Petr Jelinek wrote: >> On 16/03/17 04:42, Andres Freund wrote: >>> On 2017-03-15 20:28:33 -0700, Andres Freund wrote: >>>> Hi, >>>> >>>> I just unstuck a bunch of my

Re: [HACKERS] logical replication launcher crash on buildfarm

2017-03-16 Thread Petr Jelinek
On 16/03/17 09:53, Andres Freund wrote: > On 2017-03-16 09:40:48 +0100, Petr Jelinek wrote: >> On 16/03/17 04:42, Andres Freund wrote: >>> On 2017-03-15 20:28:33 -0700, Andres Freund wrote: >>>> Hi, >>>> >>>> I just unstuck a bunch of my

Re: [HACKERS] [COMMITTERS] pgsql: Use asynchronous connect API in libpqwalreceiver

2017-03-16 Thread Petr Jelinek
ows machine and can reproduce the issue. I played around a bit and could not really find a fix other than adding WL_TIMEOUT and short timeout to WaitLatchOrSocket (it does wait a very long time on the WaitLatchOrSocket otherwise before failing). So I wonder if this is the same issue that caused us

Re: [HACKERS] [COMMITTERS] pgsql: Use asynchronous connect API in libpqwalreceiver

2017-03-17 Thread Petr Jelinek
d it indeed solves the issue my windows test machine. Now the coding/placement probably isn't the best (and there are no comments), but maybe somebody will find proper place for this now that we know the cause. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Develop

Re: [HACKERS] [COMMITTERS] pgsql: Use asynchronous connect API in libpqwalreceiver

2017-03-17 Thread Petr Jelinek
On 17/03/17 17:28, Tom Lane wrote: > Petr Jelinek writes: >> Now the documentation for WSAEventSelect says "The FD_WRITE network >> event is handled slightly differently. An FD_WRITE network event is >> recorded when a socket is first connected with a call to the connect

Re: [HACKERS] logical replication apply to run with sync commit off by default

2017-03-18 Thread Petr Jelinek
On 07/03/17 06:23, Petr Jelinek wrote: > Hi, > > there has been discussion at the logical replication initial copy thread > [1] about making apply work with sync commit off by default for > performance reasons and adding option to change that per subscription. > > He

Re: [HACKERS] logical replication access control patches

2017-03-18 Thread Petr Jelinek
ing like SUBSCRIBE would be more fitting for publish side (to which you subscriber) so don't really have a better name. I like that the patches cache the acl result so performance impact should be negligible. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Develo

Re: [HACKERS] background sessions

2017-03-18 Thread Petr Jelinek
is stuck trying to send us a long error > message, we will have an undetected deadlock. That's why > pg_background() puts the string being passed to the worker into the > DSM segment in its entirety, rather than sending it through a shm_mq. > Yeah I think this will need to use

Re: [HACKERS] PoC plpgsql - possibility to force custom or generic plan

2017-03-18 Thread Petr Jelinek
pace code. In namespace since they are used for other thing there would be bit of unnecessary propagation but it's 8bytes per namespace, does not seem all that much. My preference would be to add it to PLpgSQL_stmt_block (unless we plan to add posibility to add pragmas for other loops and othe

Re: [HACKERS] logical decoding of two-phase transactions

2017-03-19 Thread Petr Jelinek
decoded changed catalogs (we already have global knowledge of the first one, and it would be trivial to add the second one as we have local knowledge of that as well). What I think is better strategy than filtering out by xmax would be filtering "in" by xmin though. Meaning that first s

Re: [HACKERS] PoC plpgsql - possibility to force custom or generic plan

2017-03-19 Thread Petr Jelinek
On 19/03/17 12:32, Pavel Stehule wrote: > > > 2017-03-18 19:30 GMT+01:00 Petr Jelinek <mailto:petr.jeli...@2ndquadrant.com>>: > > On 16/03/17 17:15, David Steele wrote: > > On 2/1/17 3:59 PM, Pavel Stehule wrote: > >> Hi > >> &

Re: [HACKERS] Logical decoding on standby

2017-03-19 Thread Petr Jelinek
bh I don't follow what's happening there - comment says about not doing anything, but the code inside the if block are only Asserts. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hacker

Re: [HACKERS] logical decoding of two-phase transactions

2017-03-20 Thread Petr Jelinek
On 20/03/17 09:32, Craig Ringer wrote: > On 19 March 2017 at 21:26, Petr Jelinek wrote: > >> I think only genam would need changes to do two-phase scan for this as >> the catalog scans should ultimately go there. It's going to slow down >> things but we could limi

Re: [HACKERS] logical replication access control patches

2017-03-20 Thread Petr Jelinek
On 20/03/17 13:32, Peter Eisentraut wrote: > On 3/18/17 09:31, Petr Jelinek wrote: >>> 0003 Add USAGE privilege for publications >>> >>> a way to control who can subscribe to a publication >>> >> Hmm IIUC this removes ability of REPLICATION role to subs

Re: [HACKERS] logical replication apply to run with sync commit off by default

2017-03-20 Thread Petr Jelinek
On 18/03/17 13:31, Petr Jelinek wrote: > On 07/03/17 06:23, Petr Jelinek wrote: >> Hi, >> >> there has been discussion at the logical replication initial copy thread >> [1] about making apply work with sync commit off by default for >> performance reasons and

[HACKERS] Built-in binning functions

2014-06-13 Thread Petr Jelinek
18633 -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 5c906f3..ceff2ae 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -920,

[HACKERS] Set new system identifier using pg_resetxlog

2014-06-13 Thread Petr Jelinek
ation setup where you need to easily identify each node (it's used by Bidirectional Replication for example). -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services diff --git a/doc/src/sgml/ref/pg_resetxlog.sgml b/do

Re: [HACKERS] Built-in binning functions

2014-06-17 Thread Petr Jelinek
On 17/06/14 16:15, Robert Haas wrote: On Fri, Jun 13, 2014 at 8:22 PM, Petr Jelinek wrote: here is a patch implementing varwidth_bucket (naming is up for discussion) function which does binning with variable bucket width. The use-cases are same as for width_bucket (=data analytics, mainly

Re: [HACKERS] Set new system identifier using pg_resetxlog

2014-06-17 Thread Petr Jelinek
On 17/06/14 16:18, Robert Haas wrote: On Fri, Jun 13, 2014 at 8:31 PM, Petr Jelinek wrote: attached is a simple patch which makes it possible to change the system identifier of the cluster in pg_control. This is useful for individualization of the instance that is started on top of data

Re: [HACKERS] Atomics hardware support table & supported architectures

2014-06-17 Thread Petr Jelinek
Sustaining Support (same as 8 is in now). -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresq

Re: [HACKERS] Set new system identifier using pg_resetxlog

2014-06-18 Thread Petr Jelinek
re system id before you set it, which is useful for automating some things... -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to

Re: [HACKERS] Atomics hardware support table & supported architectures

2014-06-19 Thread Petr Jelinek
year old 200Mhz (or something) machine and doing something meaningful with it. It's not like we are removing supported platforms from old releases, but this basically means we are going to support some of the obscure almost dead platforms till at least 2020, in some cases longer than t

Re: [HACKERS] review: Built-in binning functions

2014-06-22 Thread Petr Jelinek
onder if it would be acceptable to just create pg_proc entry and point it to generic implementation (that's what I originally had, then I changed pg_proc entry to polymorphic types...) -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Trai

Re: [HACKERS] Set new system identifier using pg_resetxlog

2014-06-26 Thread Petr Jelinek
ystem identifier input to ULONG_MAX-1 and reports error if it's bigger. I also added some additional input validation. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services diff --git a/doc/src/sgml/ref/pg_resetxlog.sgml b/

Re: [Fwd: Re: [HACKERS] proposal: new long psql parameter --on-error-stop]

2014-06-26 Thread Petr Jelinek
Hello, I went through the patch, seems mostly fine, I adjusted some wording, removed the default .pgpass file info since it's not accurate, and replaced couple of phrases with (hopefully) more informative ones. -- Petr Jelinek http://www.2ndQuadrant.com/ Postg

Re: [HACKERS] review: Built-in binning functions

2014-06-26 Thread Petr Jelinek
Here is v2, with fixed documentation and numeric version of the implementation. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index f475458..38330d4 10

Re: [HACKERS] Set new system identifier using pg_resetxlog

2014-06-26 Thread Petr Jelinek
hrow error, which it does now. Also based on Alvaro's comment, I replaced the scanf parsing code with strtoul(l) function. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services diff --git a/doc/src/sgml/ref/pg_reset

Re: [HACKERS] Built-in binning functions

2014-07-16 Thread Petr Jelinek
On 08/07/14 02:14, Tom Lane wrote: Petr Jelinek writes: here is a patch implementing varwidth_bucket (naming is up for discussion) function which does binning with variable bucket width. I didn't see any discussion of the naming question in this thread. I'd like to propose that it

Re: [HACKERS] Set new system identifier using pg_resetxlog

2014-07-18 Thread Petr Jelinek
On 18/07/14 00:41, Andres Freund wrote: On 2014-06-27 00:51:02 +0200, Petr Jelinek wrote: { switch (c) { @@ -227,6 +229,33 @@ main(int argc, char *argv[]) XLogFromFileName(optarg, &minXlogTli, &minX

Re: [HACKERS] Set new system identifier using pg_resetxlog

2014-07-18 Thread Petr Jelinek
On 18/07/14 13:18, Andres Freund wrote: On 2014-07-18 13:08:24 +0200, Petr Jelinek wrote: On 18/07/14 00:41, Andres Freund wrote: Wouldn't it be better to use sscanf()? That should work with large inputs across platforms. Well, sscanf does not do much validation and silently truncate

Re: [HACKERS] Built-in binning functions

2014-07-18 Thread Petr Jelinek
able parameters. Just to clarify, the ~20% performance difference is with separate generic implementation for fixed width types (most of the time seems to be spent in the FunctionCallInvoke part, I even tryed to use sortsupport instead but it does not seem to help much). -

Re: [HACKERS] Built-in binning functions

2014-07-18 Thread Petr Jelinek
t has only support for float8 and numeric, maybe it's enough for this one also... What's your opinion on that? -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hac

Re: [HACKERS] Minmax indexes

2014-08-07 Thread Petr Jelinek
On 07/08/14 16:16, Simon Riggs wrote: A better description would be "block range index" since we are indexing a range of blocks (not just one block). Perhaps a better one would be simply "range index", which we could abbreviate to RIN or BRIN. +1 for block range in

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-21 Thread Petr Jelinek
ul for a lot > of users. Many users can't realistically upgrade using pg_dump, ever. > So they'll be stuck on the release before the one that breaks > compatibility for a very long time. > This is why I like the idea of pluggable storage, if we ever get that it would buy us a

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-21 Thread Petr Jelinek
that breaks >> compatibility for a very long time. > > Right. If we weren't setting tuple and tid bits we could imrpove it > easily in PG 11, but if we use them for a single-change WARM chain for > PG 10, we might need bits that are not available to improve it later. >

Re: [HACKERS] logical replication apply to run with sync commit off by default

2017-03-21 Thread Petr Jelinek
On 21/03/17 18:54, Robert Haas wrote: > On Mon, Mar 20, 2017 at 7:56 PM, Petr Jelinek > wrote: >> On 18/03/17 13:31, Petr Jelinek wrote: >>> On 07/03/17 06:23, Petr Jelinek wrote: >>>> there has been discussion at the logical replication initial copy thread >

Re: [HACKERS] logical replication access control patches

2017-03-22 Thread Petr Jelinek
On 22/03/17 03:38, Peter Eisentraut wrote: > On 3/20/17 15:10, Petr Jelinek wrote: >> Hmm but REPLICATION role can do basebackup/consume wal, so how does >> giving it limited publication access help? Wouldn't we need some >> SUBSCRIPTION role/grant used instead for logic

Re: [HACKERS] Logical replication existing data copy

2017-03-22 Thread Petr Jelinek
-subscription-connect that it dumps subscriptions as CREATE SUBSCRIPTION ... WITH(NOCONNECT). -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make

Re: [HACKERS] Logical replication existing data copy

2017-03-23 Thread Petr Jelinek
ce that patch went in. > Thanks for report. Looks like we don't handle correctly empty result set when fetching table list. Will post fix tomorrow. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Logical replication existing data copy

2017-03-23 Thread Petr Jelinek
shly initdb'ed instances, the master has a size 100 > pgbench schema. > > I'm guessing this is a different bug from the segfault also reported > Yes, I also forgot to check if the table actually exists on subscriber when fetching them in CREATE SUBSCRIPTION (we have check duri

Re: [HACKERS] Logical replication existing data copy

2017-03-24 Thread Petr Jelinek
ht be one reason, see https://www.postgresql.org/message-id/08b7053b-5679-0733-3af7-00b8cde62...@2ndquadrant.com (although it probably needs rebase now) Also don't forget that the snapbuild performance fixes haven't been committed yet so it can take long time to even start. -- Pet

Re: [HACKERS] Logical replication existing data copy

2017-03-24 Thread Petr Jelinek
On 24/03/17 15:05, Peter Eisentraut wrote: > On 3/23/17 19:32, Petr Jelinek wrote: >> Yes, I also forgot to check if the table actually exists on subscriber >> when fetching them in CREATE SUBSCRIPTION (we have check during >> replication but not there). > > I think fo

[HACKERS] Fix slot name change handling for subscriptions

2017-03-24 Thread Petr Jelinek
Hi, ALTER SUBSCRIPTION ... WITH (SLOT NAME = foo) will make the worker dies on error about unexpected subscription changed. It's my oversight in the original logical replication patch-set. Attached patch fixes it to behave same way as other changes to subscription options. -- Petr Je

Re: [HACKERS] Logical replication origin tracking fix

2017-03-24 Thread Petr Jelinek
On 10/03/17 05:59, Petr Jelinek wrote: > Hi, > > while discussing with Craig issues around restarting logical replication > stream related to the patch he posted [1], I realized that we track > wrong origin LSN in the logical replication apply. > > We currently track commi

Re: [HACKERS] logical replication apply to run with sync commit off by default

2017-03-24 Thread Petr Jelinek
On 21/03/17 22:37, Petr Jelinek wrote: > On 21/03/17 18:54, Robert Haas wrote: >> On Mon, Mar 20, 2017 at 7:56 PM, Petr Jelinek >> wrote: >>> On 18/03/17 13:31, Petr Jelinek wrote: >>>> On 07/03/17 06:23, Petr Jelinek wrote: >>>>> there has been

Re: [HACKERS] logical replication launcher crash on buildfarm

2017-03-27 Thread Petr Jelinek
nternalBgWorkers thing that was added with logical replication to handle this in similar fashion how we do fmgrtab. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hac

Re: [HACKERS] logical replication launcher crash on buildfarm

2017-03-27 Thread Petr Jelinek
On 28/03/17 03:31, Petr Jelinek wrote: > On 27/03/17 19:01, Robert Haas wrote: >> On Mon, Mar 27, 2017 at 12:50 PM, Andres Freund wrote: >>> Robert, Petr, either of you planning to fix this (as outlined elsewhere >>> in the thred)? >> >> Oh, I didn't r

Re: [HACKERS] logical replication launcher crash on buildfarm

2017-03-27 Thread Petr Jelinek
ainly better for the back-branches, > because it doesn't cause an ABI break. But I think it would also be > fine for master. > I had something slightly more complex like the attached in mind. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development,

Re: [HACKERS] logical replication launcher crash on buildfarm

2017-03-28 Thread Petr Jelinek
On 28/03/17 17:55, Robert Haas wrote: > On Mon, Mar 27, 2017 at 11:20 PM, Petr Jelinek > wrote: >> On 28/03/17 04:46, Robert Haas wrote: >>> On Mon, Mar 27, 2017 at 10:04 PM, Andres Freund wrote: >>>>> Btw now that I look at the code, I guess we'll want t

Re: [HACKERS] PoC plpgsql - possibility to force custom or generic plan

2017-03-28 Thread Petr Jelinek
On 28/03/17 19:43, Pavel Stehule wrote: > Hi > > rebased due last changes in pg_exec.c > Thanks, I went over this and worked over the documentation/comments a bit (attached updated version of the patch with my changes). >From my side this can go to committer. --

Re: [HACKERS] logical replication launcher crash on buildfarm

2017-03-28 Thread Petr Jelinek
On 28/03/17 18:05, Petr Jelinek wrote: > On 28/03/17 17:55, Robert Haas wrote: >> On Mon, Mar 27, 2017 at 11:20 PM, Petr Jelinek >> wrote: >>> On 28/03/17 04:46, Robert Haas wrote: >>>> On Mon, Mar 27, 2017 at 10:04 PM, Andres Freund wrote: >>>>&

Re: [HACKERS] logical replication launcher crash on buildfarm

2017-03-28 Thread Petr Jelinek
On 29/03/17 01:29, Petr Jelinek wrote: > On 28/03/17 18:05, Petr Jelinek wrote: >> On 28/03/17 17:55, Robert Haas wrote: >>> On Mon, Mar 27, 2017 at 11:20 PM, Petr Jelinek >>> wrote: >>>> On 28/03/17 04:46, Robert Haas wrote: >>>>> On Mon,

Re: [HACKERS] Logical replication existing data copy

2017-03-29 Thread Petr Jelinek
ehere nearer the logical replication treatment. > > ( I read about wal_sender_timeout and keepalive ping, perhaps there's > (still) something amiss there? Just a guess, I don't know ) > > As I said, I saw no more failures with the higher 3 minute setting, with >

Re: [HACKERS] logical replication access control patches

2017-03-29 Thread Petr Jelinek
person to create new objects in the database which standard CREATE privilege would allow. So I think it makes sense to push this approach. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-30 Thread Petr Jelinek
M tries to solve mostly on timestamp or boolean values and sometimes counters so it would still be helpful to quite a lot of people even if we didn't do TOAST and compressed values in v1. It's not like not doing WARM sometimes is somehow terrible, we'll just fall back to current be

Re: [HACKERS] Schedule and Release Management Team for PG10

2017-03-30 Thread Petr Jelinek
d of the current CF. > +1, the conference makes it bit inconvenient to have freeze on exactly 31st. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.or

Re: [HACKERS] Somebody has not thought through subscription locking considerations

2017-03-30 Thread Petr Jelinek
ULL of pg_class would touch the pg_subscription_rel though, I'll have to dig into that. I can see that locking for example pg_trigger or pg_depend in ShareRowExclusiveLock will also block VACUUM FULL on pg_class (and other related tables like pg_attribute). So maybe we just need to be careful

Re: [HACKERS] Somebody has not thought through subscription locking considerations

2017-03-31 Thread Petr Jelinek
On 31/03/17 19:35, Tom Lane wrote: > Masahiko Sawada writes: >> On Fri, Mar 31, 2017 at 9:53 AM, Petr Jelinek >> wrote: >>> On 30/03/17 07:25, Tom Lane wrote: >>>> I await with interest an explanation of what "VACUUM FULL pg_class" is >>

Re: [HACKERS] Somebody has not thought through subscription locking considerations

2017-03-31 Thread Petr Jelinek
On 31/03/17 20:23, Tom Lane wrote: > Petr Jelinek writes: >> On 31/03/17 19:35, Tom Lane wrote: >>> At the very least, it would be a good idea to exclude the system >>> catalogs from logical replication, wouldn't it? > >> They are excluded. > >

Re: [HACKERS] Somebody has not thought through subscription locking considerations

2017-03-31 Thread Petr Jelinek
On 31/03/17 21:00, Tom Lane wrote: > Petr Jelinek writes: >> On 31/03/17 20:23, Tom Lane wrote: >>> No, the problematic part is that there is any heap_open happening at >>> all. That open could very easily result in a recursive attempt to read >>> pg_clas

Re: [HACKERS] Somebody has not thought through subscription locking considerations

2017-03-31 Thread Petr Jelinek
On 01/04/17 00:52, Tom Lane wrote: > Petr Jelinek writes: >> On 31/03/17 21:00, Tom Lane wrote: >>> Looking at dependency info isn't going to fix this, it only moves the >>> unsafe catalog access somewhere else (ie pg_depend instead of >>> pg_subscriptio

Re: [HACKERS] Somebody has not thought through subscription locking considerations

2017-03-31 Thread Petr Jelinek
On 01/04/17 01:20, Tom Lane wrote: > Petr Jelinek writes: >> But the pg_subscription_rel is also not accessed on heap_open, the >> problematic code is called from heap_drop_with_catalog. And VACUUM FULL >> pg_class calls heap_drop_with_catalog() when doing the heap swap (a

Re: [HACKERS] Somebody has not thought through subscription locking considerations

2017-03-31 Thread Petr Jelinek
On 01/04/17 01:57, Petr Jelinek wrote: > On 01/04/17 01:20, Tom Lane wrote: >> Petr Jelinek writes: >>> But the pg_subscription_rel is also not accessed on heap_open, the >>> problematic code is called from heap_drop_with_catalog. And VACUUM FULL >>> pg_class

Re: [HACKERS] Somebody has not thought through subscription locking considerations

2017-03-31 Thread Petr Jelinek
On 01/04/17 01:57, Petr Jelinek wrote: > On 01/04/17 01:20, Tom Lane wrote: >> Petr Jelinek writes: >>> But the pg_subscription_rel is also not accessed on heap_open, the >>> problematic code is called from heap_drop_with_catalog. And VACUUM FULL >>> pg_class

Re: [HACKERS] logical replication launcher crash on buildfarm

2017-03-31 Thread Petr Jelinek
On 31/03/17 15:42, Robert Haas wrote: > On Tue, Mar 28, 2017 at 7:39 PM, Petr Jelinek > wrote: >> Sigh, forgot git add for the docs, so one more try... > > +if (strncmp(worker->bgw_library_name, "postgres", 8) != 0) > +return NULL; > > I th

Re: [HACKERS] logical replication launcher crash on buildfarm

2017-03-31 Thread Petr Jelinek
On 01/04/17 02:53, Robert Haas wrote: > On Fri, Mar 31, 2017 at 8:32 PM, Petr Jelinek > wrote: >> Right, changed to BGW_MAXLEN. > > Hmm, I don't know if there's any good reason not to just use strcmp(), > but sure, OK. Committed and back-patched. > Hmm culici

Re: [HACKERS] logical replication launcher crash on buildfarm

2017-03-31 Thread Petr Jelinek
On 01/04/17 03:44, Andres Freund wrote: > On 2017-04-01 03:26:01 +0200, Petr Jelinek wrote: >> On 01/04/17 02:53, Robert Haas wrote: >>> On Fri, Mar 31, 2017 at 8:32 PM, Petr Jelinek >>> wrote: >>>> Right, changed to BGW_MAXLEN. >>> >>> Hmm

Re: [HACKERS] logical replication launcher crash on buildfarm

2017-03-31 Thread Petr Jelinek
On 01/04/17 04:19, Andres Freund wrote: > On 2017-03-31 21:30:17 -0400, Robert Haas wrote: >> On Fri, Mar 31, 2017 at 9:26 PM, Petr Jelinek >> wrote: >>>> Hmm, I don't know if there's any good reason not to just use strcmp(), >>>> but sure, OK.

Re: [HACKERS] PATCH: two slab-like memory allocators

2017-04-02 Thread Petr Jelinek
Hi, this patch is marked as committed in CF application but the second part (generational allocator) was AFAICS never committed. Does anybody plan to push this forward? -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Serv

Re: [HACKERS] logical replication launcher crash on buildfarm

2017-04-04 Thread Petr Jelinek
x27;s better to share what I have so far. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services diff --git a/src/backend/access/transam/README.parallel b/src/backend/access/transam/README.parallel index db9ac3d..b360887 10064

Re: [HACKERS] Quorum commit for multiple synchronous replication.

2017-04-05 Thread Petr Jelinek
many new features including quorum-based syncrep. >> Then if many of them complain about (1) and (3), we can change the code >> at that timing. So we need more time that users can try the feature. > > I've moved (1) to a new section for things to revisit during beta. If

Re: [HACKERS] PoC plpgsql - possibility to force custom or generic plan

2017-04-06 Thread Petr Jelinek
ncache at user level. > The heuristics it uses certainly need work, That's an understatement, there are thousands of plpgsql functions in large installations of PostgreSQL which use EXECUTE everywhere just to avoid current behavior (and that's just what I've seen). --

Re: [HACKERS] subscription worker doesn't start immediately on eabled

2017-04-06 Thread Petr Jelinek
on't be a matter. > Makes sense, I think this got lost in all the refactoring, thanks. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers

Re: [HACKERS] Interval for launching the table sync worker

2017-04-06 Thread Petr Jelinek
good solution, you'd lose replication on every network glitch, upstream server restart, etc. >>> >>>>> Interval can reduce >>>>> the frequence of reconnecting, but I think that walreciever >>>>> should refrain from reconnecting on unrecoverable(or repeating) >>>>> error in walsender. >>>>> >>>> >>>> Yeah, that's also considerable issue. >>> >>> But not to do now. >> > > I've added this as an open item, and sent a patch for this. > I am not exactly sure what's the open item from this thread. To use the wal_retrieve_interval to limit table sync restarts? -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

<    1   2   3   4   5   6   7   8   9   10   >