Re: [HACKERS] [PATCH] Transaction traceability - txid_status(bigint)

2016-08-23 Thread Petr Jelinek
On 23/08/16 11:27, Craig Ringer wrote: On 23 Aug 2016 16:02, "Petr Jelinek" mailto:p...@2ndquadrant.com>> wrote: On 23/08/16 02:55, Craig Ringer wrote: On 23 August 2016 at 01:03, Robert Haas <mailto:robertmh...@gmail.com> <mailto:robertmh...@gmail.com <

Re: [HACKERS] Logical decoding of sequence advances, part II

2016-08-23 Thread Petr Jelinek
r than trying to make the nextval() operation sometimes transactional and other times non-transactional. Well, that's what Craig is trying to do by tracking if the transactional change has happend on a sequence in current transaction, no? -- Petr Jelinek http://www.2ndQu

Re: [HACKERS] Renaming of pg_xlog and pg_clog

2016-08-29 Thread Petr Jelinek
rans (don't care really which one) And +1 for renaming pg_logical to something more reasonable. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@

Re: [HACKERS] pg_sequence catalog

2016-08-31 Thread Petr Jelinek
le built-in logical replication functionality. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] autonomous transactions

2016-08-31 Thread Petr Jelinek
be viable. Caches alone. Peter will be able to explain more, I'm sure. I can also see some advantages in bgworker approach. For example it could be used for "fire and forget" type of interface in the future, where you return as soon as you send exec and don't care

Re: [HACKERS] Logical decoding slots can go backwards when used from SQL, docs are wrong

2016-09-02 Thread Petr Jelinek
ink it's worth the bother, since in the end callers have to be prepared for repeated data on crash anyway. Correct me if I am wrong but I think the only situation where it would matter is on server that restarts often or crashes often (as the logical decoding then has to do the wo

Re: [HACKERS] [PATCH] Transaction traceability - txid_status(bigint)

2016-09-02 Thread Petr Jelinek
ussed upthread with 'xid' exposed to users. You could put it to txid.c where all the other txid stuff is in? -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-ha

Re: [HACKERS] Logical Replication WIP

2016-09-03 Thread Petr Jelinek
ROLE, but I am open to changes. I also found ALTER PUBLICATION FOR TABLE / FOR ALL TABLES confusing. Maybe that should be SET TABLE or something. Yeah I am not sure what is the best option there. SET was also what I was thinking but then it does not map well to the CREATE PUBLICATI

Re: [HACKERS] Logical Replication WIP

2016-09-03 Thread Petr Jelinek
On 03/09/16 18:04, Tom Lane wrote: Petr Jelinek writes: On 02/09/16 22:57, Peter Eisentraut wrote: The new system catalog pg_publication_rel has columns pubid, relid, and does not use the customary column name prefixes. Maybe that is OK here. I can't actually think of a naming scheme

Re: [HACKERS] Logical decoding slots can go backwards when used from SQL, docs are wrong

2016-09-05 Thread Petr Jelinek
LSN. But it's not convenient or easy for SQL callers to extract the last commit LSN received from the last call to pass it to the next one, so this isn't simple, and is probably best tackled as part of the SQL interface API change Petr and Andres discussed in this thread. It isn't? I

Re: [HACKERS] Proposal for changes to recovery.conf API

2016-09-06 Thread Petr Jelinek
hen it was in recovery.conf). We support one recovery_target at a time so it would make sense to have single config option for it IMHO. So +1 on the recovery_target = 'xid:xxx' idea. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support,

Re: [HACKERS] Logical Replication WIP

2016-09-06 Thread Petr Jelinek
On 05/09/16 21:58, Steve Singer wrote: On 08/31/2016 04:51 PM, Petr Jelinek wrote: Hi, and one more version with bug fixes, improved code docs and couple more tests, some general cleanup and also rebased on current master for the start of CF. To get the 'subscription' TAP tests

Re: [HACKERS] Logical Replication WIP

2016-09-06 Thread Petr Jelinek
On 05/09/16 23:35, Steve Singer wrote: On 09/05/2016 03:58 PM, Steve Singer wrote: On 08/31/2016 04:51 PM, Petr Jelinek wrote: Hi, and one more version with bug fixes, improved code docs and couple more tests, some general cleanup and also rebased on current master for the start of CF

Re: [HACKERS] Proposal for changes to recovery.conf API

2016-09-06 Thread Petr Jelinek
On 06/09/16 13:52, David Steele wrote: On 9/6/16 4:56 AM, Petr Jelinek wrote: On 06/09/16 07:18, Abhijit Menon-Sen wrote: Do we want something like this (easy to implement and document, perhaps not especially convenient to use): recovery_target = 'xid' # or 'ti

Re: [HACKERS] Logical Replication WIP

2016-09-06 Thread Petr Jelinek
On 01/09/16 08:29, Erik Rijkers wrote: On 2016-09-01 01:04, Erik Rijkers wrote: On 2016-08-31 22:51, Petr Jelinek wrote: Here are some small changes to logical-replication.sgml ... and other .sgml files. Thanks I'll integrate these into next iteration of the patch, -- Petr Je

Re: [HACKERS] Logical Replication WIP

2016-09-06 Thread Petr Jelinek
On 06/09/16 20:14, Peter Eisentraut wrote: On 9/3/16 5:14 AM, Petr Jelinek wrote: What are the BKI_ROWTYPE_OID assignments for? Are they necessary here? (Maybe this was just copied from pg_subscription?) Yes they are. Please explain/document why. It does not match other catalogs, which

Re: [HACKERS] Logical Replication WIP

2016-09-07 Thread Petr Jelinek
formed ideas about this, though. Yes it's potential issue, don't have good solution for it either. It's loosely coupled system so we can't have 100% control over everything. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Supp

Re: [HACKERS] Logical Replication WIP

2016-09-08 Thread Petr Jelinek
Hi, On 07/09/16 14:10, Erik Rijkers wrote: On 2016-08-31 22:51, Petr Jelinek wrote: and one more version with bug fixes, improved code docs and couple I am not able to get the replication to work. Would you (or anyone) be so kind to point out what I am doing wrong? Patches applied

Re: [HACKERS] Quorum commit for multiple synchronous replication.

2016-09-09 Thread Petr Jelinek
is going to be GA soon. Indeed. I'll vote for pulling a fast one on 9.6 for this. +1 -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@post

Re: [HACKERS] Logical Replication WIP

2016-09-12 Thread Petr Jelinek
ages using pq_beginmessage()/pq_endmessage(), and we're not using pq_getmsgend() when reading. The "proper" way to do this is probably to define a custom set of PQcommMethods. (low priority) If we change that, I'd probably rather go with direct use of StringInfo functions.

Re: [HACKERS] Logical Replication WIP

2016-09-12 Thread Petr Jelinek
On 12/09/16 21:54, Andres Freund wrote: On 2016-09-12 21:47:08 +0200, Petr Jelinek wrote: On 09/09/16 06:33, Peter Eisentraut wrote: The start_replication option pg_version option is not documented and not used in any later patch. We can probably do without it and just rely on the protocol

Re: [HACKERS] Logical Replication WIP

2016-09-12 Thread Petr Jelinek
On 12/09/16 22:21, Andres Freund wrote: On 2016-09-12 21:57:39 +0200, Petr Jelinek wrote: On 12/09/16 21:54, Andres Freund wrote: On 2016-09-12 21:47:08 +0200, Petr Jelinek wrote: On 09/09/16 06:33, Peter Eisentraut wrote: The start_replication option pg_version option is not documented

Re: [HACKERS] Logical Replication WIP

2016-09-13 Thread Petr Jelinek
On 13/09/16 02:55, Craig Ringer wrote: On 13 September 2016 at 06:03, Petr Jelinek wrote: Oh sure, I don't see that as big problem, the TupleData already contains type of the data it sends (to distinguish between nulls and text data) so that's mostly about adding some different type

Re: [HACKERS] autonomous transactions

2016-09-13 Thread Petr Jelinek
ght be better. It's definitely useful functionality but the naming is clearly contentious. It won't stop people using it for same use-cases as autonomous transactions though (which is fine). -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 2

Re: [HACKERS] Logical Replication WIP

2016-09-14 Thread Petr Jelinek
On 14/09/16 00:48, Andres Freund wrote: First read through the current version. Hence no real architectural comments. Hi, Thanks for looking! On 2016-09-09 00:59:26 +0200, Petr Jelinek wrote: diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c

Re: [HACKERS] Logical Replication WIP

2016-09-14 Thread Petr Jelinek
reinvent new versions of these that perform better but also behave correctly (in terms of triggers and stuff for example). -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-

Re: [HACKERS] Logical Replication WIP

2016-09-14 Thread Petr Jelinek
On 14/09/16 18:21, Andres Freund wrote: (continuing, uh, a bit happier) On 2016-09-09 00:59:26 +0200, Petr Jelinek wrote: +/* + * Relcache invalidation callback for our relation map cache. + */ +static void +logicalreprelmap_invalidate_cb(Datum arg, Oid reloid

Re: [HACKERS] Logical Replication WIP

2016-09-14 Thread Petr Jelinek
On 14/09/16 21:53, Andres Freund wrote: Hi, On 2016-09-14 21:17:42 +0200, Petr Jelinek wrote: +/* + * Gather Relations based o provided by RangeVar list. + * The gathered tables are locked in access share lock mode. + */ Why access share? Shouldn't we make this ShareUpdateExclusi

Re: [HACKERS] Rename max_parallel_degree?

2016-09-20 Thread Petr Jelinek
ason for wanting is that I suspect there will be a similar > desire to limit the number of replication workers at some point. > Yes there definitely is desire to use this for replication workers as well. The logical replication launcher currently handles the limit by itself but I'

Re: [HACKERS] Logical Replication WIP

2016-09-21 Thread Petr Jelinek
though because the slot creation needs snapshot and it will wait forever for the current query to finish. So it will never really work. The hanging query is now abortable though. Question is if doing the logical snapshot is really required since we don't really use the snapshot for anythin

Re: [HACKERS] Quorum commit for multiple synchronous replication.

2016-09-21 Thread Petr Jelinek
[1]: >> https://www.postgresql.org/message-id/CAB7nPqRDvJn18e54ccNpOP1A2_iUN6-iU=4njgmmgiagvcs...@mail.gmail.com > > I misunderstood your intent, then. But I still stand by what I did > understand, namely that 'k (...)' should mean 'any k (...)'. It's m

Re: [HACKERS] ICU integration

2016-09-21 Thread Petr Jelinek
there might be check constraints affected, etc. If nothing else it deserves explanation in the docs. The icu conversion functions could use some docs. I wonder if some of the pieces of the code with following pattern: if (mylocale->provider == COLLPROVIDER_ICU) do lot's of ic

Re: [HACKERS] Logical Replication WIP

2016-09-23 Thread Petr Jelinek
(The term > "cache" is perhaps inappropriate for LogicalRepRelMap, because it > implies that the value can be gotten from elsewhere if it's not in the > cache. In this case it's really session state that cannot be recovered > easily.) > Yeah I have differ

Re: [HACKERS] PATCH: two slab-like memory allocators

2016-09-25 Thread Petr Jelinek
simple wrapper it looks mostly ok to me. The only issue I have here is that I am not quite sure about those FIXME functions (Free, Realloc, GetChunkSpace). It's slightly weird to call to mcxt but I guess the alternative there is to error out so this is probably preferable. Would want to hear other opinions here. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] PATCH: two slab-like memory allocators

2016-09-25 Thread Petr Jelinek
On 25/09/16 22:17, Tomas Vondra wrote: > On 09/25/2016 08:48 PM, Petr Jelinek wrote: > >> Slab: >> In general it seems understandable, the initial description helps to >> understand what's happening well enough. >> >> One thing I don't understand ho

Re: [HACKERS] [sqlsmith] stuck spinlock in pg_stat_get_wal_receiver after OOM

2017-10-03 Thread Petr Jelinek
andardize on the procLatch. If we indeed wanted to change the decision about this I think we can simply give latch as a parameter to libpqrcv_connect and store it in the WalReceiverConn struct. The connection does not need to live past the latch (although it currently does, but that's just

Re: [HACKERS] [sqlsmith] stuck spinlock in pg_stat_get_wal_receiver after OOM

2017-10-03 Thread Petr Jelinek
On 03/10/17 22:01, Thomas Munro wrote: > On Wed, Oct 4, 2017 at 7:32 AM, Petr Jelinek > wrote: >> On 03/10/17 19:44, Tom Lane wrote: >>> I wrote: >>>>> So that's trouble waiting to happen, for sure. At the very least we >>>>> need to do a s

Re: [HACKERS] Issues with logical replication

2017-10-03 Thread Petr Jelinek
On 02/10/17 18:59, Petr Jelinek wrote: >> >> Now fix the trigger function: >> CREATE OR REPLACE FUNCTION replication_trigger_proc() RETURNS TRIGGER AS $$ >> BEGIN >>   RETURN NEW; >> END $$ LANGUAGE plpgsql; >> >> And manually perform at master tw

Re: [HACKERS] Issue with logical replication: MyPgXact->xmin already is valid

2017-10-06 Thread Petr Jelinek
I wonder if it is normal situation or something goes wrong? > Hi, no it's not normal situation, it seems you are doing something that assigns xid before you run the CREATE_REPLICATION_SLOT command on that connection. -- Petr Jelinek http://www.2ndQuadrant.com/ P

Re: [HACKERS] Discussion on missing optimizations

2017-10-07 Thread Petr Jelinek
'll just waste equal() call and if it is there the optimization seems worth it as it will lead to orders of magnitude better estimation in many cases. So I wrote prototype of achieving this optimization and it seems to be really quite simple code-wise (see attached). I did only a limit

Re: [HACKERS] Issue with logical replication: MyPgXact->xmin already is valid

2017-10-07 Thread Petr Jelinek
On 06/10/17 16:46, Konstantin Knizhnik wrote: > > > On 06.10.2017 15:29, Petr Jelinek wrote: >> On 06/10/17 12:16, Konstantin Knizhnik wrote: >>> When creating logical replication slots we quite often get the following >>> error: >>> >>>

Re: [HACKERS] Issue with logical replication: MyPgXact->xmin already is valid

2017-10-07 Thread Petr Jelinek
On 07/10/17 18:23, Konstantin Knizhnik wrote: > On 10/07/2017 04:26 PM, Petr Jelinek wrote: >> >> Hmm so you start transaction (you have to when running >> CREATE_REPLICATION_SLOT with USE_SNAPSHOT parameter). And while the slot >> is being created the config is reloade

Re: [HACKERS] Discussion on missing optimizations

2017-10-07 Thread Petr Jelinek
On 07/10/17 18:15, Tom Lane wrote: > Petr Jelinek writes: >> On 07/10/17 04:19, Tom Lane wrote: >>> (edit: a few minutes later, I seem to remember that equivclass.c has >>> to do something special with the X=X case, so maybe it could do >>> something else speci

Re: [HACKERS] Discussion on missing optimizations

2017-10-07 Thread Petr Jelinek
On 07/10/17 19:59, Tom Lane wrote: > Petr Jelinek writes: >> On 07/10/17 18:15, Tom Lane wrote: >>> No, I'm afraid you didn't read that comment closely enough. This will >>> flat out fail for cases like "select ... where x=x order by x", because &g

[HACKERS] Re: [BUGS] 10.0: Logical replication doesn't execute BEFORE UPDATE OF trigger

2017-10-10 Thread Petr Jelinek
of this issue is that the apply worker doesn't set >> updatedCols of RangeTblEntry when applying updates. So TriggerEnabled >> always ends up with false. I'll make a patch and submit. >> > > Attached patch store the updated columns bitmap set to RangeTblEntry. >

[HACKERS] Re: [BUGS] 10.0: Logical replication doesn't execute BEFORE UPDATE OF trigger

2017-10-10 Thread Petr Jelinek
specified to be triggered for columns listed in the statement, not for what has been actually changed by the execution) I don't really see the difference between this and statement triggers except that statement trigger behavior is documented. I understand that it's somewhat confusin

Re: [HACKERS] Help required to debug pg_repack breaking logical replication

2017-10-10 Thread Petr Jelinek
_repack/issues/135 >> - https://github.com/2ndQuadrant/pglogical/issues/113 > > Yeah, I was going to say I've seen reports of this with pglogical, but > I see you've linked to them. > > I haven't had a chance to look into it though, and haven't had a >

Re: [HACKERS] Fix a typo in execReplication.c

2017-10-12 Thread Petr Jelinek
her copy-pasto. > > Thank you! > I think you're right. It should be trig_delete_before_row and be > back-patched to 10. Attached patch fixes it. > Thanks for the patch, looks correct to me. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreS

[HACKERS] Subscriber resets additional columns to NULL on UPDATE

2017-10-26 Thread Petr Jelinek
slot_modify_cstrings() so the extra columns will fall through to the else block which sets the value to NULL. Attached patch fixes it and adds couple of tests for this scenario. This is rather serious issue so it would be good if we could get it fixed in 10.1. -- Petr Jelinek http://www

Re: [HACKERS] Walsender timeouts and large transactions

2017-11-01 Thread Petr Jelinek
e is specific to logical decoding walsender interface so it only happens for logical decoding/replication (which means it should be backported all the way to 9.4). The physical one These two issues happen quite normally in the wild as all we need is big data load in single transaction, or upda

Re: [HACKERS] Walsender timeouts and large transactions

2017-11-03 Thread Petr Jelinek
Hi, thanks for checking. On 02/11/17 10:00, Robert Haas wrote: > On Wed, Nov 1, 2017 at 8:19 PM, Petr Jelinek > wrote: >> sorry for the delay but I didn't have much time in past weeks to follow >> this thread. > > +TimestampTz now = GetCurrentTimestamp(); &g

Re: [HACKERS] Exclude pg_internal.init from base backup

2017-11-03 Thread Petr Jelinek
> + > + Not specific problem to this patch, but I wonder if it should be made more clear that those files (there are couple above of what you added) are skipped no matter which directory they reside in. -- Petr Jelinek http://www.2ndQuadrant.com/ Pos

Re: [HACKERS] psql casts aspersions on server reliability

2016-09-28 Thread Petr Jelinek
lly doesn't. It's neither helpful nor >> good PR for libpq to guess that the most likely cause of a server >> disconnection is server unreliability. >> >> I have seen actual instances of customers getting upset by this >> message even though the server h

Re: [HACKERS] Handling dropped attributes in pglogical_proto

2016-09-28 Thread Petr Jelinek
should never contain attisdropped attributes. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] PATCH: two slab-like memory allocators

2016-10-04 Thread Petr Jelinek
xported and used by GenSlab directly instead of using child contexts. Then all the calls would go to GenSlab which could decide what to do (and move the chunks between the allocators). But honestly given the usecases for GenSlab, I would at the moment prefer just to have predictable error as it can b

Re: [HACKERS] Declarative partitioning - another take

2016-10-04 Thread Petr Jelinek
told you to document "note that if a > partition being detached is unlogged, it will still be unlogged". > I think this is bit different though, it basically means you are detaching whole branch from the rest of the partitioning tree, not just that one partition. To me that'

Re: [HACKERS] autonomous transactions

2016-10-06 Thread Petr Jelinek
I suggest we commit Background Transactions first and then in a > fairly thin commit, implement Autonomous Transactions on top of it for > now and if we get a better one, switch it over. > +1 to this -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Supp

Re: [HACKERS] autonomous transactions

2016-10-11 Thread Petr Jelinek
ust one exists. > > -1 on implementing both autonomous and background transactions. This > will confuse everyone. > I personally care much more about having background transactions than autonomous ones (as I only ever had use-cases for the background ones) so don't agree

Re: [HACKERS] autonomous transactions

2016-10-11 Thread Petr Jelinek
On 11/10/16 21:54, Merlin Moncure wrote: > On Tue, Oct 11, 2016 at 10:06 AM, Petr Jelinek wrote: >> On 10/10/16 16:44, Merlin Moncure wrote: >>> On Thu, Oct 6, 2016 at 3:53 PM, Simon Riggs wrote: >>>> On 6 October 2016 at 21:27, Robert Haas wrote: >>>&g

Re: [HACKERS] logical replication connection information management

2016-10-14 Thread Petr Jelinek
require superuserness. > Yes, superuser in the beginning, especially for subscriptions as it will be bit harder to do proper checks without loss of performance. The good part is that if we do superuser initially we can always add some GRANT or ROLE later to lift the limitation so we don&#x

Re: [HACKERS] PATCH: two slab-like memory allocators

2016-10-18 Thread Petr Jelinek
ich would use some optimized algorithm for fixed sized objects in our current allocator. The advantage of that would be that we could for example use that for things like ListCell easily (memory management of which I see quite often in profiles). -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Indirect indexes

2016-10-20 Thread Petr Jelinek
er who does not have performance problem by 1% is very different from helping user who has performance problem by 50% even if she needs to think about the solution a bit. WARM can do WARM update 50% of time, indirect index can do HOT update 100% of time (provided the column is not chan

Re: [HACKERS] Indirect indexes

2016-10-20 Thread Petr Jelinek
On 20/10/16 17:24, Bruce Momjian wrote: > On Thu, Oct 20, 2016 at 05:14:51PM +0200, Petr Jelinek wrote: >>> Also, it seems indirect indexes would be useful for indexing columns >>> that are not updated frequently on tables that are updated frequently, >>> and wh

Re: [HACKERS] PATCH: two slab-like memory allocators

2016-10-23 Thread Petr Jelinek
rderbuffer, from what I've seen in practice, I'd prefer proper freeing to 5% performance gain as I seen walsenders taking GBs of memory dues to reoderbuffer allocations that are never properly freed. About your actual patch. I do like both the Slab and the Gen allocators and think t

Re: [HACKERS] [PATCH] Send catalog_xmin separately in hot standby feedback

2016-10-24 Thread Petr Jelinek
tXmin() that > currently don't retain it. I could also convert the current boolean > param ignoreVacuum into a flags argument instead of adding another > boolean. No real preference from me. > I would honestly prefer the change to GetOldestXmin to return the catalog_xmin. It seem

Re: [HACKERS] [PATCH] Logical decoding timeline following take II

2016-10-24 Thread Petr Jelinek
e it should be committed. Yeah as I said above I split out the test harness for this reason. The good thing is that when the followup patches get in the test harness should be easy to removed as the changes are very localized. -- Petr Jelinek http://www.2ndQuadrant.com/ Postg

Re: [HACKERS] [COMMITTERS] pgsql: Remove extra comma at end of enum list

2016-10-24 Thread Petr Jelinek
ng comments and /* for all other uses. We'd >> have to teach pgindent about that, and I dunno how hard that is. > > I'm not sure it's worth the resulting mess. > I think the current commenting style stylistically nicer. -- Petr Jelinek http://w

Re: [HACKERS] Fast Default WIP patch for discussion

2016-10-26 Thread Petr Jelinek
r bigger (or complex) patches or patches where there is code cooperation between multiple people. Wiki page for patch sometimes contains basically what your first email said. BTW it's customary to bottom post, not top post on this list. -- Petr Jelinek http://www.2ndQuadran

Re: [HACKERS] Logical Replication WIP

2016-11-03 Thread Petr Jelinek
On 02/11/16 17:22, Peter Eisentraut wrote: > On 10/24/16 9:22 AM, Petr Jelinek wrote: >> I added one more prerequisite patch (the first one) which adds ephemeral >> slots (or well implements UI on top of the code that was mostly already >> there). The ephemeral slots are diff

Re: [HACKERS] Logical Replication WIP

2016-11-10 Thread Petr Jelinek
ch is not very pretty from api perspective but I don't have better idea here. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Logical Replication WIP

2016-11-11 Thread Petr Jelinek
tCleanup), mainly because it complicates the interaction with ReplicationSlotDropPtr and it seems to me that locking there is not really needed there as other backends will never change active_pid to our backend pid and then the ReplicationSlotDropPtr does exclusive lock when resetting it. --

Re: [HACKERS] Logical Replication WIP

2016-11-11 Thread Petr Jelinek
Probably, I wanted to do that as follow-up patch originally, but looks like it should be in initial version. > I also wonder if we want an easier to > extend form of pubinsert/update/delete (say to add pubddl, pubtruncate, > pub ... without changing the schema). > So like, text a

Re: [HACKERS] Logical Replication WIP

2016-11-11 Thread Petr Jelinek
+ /* > + * We must ignore errors here as that would make it impossible to drop > + * subscription when publisher is down. > + */ > > I'm not convinced. Leaving a slot around without a "record" of it on > the creating side isn't nice e

Re: [HACKERS] Logical Replication WIP

2016-11-11 Thread Petr Jelinek
> and context management. But RelationData appears to favor rich data > structures, so maybe that is fine. > Thanks for these, some of it is result of various rebases that I did (the sync patch makes rebasing bit complicated as it touches everything) and it's easy for me to overlook

Re: [HACKERS] Why PostgreSQL doesn't implement a semi sync replication?

2016-11-11 Thread Petr Jelinek
n problems)" than "we lose data when 1 server dies" and it's also more acceptable to say "we stop answering when we lose 2 servers" but not "we stop answering when we lose 1 server", and semisync replication works for combination of these two. -- Petr Je

Re: [HACKERS] Logical Replication WIP

2016-11-12 Thread Petr Jelinek
On 12/11/16 20:19, Andres Freund wrote: > On 2016-11-10 23:31:27 +0100, Petr Jelinek wrote: >> On 04/11/16 13:15, Andres Freund wrote: >>> >>> /* Prototypes for private functions */ >>> -static bool libpq_select(int timeout_ms); >>>

Re: [HACKERS] Logical Replication WIP

2016-11-14 Thread Petr Jelinek
that up when that happens). And that won't change if we convert this into array. Besides, shouldn't the code then anyway check that we only have expected data in that array otherwise we might miss corruption? So if the main reason for turning this into array is extendability fo

Re: [HACKERS] Logical Replication WIP

2016-11-20 Thread Petr Jelinek
On 13/11/16 23:02, Steve Singer wrote: > On 10/31/2016 06:38 AM, Petr Jelinek wrote: >> On 31/10/16 00:52, Steve Singer wrote: >> There are some fundamental issues with initial sync that need to be >> discussed on list but this one is not known. I'll try to convert this &

Re: [HACKERS] Logical replication existing data copy

2017-02-18 Thread Petr Jelinek
s interesting given that it's not even committed yet :) -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Logical replication existing data copy

2017-02-18 Thread Petr Jelinek
On 18/02/17 14:17, Erik Rijkers wrote: > On 2017-02-16 00:43, Petr Jelinek wrote: >> On 13/02/17 14:51, Erik Rijkers wrote: >>> On 2017-02-11 11:16, Erik Rijkers wrote: >>>> On 2017-02-08 23:25, Petr Jelinek wrote: >>>> >>>>> 0001-

Re: [HACKERS] Provide list of subscriptions and publications in psql's completion

2017-02-18 Thread Petr Jelinek
ceiver, which also sets to > NULL fields for non-superusers and even clobbered values in the > printed connection string for superusers, but pg_subscription fails on > those points. > I am not following here, pg_subscription is currently superuser only catalog, similarly to pg_user_map

Re: [HACKERS] Provide list of subscriptions and publications in psql's completion

2017-02-18 Thread Petr Jelinek
On 19/02/17 00:02, Michael Paquier wrote: > On Sat, Feb 18, 2017 at 11:57 PM, Petr Jelinek > wrote: >> On 15/02/17 05:56, Michael Paquier wrote: >>> I thought that this was correctly clobbered... But... No that's not >>> the case by looking at the co

Re: [HACKERS] Provide list of subscriptions and publications in psql's completion

2017-02-19 Thread Petr Jelinek
ptions will happily use .pgpass for example so it's not like users are forced to put password to catalog (well barring some DBaaS solutions). But I guess it would not hurt to give extra notice in docs about dangers of the various catalogs storing passwords. -- Petr Jelinek h

Re: [HACKERS] Replication vs. float timestamps is a disaster

2017-02-20 Thread Petr Jelinek
d that hard to add a SystemToIntTimestamp() function, > given it only needs to do something if float timestamps are enabled? > It's definitely not hard, we already have IntegerTimestampToTimestampTz() which does the opposite conversion anyway. That being said, I did wonder myself

Re: [HACKERS] Replication vs. float timestamps is a disaster

2017-02-20 Thread Petr Jelinek
On 20/02/17 12:04, Andres Freund wrote: > On 2017-02-20 11:58:12 +0100, Petr Jelinek wrote: >> That being said, I did wonder myself if we should just deprecate float >> timestamps as well. > > I think we need a proper deprecation period for that, given that the > convers

Re: [HACKERS] Provide list of subscriptions and publications in psql's completion

2017-02-21 Thread Petr Jelinek
in CREATE/ALTER SUBSCRIPTION as > this refers to remote objects defined in subconninfo. > - authorize read access to public for all columns of pg_subscription > except subconninfo > Thoughts? > Maybe it would make sense to also have pg_subscriptions view which selects everything but s

Re: [HACKERS] snapbuild woes

2017-02-21 Thread Petr Jelinek
On 13/12/16 00:38, Petr Jelinek wrote: > On 12/12/16 23:33, Andres Freund wrote: >> On 2016-12-12 23:27:30 +0100, Petr Jelinek wrote: >>> On 12/12/16 22:42, Andres Freund wrote: >>>> Hi, >>>> >>>> On 2016-12-10 23:10:19 +0100, Petr Jelinek w

Re: [HACKERS] Logical Replication WIP

2017-02-22 Thread Petr Jelinek
Hi, I updated these patches for current HEAD and removed the string initialization in walsender as Fuji Masao committed similar fix in meantime. I also found typo/thinko in the first patch which is now fixed. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL

Re: [HACKERS] Logical Replication WIP

2017-02-22 Thread Petr Jelinek
On 22/02/17 12:24, Petr Jelinek wrote: > Hi, > > I updated these patches for current HEAD and removed the string > initialization in walsender as Fuji Masao committed similar fix in meantime. > > I also found typo/thinko in the first patch which is now fixed. > And of cou

Re: [HACKERS] snapbuild woes

2017-02-22 Thread Petr Jelinek
On 22/02/17 11:29, Erik Rijkers wrote: > On 2017-02-22 03:05, Petr Jelinek wrote: >> >> So to summarize attached patches: >> 0001 - Fixes performance issue where we build tons of snapshots that we >> don't need which kills CPU. >> >> 0002 - Disable

Re: [HACKERS] Logical replication existing data copy

2017-02-22 Thread Petr Jelinek
On 23/02/17 01:02, Erik Rijkers wrote: > On 2017-02-22 18:13, Erik Rijkers wrote: >> On 2017-02-22 14:48, Erik Rijkers wrote: >>> On 2017-02-22 13:03, Petr Jelinek wrote: >>> >>>> 0001-Skip-unnecessary-snapshot-builds.patch >>>> 0002-Don-t-u

Re: [HACKERS] snapbuild woes

2017-02-24 Thread Petr Jelinek
On 22/02/17 03:05, Petr Jelinek wrote: > On 13/12/16 00:38, Petr Jelinek wrote: >> On 12/12/16 23:33, Andres Freund wrote: >>> On 2016-12-12 23:27:30 +0100, Petr Jelinek wrote: >>>> On 12/12/16 22:42, Andres Freund wrote: >>>>> Hi, >>>>

Re: [HACKERS] Logical replication existing data copy

2017-02-24 Thread Petr Jelinek
On 25/02/17 00:05, Erik Rijkers wrote: > On 2017-02-24 22:58, Petr Jelinek wrote: >> On 23/02/17 01:41, Petr Jelinek wrote: >>> On 23/02/17 01:02, Erik Rijkers wrote: >>>> On 2017-02-22 18:13, Erik Rijkers wrote: >>>>> On 2017-02-22 14:48, Erik Ri

Re: [HACKERS] Logical replication existing data copy

2017-02-24 Thread Petr Jelinek
On 25/02/17 00:39, Erik Rijkers wrote: > On 2017-02-25 00:08, Petr Jelinek wrote: >> >> There is now a lot of fixes for existing code that this patch depends >> on. Hopefully some of the fixes get committed soonish. > > Indeed - could you look over the below list of 8 p

Re: [HACKERS] snapbuild woes

2017-02-25 Thread Petr Jelinek
On 24/02/17 22:56, Petr Jelinek wrote: > On 22/02/17 03:05, Petr Jelinek wrote: >> On 13/12/16 00:38, Petr Jelinek wrote: >>> On 12/12/16 23:33, Andres Freund wrote: >>>> On 2016-12-12 23:27:30 +0100, Petr Jelinek wrote: >>>>> On 12/12/16 22:42, And

Re: [HACKERS] Logical replication existing data copy

2017-02-25 Thread Petr Jelinek
On 25/02/17 09:40, Erik Rijkers wrote: > On 2017-02-25 00:40, Petr Jelinek wrote: > >> 0001-Use-asynchronous-connect-API-in-libpqwalreceiver.patch >> 0002-Fix-after-trigger-execution-in-logical-replication.patch >> 0003-Add-RENAME-support-for-PUBLICATIONs-and-SUBSCRIPTI

Re: [HACKERS] Documentation improvements for partitioning

2017-02-26 Thread Petr Jelinek
hould probably work for different hierarchies when we enable it and I am not quite sure that's doable before start of the CF at this point. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsq

Re: [HACKERS] PUBLICATIONS and pg_dump

2017-02-26 Thread Petr Jelinek
rship was supposed to be dumped only when table filter is not used. I mistakenly made it check for data_only instead of using the selectDumpableObject machinery. Fix attached. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Servic

Re: [HACKERS] Logical replication existing data copy

2017-02-27 Thread Petr Jelinek
6 (from > default 2) but it doesn't help much (at all). > > To be continued... > Thanks, The performance was why in original patch I wanted the apply process to default to synchronous_commit = off as without it the apply performance (due to applying transactions individually and in

Re: [HACKERS] PATCH: two slab-like memory allocators

2017-02-27 Thread Petr Jelinek
e failure doesn't look > integer timestamp related (assert failure is clearly about this, and set > of changed commits *only* include slab related commits). > termite is ppc64 but with 4 byte pointer size according to configure so it might be related to that perhaps? -- Petr Jelin

<    1   2   3   4   5   6   7   8   9   10   >