Re: [Patch] Optimize dropping of relation buffers using dlist

2020-12-23 Thread Amit Kapila
On Wed, Dec 23, 2020 at 10:42 AM Kyotaro Horiguchi wrote: > > At Wed, 23 Dec 2020 04:22:19 +, "tsunakawa.ta...@fujitsu.com" > wrote in > > From: Amit Kapila > > > + /* Get the number of blocks for a relation's fork */ > > > + bl

Re: Single transaction in the tablesync worker?

2020-12-23 Thread Amit Kapila
start decoding from. I think all these should be done in a single transaction. -- With Regards, Amit Kapila.

Re: Single transaction in the tablesync worker?

2020-12-23 Thread Amit Kapila
On Tue, Dec 22, 2020 at 4:58 PM Peter Smith wrote: > > On Mon, Dec 21, 2020 at 11:36 PM Amit Kapila wrote: > > > > On Mon, Dec 21, 2020 at 3:17 PM Peter Smith wrote: > > > > > > On Mon, Dec 21, 2020 at 4:23 PM Amit Kapila > >

Re: Logical decoding without slots: decoding in lockstep with recovery

2020-12-23 Thread Amit Kapila
ubscriber restart? I think you need some way to remember confirmed_flush_lsn and restart_lsn and then need to teach WAL machinery to restart from some previous point. -- With Regards, Amit Kapila.

Re: [HACKERS] logical decoding of two-phase transactions

2020-12-23 Thread Amit Kapila
001-Extend-the-output-plugin-API-to-allow-decoding-o is good to go. So, I am planning to push the first patch (0001*) in next week sometime unless you or someone else has any comments on it. -- With Regards, Amit Kapila.

Re: Parallel Inserts in CREATE TABLE AS

2020-12-23 Thread Amit Kapila
her which is the purpose of this patch. I think this is similar to what the Parallel Insert patch [1] has for a similar plan. [1] - https://commitfest.postgresql.org/31/2844/ -- With Regards, Amit Kapila.

Re: Cannot ship records to subscriber for partition tables using logical replication (publish_via_partition_root=false)

2020-12-23 Thread Amit Kapila
> (1 row) > ``` > > I think it looks strange. > The current behavior of partitioned tables is the same as for regular tables. We don't automatically replicate the newly added tables to the existing publication. So, if you try Alter Subscription my_test_sub Refresh Publication;, it will replicate the newly added partition. -- With Regards, Amit Kapila.

Re: Movement of restart_lsn position movement of logical replication slots is very slow

2020-12-23 Thread Amit Kapila
ding of XLOG_RUNNING_XACTS record. So, it is not clear to me how in your case restart_lsn is getting advanced without decode? I think if you add some elogs in the code to track the values of candidate_restart_valid, confirmed_flush_lsn, and restart_lsn, you might get some clue. -- With Regards, Amit Kapila.

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-12-24 Thread Amit Kapila
re: > Below is test table amount for different shared buffers. Each table size is > 8k, > Table size should be more than 8k to get all this data because 8k means just one block. I guess either it is a typo or some other mistake. -- With Regards, Amit Kapila.

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-12-24 Thread Amit Kapila
duplicate code/check in smgrnblocks. I have changed the order of patches (moved Assert related patch to last because as mentioned earlier, I am not sure if we want to commit it.). We might still have to change the scan threshold value based on your and Tang-San's results. -- With Regards, Ami

Re: Movement of restart_lsn position movement of logical replication slots is very slow

2020-12-24 Thread Amit Kapila
sn required by the oldesttxn. But it is strange that it affects only one of the slots. -- With Regards, Amit Kapila.

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-12-24 Thread Amit Kapila
t for the vacuum test. I think with more number of tables there is a greater chance of variation. We have previously used multiple tables in one of the tests because of the Truncate operation (which uses DropRelFileNodesAllBuffers that takes multiple relations as input) and that is not true for Vacuum operation which I suppose you are testing here. -- With Regards, Amit Kapila.

Re: Parallel Inserts in CREATE TABLE AS

2020-12-24 Thread Amit Kapila
On Fri, Dec 25, 2020 at 9:54 AM Bharath Rupireddy wrote: > > On Fri, Dec 25, 2020 at 7:12 AM vignesh C wrote: > > On Thu, Dec 24, 2020 at 11:29 AM Amit Kapila > > wrote: > > > > > > On Thu, Dec 24, 2020 at 10:25 AM vignesh C wrote: > > > >

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-12-30 Thread Amit Kapila
be reliable because before you run the test via Vacuum command autovacuum would have done that work? -- With Regards, Amit Kapila.

Re: [HACKERS] logical decoding of two-phase transactions

2020-12-30 Thread Amit Kapila
ch-0009 > Thanks, I will look into this. -- With Regards, Amit Kapila.

Re: Added missing copy related data structures to typedefs.list

2020-12-30 Thread Amit Kapila
atches that require changes in typedefs. It is not a norm and I am not sure how much value it adds to do it separately for the missing ones unless you are making changes in the same file they are used and you are facing unrelated diffs due to those missing ones. -- With Regards, Amit Kapila.

Re: [HACKERS] logical decoding of two-phase transactions

2020-12-30 Thread Amit Kapila
On Thu, Dec 31, 2020 at 10:48 AM Amit Kapila wrote: > > On Wed, Dec 30, 2020 at 6:49 PM Amit Kapila wrote: > > > > On Tue, Dec 29, 2020 at 3:15 PM Ajin Cherian wrote: > > > > > > Hi Sawada-san, > > > > > > I think Amit has a plan to commit

Re: faster ETL / bulk data load for heap tables

2021-01-01 Thread Amit Kapila
ent way. -- With Regards, Amit Kapila.

Re: [Patch] Optimize dropping of relation buffers using dlist

2021-01-03 Thread Amit Kapila
an share scripts or exact steps used to measure the time and other can use the same technique and see if we are getting consistent results. -- With Regards, Amit Kapila.

Re: faster ETL / bulk data load for heap tables

2021-01-03 Thread Amit Kapila
On Sat, Jan 2, 2021 at 3:34 PM Luc Vlaming wrote: > > On 02-01-2021 08:36, Amit Kapila wrote: > > On Fri, Jan 1, 2021 at 7:37 PM Luc Vlaming wrote: > >> > >> Hi, > >> > >> In an effort to speed up bulk data loading/transforming I noticed that

Re: [HACKERS] logical decoding of two-phase transactions

2021-01-04 Thread Amit Kapila
On Thu, Dec 31, 2020 at 12:31 PM Ajin Cherian wrote: > > On Thu, Dec 31, 2020 at 4:16 PM Amit Kapila wrote: > > > 3. Merged the doc changes patch after some changes mostly cosmetic. > Some minor comments here: > > v35-0001 - logicaldecoding.sgml > > In the

Re: Single transaction in the tablesync worker?

2021-01-04 Thread Amit Kapila
On Wed, Dec 30, 2020 at 11:51 AM Peter Smith wrote: > > On Wed, Dec 23, 2020 at 8:43 PM Amit Kapila wrote: > > > > 1. > > + * Rarely, the DropSubscription may be issued when a tablesync still > > + * is in SYNCDONE but not yet in READY state. If this happens then &g

Re: Single transaction in the tablesync worker?

2021-01-04 Thread Amit Kapila
On Mon, Jan 4, 2021 at 2:38 PM Amit Kapila wrote: > > Few other comments: > = > Few more comments on v9: == 1. + /* Drop the tablesync slot. */ + { + char *syncslotname = ReplicationSlotNameForTablesync(subid, relid); + + /* + * If the subscription

Re: fix typo in ReorderBufferProcessTXN

2021-01-04 Thread Amit Kapila
. > +1. I will push this in some time. -- With Regards, Amit Kapila.

Re: fix typo in ReorderBufferProcessTXN

2021-01-04 Thread Amit Kapila
On Tue, Jan 5, 2021 at 7:45 AM Amit Kapila wrote: > > On Tue, Jan 5, 2021 at 7:13 AM Masahiko Sawada wrote: > > > > On Tue, Jan 5, 2021 at 9:54 AM Hou, Zhijie > > wrote: > > > > > in contrast to normal the normal catalog during decoding == >> in

Re: Fix typo in comment

2021-01-04 Thread Amit Kapila
On Tue, Jan 5, 2021 at 5:36 AM Peter Smith wrote: > > PSA a trivial patch to correct a comment typo. > LGTM. Pushed! -- With Regards, Amit Kapila.

Track replica origin progress for Rollback Prepared

2021-01-04 Thread Amit Kapila
records when wal_level = logical. This allows logical decoding to send the SAME gid to subscribers across restarts of logical replication. Track relica origin replay progress for 2PC. -- With Regards, Amit Kapila. v1-0001-Track-replication-origin-progress-for-rollbacks.patch Description: Binary

Re: [HACKERS] logical decoding of two-phase transactions

2021-01-04 Thread Amit Kapila
. +step "s2c" { COMMIT; } ... +step "s2prepare" { PREPARE TRANSACTION 'test1'; } +step "s2commit" { COMMIT PREPARED 'test1'; } s2c and s2commit seem to be confusing names as both sounds like doing the same thing. How about changing s2commit to s2cp and s2prepare to s2p? -- With Regards, Amit Kapila.

Re: Track replica origin progress for Rollback Prepared

2021-01-05 Thread Amit Kapila
On Tue, Jan 5, 2021 at 12:32 PM Michael Paquier wrote: > > On Tue, Jan 05, 2021 at 09:35:21AM +0530, Amit Kapila wrote: > > As noted in [1], without this the subscriber might again ask for > > rollback prepared lsn after restart. > > > > Attached patch addresses th

Re: [HACKERS] logical decoding of two-phase transactions

2021-01-05 Thread Amit Kapila
On Tue, Jan 5, 2021 at 2:11 PM Ajin Cherian wrote: > > > I've addressed the above comments and the patch is attached. I've > called it v36-0007. > Thanks, I have pushed this after minor wordsmithing. -- With Regards, Amit Kapila.

Re: Single transaction in the tablesync worker?

2021-01-05 Thread Amit Kapila
On Tue, Jan 5, 2021 at 3:32 PM Peter Smith wrote: > > On Mon, Jan 4, 2021 at 10:48 PM Amit Kapila wrote: > > > > Few more comments on v9: > > == > > 1. > > + /* Drop the tablesync slot. */ > > + { > > + char *syncslotnam

Re: Single transaction in the tablesync worker?

2021-01-05 Thread Amit Kapila
On Wed, Jan 6, 2021 at 4:32 AM Peter Smith wrote: > > On Tue, Jan 5, 2021 at 10:41 PM Amit Kapila wrote: > > > > 1. > > > > + /* Drop the tablesync slot. */ > > > > + { > > > > + char *syncslotname = ReplicationSlotNameForTablesync(su

Re: [HACKERS] logical decoding of two-phase transactions

2021-01-05 Thread Amit Kapila
On Tue, Dec 22, 2020 at 3:31 PM Amit Kapila wrote: > > On Tue, Dec 22, 2020 at 2:51 PM Ajin Cherian wrote: > > > > > Other than this I've noticed a few typos that are not in the patch but > > in the surrounding code. > > logical.c: 1383: Comme

Re: Single transaction in the tablesync worker?

2021-01-05 Thread Amit Kapila
On Tue, Jan 5, 2021 at 3:32 PM Peter Smith wrote: > > On Mon, Jan 4, 2021 at 10:48 PM Amit Kapila wrote: > > > > > 5. Is it possible to write a testcase where we fail (say due to pk > > violation or some other error) after the initial copy is done, then > > remov

Re: When (and whether) should we improve the chapter on parallel query to accommodate parallel data updates?

2021-01-06 Thread Amit Kapila
e improving overall docs for parallel execution as you are suggesting then it can be done separately as well. I think you have a valid point but maybe it would be better to do the improvements you are suggesting once the parallel inserts related work is committed. [1] - https://www.postgresql.org/docs/devel/sql-vacuum.html -- With Regards, Amit Kapila.

Re: Single transaction in the tablesync worker?

2021-01-06 Thread Amit Kapila
lated information is present in pg_dump doc page [1], see from "When dumping logical replication subscriptions ". [1] - https://www.postgresql.org/docs/devel/app-pgdump.html -- With Regards, Amit Kapila.

Re: Track replica origin progress for Rollback Prepared

2021-01-06 Thread Amit Kapila
On Wed, Jan 6, 2021 at 5:18 PM Michael Paquier wrote: > > On Tue, Jan 05, 2021 at 04:24:21PM +0530, Amit Kapila wrote: > > There are already tests [1] in one of the upcoming patches for logical > > decoding of 2PC which covers this code using which I have found this > >

Re: Single transaction in the tablesync worker?

2021-01-06 Thread Amit Kapila
On Wed, Jan 6, 2021 at 3:39 PM Amit Kapila wrote: > > On Wed, Jan 6, 2021 at 2:13 PM Peter Smith wrote: > > > > I think it makes sense. If there can be a race between the tablesync > > re-launching (after error), and the AlterSubscription_refresh removing > &g

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-06 Thread Amit Kapila
e checksums. [1] - https://www.postgresql.org/message-id/20190330192543.GH4719%40development -- With Regards, Amit Kapila.

Re: [Patch] Optimize dropping of relation buffers using dlist

2021-01-07 Thread Amit Kapila
r such cases, we find the buffers by doing lookups in BufMapping table. This improves the performance by more than 100 times in many cases when several small tables (tested with 1000 relations) are truncated and where the server is configured with a large value of shared buffers (greater than 100GB)." -- With Regards, Amit Kapila.

Re: Parallel INSERT (INTO ... SELECT ...)

2021-01-07 Thread Amit Kapila
sion is that the "INSERT INTO ... SELECT > ..." statement consumes XID even if the SELECT eventually does not return any > row. However, if the same query is processed w/o parallelism, the XID is only > allocated if at least one tuple needs to be inserted. > Yeah, that is true but I think this can happen w/o parallelism for updates and deletes where by the time we try to modify the row, it got modified by a concurrent session and the first session will needlessly allocate XID. -- With Regards, Amit Kapila.

Re: [PATCH] Simple progress reporting for COPY command

2021-01-07 Thread Amit Kapila
> How about including a column for command similar to pg_stat_progress_create_index and pg_stat_progress_cluster? It seems that command will be useful in the context of COPY as there are many variants of COPY. -- With Regards, Amit Kapila.

Re: Single transaction in the tablesync worker?

2021-01-07 Thread Amit Kapila
On Fri, Jan 8, 2021 at 7:14 AM Peter Smith wrote: > > On Thu, Jan 7, 2021 at 3:20 PM Amit Kapila wrote: > > > > On Wed, Jan 6, 2021 at 3:39 PM Amit Kapila wrote: > > > > > > On Wed, Jan 6, 2021 at 2:13 PM Peter Smith wrote: > > > > > > >

Re: [PATCH] Simple progress reporting for COPY command

2021-01-07 Thread Amit Kapila
On Thu, Jan 7, 2021 at 7:02 PM Josef Šimánek wrote: > > čt 7. 1. 2021 v 14:08 odesílatel Amit Kapila napsal: > > > > On Thu, Jan 7, 2021 at 3:15 AM Tomas Vondra > > wrote: > > > > > > On 1/5/21 11:02 AM, Josef Šimánek wrote: > > > > I&

Re: [PATCH] Simple progress reporting for COPY command

2021-01-07 Thread Amit Kapila
On Fri, Jan 8, 2021 at 8:42 AM Josef Šimánek wrote: > > pá 8. 1. 2021 v 3:55 odesílatel Amit Kapila napsal: > > > > > > Can't we display the entire COPY command? I checked that > > pg_stat_statements display the query so there shouldn't be a problem &

Re: [PATCH] Simple progress reporting for COPY command

2021-01-07 Thread Amit Kapila
On Fri, Jan 8, 2021 at 9:45 AM Josef Šimánek wrote: > > pá 8. 1. 2021 v 5:03 odesílatel Amit Kapila napsal: > > > > On Fri, Jan 8, 2021 at 8:42 AM Josef Šimánek > > wrote: > > > > > > pá 8. 1. 2021 v 3:55 odesílatel Amit Kapila > > > n

Re: Parallel INSERT (INTO ... SELECT ...)

2021-01-08 Thread Amit Kapila
.c. This sounds worth investigating. > Moreover, ExecInitModifyTable() would no longer need to set the > targetlist. > I am not sure if we need to do anything about ExecInitModifyTable. If we want to unify what setrefs.c does with ExecInitModifyTable, then we can start a separate thread. Thanks for all the reviews. I would like to emphasize what I said earlier in this thread that it is better to first focus on Parallelising Selects for Insert (aka what v11-0001-Enable-parallel-SELECT-for-INSERT-INTO-.-SELECT does) as that in itself is a step towards achieving parallel inserts, doing both 0001 and 0003 at the same time can take much more time as both touches quite intricate parts of the code. -- With Regards, Amit Kapila.

Re: Parallel INSERT (INTO ... SELECT ...)

2021-01-08 Thread Amit Kapila
On Fri, Jan 8, 2021 at 12:21 PM Antonin Houska wrote: > > Amit Kapila wrote: > > > > As an alternative, have you considered allocation of the XID even in > > > parallel > > > mode? I imagine that the first parallel worker that needs the XID for > > &g

Re: Added schema level support for publication.

2021-01-08 Thread Amit Kapila
we do if the user Drops the schema? Do we automatically remove it from the publication? I see some use of such a feature but you haven't described the use case or how did you arrive at the conclusion that it would be quite useful? -- With Regards, Amit Kapila.

Re: Single transaction in the tablesync worker?

2021-01-08 Thread Amit Kapila
ublsn NULL) */ #define SUBREL_STATE_SYNCDONE 's' /* synchronization finished in front of * apply (sublsn set) */ I am not very happy with the new state name SUBREL_STATE_TCOPYDONE as it is quite different from other adjoining state names and somehow not going well with the code. How about SUBREL_STATE_ENDCOPY 'e' or SUBREL_STATE_FINISHEDCOPY 'f'? -- With Regards, Amit Kapila.

Re: [HACKERS] logical decoding of two-phase transactions

2021-01-08 Thread Amit Kapila
On Tue, Jan 5, 2021 at 4:26 PM Amit Kapila wrote: > > On Tue, Jan 5, 2021 at 2:11 PM Ajin Cherian wrote: > > > > > > I've addressed the above comments and the patch is attached. I've > > called it v36-0007. > > > > Thanks, I have pushed this a

Re: [HACKERS] logical decoding of two-phase transactions

2021-01-09 Thread Amit Kapila
On Sat, Jan 9, 2021 at 12:57 PM Amit Kapila wrote: > > On Tue, Jan 5, 2021 at 4:26 PM Amit Kapila wrote: > > > > On Tue, Jan 5, 2021 at 2:11 PM Ajin Cherian wrote: > > > > > > > > > I've addressed the above comments and the patch is attached. I

Re: Single transaction in the tablesync worker?

2021-02-11 Thread Amit Kapila
On Fri, Feb 12, 2021 at 7:18 AM Ajin Cherian wrote: > > On Thu, Feb 11, 2021 at 10:38 PM Amit Kapila wrote: > > > Okay, attached an updated patch with only that change. > > I ran Erik's test suite [1] on this patch overnight and found no > errors. No more comments

Re: Single transaction in the tablesync worker?

2021-02-11 Thread Amit Kapila
On Fri, Feb 12, 2021 at 10:08 AM Ajin Cherian wrote: > > On Fri, Feb 12, 2021 at 2:46 PM Amit Kapila wrote: > > > > > Thanks, I have pushed the patch but getting one failure: > > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=thorntail&dt=2021-02-12%20

Re: repeated decoding of prepared transactions

2021-02-12 Thread Amit Kapila
On Fri, Feb 12, 2021 at 1:10 AM Robert Haas wrote: > > On Thu, Feb 11, 2021 at 5:37 AM Amit Kapila wrote: > > to explain the exact case to you which is not very apparent. The basic > > idea is that we ship/replay all transactions where commit happens > > after the snapsho

Re: [HACKERS] logical decoding of two-phase transactions

2021-02-12 Thread Amit Kapila
as a lot of timing-dependent tests and I am not sure if it is valuable enough at this stage. So, we can ignore cosmetic comments in this patch for now. -- With Regards, Amit Kapila.

Re: logical replication seems broken

2021-02-12 Thread Amit Kapila
dy. perhaps one of you see the light > immediately... > That would be really helpful. -- With Regards, Amit Kapila.

Re: logical replication seems broken

2021-02-13 Thread Amit Kapila
On Fri, Feb 12, 2021 at 10:00 PM wrote: > > > On 02/12/2021 1:51 PM Amit Kapila wrote: > > > > > > On Fri, Feb 12, 2021 at 6:04 PM Erik Rijkers wrote: > > > > > > Hello, > > > > > > I am seeing errors in replication in a test p

Re: repeated decoding of prepared transactions

2021-02-14 Thread Amit Kapila
t, the origin's lsn location is also WAL-logged. That helps us to send the start location for a slot after the restart. As far as I understand this is how it works from the apply-worker side. I am not sure if I am missing something here? -- With Regards, Amit Kapila.

Re: logical replication seems broken

2021-02-15 Thread Amit Kapila
this prior to commit ce0fdbfe97 and we haven't received any complaints so far. What do you think? -- With Regards, Amit Kapila. fix_origin_message_2.patch Description: Binary data

Re: logical replication seems broken

2021-02-15 Thread Amit Kapila
On Mon, Feb 15, 2021 at 7:50 PM vignesh C wrote: > > On Mon, Feb 15, 2021 at 6:14 PM wrote: > > > > > > > On 2021.02.15. 12:31 Amit Kapila wrote: > > > On Mon, Feb 15, 2021 at 11:53 AM vignesh C wrote: > > > > On Sat, Feb 13, 2021 at 5:58 PM Erik

Re: repeated decoding of prepared transactions

2021-02-15 Thread Amit Kapila
On Thu, Feb 11, 2021 at 4:06 PM Amit Kapila wrote: > > On Mon, Feb 8, 2021 at 2:01 PM Markus Wanner > wrote: > > > Now, coming back to the restart case where the prepared transaction > can be sent again by the publisher. I understand yours and others > point that we

Re: [PATCH] pg_hba.conf error messages for logical replication connections

2021-02-16 Thread Amit Kapila
On Tue, Feb 2, 2021 at 1:43 AM Paul Martinez wrote: > > On Fri, Jan 29, 2021 at 8:41 PM Amit Kapila wrote: > > > > Yeah, hints or more details might improve the situation but I am not > > sure we want to add more branching here. Can we write something > > similar

Re: [PATCH] pg_hba.conf error messages for logical replication connections

2021-02-17 Thread Amit Kapila
On Tue, Feb 16, 2021 at 10:40 PM Paul Martinez wrote: > > On Tue, Feb 16, 2021 at 2:22 AM Amit Kapila wrote: > > > > I don't think we need to update the error messages, it makes the code > > a bit difficult to parse without much benefit. How about just adding > &

Re: repeated decoding of prepared transactions

2021-02-18 Thread Amit Kapila
On Tue, Feb 16, 2021 at 9:43 AM Amit Kapila wrote: > > On Thu, Feb 11, 2021 at 4:06 PM Amit Kapila wrote: > > > > On Mon, Feb 8, 2021 at 2:01 PM Markus Wanner > > wrote: > > > > > Now, coming back to the restart case where the prepared transaction &g

Re: [HACKERS] logical decoding of two-phase transactions

2021-02-18 Thread Amit Kapila
that we can process commits if any, and write prepared transactions to file. For prepared transactions, we can take decisions only once the commit prepared for them has arrived. -- With Regards, Amit Kapila.

Re: [PATCH] pg_hba.conf error messages for logical replication connections

2021-02-18 Thread Amit Kapila
On Thu, Feb 18, 2021 at 5:59 AM Euler Taveira wrote: > > On Wed, Feb 17, 2021, at 8:01 AM, Amit Kapila wrote: > > Before fix > ERROR: could not connect to the publisher: connection to server at > "localhost" (::1), port 5432 failed: FATAL: pg_hba.conf rejects >

Re: Parallel INSERT (INTO ... SELECT ...)

2021-02-19 Thread Amit Kapila
003-Enable-parallel-INSERT-and-or-SELECT-for-INSERT-INTO. Because even if we get the first patch that is good enough for some users. What do you think? [1] - https://www.postgresql.org/message-id/CAA4eK1K-cW7svLC2D7DHoGHxdAdg3P37BLgebqBOC2ZLc9a6QQ%40mail.gmail.com -- With Regards, Amit Kapila.

Re: Parallel INSERT (INTO ... SELECT ...)

2021-02-19 Thread Amit Kapila
y. > This could be a problem for analytics use cases where the table is just > inserted and read only afterwards. We could advise the user to run REINDEX > CONCURRENTLY after loading data, but what about tables? > I think here you are talking about the third patch (Parallel Inserts). I guess if one has run inserts parallelly from psql then also similar behavior would have been observed. For tables, it might lead to better results once we have the patch discussed at [1]. Actually, this needs more investigation. [1] - https://www.postgresql.org/message-id/20200508072545.GA9701%40telsasoft.com -- With Regards, Amit Kapila.

Re: repeated decoding of prepared transactions

2021-02-19 Thread Amit Kapila
this assumption. During the initial synchronization, this transaction won't be visible to snapshot and we won't copy it. Then later if we won't decode and send it then the replica will be out of sync. Such a problem won't happen with Ajin's patch. -- With Regards, Amit Kapila.

Re: repeated decoding of prepared transactions

2021-02-19 Thread Amit Kapila
t pg_logical_slot_get_changes is called with two-phase-commit off, > 2nd is called with two-phase-commit on. You can see that the > transaction is not decoded at all. > For this, I am planning to change the semantics such that > two-phase-commit can only be specified while creating the slot using > pg_create_logical_replication_slot() > and not in pg_logical_slot_get_changes, thus preventing > two-phase-commit flag from being toggled between restarts of the > decoder. > +1. -- With Regards, Amit Kapila.

Re: repeated decoding of prepared transactions

2021-02-20 Thread Amit Kapila
On Sat, Feb 20, 2021 at 9:46 AM Amit Kapila wrote: > > On Fri, Feb 19, 2021 at 8:21 AM Ajin Cherian wrote: > > > > Based on this suggestion, I have created a patch on HEAD which now > > does not allow repeated decoding > > of prepared transactions. F

Re: [PATCH] pg_hba.conf error messages for logical replication connections

2021-02-20 Thread Amit Kapila
On Thu, Feb 18, 2021 at 2:42 PM Amit Kapila wrote: > > On Thu, Feb 18, 2021 at 5:59 AM Euler Taveira wrote: > > > > On Wed, Feb 17, 2021, at 8:01 AM, Amit Kapila wrote: > > > > Before fix > > ERROR: could not connect to the publisher: connection to server

Re: [PATCH] Present all committed transaction to the output plugin

2021-02-20 Thread Amit Kapila
www.postgresql.org/message-id/CAMkU%3D1yohp9-dv48FLoSPrMqYEyyS5ZWkaZGD41RJr10xiNo_Q%40mail.gmail.com -- With Regards, Amit Kapila.

Re: repeated decoding of prepared transactions

2021-02-20 Thread Amit Kapila
On Sat, Feb 20, 2021 at 4:25 PM Markus Wanner wrote: > > On 20.02.21 04:38, Amit Kapila wrote: > > I see a problem with this assumption. During the initial > > synchronization, this transaction won't be visible to snapshot and we > > won't copy it. Then later i

Re: repeated decoding of prepared transactions

2021-02-20 Thread Amit Kapila
On Sat, Feb 20, 2021 at 10:26 PM Andres Freund wrote: > > Hi, > > On Fri, Feb 19, 2021, at 19:38, Amit Kapila wrote: > > On Fri, Feb 19, 2021 at 8:23 PM Markus Wanner > > wrote: > > > > > > With that line of thinking, the point in time (or in WAL) of th

Re: repeated decoding of prepared transactions

2021-02-21 Thread Amit Kapila
On Mon, Feb 22, 2021 at 3:56 AM Andres Freund wrote: > > On 2021-02-21 11:32:29 +0530, Amit Kapila wrote: > > Here, I am assuming you are asking to disable 2PC both via > > apply-worker and tablesync worker till the initial sync (aka all > > tables are in SUBREL_STA

Re: repeated decoding of prepared transactions

2021-02-22 Thread Amit Kapila
On Mon, Feb 22, 2021 at 9:39 AM Andres Freund wrote: > > Hi, > > On 2021-02-22 08:22:35 +0530, Amit Kapila wrote: > > On Mon, Feb 22, 2021 at 3:56 AM Andres Freund wrote: > > > > > > On 2021-02-21 11:32:29 +0530, Amit Kapila wrote: > > > > Here,

Re: repeated decoding of prepared transactions

2021-02-22 Thread Amit Kapila
e's no easy way to just use the initial slot. > +1. The solution proposed by Andres seems to be better than other ideas we have discussed so far. -- With Regards, Amit Kapila.

Re: Parallel INSERT (INTO ... SELECT ...)

2021-02-22 Thread Amit Kapila
On Mon, Feb 22, 2021 at 8:46 AM Amit Langote wrote: > > On Fri, Feb 19, 2021 at 7:38 PM Amit Kapila wrote: > > On Thu, Feb 18, 2021 at 11:05 AM Amit Langote > > wrote: > > > > > > > > It also occurred to me that we can avoid pointless adding of > &

Re: Parallel INSERT (INTO ... SELECT ...)

2021-02-22 Thread Amit Kapila
On Mon, Feb 22, 2021 at 8:41 AM Greg Nancarrow wrote: > > On Fri, Feb 19, 2021 at 9:38 PM Amit Kapila wrote: > > > > On Thu, Feb 18, 2021 at 11:05 AM Amit Langote > > wrote: > > > > > > > > It also occurred to me that we can avoid pointless

Re: [HACKERS] logical decoding of two-phase transactions

2021-02-22 Thread Amit Kapila
On Mon, Feb 22, 2021 at 11:04 PM Markus Wanner wrote: > > On 04.01.21 09:18, Amit Kapila wrote: > > Thanks, I have pushed the 0001* patch after making the above and a few > > other cosmetic modifications. > > That commit added the following snippet to the top of >

Re: locking [user] catalog tables vs 2pc vs logical rep

2021-02-22 Thread Amit Kapila
/message-id/CAMGcDxf83P5SGnGH52=_0wrp9po6urwcmrwaa0nxktzvir2...@mail.gmail.com -- With Regards, Amit Kapila.

Re: Parallel INSERT (INTO ... SELECT ...)

2021-02-22 Thread Amit Kapila
On Tue, Feb 23, 2021 at 6:37 AM Greg Nancarrow wrote: > > On Tue, Feb 23, 2021 at 12:33 AM Amit Kapila wrote: > > > > On Mon, Feb 22, 2021 at 8:41 AM Greg Nancarrow wrote: > > > > > > On Fri, Feb 19, 2021 at 9:38 PM Amit Kapila > > > wrote: >

Re: locking [user] catalog tables vs 2pc vs logical rep

2021-02-22 Thread Amit Kapila
On Tue, Feb 23, 2021 at 9:09 AM Andres Freund wrote: > > On 2021-02-23 08:56:39 +0530, Amit Kapila wrote: > > On Tue, Feb 23, 2021 at 3:58 AM Andres Freund wrote: > > > Perhaps all that we need to do is to disallow 2PC prepare if [user] > > > catalog table

Re: locking [user] catalog tables vs 2pc vs logical rep

2021-02-22 Thread Amit Kapila
On Tue, Feb 23, 2021 at 9:33 AM Andres Freund wrote: > > On 2021-02-23 09:24:18 +0530, Amit Kapila wrote: > > Okay, so is it sufficient to add comments in code, or do we want to > > add something in docs? I am not completely sure if we need to add in > > docs till we hav

Re: [HACKERS] logical decoding of two-phase transactions

2021-02-23 Thread Amit Kapila
On Tue, Feb 23, 2021 at 7:43 AM Amit Kapila wrote: > > On Mon, Feb 22, 2021 at 11:04 PM Markus Wanner wrote: > > > > On 04.01.21 09:18, Amit Kapila wrote: > > > Thanks, I have pushed the 0001* patch after making the above and a few > > > other cosmetic modi

Re: [PATCH] pg_hba.conf error messages for logical replication connections

2021-02-23 Thread Amit Kapila
On Mon, Feb 22, 2021 at 6:08 PM Euler Taveira wrote: > > On Sat, Feb 20, 2021, at 7:33 AM, Amit Kapila wrote: > > I have used a bit of different wording here to make things clear. > > Let me know what you think of the attached? > > WFM. > Thanks, Pushed! -- With Regards, Amit Kapila.

Re: repeated decoding of prepared transactions

2021-02-23 Thread Amit Kapila
/CAA4eK1L%3DdhuCRvyDvrXX5wZgc7s1hLRD29CKCK6oaHtVCPgiFA%40mail.gmail.com [2] - https://www.postgresql.org/message-id/CAFPTHDbbth0XVwf%3DWXcmp%3D_2nU5oNaK4CxetUr22qi1UM5v6rw%40mail.gmail.com -- With Regards, Amit Kapila.

Re: Parallel INSERT (INTO ... SELECT ...)

2021-02-23 Thread Amit Kapila
On Tue, Feb 23, 2021 at 4:47 PM Greg Nancarrow wrote: > > On Tue, Feb 23, 2021 at 2:30 PM Amit Kapila wrote: > > > > On Tue, Feb 23, 2021 at 6:37 AM Greg Nancarrow wrote: > > > > > > On Tue, Feb 23, 2021 at 12:33 AM Amit Kapila > > > wrote: > &g

Re: Parallel INSERT (INTO ... SELECT ...)

2021-02-23 Thread Amit Kapila
On Wed, Feb 24, 2021 at 8:41 AM Greg Nancarrow wrote: > > On Tue, Feb 23, 2021 at 10:53 PM Amit Kapila wrote: > > > > > But the non-parallel plan was chosen (instead of a parallel plan) > > > because of parallel-safety checks on the partitions, which found >

Re: Parallel INSERT (INTO ... SELECT ...)

2021-02-24 Thread Amit Kapila
On Wed, Feb 24, 2021 at 2:14 PM Greg Nancarrow wrote: > > On Wed, Feb 24, 2021 at 3:12 PM Amit Kapila wrote: > > > > On Wed, Feb 24, 2021 at 8:41 AM Greg Nancarrow wrote: > > > > > > On Tue, Feb 23, 2021 at 10:53 PM Amit Kapila > > > wrote: &g

Re: Parallel INSERT (INTO ... SELECT ...)

2021-02-24 Thread Amit Kapila
llel-SELECT-for-INSERT-INTO-.-SELECT. -- With Regards, Amit Kapila. fix_modifyingcte_parallel_safety_v1.patch Description: Binary data

Re: Parallel INSERT (INTO ... SELECT ...)

2021-02-24 Thread Amit Kapila
On Wed, Feb 24, 2021 at 4:30 PM Greg Nancarrow wrote: > > On Wed, Feb 24, 2021 at 8:39 PM Amit Kapila wrote: > > > > On Fri, Feb 19, 2021 at 6:56 AM Greg Nancarrow wrote: > > > > > > Posting a new version of the patches, with the following updates: > &g

Re: Single transaction in the tablesync worker?

2021-02-24 Thread Amit Kapila
On Wed, Feb 24, 2021 at 12:47 PM Masahiko Sawada wrote: > > On Fri, Feb 12, 2021 at 2:49 PM Amit Kapila wrote: > > > > > > Thanks, I have pushed the fix and the latest run of 'thorntail' has passed. > > I got the following WARNING message from a logic

Re: Parallel INSERT (INTO ... SELECT ...)

2021-02-24 Thread Amit Kapila
On Wed, Feb 24, 2021 at 6:21 PM Greg Nancarrow wrote: > > On Wed, Feb 24, 2021 at 10:38 PM Amit Kapila wrote: > > > > > > Thanks, I'll try it. > > > I did, however, notice a few concerns with your suggested alternative fix: > > > - It

Re: Single transaction in the tablesync worker?

2021-02-24 Thread Amit Kapila
On Wed, Feb 24, 2021 at 5:55 PM Amit Kapila wrote: > > On Wed, Feb 24, 2021 at 12:47 PM Masahiko Sawada > wrote: > > > > On Fri, Feb 12, 2021 at 2:49 PM Amit Kapila wrote: > > > > > > > > > Thanks, I have pushed the fix and the latest run of

Re: repeated decoding of prepared transactions

2021-02-25 Thread Amit Kapila
you can include these in the main patch. Apart from the above, I think the comments related to docs in my previous email [1] are still valid, can you please take care of those. [1] - https://www.postgresql.org/message-id/CAA4eK1Kr34_TiREr57Wpd%3D3%3D03x%3D1n55DAjwJPGpHAEc4dWfUQ%40mail.gmail.com --

Re: [PATCH] Note effect of max_replication_slots on subscriber side in documentation.

2021-02-25 Thread Amit Kapila
ing. + Why can't we just mention pg_replication_origin above? [1] - https://www.postgresql.org/docs/13/logical-replication-config.html -- With Regards, Amit Kapila.

Update docs of logical replication for commit ce0fdbfe97.

2021-02-25 Thread Amit Kapila
We forgot to update the logical replication configuration settings page in commit ce0fdbfe97. After commit ce0fdbfe97, table synchronization workers also started using replication origins to track the progress and the same should be reflected in docs. Attached patch for the same. v1-0001-Update-

<    1   2   3   4   5   6   7   8   9   10   >