Re: logical replication empty transactions

2022-02-22 Thread Ajin Cherian
On Thu, Feb 17, 2022 at 9:42 PM Amit Kapila wrote: > > On Mon, Jan 31, 2022 at 6:18 PM Ajin Cherian wrote: > > > > Few comments: > = > 1. Is there any particular why the patch is not skipping empty xacts > for streaming (in-progress) transactions as noted

Re: logical replication empty transactions

2022-02-25 Thread Ajin Cherian
ts from Peter and Wang in my next patch update. regards, Ajin Cherian Fujitsu Australia v20-0001-Skip-empty-transactions-for-logical-replication.patch Description: Binary data

Re: logical replication empty transactions

2022-02-28 Thread Ajin Cherian
>output_plugin_private = NULL; > + > + /* If no changes were part of this transaction then drop the commit */ > + if (!sent_begin_txn) > + { > + elog(DEBUG1, "Skipping replication of an empty transaction in stream > commit"); > + return; > + } > > (Same as

Re: logical replication empty transactions

2022-03-02 Thread Ajin Cherian
worker as it tries to find the non-existent streamed file. We could add logic to silently ignore a spurious "stream prepare" but that might not be ideal. Any thoughts on how to address this? Or else, we will need to avoid skipping streamed transactions as well. regards, Ajin Cherian Fujitsu Australia

Re: logical replication empty transactions

2022-03-02 Thread Ajin Cherian
ng and resetting our own context */ > old = MemoryContextSwitchTo(data->context); > > > I am not sure if it is suitable to send begin or stream_start here, because > the > row filter is not checked yet. That means, empty transactions caused by row > filter are n

Re: logical replication empty transactions

2022-03-03 Thread Ajin Cherian
I have split the patch into two. I have kept the logic of skipping streaming changes in the second patch. I will work on the second patch once we can figure out a solution for the COMMIT PREPARED after restart problem. regards, Ajin Cherian v23-0001-Skip-empty-transactions-for-logical

Re: logical replication empty transactions

2022-03-07 Thread Ajin Cherian
On Mon, Mar 7, 2022 at 7:50 PM shiy.f...@fujitsu.com wrote: > > On Fri, Mar 4, 2022 9:41 AM Ajin Cherian wrote: > > > > I have split the patch into two. I have kept the logic of skipping > > streaming changes in the second patch. > > I will work on the second

Re: Logical replication timeout problem

2022-03-07 Thread Ajin Cherian
paliveIfNecessary(ctx, false); + This reset is called too early, this function might go on to skip changes because of the row filter, so this reset fits better once we know for sure that a change is sent out. You will also need to send keep alive when the change is skipped due to the row filter.

Re: logical replication empty transactions

2022-03-16 Thread Ajin Cherian
On Mon, Mar 7, 2022 at 11:44 PM Ajin Cherian wrote: > > Fixed. > > regards, > Ajin Cherian > Fujitsu Australia Rebased the patch and fixed some whitespace errors. regards, Ajin Cherian Fujitsu Australia v25-0001-Skip-empty-transactions-for-logical-replication.patch Descri

Re: logical replication empty transactions

2022-03-18 Thread Ajin Cherian
in one of the existing test > files(say in 001_rep_changes.pl)? added a simple test. > 4. I think we can drop the skip streaming patch as we can't do that for now. Dropped, In addition, I have also added a few more comments explaining why the begin send is delayed in pgoutput_change

Re: [HACKERS] logical decoding of two-phase transactions

2020-11-30 Thread Ajin Cherian
sistent point, it does not satisfy DecodingContextReady. Which is why I suggested using the check DecodingContextReady to mark the prepare as 'Not decoded". regards, Ajin Cherian Fujitsu Australia

Re: [HACKERS] logical decoding of two-phase transactions

2020-12-01 Thread Ajin Cherian
n when using the pgoutput and real world scenarios of pub/sub. The second approach is a bit more involved requiring file creation and manipulation as well as the overhead of having to write to a file on every prepare which might be a performance bottleneck. Let me know what you think. regards, Ajin Cherian Fujitsu Australia

Re: [HACKERS] logical decoding of two-phase transactions

2020-12-16 Thread Ajin Cherian
> v31-0009-Support-2PC-txn-Subscription-option > 1. > --- a/src/include/catalog/catversion.h > +++ b/src/include/catalog/catversion.h > @@ -53,6 +53,6 @@ > */ > > /* mmddN */ > -#define CATALOG_VERSION_NO 202011251 > +#define CATALOG_VERSION_NO 202011271 > > No need to change catversion as t

Re: [HACKERS] logical decoding of two-phase transactions

2020-12-16 Thread Ajin Cherian
soon as we reach CONSISTENT_STATE? Did you observe any problem with > this? > Yes, it did not stream any xact during initialization but I was surprised that the DecodePrepare code was invoked even though I hadn't created the subscription with twophase enabled. No problem was observed. regards, Ajin Cherian Fujitsu Australia

Re: [HACKERS] logical decoding of two-phase transactions

2020-12-16 Thread Ajin Cherian
Adding a test case that tests that when a consistent snapshot is formed after a prepared transaction but before it has been COMMIT PREPARED. This test makes sure that in this case, the entire transaction is decoded on a COMMIT PREPARED. This patch applies on top of v31. regards, Ajin Cherian

Re: [HACKERS] logical decoding of two-phase transactions

2020-12-16 Thread Ajin Cherian
eparately (not as part of > this patch) we can investigate whether it is a good idea to prohibit > prepare for transactions that acquire exclusive locks on catalog > relations. > > Thoughts? I agree with the documentation option. If we choose to disable two-phase on timeout, we still need to decide what to do with already prepared transactions. regards, Ajin Cherian Fujitsu Australia

Re: [HACKERS] logical decoding of two-phase transactions

2020-12-17 Thread Ajin Cherian
eused which could lead to this. Yes, I think we could change rollback_prepare to send out prepare_lsn and prepare_time as well, just to be safe. regards, Ajin Cherian Fujitsu Australia.

Re: [HACKERS] logical decoding of two-phase transactions

2020-12-22 Thread Ajin Cherian
ve noticed a few typos that are not in the patch but in the surrounding code. logical.c: 1383: Comment should mention stream_commit_cb not stream_abort_cb. decode.c: 686 - Extra "it's" here: "because it's it happened" regards, Ajin Cherian Fujitsu Australia

Re: [HACKERS] logical decoding of two-phase transactions

2020-12-29 Thread Ajin Cherian
regards, Ajin Cherian Fujitsu Australia v34-0009-Support-2PC-consistent-snapshot-isolation-tests.patch Description: Binary data

Re: [HACKERS] logical decoding of two-phase transactions

2020-12-30 Thread Ajin Cherian
additional callbacks." to "To support streaming of two-phase commands, an output plugin needs to provide additional callbacks." Other than that, I have no more comments. regards, Ajin Cherian Fujitsu Australia

Re: Single transaction in the tablesync worker?

2021-01-04 Thread Ajin Cherian
ontents of the tables are compared at the end of each run to make sure they are identical. I have run it for around 12 hours, and it worked without any errors. Attaching the script I used. regards, Ajin Cherian Fujitsu Australia [1]- https://www.postgresql.org/message-id/93d02794068482f96d

Re: [HACKERS] logical decoding of two-phase transactions

2021-01-05 Thread Ajin Cherian
h > important steps help us to realize the required scenario. See > subxact_without_top.spec for reference. Added more comments to explain the state change of logical decoding. > 4. > +step "s2c" { COMMIT; } > ... > +step "s2prepare" { PREPARE TRANSACTION '

Re: Single transaction in the tablesync worker?

2021-02-11 Thread Ajin Cherian
On Thu, Feb 11, 2021 at 10:38 PM Amit Kapila wrote: > Okay, attached an updated patch with only that change. I ran Erik's test suite [1] on this patch overnight and found no errors. No more comments from me. The patch looks good. regards, Ajin Cherian Fujitsu Australia [1]

Re: Single transaction in the tablesync worker?

2021-02-11 Thread Ajin Cherian
max_wal_senders is set to zero. I think we can write this without > trying to connect. The attached patch fixes the problem for me. What > do you think? Verified this with installcheck and modified configuration to have wal_level = minimal and max_wal_senders = 0. Tests passed. The changes look

Re: repeated decoding of prepared transactions

2021-02-18 Thread Ajin Cherian
u can see that the transaction is not decoded at all. For this, I am planning to change the semantics such that two-phase-commit can only be specified while creating the slot using pg_create_logical_replication_slot() and not in pg_logical_slot_get_changes, thus preventing two-phase-commit flag from

Re: repeated decoding of prepared transactions

2021-02-23 Thread Ajin Cherian
On Mon, Feb 22, 2021 at 8:27 PM Andres Freund wrote: > > Yeah, we need to probably store this new point as slot's persistent > > information. > > Should be fine I think... This idea looks convincing. I'll write up a patch incorporating these changes. regards, Ajin Cherian Fujitsu Australia

Re: repeated decoding of prepared transactions

2021-02-23 Thread Ajin Cherian
On Tue, Feb 23, 2021 at 8:54 PM Amit Kapila wrote: > 1. With respect to SQL APIs, currently 'two-phase-commit' is a plugin > option so it is possible that the first time when it gets changes > (pg_logical_slot_get_changes) *without* 2PC enabled it will not get > the prepared even though prepare i

Re: repeated decoding of prepared transactions

2021-02-24 Thread Ajin Cherian
On Wed, Feb 24, 2021 at 4:48 PM Ajin Cherian wrote: > I plan to split this into two patches next. But do review and let me > know if you have any comments. Attaching an updated patch-set with the changes for snapshot_was_exported_at_lsn separated out from the changes for th

Re: repeated decoding of prepared transactions

2021-02-26 Thread Ajin Cherian
yed. Is this intentional? If not intentional we can >include snapshot_was_exported_at_lsn in pg_replication_slots. I've updated snapshot_was_exported_at_ member to pg_replication_slots as well. Do have a look and let me know if there are any comments. regards, Ajin Cherian Fujitsu Australia v4-0001-Avoid-repeated-decoding-of-prepared-transactions.patch Description: Binary data

Re: repeated decoding of prepared transactions

2021-02-26 Thread Ajin Cherian
On Fri, Feb 26, 2021 at 7:47 PM Ajin Cherian wrote: > I've updated snapshot_was_exported_at_ member to pg_replication_slots as > well. > Do have a look and let me know if there are any comments. Update with both patches. regards, Ajin Cherian Fujitsu Australia v4-0002-Add-op

Re: repeated decoding of prepared transactions

2021-02-26 Thread Ajin Cherian
via SQL APIs. I am not sure why Ajin neither changed the > name nor responded to that comment. What is your opinion? > I am fine with the name initial_consistency_at. I am also fine with not showing this in the pg_replication_slot view and keeping this internal. Regards, Ajin Cherian Fujitsu Australia > >

Re: repeated decoding of prepared transactions

2021-02-28 Thread Ajin Cherian
t; when we support it in CreateSubscription). Right now, there is nothing > to test all the code you have added in repl_gram.y. > Removed that. > 4. I think we can expose this new option via pg_replication_slots. > Done. Added, regards, Ajin Cherian Fujitsu Australia v7-0001-Avoid-re

Re: repeated decoding of prepared transactions

2021-03-01 Thread Ajin Cherian
text(XLogRecPtr start_lsn, > > ctx->reorder->output_rewrites = ctx->options.receive_rewrites; > > + /* > + * If twophase is set on the slot at create time, then > + * make sure the field in the context is also updated. > + */ > + ctx->twophase &= MyRepl

Re: repeated decoding of prepared transactions

2021-03-01 Thread Ajin Cherian
, so this might not be a problem. Other than that no more comments about the patch. Looks good. regards, Ajin Cherian Fujitsu Australia

Make stream_prepare an optional callback

2021-03-08 Thread Ajin Cherian
d two_phase commits. Plugins can and should be allowed to disallow this combination of allowing both streaming and two_phase at the same time. In which case, stream_prepare should be an optional callback. Attaching a patch that makes this change. Let me know if you have any comments. regards, Ajin Cher

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-09 Thread Ajin Cherian
minute. Attaching the test script and the test run log. regards, Ajin Cherian Fujitsu Australia #!/bin/bash # I compile postgres server versions into dir: # $HOME/pg_stuff/pg_installations/pgsql.$project ( where project is a name ) # # This script assumes that projects HEAD and head0 are

Re: [HACKERS] logical decoding of two-phase transactions

2020-11-04 Thread Ajin Cherian
On Fri, Oct 30, 2020 at 9:51 PM Amit Kapila wrote: > > On Fri, Oct 30, 2020 at 2:46 PM Ajin Cherian wrote: > > > > On Thu, Oct 29, 2020 at 11:19 PM Amit Kapila > > wrote: > > > > > > > > > 6. > > > +pg_decode_stream_prepar

Re: [HACKERS] logical decoding of two-phase transactions

2020-11-04 Thread Ajin Cherian
On Mon, Nov 2, 2020 at 9:40 PM Amit Kapila wrote: > > On Wed, Oct 28, 2020 at 10:50 AM Peter Smith wrote: > > > > Hi Ajin. > > > > I have re-checked the v13 patches for how my remaining review comments > > have been addressed. > > > > On

Re: [HACKERS] logical decoding of two-phase transactions

2020-11-04 Thread Ajin Cherian
On Wed, Nov 4, 2020 at 9:02 PM Amit Kapila wrote: > > On Wed, Nov 4, 2020 at 3:01 PM Ajin Cherian wrote: > > > > On Mon, Nov 2, 2020 at 9:40 PM Amit Kapila wrote: > > > > > > On Wed, Oct 28, 2020 at 10:50 AM Peter Smith > > > wrote: > > &g

Re: [HACKERS] logical decoding of two-phase transactions

2020-11-05 Thread Ajin Cherian
repared xact. Updated to execute invalidations while skipping prepared transactions. Also ran pgindent on the source files with updated typedefs. Attaching v17 with 1,2 and 3. regards, Ajin Cherian Fujitsu Australia v17-0001-Support-2PC-txn-base.patch Description: Binary data v17-0002-Support-2PC-txn-b

Re: [HACKERS] logical decoding of two-phase transactions

2020-11-10 Thread Ajin Cherian
'skip-empty-xacts', '1', 'stream-changes', '1'); ROLLBACK PREPARED 'test1'; == Here, what is seen is that the transaction is streamed twice, first when it crosses the memory threshold and is streamed (usually only in the 2

Re: [HACKERS] logical decoding of two-phase transactions

2020-11-11 Thread Ajin Cherian
ges', '1'); ** see the same streamed output here ** I think this is because since the transaction has not been committed, SnapBuildCommitTxn is not called which is what moves the "builder->start_decoding_at", and as a result later calls to pg_logical_slot_get_cha

Re: [HACKERS] logical decoding of two-phase transactions

2020-11-12 Thread Ajin Cherian
se_gid)); if (info == XLOG_XACT_COMMIT_PREPARED && ctx->twophase && skip_twophase) already_decoded = true; regards, Ajin Cherian Fujitsu Australia

Re: [HACKERS] logical decoding of two-phase transactions

2020-11-13 Thread Ajin Cherian
On Fri, Nov 13, 2020 at 9:44 PM Amit Kapila wrote: > > On Thu, Nov 12, 2020 at 2:28 PM Ajin Cherian wrote: > Hmm, introducing an additional boolean variable for this doesn't seem > like a good idea neither the other alternative suggested by you. How > about if we change th

Re: [HACKERS] logical decoding of two-phase transactions

2020-11-17 Thread Ajin Cherian
dated in GlobalTransaction } xl_xact_prepare; regards, Ajin Cherian Fujitsu Australia

Re: [HACKERS] logical decoding of two-phase transactions

2020-11-19 Thread Ajin Cherian
On Fri, Nov 20, 2020 at 12:23 AM Amit Kapila wrote: > > On Thu, Nov 19, 2020 at 2:52 PM Ajin Cherian wrote: > > > > On Thu, Nov 19, 2020 at 5:06 PM Amit Kapila wrote: > > > > > I think the same check should be there in truncate as well to make the > > &g

Re: [HACKERS] logical decoding of two-phase transactions

2020-11-23 Thread Ajin Cherian
e testing and reviewing the changes. regards, Ajin Cherian Fujitsu Australia

Re: [HACKERS] logical decoding of two-phase transactions

2020-11-24 Thread Ajin Cherian
On Mon, Nov 23, 2020 at 10:35 PM Amit Kapila wrote: > For the first two, as the xact is still not visible to others so we > don't need to make it behave like a committed txn. To make the (DDL) > changes visible to the current txn, the message > REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID copies the

Re: [HACKERS] logical decoding of two-phase transactions

2020-11-26 Thread Ajin Cherian
o create a replication slot using pg_create_logical_replication_slot when a transaction has been prepared but not yet committed , it just hangs till the transaction is committed/aborted. regards, Ajin Cherian Fujitsu Australia

Re: [HACKERS] logical decoding of two-phase transactions

2020-11-27 Thread Ajin Cherian
oints, have made sure the XLOG_RUNNING_XACTS are being inserted., but while iterating into SnapBuildFindSnapshot with a ,new XLOG_RUNNING_XACTS:record, the oldestRunningXid is being incremented at one xid at a time, which will eventually make it catch up running->nextXid and reach a SNAPBUILD_CONSIS

Re: [HACKERS] logical decoding of two-phase transactions

2020-11-30 Thread Ajin Cherian
the subscriber side does not need to try and handle a prepared transaction that does not exist on its side. Let me know what you think of this? regards, Ajin Cherian Fujitsu Australia

Re: Single transaction in the tablesync worker?

2021-01-11 Thread Ajin Cherian
ub} (1 row) and looking at the pg_subscription_rel: postgres=# select * from pg_subscription_rel; srsubid | srrelid | srsubstate | srsublsn -+-++-- (0 rows) As can be seen, none of the data in the pg_subscription_rel has been copied over. Same behaviour is seen with the patch and without the patch. regards, Ajin Cherian Fujitsu Australia

Re: Track replica origin progress for Rollback Prepared

2021-01-12 Thread Ajin Cherian
uot; LOGICAL 0/16054E0 (proto_version '2', publication_names '"tap_pub"') 2021-01-12 04:01:49.235 EST [4195] LOG: change replorigin after a rollback prepared Also restarted server2 and confirmed that the rollback prepared was not re-sent again. regards, Ajin Cherian Fujitsu Australia [1] - https://www.postgresql.org/message-id/CAA4eK1J7TiOQSiDq9MS_4k7zr4Jca7SnO8Vy3L4wtc7t1X9zsQ%40mail.gmail.com

Re: Track replica origin progress for Rollback Prepared

2021-01-12 Thread Ajin Cherian
3 02:14:40.868 EST [4851] LOG: database system is ready to accept connections 2021-01-13 02:14:40.897 EST [4887] LOG: logical replication apply worker for subscription "tap_sub" has started I see the same logs are seen in the test cases that have been added as part of https://www.postgresql.org/message-id/caa4ek1l3p4z+9wtk77mbdpkagr4gs2y3r1je7zevlqvf9ga...@mail.gmail.com regards, Ajin Cherian Fujitsu Australia .

Re: Track replica origin progress for Rollback Prepared

2021-01-13 Thread Ajin Cherian
On Wed, Jan 13, 2021 at 6:28 PM Ajin Cherian wrote: > I added the below log: Small correction, I sent the wrong code change for logging, this was the correct one: @@ -5976,6 +5977,14 @@ xact_redo_abort(xl_xact_parsed_abort *parsed, TransactionId

Re: Single transaction in the tablesync worker?

2021-01-22 Thread Ajin Cherian
die "Timed out while waiting for subscriber to start sync"; > > Is there a reason why we can't use the existing way to check for > failure in this case? Since the new design now uses temporary slots, is this test case still required?. If required, I can change it accordingly. regards, Ajin Cherian Fujitsu Australia

Re: Single transaction in the tablesync worker?

2021-01-22 Thread Ajin Cherian
On Sat, Jan 23, 2021 at 3:16 PM Amit Kapila wrote: > > I think so. But do you have any reason to believe that it won't be > required anymore? A temporary slot will not clash with a permanent slot of the same name, regards, Ajin Cherian Fujitsu

Re: Single transaction in the tablesync worker?

2021-02-01 Thread Ajin Cherian
the tablesync slot is not dropped. Hence causing the slot create to fail in the next restart. I think this can be avoided. We could either attempt a rollback only on specific failures and drop slot prior to erroring out. regards, Ajin Cherian Fujitsu Australia

Re: Single transaction in the tablesync worker?

2021-02-01 Thread Ajin Cherian
meant that after the tablesync is restarted, the worker has no idea about the old slot created as its name uses the relid of the dropped table. regards, Ajin Cherian Fujitsu Australia

Re: Single transaction in the tablesync worker?

2021-02-02 Thread Ajin Cherian
On Tue, Feb 2, 2021 at 7:40 PM Amit Kapila wrote: > > On Tue, Feb 2, 2021 at 10:34 AM Ajin Cherian wrote: > > > > On Mon, Feb 1, 2021 at 11:26 PM Amit Kapila wrote: > > > I have updated the patch to display WARNING for each of the tablesync > > > slots duri

Re: Single transaction in the tablesync worker?

2021-02-02 Thread Ajin Cherian
g for subscription "tap_sub" DETAIL: Table synchronization for relation "tab_rep" is in progress and is in state "f". HINT: Use ALTER SUBSCRIPTION ... ENABLE to enable subscription if not already enabled or use DROP SUBSCRIPTION ... to drop the subscription. regards, Ajin Cherian Fujitsu Australia

Re: Single transaction in the tablesync worker?

2021-02-02 Thread Ajin Cherian
On Tue, Feb 2, 2021 at 9:03 PM Ajin Cherian wrote: > I am sorry, my above steps were not correct. I think the reason for > the failure I was seeing were some other steps I did prior to this. I > will recreate this and update you with the appropriate steps. The correct steps are a

Re: Single transaction in the tablesync worker?

2021-02-03 Thread Ajin Cherian
ix this? I have verified that the problem is not seen after this patch. I also agree with the approach taken for the fix, regards, Ajin Cherian Fujitsu Australia

Re: Single transaction in the tablesync worker?

2021-02-05 Thread Ajin Cherian
I did some basic cross-version testing, publisher on PG13 and subscriber on PG14 and publisher on PG14 and subscriber on PG13. Did some basic operations, CREATE, ALTER and STOP subscriptions and it seemed to work fine, no errors. regards, Ajin Cherian Fujitsu Australia.

Re: repeated decoding of prepared transactions

2021-02-08 Thread Ajin Cherian
emselves. > Also this will cause performance degradation. So if we can avoid > resending prepared transactions twice that will help. One of this scenario is explained in the test case in postgres/contrib/test_decoding/specs/twophase_snapshot.spec regards, Ajin Cherian Fujitsu Australia

Re: Single transaction in the tablesync worker?

2021-02-09 Thread Ajin Cherian
16401_sync_16389_6927117142022745645" on publisher DETAIL: ERROR: replication slot "pg_16401_sync_16389_6927117142022745645" does not exist regards, Ajin Cherian Fujitsu Australia

Re: repeated decoding of prepared transactions

2021-02-09 Thread Ajin Cherian
nly happen if the user restarts the decoding after receiving the prepare for a transaction but before receiving the commit prepared say because of some error." The pgoutput plugin is also being updated to be able to handle this situation of duplicate prepared transactions. regards, Ajin Cherian Fujitsu Australia

Re: row filtering for logical replication

2021-12-01 Thread Ajin Cherian
e can check the transformed where clause and see if the where > clause is the same or not. If you are ok with this approach you could > make similar changes. thanks for your patch, I have used the same logic with minor changes and shared it with Peter for v44. regards, Ajin Cherian Fujitsu Australia

Re: row filtering for logical replication

2021-12-17 Thread Ajin Cherian
decide if the update was previously published or not? I think it's futile for the publisher side to try and figure out the history of published rows. In fact, if this level of logic is required then it is best implemented on the subscriber side, which then defeats the purpose of a publication filter. regards, Ajin Cherian Fujitsu Australia

Re: row filtering for logical replication

2021-12-19 Thread Ajin Cherian
7;s better to give users more flexibility while warning him on what the consequences are rather than restricting him with constraints. We could explain this in the documentation so that users can better predict the effect of having pubaction specific filters. regards, Ajin Cherian Fujitsu Australia

Re: row filtering for logical replication

2021-12-20 Thread Ajin Cherian
t can directly get from the value >from the virtual tuple. Storing the old tuple/new tuple in a slot and re-using the slot avoids the overhead of continuous deforming of tuple at multiple levels in the code. regards, Ajin Cherian [1] - https://www.postgresql.org/message-id/CAFiTN-vwBjy+eR+iodkO5UVN5cPv_xx1=s8ehzgcrjza+az...@mail.gmail.com

Re: logical replication empty transactions

2022-01-11 Thread Ajin Cherian
On Wed, Sep 1, 2021 at 8:57 PM Ajin Cherian wrote: > > Thanks for the comments. Addressed them in the attached patch. > > regards, > Ajin Cherian > Fujitsu Australia Minor update to rebase the patch so that it applies clean on HEAD. regards, Ajin Cherian regards, Ajin Cheri

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-18 Thread Ajin Cherian
le-synchronization phase to handle two-phase from the start. regards, Ajin Cherian Fujitsu Australia v62-0002-Support-2PC-txn-subscriber-tests.patch Description: Binary data v62-0003-Fix-apply-worker-dev-logs.patch Description: Binary data

Re: Logical Replication vs. 2PC

2021-03-18 Thread Ajin Cherian
les across 2 subscriptions which are on the same server? Maybe it could be documented warning users from having such a setup. Do we really want to create a solution for an impractical scenario? regards, Ajin Cherian Fujitsu Australia

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-19 Thread Ajin Cherian
On Sat, Mar 20, 2021 at 1:35 AM Amit Kapila wrote: > On Fri, Mar 19, 2021 at 5:03 AM Ajin Cherian wrote: > > > > Missed the patch - 0001, resending. > > > > > @@ -538,10 +550,21 @@ CreateDecodingContext(XLogRecPtr start_lsn, > .. > + /* Set two_phase_at LS

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-19 Thread Ajin Cherian
On Sat, Mar 20, 2021 at 1:35 AM Amit Kapila wrote: > On Fri, Mar 19, 2021 at 5:03 AM Ajin Cherian wrote: > > > > Missed the patch - 0001, resending. > > > > I have made miscellaneous changes in the patch which includes > improving comments, error mess

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-23 Thread Ajin Cherian
st log attached. regards, Ajin Cherian Fujitsu Australia 5 Description: Binary data

Re: [PATCH] add concurrent_abort callback for output plugin

2021-03-29 Thread Ajin Cherian
can know about it when rollback prepared arrives? > > Nice catch, Markus. Interesting suggestion Amit. Let me try and code this. regards, Ajin Cherian Fujitsu Australia

Re: [PATCH] add concurrent_abort callback for output plugin

2021-03-29 Thread Ajin Cherian
On Mon, Mar 29, 2021 at 10:09 PM Markus Wanner < markus.wan...@enterprisedb.com> wrote: > On 29.03.21 13:02, Ajin Cherian wrote: > > Nice catch, Markus. > > Interesting suggestion Amit. Let me try and code this. > > Thanks, Ajin. Please consider this concurrent_abort

Re: [PATCH] add concurrent_abort callback for output plugin

2021-03-30 Thread Ajin Cherian
On Tue, Mar 30, 2021 at 5:30 PM Markus Wanner < markus.wan...@enterprisedb.com> wrote: > Hello Ajin, > > On 30.03.21 06:48, Ajin Cherian wrote: > > For now, I've created a patch that addresses the problem reported using > > the existing callbacks. > > Than

Re: [PATCH] add concurrent_abort callback for output plugin

2021-03-30 Thread Ajin Cherian
On Tue, Mar 30, 2021 at 7:10 PM Markus Wanner < markus.wan...@enterprisedb.com> wrote: > On 30.03.21 09:39, Ajin Cherian wrote: > > Where do you suggest this be documented? From an externally visible > > point of view, I dont see much of a surprise. > > > > S

Re: [PATCH] add concurrent_abort callback for output plugin

2021-03-30 Thread Ajin Cherian
sal? > > > + This is done so that eventually when the ROLLBACK > PREPARED > > + is decoded, there is a corresponding prepared transaction with a > matching gid. > > > > > > > > Everything else sounds good to me. > Updated. regards, Ajin Cherian Fujitsu Australia v3-0001-Make-sure-a-prepare-is-sent-when-decoder-detects-.patch Description: Binary data

Re: locking [user] catalog tables vs 2pc vs logical rep

2021-03-31 Thread Ajin Cherian
d transaction. > I have made a patch to make the prepare transaction fail in this > scenario. Attached the patch for the same. > Thoughts? > > The patch applies fine on HEAD and "make check" passes fine. No major comments on the patch, just a minor comment: If you could change the error from, " cannot PREPARE a transaction that has a lock on user catalog/system table(s)" to "cannot PREPARE a transaction that has an *exclusive lock* on user catalog/system table(s)" that would be a more accurate instruction to the user. regards, Ajin Cherian Fujitsu Australia

Re: [PATCH] add concurrent_abort callback for output plugin

2021-03-31 Thread Ajin Cherian
otherwise missing earlier in the > docs (similar to how the commit message describes it). > > > I'm fine with Amit's changes and like Markus's last suggestion as well. regards, Ajin Cherian Fujitsu Australia

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-31 Thread Ajin Cherian
some more work and testing is required. We > can try to make it work with the streaming option and do something > about empty prepare transactions to reduce the need for users to set a > much higher value for max_prepared_xacts on subscribers. So, I propose > to move it to the next CF, what

missing documentation for streaming in-progress transactions

2021-04-07 Thread Ajin Cherian
Hi, Found that some documentation hasn't been updated for the changes made as part of streaming large in-progress transactions. Attached a patch that adds the missing changes. Let me know if anything more needs to be added or any comments on this change. regards, Ajin Cherian Fujitsu Aust

Re: missing documentation for streaming in-progress transactions

2021-04-08 Thread Ajin Cherian
"Xid of the transaction" > I think there should be consistent terminology used on this page > Updated. regards, Ajin Cherian Fujitsu Australia v3-0001-doc-Update-information-of-new-messages-for-logica.patch Description: Binary data

Re: missing documentation for streaming in-progress transactions

2021-04-08 Thread Ajin Cherian
; > Although, sub-transaction is also used in the documentation, I suggest to > use > subtransaction. Maybe change the other sub-transaction occurrences too. > Updated. regards, Ajin Cherian Fujitsu Australia v4-0001-doc-Update-information-of-new-messages-for-logica.patch Description: Binary data

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-07-20 Thread Ajin Cherian
this as the stats framework is updated. Let me know if you have any comments. regards, Ajin Cherian Fujitsu Australia [1] - https://www.postgresql.org/message-id/flat/CA%2Bfd4k5_pPAYRTDrO2PbtTOe0eHQpBvuqmCr8ic39uTNmR49Eg%40mail.gmail.com v1_streaming_stats_update.patch Description: Binary data

Re: logical replication empty transactions

2020-07-24 Thread Ajin Cherian
The patch no longer applies, because of additions in the test source. Otherwise, I have tested the patch and confirmed that updates and deletes on tables with deferred primary keys work with logical replication. The new status of this patch is: Waiting on Author

Re: logical replication empty transactions

2020-07-24 Thread Ajin Cherian
Sorry, I replied in the wrong thread. Please ignore above mail. > >

Re: deferred primary key and logical replication

2020-07-24 Thread Ajin Cherian
The patch no longer applies, because of additions in the test source. Otherwise, I have tested the patch and confirmed that updates and deletes on tables with deferred primary keys work with logical replication. The new status of this patch is: Waiting on Author

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-07-29 Thread Ajin Cherian
act the TPS. But what is good is that, enabling streaming improves the TPS (about 58% improvement) [1] - https://www.postgresql.org/message-id/flat/CAMsr%2BYE6aE6Re6smrMr-xCabRmCr%3DyzXEf2Yuv5upEDY5nMX8g%40mail.gmail.com#dbe51a181dd735eec8bb36f8a07bacf5 regards, Ajin Cherian Fujitsu Australia

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-07-31 Thread Ajin Cherian
ied easily on top of > v44. regards, Ajin Cherian Fujitsu Australia streaming_stats_update.patch Description: Binary data

Re: deferred primary key and logical replication

2020-08-03 Thread Ajin Cherian
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: not tested Documentation:not tested Patch applies cleanly. Tested that update/delete of tables with defer

Re: [HACKERS] logical decoding of two-phase transactions

2020-09-06 Thread Ajin Cherian
, like the handling of concurrent aborts, so all that logic has been left out. I think some of the earlier comments have already been addressed or are no longer relevant. Do have a look at the patch and let me know what you think.I will try and address any pending issues going forward. regards, Aj

Re: [HACKERS] logical decoding of two-phase transactions

2020-09-09 Thread Ajin Cherian
tream-changes' > option in commit 7259736a6e? Just refer how we set ctx->streaming and > you can use a similar way to set this parameter. > Done, I've moved the checks for callbacks to inside the corresponding wrappers. Regards, Ajin Cherian Fujitsu Australia 0001-Support-dec

Re: row filtering for logical replication

2021-10-11 Thread Ajin Cherian
On Tue, Oct 12, 2021 at 1:37 AM Dilip Kumar wrote: > > On Wed, Oct 6, 2021 at 2:33 PM Ajin Cherian wrote: > > > > On Sat, Oct 2, 2021 at 5:44 PM Ajin Cherian wrote: > > > > > > I have for now also rebased the patch and merged the first 5 patches > > &g

Re: Zedstore - compressed in-core columnar storage

2019-05-23 Thread Ajin Cherian
hink. regards, Ajin Cherian Fujitsu Australia On Thu, May 23, 2019 at 3:56 PM Ashwin Agrawal wrote: > > We (Heikki, me and Melanie) are continuing to build Zedstore. Wish to > share the recent additions and modifications. Attaching a patch > with the latest code. Link to github branch

Re: Synchronizing slots from primary to standby

2024-01-23 Thread Ajin Cherian
n_slot('lsub2_slot', 'pgoutput', false, false, true); ?column? -- init (1 row) Time: 36.125 ms postgres=# SELECT 'init' FROM pg_create_logical_replication_slot('lsub1_slot', 'pgoutput', false, false, false); ?column? -- init (1 row) Time: 53.981 ms regards, Ajin Cherian Fujitsu Australia

  1   2   3   >