RE: Conflict detection for update_deleted in logical replication

2025-09-20 Thread Zhijie Hou (Fujitsu)
On Friday, September 12, 2025 4:43 PM Amit Kapila wrote: > > On Fri, Sep 12, 2025 at 8:55 AM Zhijie Hou (Fujitsu) > wrote: > > > > I agree. Here is a V73 patch that will restart the worker if the > > retention resumes. I also addressed other comments posted by Amit[1]. > > > > Few comments: >

RE: Conflict detection for update_deleted in logical replication

2025-09-20 Thread Zhijie Hou (Fujitsu)
On Wednesday, September 3, 2025 12:19 PM shveta malik wrote: > On Tue, Sep 2, 2025 at 3:30 PM shveta malik > wrote: > > > > > > > > > > > > > Here is V70 patch set. > > > > > > > > > Please find a few comments on v70-003: > > 1) > Doc of dead_tuple_retention_active says: > True if retain_dead_

RE: Conflict detection for update_deleted in logical replication

2025-09-20 Thread Zhijie Hou (Fujitsu)
On Monday, September 8, 2025 7:21 PM Zhijie Hou (Fujitsu) wrote: > > On Monday, September 8, 2025 3:13 PM Amit Kapila > wrote: > > > > On Fri, Sep 5, 2025 at 5:03 PM Zhijie Hou (Fujitsu) > > > > wrote: > > > > > > Here are v2 patches which addressed above comments. > > > > > > > I have pushed

Re: Conflict detection for update_deleted in logical replication

2025-09-17 Thread Masahiko Sawada
On Thu, Sep 11, 2025 at 3:54 AM Dilip Kumar wrote: > > On Wed, Sep 10, 2025 at 2:09 PM Amit Kapila wrote: > > > > On Wed, Sep 10, 2025 at 9:08 AM Zhijie Hou (Fujitsu) > > wrote: > > > > > > On Tuesday, September 9, 2025 7:01 PM Amit Kapila > > > wrote: > > > > On Tue, Sep 9, 2025 at 11:47 AM Z

Re: Conflict detection for update_deleted in logical replication

2025-09-16 Thread Amit Kapila
On Tue, Sep 16, 2025 at 11:13 AM shveta malik wrote: > > > > > I updated the patch to fix this as well. > > > > Thank You for the patch. Fix looks good. > > Shall we update the comment: > +# Drop the physical slot and reset the synchronized_standby_slots setting. We > +# change this after setting

Re: Conflict detection for update_deleted in logical replication

2025-09-15 Thread shveta malik
On Tue, Sep 16, 2025 at 10:08 AM Zhijie Hou (Fujitsu) wrote: > > On Tuesday, September 16, 2025 11:54 AM Zhijie Hou (Fujitsu) > wrote: > > > > On Monday, September 15, 2025 8:11 PM Amit Kapila > > wrote: > > > > > > On Mon, Sep 15, 2025 at 1:07 PM Zhijie Hou (Fujitsu) > > > wrote: > > > > > >

RE: Conflict detection for update_deleted in logical replication

2025-09-15 Thread Zhijie Hou (Fujitsu)
On Tuesday, September 16, 2025 11:54 AM Zhijie Hou (Fujitsu) wrote: > > On Monday, September 15, 2025 8:11 PM Amit Kapila > wrote: > > > > On Mon, Sep 15, 2025 at 1:07 PM Zhijie Hou (Fujitsu) > > wrote: > > > > > > Thanks, the changes look good to me. I have merged them in V75 patch. > > > > >

RE: Conflict detection for update_deleted in logical replication

2025-09-15 Thread Zhijie Hou (Fujitsu)
On Monday, September 15, 2025 8:11 PM Amit Kapila wrote: > > On Mon, Sep 15, 2025 at 1:07 PM Zhijie Hou (Fujitsu) > wrote: > > > > Thanks, the changes look good to me. I have merged them in V75 patch. > > > > Pushed. I see a BF which is not related with this commit but a previous commit > for

Re: Conflict detection for update_deleted in logical replication

2025-09-15 Thread Amit Kapila
On Mon, Sep 15, 2025 at 1:07 PM Zhijie Hou (Fujitsu) wrote: > > Thanks, the changes look good to me. I have merged them in V75 patch. > Pushed. I see a BF which is not related with this commit but a previous commit for the work in this thread. See LOGs [1]: regress_log_035_conflicts

RE: Conflict detection for update_deleted in logical replication

2025-09-15 Thread Zhijie Hou (Fujitsu)
On Monday, September 15, 2025 12:52 PM Amit Kapila wrote: > > On Fri, Sep 12, 2025 at 3:39 PM Zhijie Hou (Fujitsu) > wrote: > > > > Here is the V74 patch which addressed all comments. > > > > + ereport(LOG, > + errmsg("logical replication worker for subscription \"%s\" will > resume retaining

RE: Conflict detection for update_deleted in logical replication

2025-09-15 Thread Zhijie Hou (Fujitsu)
On Monday, September 15, 2025 12:55 PM shveta malik wrote: > > One concern: > > if (should_stop_conflict_info_retention(rdt_data)) > + { > + /* > + * Stop retention if not yet. Otherwise, reset to the initial phase to > + * retry all phases. This is required to recalculate the current wait >

Re: Conflict detection for update_deleted in logical replication

2025-09-14 Thread shveta malik
One concern: if (should_stop_conflict_info_retention(rdt_data)) + { + /* + * Stop retention if not yet. Otherwise, reset to the initial phase to + * retry all phases. This is required to recalculate the current wait + * time and resume retention if the time falls within + * max_retention_duratio

Re: Conflict detection for update_deleted in logical replication

2025-09-14 Thread Amit Kapila
On Fri, Sep 12, 2025 at 3:39 PM Zhijie Hou (Fujitsu) wrote: > > Here is the V74 patch which addressed all comments. > + ereport(LOG, + errmsg("logical replication worker for subscription \"%s\" will resume retaining the information for detecting conflicts", +MySubscription->name), + MySubscri

RE: Conflict detection for update_deleted in logical replication

2025-09-12 Thread Zhijie Hou (Fujitsu)
On Friday, September 12, 2025 4:48 PM shveta malik wrote: > On Fri, Sep 12, 2025 at 8:55 AM Zhijie Hou (Fujitsu) > wrote: > > > > > > I agree. Here is a V73 patch that will restart the worker if the > > retention resumes. I also addressed other comments posted by Amit[1]. > > > > Thanks for the

Re: Conflict detection for update_deleted in logical replication

2025-09-12 Thread shveta malik
On Fri, Sep 12, 2025 at 8:55 AM Zhijie Hou (Fujitsu) wrote: > > > I agree. Here is a V73 patch that will restart the worker if the retention > resumes. I also addressed other comments posted by Amit[1]. > Thanks for the patch. Few comments: 1) There is a small window where worker can exit while

Re: Conflict detection for update_deleted in logical replication

2025-09-12 Thread Amit Kapila
On Fri, Sep 12, 2025 at 8:55 AM Zhijie Hou (Fujitsu) wrote: > > I agree. Here is a V73 patch that will restart the worker if the retention > resumes. I also addressed other comments posted by Amit[1]. > Few comments: * In adjust_xid_advance_interval(), for the case when retention is

RE: Conflict detection for update_deleted in logical replication

2025-09-12 Thread Zhijie Hou (Fujitsu)
On Tuesday, September 2, 2025 6:00 PM shveta malik wrote: > > On Mon, Sep 1, 2025 at 5:45 PM shveta malik > wrote: > > > > > > > > Here is V70 patch set. > > > > > > > The patch v70-0001 looks good to me. Verified, all the old issues are > > resolved. > > > > Will resume review of v70-0002 now

Re: Conflict detection for update_deleted in logical replication

2025-09-11 Thread Amit Kapila
On Fri, Sep 5, 2025 at 5:03 PM Zhijie Hou (Fujitsu) wrote: > > Here are v2 patches which addressed above comments. > I have pushed the first patch. I find that the test can't reliably fail without a fix. Can you please investigate it? -- With Regards, Amit Kapila.

RE: Conflict detection for update_deleted in logical replication

2025-09-11 Thread Zhijie Hou (Fujitsu)
On Friday, September 12, 2025 2:39 AM Masahiko Sawada wrote: > > On Thu, Sep 11, 2025 at 3:54 AM Dilip Kumar wrote: > > > > On Wed, Sep 10, 2025 at 2:09 PM Amit Kapila > wrote: > > > > > > On Wed, Sep 10, 2025 at 9:08 AM Zhijie Hou (Fujitsu) > > > wrote: > > > > > > > > On Tuesday, September

Re: Conflict detection for update_deleted in logical replication

2025-09-11 Thread Dilip Kumar
On Wed, Sep 10, 2025 at 2:09 PM Amit Kapila wrote: > > On Wed, Sep 10, 2025 at 9:08 AM Zhijie Hou (Fujitsu) > wrote: > > > > On Tuesday, September 9, 2025 7:01 PM Amit Kapila > > wrote: > > > On Tue, Sep 9, 2025 at 11:47 AM Zhijie Hou (Fujitsu) > > > > > > wrote: > > > > > > Few other comment

Re: Conflict detection for update_deleted in logical replication

2025-09-11 Thread shveta malik
On Thu, Sep 11, 2025 at 2:29 PM Zhijie Hou (Fujitsu) wrote: > > On Monday, September 8, 2025 7:21 PM Zhijie Hou (Fujitsu) > wrote: > > > > On Monday, September 8, 2025 3:13 PM Amit Kapila > > wrote: > > > > > > On Fri, Sep 5, 2025 at 5:03 PM Zhijie Hou (Fujitsu) > > > > > > wrote: > > > > > >

Re: Conflict detection for update_deleted in logical replication

2025-09-10 Thread Amit Kapila
On Wed, Sep 10, 2025 at 9:08 AM Zhijie Hou (Fujitsu) wrote: > > On Tuesday, September 9, 2025 7:01 PM Amit Kapila > wrote: > > On Tue, Sep 9, 2025 at 11:47 AM Zhijie Hou (Fujitsu) > > > > wrote: > > > > Few other comments: > > 1. > > + /* > > + * Return if the launcher has not initialized olde

RE: Conflict detection for update_deleted in logical replication

2025-09-09 Thread Zhijie Hou (Fujitsu)
On Tuesday, September 9, 2025 7:01 PM Amit Kapila wrote: > On Tue, Sep 9, 2025 at 11:47 AM Zhijie Hou (Fujitsu) > wrote: > > > > Here is V71 patch set which addressed above comments and [1]. > > > > IIUC, this patch after stopping the retention, it immediately starts retrying > to > resume by

RE: Conflict detection for update_deleted in logical replication

2025-09-09 Thread Zhijie Hou (Fujitsu)
On Tuesday, September 9, 2025 5:17 PM shveta malik wrote: > > On Tue, Sep 9, 2025 at 11:47 AM Zhijie Hou (Fujitsu) > wrote: > > > > > > Here is V71 patch set which addressed above comments and [1]. > > > > Thank You for the patches. Please find a few comments on 001: > > 1) > In compute_min_n

Re: Conflict detection for update_deleted in logical replication

2025-09-09 Thread Amit Kapila
On Tue, Sep 9, 2025 at 11:47 AM Zhijie Hou (Fujitsu) wrote: > > Here is V71 patch set which addressed above comments and [1]. > IIUC, this patch after stopping the retention, it immediately starts retrying to resume by transitioning through various phases. This can consume CPU and network resourc

Re: Conflict detection for update_deleted in logical replication

2025-09-09 Thread shveta malik
On Tue, Sep 9, 2025 at 11:47 AM Zhijie Hou (Fujitsu) wrote: > > > Here is V71 patch set which addressed above comments and [1]. > Thank You for the patches. Please find a few comments on 001: 1) In compute_min_nonremovable_xid, when 'wait_for_xid' is true, we should have Assert(!worker->oldest_n

RE: Conflict detection for update_deleted in logical replication

2025-09-08 Thread Zhijie Hou (Fujitsu)
On Monday, September 8, 2025 3:13 PM Amit Kapila wrote: > > On Fri, Sep 5, 2025 at 5:03 PM Zhijie Hou (Fujitsu) > wrote: > > > > Here are v2 patches which addressed above comments. > > > > I have pushed the first patch. I find that the test can't reliably fail > without a fix. > Can you pleas

Re: Conflict detection for update_deleted in logical replication

2025-09-08 Thread shveta malik
On Mon, Sep 8, 2025 at 3:06 AM Tom Lane wrote: > > Coverity is not happy with commit a850be2fe: > > /srv/coverity/git/pgsql-git/postgresql/src/backend/replication/logical/worker.c: > 3276 in FindDeletedTupleInLocalRel() > 3270 * maybe_advance_nonremovable_xid() for

Re: Conflict detection for update_deleted in logical replication

2025-09-07 Thread Tom Lane
Coverity is not happy with commit a850be2fe: /srv/coverity/git/pgsql-git/postgresql/src/backend/replication/logical/worker.c: 3276 in FindDeletedTupleInLocalRel() 3270 * maybe_advance_nonremovable_xid() for details). 3271 */ 3272

RE: Conflict detection for update_deleted in logical replication

2025-09-05 Thread Zhijie Hou (Fujitsu)
On Friday, September 5, 2025 2:01 PM shveta malik wrote: > > On Thu, Sep 4, 2025 at 3:30 PM Zhijie Hou (Fujitsu) > wrote: > > > > Hi, > > > > As reported by Robert[1], it is worth adding a test for the race condition > > in > > the RecordTransactionCommitPrepared() function to reduce the risk

Re: Conflict detection for update_deleted in logical replication

2025-09-04 Thread shveta malik
On Thu, Sep 4, 2025 at 3:30 PM Zhijie Hou (Fujitsu) wrote: > > Hi, > > As reported by Robert[1], it is worth adding a test for the race condition in > the RecordTransactionCommitPrepared() function to reduce the risk of future > code > changes: > > /* > * Note it is important to

RE: Conflict detection for update_deleted in logical replication

2025-09-04 Thread Zhijie Hou (Fujitsu)
Hi, As reported by Robert[1], it is worth adding a test for the race condition in the RecordTransactionCommitPrepared() function to reduce the risk of future code changes: /* * Note it is important to set committs value after marking ourselves as * in the commit critical

Re: Conflict detection for update_deleted in logical replication

2025-09-02 Thread shveta malik
On Tue, Sep 2, 2025 at 3:30 PM shveta malik wrote: > > > > > > > > > Here is V70 patch set. > > > > > Please find a few comments on v70-003: 1) Doc of dead_tuple_retention_active says: True if retain_dead_tuples is enabled and the retention duration for information used in conflict detection is

Re: Conflict detection for update_deleted in logical replication

2025-09-02 Thread shveta malik
On Mon, Sep 1, 2025 at 5:45 PM shveta malik wrote: > > > > > Here is V70 patch set. > > > > The patch v70-0001 looks good to me. Verified, all the old issues are > resolved. > > Will resume review of v70-0002 now. > Please find a few comments on v70-0002: 1) - * Note: Retention won't be resumed

Re: Conflict detection for update_deleted in logical replication

2025-09-01 Thread shveta malik
On Mon, Sep 1, 2025 at 5:07 PM Zhijie Hou (Fujitsu) wrote: > > > I reviewed the patch internally and tweaked a small detail of the apply worker > to reduce the waiting time in the main loop when max_retention_duration is > defined (set wait_time = min(wait_time, max_retention_duration)). Also, I

RE: Conflict detection for update_deleted in logical replication

2025-08-31 Thread Zhijie Hou (Fujitsu)
On Saturday, August 30, 2025 12:48 PM Nisha Moond wrote: > > On Fri, Aug 29, 2025 at 11:49 AM Zhijie Hou (Fujitsu) > wrote: > > > > Here is the new version patch set which also addressed Shveta's > comments[1]. > > > > Thanks for the patches here, I tested the v68-001 patch alone, please find

Re: Conflict detection for update_deleted in logical replication

2025-08-31 Thread Nisha Moond
On Thu, Aug 28, 2025 at 6:00 PM Nisha Moond wrote: > > Test-04. pgbench on both side, and max_conflict_retention_duration was tuned > > Setup: > --- > Pub --> Sub > - setup is same as Test-03(above) > - Additionally, s

Re: Conflict detection for update_deleted in logical replication

2025-08-30 Thread Nisha Moond
On Sat, Aug 30, 2025 at 10:17 AM Nisha Moond wrote: > > On Fri, Aug 29, 2025 at 11:49 AM Zhijie Hou (Fujitsu) > wrote: > > > > Here is the new version patch set which also addressed Shveta's comments[1]. > > > > Thanks for the patches here, I tested the v68-001 patch alone, please > find review c

Re: Conflict detection for update_deleted in logical replication

2025-08-29 Thread Nisha Moond
On Fri, Aug 29, 2025 at 11:49 AM Zhijie Hou (Fujitsu) wrote: > > Here is the new version patch set which also addressed Shveta's comments[1]. > Thanks for the patches here, I tested the v68-001 patch alone, please find review comments - 1) If a sub is created with retain_dead_tuples=on but disab

Re: Conflict detection for update_deleted in logical replication

2025-08-29 Thread shveta malik
On Fri, Aug 29, 2025 at 11:49 AM Zhijie Hou (Fujitsu) wrote: > > Here is the new version patch set which also addressed Shveta's comments[1]. > Thanks for the patch. On 001 alone, I’m observing a behavior where, if sub1 has stopped retention, and I then create a new subscription sub2, the worker

RE: Conflict detection for update_deleted in logical replication

2025-08-28 Thread Zhijie Hou (Fujitsu)
On Thursday, August 28, 2025 6:07 PM shveta malik wrote: > On Thu, Aug 28, 2025 at 8:02 AM Zhijie Hou (Fujitsu) > wrote: > > > > > > I noticed that Cfbot failed to compile the document due to a typo > > after renaming the subscription option. Here are the updated V67 > > patches to fix that, onl

Re: Conflict detection for update_deleted in logical replication

2025-08-28 Thread Nisha Moond
Hi, As per the test results in [1], the TPS drop observed on the subscriber with update_deleted enabled was mainly because only a single apply worker was handling the replication workload from multiple concurrent publisher clients. The following performance benchmarks were conducted to evaluate th

Re: Conflict detection for update_deleted in logical replication

2025-08-28 Thread Amit Kapila
On Thu, Aug 28, 2025 at 4:39 PM Amit Kapila wrote: > > On Thu, Aug 28, 2025 at 8:02 AM Zhijie Hou (Fujitsu) > wrote: > > > > I noticed that Cfbot failed to compile the document due to a typo after > > renaming > > the subscription option. Here are the updated V67 patches to fix that, only > > t

Re: Conflict detection for update_deleted in logical replication

2025-08-28 Thread Amit Kapila
On Thu, Aug 28, 2025 at 8:02 AM Zhijie Hou (Fujitsu) wrote: > > I noticed that Cfbot failed to compile the document due to a typo after > renaming > the subscription option. Here are the updated V67 patches to fix that, only > the doc > in 0001 is modified. > I have made a number of cosmetic an

Re: Conflict detection for update_deleted in logical replication

2025-08-28 Thread shveta malik
On Thu, Aug 28, 2025 at 8:02 AM Zhijie Hou (Fujitsu) wrote: > > > I noticed that Cfbot failed to compile the document due to a typo after > renaming > the subscription option. Here are the updated V67 patches to fix that, only > the doc > in 0001 is modified. > Please find a few comments: pat

RE: Conflict detection for update_deleted in logical replication

2025-08-26 Thread Zhijie Hou (Fujitsu)
On Tuesday, August 26, 2025 2:45 PM Amit Kapila wrote: > On Mon, Aug 25, 2025 at 5:05 PM Amit Kapila > wrote: > > > > A few comments on 0001: > > > > Some more comments: Thanks for the comments! > 1. > + /* > + * Return false if the leader apply worker has stopped retaining > + * information f

Re: Conflict detection for update_deleted in logical replication

2025-08-26 Thread shveta malik
Please find some more comments: 1) In CheckSubDeadTupleRetention(), shall we have below instead of retain_dead_tuples check in all conditions? if (retain_dead_tuples) guc checks (wal_level and tracl_commit) else max retention check 2) Currently stop and resume messages are: ~~ LOG: logic

Re: Conflict detection for update_deleted in logical replication

2025-08-26 Thread Dilip Kumar
On Tue, Aug 26, 2025 at 12:15 PM Amit Kapila wrote: > > On Mon, Aug 25, 2025 at 5:05 PM Amit Kapila wrote: > > Some comments on latest patch 0001: 1. + + Note that setting a non-zero value for this option could lead to + information for conflict detection being rem

Re: Conflict detection for update_deleted in logical replication

2025-08-25 Thread Amit Kapila
On Mon, Aug 25, 2025 at 5:05 PM Amit Kapila wrote: > > A few comments on 0001: > Some more comments: 1. + /* + * Return false if the leader apply worker has stopped retaining + * information for detecting conflicts. This implies that update_deleted + * can no longer be reliably detected. + */ + i

Re: Conflict detection for update_deleted in logical replication

2025-08-25 Thread Amit Kapila
On Mon, Aug 25, 2025 at 10:06 AM Zhijie Hou (Fujitsu) wrote: > > Attach the V65 patch set which addressed above and > Shveta's comments[1]. > A few comments on 0001: 1. - if (opts.retaindeadtuples) - CheckSubDeadTupleRetention(true, !sub->enabled, NOTICE); + CheckSubDeadTupleRetention(true, !sub-

Re: Conflict detection for update_deleted in logical replication

2025-08-25 Thread shveta malik
On Mon, Aug 25, 2025 at 12:09 PM shveta malik wrote: > > On Mon, Aug 25, 2025 at 10:06 AM Zhijie Hou (Fujitsu) > wrote: > > > > Attach the V65 patch set which addressed above and > > Shveta's comments[1]. > > > > Thank You for the patches, please find a few comments on v64 itself (I > think valid

Re: Conflict detection for update_deleted in logical replication

2025-08-24 Thread shveta malik
On Mon, Aug 25, 2025 at 10:06 AM Zhijie Hou (Fujitsu) wrote: > > Attach the V65 patch set which addressed above and > Shveta's comments[1]. > Thank You for the patches, please find a few comments on v64 itself (I think valid on v65 as well): 1) in resume_conflict_info_retention(), shall we rewri

Re: Conflict detection for update_deleted in logical replication

2025-08-22 Thread shveta malik
On Fri, Aug 22, 2025 at 5:10 PM Amit Kapila wrote: > > On Mon, Aug 4, 2025 at 3:11 PM Amit Kapila wrote: > > > > On Mon, Aug 4, 2025 at 11:46 AM shveta malik wrote: > > > > > > 7) > > > Shall we rename 'max_conflict_retention_duration' to > > > 'max_conflict_info_retention_duration' as the latte

Re: Conflict detection for update_deleted in logical replication

2025-08-22 Thread Amit Kapila
On Mon, Aug 4, 2025 at 3:11 PM Amit Kapila wrote: > > On Mon, Aug 4, 2025 at 11:46 AM shveta malik wrote: > > > > 7) > > Shall we rename 'max_conflict_retention_duration' to > > 'max_conflict_info_retention_duration' as the latter one is more > > clear? > > > > Before bikeshedding on the name of

Re: Conflict detection for update_deleted in logical replication

2025-08-22 Thread Amit Kapila
On Thu, Aug 21, 2025 at 2:01 PM Zhijie Hou (Fujitsu) wrote: > > Here is the V64 patch set addressing this concern. > Few minor comments: 1. static void process_syncing_tables_for_sync(XLogRecPtr current_lsn) { - SpinLockAcquire(&MyLogicalRepWorker->relmutex); + SpinLockAcquire(&MyLogicalRepWork

Re: Conflict detection for update_deleted in logical replication

2025-08-21 Thread shveta malik
On Thu, Aug 21, 2025 at 2:09 PM Zhijie Hou (Fujitsu) wrote: > > On Thursday, August 21, 2025 2:01 PM shveta malik > wrote: > > > > On Wed, Aug 20, 2025 at 12:12 PM Zhijie Hou (Fujitsu) > > wrote: > > > > > > > > > I agree. Here is V63 version which implements this approach. > > > > > > > Thank

Re: Conflict detection for update_deleted in logical replication

2025-08-21 Thread Mihail Nikalayeu
Hello! Sorry for being noisy - just want to remind: conflict detection system (including new updated_deleted) is giving invalid reports because of the issue related to SnapshotDirty vs btree. So, it is not possible to rely on that at the moment. You may check TAP tests here [0] and some explanati

RE: Conflict detection for update_deleted in logical replication

2025-08-21 Thread Zhijie Hou (Fujitsu)
On Thursday, August 21, 2025 2:01 PM shveta malik wrote: > > On Wed, Aug 20, 2025 at 12:12 PM Zhijie Hou (Fujitsu) > wrote: > > > > > > I agree. Here is V63 version which implements this approach. > > > > Thank You for the patches. > > > The retention status is recorded in the pg_subscription

Re: Conflict detection for update_deleted in logical replication

2025-08-21 Thread Nisha Moond
On Wed, Aug 20, 2025 at 12:12 PM Zhijie Hou (Fujitsu) wrote: > > I agree. Here is V63 version which implements this approach. > Thank you Hou-san for the patches. Here are couple of comments: 1) Once retention is stopped for all subscriptions and conflict_slot.xmin is reset to NULL, we are no lo

Re: Conflict detection for update_deleted in logical replication

2025-08-20 Thread shveta malik
On Wed, Aug 20, 2025 at 12:12 PM Zhijie Hou (Fujitsu) wrote: > > > I agree. Here is V63 version which implements this approach. > Thank You for the patches. > The retention status is recorded in the pg_subscription catalog > (subretentionactive) to prevent unnecessary retention initiation upon s

Re: Conflict detection for update_deleted in logical replication

2025-08-19 Thread Masahiko Sawada
On Sun, Aug 17, 2025 at 10:06 PM Amit Kapila wrote: > > On Sat, Aug 16, 2025 at 5:15 AM Masahiko Sawada wrote: > > > > Regarding the subscription-level option vs. GUC, I don't disagree with > > the current approach. > > > > For the record, while I agree that the subscription-level option is > > m

Re: Conflict detection for update_deleted in logical replication

2025-08-18 Thread Amit Kapila
On Mon, Aug 18, 2025 at 5:05 PM Zhijie Hou (Fujitsu) wrote: > > On Monday, August 18, 2025 2:32 PM Dilip Kumar wrote: > > > > On Mon, Aug 18, 2025 at 10:36 AM Amit Kapila > > wrote: > > > > > > > --- > > > > Even if an apply worker disables retaining dead tuples due to > > > > max_conflict_reten

RE: Conflict detection for update_deleted in logical replication

2025-08-18 Thread Zhijie Hou (Fujitsu)
On Monday, August 18, 2025 2:32 PM Dilip Kumar wrote: > > On Mon, Aug 18, 2025 at 10:36 AM Amit Kapila > wrote: > > > > > --- > > > Even if an apply worker disables retaining dead tuples due to > > > max_conflict_retention_duration, it enables again after the server > > > restarts. > > > > > > >

Re: Conflict detection for update_deleted in logical replication

2025-08-17 Thread Dilip Kumar
On Mon, Aug 18, 2025 at 10:36 AM Amit Kapila wrote: > > > Given that max_conflict_retention_duration works only when > > retain_dead_tuples is enabled, why not merge these two parameters? For > > example, setting max_conflict_retention_duration=-1 means to disable > > retaining dead tuples behavi

Re: Conflict detection for update_deleted in logical replication

2025-08-17 Thread Amit Kapila
On Sat, Aug 16, 2025 at 5:15 AM Masahiko Sawada wrote: > > Regarding the subscription-level option vs. GUC, I don't disagree with > the current approach. > > For the record, while I agree that the subscription-level option is > more consistent with the existing retain_dead_tuples option and can >

RE: Conflict detection for update_deleted in logical replication

2025-08-17 Thread Zhijie Hou (Fujitsu)
On Saturday, August 16, 2025 7:44 AM Masahiko Sawada wrote: > Here are review comments on v62 patch: Thanks for the comments! > > > --- > +else if (IsSet(supported_opts, > SUBOPT_MAX_CONFLICT_RETENTION_DURATION) && > + strcmp(defel->defname, > "

Re: Conflict detection for update_deleted in logical replication

2025-08-15 Thread Masahiko Sawada
On Thu, Aug 14, 2025 at 10:46 PM Zhijie Hou (Fujitsu) wrote: > > On Thursday, August 14, 2025 11:46 AM shveta malik > wrote: > > > > On Wed, Aug 13, 2025 at 4:15 PM shveta malik > > wrote: > > > > > > On Wed, Aug 13, 2025 at 10:41 AM Zhijie Hou (Fujitsu) > > > wrote: > > > > > > > > > > > > He

Re: Conflict detection for update_deleted in logical replication

2025-08-14 Thread Amit Kapila
On Thu, Aug 14, 2025 at 9:02 AM shveta malik wrote: > > On Wed, Aug 13, 2025 at 5:42 PM Amit Kapila wrote: > > > > > > > > 4) > > > For the DETAIL part of resume and stop messages, how about these: > > > > > > The retention duration for information used in conflict detection has > > > exceeded th

Re: Conflict detection for update_deleted in logical replication

2025-08-14 Thread shveta malik
Few more comments: 1) src/sgml/monitoring.sgml: + + True if retain_dead_tuples + is enabled and the duration for which information useful for conflict + detection is retained by this apply worker does not exceed + max_conflict_retention_duration; NULL for + par

Re: Conflict detection for update_deleted in logical replication

2025-08-13 Thread shveta malik
On Wed, Aug 13, 2025 at 4:15 PM shveta malik wrote: > > On Wed, Aug 13, 2025 at 10:41 AM Zhijie Hou (Fujitsu) > wrote: > > > > > > Here is the V61 patch set which addressed above comments and the comment by > > Nisha[2]. > > > > Thank You for the patch. I tested the patch, please find a few comm

Re: Conflict detection for update_deleted in logical replication

2025-08-13 Thread shveta malik
On Wed, Aug 13, 2025 at 5:42 PM Amit Kapila wrote: > > > > > 4) > > For the DETAIL part of resume and stop messages, how about these: > > > > The retention duration for information used in conflict detection has > > exceeded the limit of xx. > > The retention duration for information used in confl

Re: Conflict detection for update_deleted in logical replication

2025-08-13 Thread Amit Kapila
On Wed, Aug 13, 2025 at 4:15 PM shveta malik wrote: > > On Wed, Aug 13, 2025 at 10:41 AM Zhijie Hou (Fujitsu) > wrote: > > > > > > Here is the V61 patch set which addressed above comments and the comment by > > Nisha[2]. > > > > Thank You for the patch. I tested the patch, please find a few comm

Re: Conflict detection for update_deleted in logical replication

2025-08-13 Thread shveta malik
On Wed, Aug 13, 2025 at 10:41 AM Zhijie Hou (Fujitsu) wrote: > > > Here is the V61 patch set which addressed above comments and the comment by > Nisha[2]. > Thank You for the patch. I tested the patch, please find a few comments: 1) Now when it stops-retention and later resumes it due to the fa

Re: Conflict detection for update_deleted in logical replication

2025-08-12 Thread Nisha Moond
On Mon, Aug 11, 2025 at 2:40 PM Zhijie Hou (Fujitsu) wrote: > > I agree. So, following the above points and some off-list discussions, I have > revised the option to be a subscription option in the V60 version. > Thanks Hou-san for the patches. I have tested the patches and are working as expecte

Re: Conflict detection for update_deleted in logical replication

2025-08-12 Thread shveta malik
On Tue, Aug 12, 2025 at 3:22 PM Dilip Kumar wrote: > > On Tue, Aug 12, 2025 at 3:02 PM Zhijie Hou (Fujitsu) > wrote: > > > > On Tuesday, August 12, 2025 5:01 PM Dilip Kumar > > wrote: > > > > Hi, > > > > > > > > On Tue, Aug 12, 2025 at 2:21 PM Amit Kapila > > > wrote: > > > > > > > > On Tue, A

Re: Conflict detection for update_deleted in logical replication

2025-08-12 Thread Dilip Kumar
On Tue, Aug 12, 2025 at 3:02 PM Zhijie Hou (Fujitsu) wrote: > > On Tuesday, August 12, 2025 5:01 PM Dilip Kumar wrote: > > Hi, > > > > > On Tue, Aug 12, 2025 at 2:21 PM Amit Kapila > > wrote: > > > > > > On Tue, Aug 12, 2025 at 2:06 PM shveta malik > > wrote: > > > > > > > > 2) > > > > postgres

RE: Conflict detection for update_deleted in logical replication

2025-08-12 Thread Zhijie Hou (Fujitsu)
On Tuesday, August 12, 2025 5:01 PM Dilip Kumar wrote: Hi, > > On Tue, Aug 12, 2025 at 2:21 PM Amit Kapila > wrote: > > > > On Tue, Aug 12, 2025 at 2:06 PM shveta malik > wrote: > > > > > > 2) > > > postgres=# create subscription sub2 connection 'dbname=postgres > > > host=localhost user=shv

Re: Conflict detection for update_deleted in logical replication

2025-08-12 Thread Dilip Kumar
On Tue, Aug 12, 2025 at 2:21 PM Amit Kapila wrote: > > On Tue, Aug 12, 2025 at 2:06 PM shveta malik wrote: > > > > 2) > > postgres=# create subscription sub2 connection 'dbname=postgres > > host=localhost user=shveta port=5433' publication pub2 WITH > > (retain_dead_tuples = false, max_conflict_

Re: Conflict detection for update_deleted in logical replication

2025-08-12 Thread Amit Kapila
On Tue, Aug 12, 2025 at 2:06 PM shveta malik wrote: > > 2) > postgres=# create subscription sub2 connection 'dbname=postgres > host=localhost user=shveta port=5433' publication pub2 WITH > (retain_dead_tuples = false, max_conflict_retention_duration=1000); > NOTICE: created replication slot "sub

Re: Conflict detection for update_deleted in logical replication

2025-08-12 Thread shveta malik
On Mon, Aug 11, 2025 at 2:40 PM Zhijie Hou (Fujitsu) wrote: > > I agree. So, following the above points and some off-list discussions, I have > revised the option to be a subscription option in the V60 version. > Thank You for the patches. Tried to test the new sub-level parameter, have few comme

RE: Conflict detection for update_deleted in logical replication

2025-08-08 Thread Zhijie Hou (Fujitsu)
On Friday, August 8, 2025 2:34 PM shveta malik wrote: > > On Thu, Aug 7, 2025 at 10:10 AM Zhijie Hou (Fujitsu) > wrote: > > > > On Tuesday, August 5, 2025 10:09 AM Zhijie Hou (Fujitsu) > wrote: > > > Here is V57 patch set which addressed most of comments. > > > > > > In this version, I also fix

Re: Conflict detection for update_deleted in logical replication

2025-08-07 Thread shveta malik
On Thu, Aug 7, 2025 at 10:10 AM Zhijie Hou (Fujitsu) wrote: > > On Tuesday, August 5, 2025 10:09 AM Zhijie Hou (Fujitsu) > wrote: > > Here is V57 patch set which addressed most of comments. > > > > In this version, I also fixed a bug that the apply worker continued to find > > dead > > tuples e

Re: Conflict detection for update_deleted in logical replication

2025-08-07 Thread shveta malik
On Thu, Aug 7, 2025 at 6:05 PM Dilip Kumar wrote: > > On Mon, Aug 4, 2025 at 3:11 PM Amit Kapila wrote: > > > > On Mon, Aug 4, 2025 at 11:46 AM shveta malik wrote: > > > > > > 7) > > > Shall we rename 'max_conflict_retention_duration' to > > > 'max_conflict_info_retention_duration' as the latter

Re: Conflict detection for update_deleted in logical replication

2025-08-07 Thread Dilip Kumar
On Mon, Aug 4, 2025 at 3:11 PM Amit Kapila wrote: > > On Mon, Aug 4, 2025 at 11:46 AM shveta malik wrote: > > > > 7) > > Shall we rename 'max_conflict_retention_duration' to > > 'max_conflict_info_retention_duration' as the latter one is more > > clear? > > > > Before bikeshedding on the name of

RE: Conflict detection for update_deleted in logical replication

2025-08-06 Thread Zhijie Hou (Fujitsu)
On Tuesday, August 5, 2025 10:09 AM Zhijie Hou (Fujitsu) wrote: > Here is V57 patch set which addressed most of comments. > > In this version, I also fixed a bug that the apply worker continued to find > dead > tuples even if it has already stop retaining dead tuples. Here is a V58 patch set w

RE: Conflict detection for update_deleted in logical replication

2025-08-04 Thread Zhijie Hou (Fujitsu)
On Monday, August 4, 2025 2:16 PM shveta malik wrote: > > On Fri, Aug 1, 2025 at 9:16 PM Zhijie Hou (Fujitsu) > wrote: > > > > > > Thanks for confirming. Here is V56 patch set which addressed all the > > comments including the comments from Amit[1] and Shveta[2]. > > > > I have merged V55-0002 i

Re: Conflict detection for update_deleted in logical replication

2025-08-04 Thread Amit Kapila
On Mon, Aug 4, 2025 at 11:46 AM shveta malik wrote: > > 7) > Shall we rename 'max_conflict_retention_duration' to > 'max_conflict_info_retention_duration' as the latter one is more > clear? > Before bikeshedding on the name of this option, I would like us to once again consider whether we should

Re: Conflict detection for update_deleted in logical replication

2025-08-03 Thread shveta malik
On Fri, Aug 1, 2025 at 9:16 PM Zhijie Hou (Fujitsu) wrote: > > > Thanks for confirming. Here is V56 patch set which addressed all the > comments including the comments from Amit[1] and Shveta[2]. > > I have merged V55-0002 into 0001 and updated the list of author > and reviewers based on my knowle

Re: Conflict detection for update_deleted in logical replication

2025-08-03 Thread Dilip Kumar
On Fri, Aug 1, 2025 at 9:16 PM Zhijie Hou (Fujitsu) wrote: > > On Friday, August 1, 2025 7:42 PM Dilip Kumar wrote: > > > > On Fri, Aug 1, 2025 at 5:02 PM Zhijie Hou (Fujitsu) > > wrote: > > > > > > > 2. > > > > > > > > + If set to true, the detection of > > > > + is enabled,

RE: Conflict detection for update_deleted in logical replication

2025-08-01 Thread Zhijie Hou (Fujitsu)
On Friday, August 1, 2025 7:42 PM Dilip Kumar wrote: > > On Fri, Aug 1, 2025 at 5:02 PM Zhijie Hou (Fujitsu) > wrote: > > > > > 2. > > > > > > + If set to true, the detection of > > > + is enabled, and > > > + a > > > physical > > > + replication slot named > > > pg_c

Re: Conflict detection for update_deleted in logical replication

2025-08-01 Thread Dilip Kumar
On Fri, Aug 1, 2025 at 5:02 PM Zhijie Hou (Fujitsu) wrote: > > > 2. > > > > + If set to true, the detection of > > + is enabled, and a > > physical > > + replication slot named > > pg_conflict_detection > >created on the subscriber to prevent the conflict in

Re: Conflict detection for update_deleted in logical replication

2025-08-01 Thread Dilip Kumar
On Fri, Aug 1, 2025 at 4:46 PM Amit Kapila wrote: > > On Fri, Aug 1, 2025 at 4:20 PM Amit Kapila wrote: > > > > On Fri, Aug 1, 2025 at 3:58 PM Dilip Kumar wrote: > > > > > > 4. > > > + /* > > > +* Instead of invoking GetOldestNonRemovableTransactionId() for > > > conflict > > > +* det

RE: Conflict detection for update_deleted in logical replication

2025-08-01 Thread Zhijie Hou (Fujitsu)
> -Original Message- > From: Dilip Kumar > Sent: Friday, August 1, 2025 6:29 PM > To: Hou, Zhijie/侯 志杰 > Cc: Amit Kapila ; shveta malik > ; Kuroda, Hayato/黒田 隼人 > ; pgsql-hackers > ; vignesh C ; Nisha > Moond ; Masahiko Sawada > > Subject: Re: Conflic

Re: Conflict detection for update_deleted in logical replication

2025-08-01 Thread Amit Kapila
On Fri, Aug 1, 2025 at 4:20 PM Amit Kapila wrote: > > On Fri, Aug 1, 2025 at 3:58 PM Dilip Kumar wrote: > > > > 4. > > + /* > > +* Instead of invoking GetOldestNonRemovableTransactionId() for conflict > > +* detection, we use the conflict detection slot.xmin. This value will > > be > >

Re: Conflict detection for update_deleted in logical replication

2025-08-01 Thread Amit Kapila
On Fri, Aug 1, 2025 at 3:58 PM Dilip Kumar wrote: > > 4. > + /* > +* Instead of invoking GetOldestNonRemovableTransactionId() for conflict > +* detection, we use the conflict detection slot.xmin. This value will be > +* greater than or equal to the other threshold and provides a more

Re: Conflict detection for update_deleted in logical replication

2025-08-01 Thread Dilip Kumar
On Thu, Jul 31, 2025 at 3:49 PM Zhijie Hou (Fujitsu) wrote: > > On Thursday, July 31, 2025 5:29 PM Amit Kapila > wrote: > > > > On Tue, Jul 29, 2025 at 10:51 AM Zhijie Hou (Fujitsu) > > > > wrote: > > > > > > This is the V54 patch set, with only patch 0001 updated to address the > > > latest c

Re: Conflict detection for update_deleted in logical replication

2025-07-31 Thread shveta malik
On Thu, Jul 31, 2025 at 3:49 PM Zhijie Hou (Fujitsu) wrote: > > On Thursday, July 31, 2025 5:26 PM shveta malik > wrote: > > > > On Tue, Jul 29, 2025 at 10:51 AM Zhijie Hou (Fujitsu) > > > > wrote: > > > > > > > > > This is the V54 patch set, with only patch 0001 updated to address the > > > l

RE: Conflict detection for update_deleted in logical replication

2025-07-31 Thread Zhijie Hou (Fujitsu)
On Thursday, July 31, 2025 5:26 PM shveta malik wrote: > > On Tue, Jul 29, 2025 at 10:51 AM Zhijie Hou (Fujitsu) > wrote: > > > > > > This is the V54 patch set, with only patch 0001 updated to address the > > latest comments. > > > > Thanks for the patch. > > While performing tests on the late

RE: Conflict detection for update_deleted in logical replication

2025-07-31 Thread Zhijie Hou (Fujitsu)
On Thursday, July 31, 2025 5:29 PM Amit Kapila wrote: > > On Tue, Jul 29, 2025 at 10:51 AM Zhijie Hou (Fujitsu) > wrote: > > > > This is the V54 patch set, with only patch 0001 updated to address the > > latest comments. > > > > Few minor comments: Thanks for the comments. > 1. > /* The row t

Re: Conflict detection for update_deleted in logical replication

2025-07-31 Thread Amit Kapila
On Tue, Jul 29, 2025 at 10:51 AM Zhijie Hou (Fujitsu) wrote: > > This is the V54 patch set, with only patch 0001 updated to address the latest > comments. > Few minor comments: 1. /* The row to be updated was deleted by a different origin */ CT_UPDATE_DELETED, /* The row to be updated was modifie

  1   2   3   4   >