RE: Conflict detection for update_deleted in logical replication

2025-07-28 Thread Zhijie Hou (Fujitsu)
On Monday, July 28, 2025 5:54 PM Amit Kapila wrote: > > On Fri, Jul 25, 2025 at 4:38 PM Zhijie Hou (Fujitsu) > wrote: > > > > Right, I think it makes sense to do with the index scan when the > > index's xmin is less than the conflict detection xmin, as that can > > ensure that all the tuples del

RE: Conflict detection for update_deleted in logical replication

2025-07-28 Thread Zhijie Hou (Fujitsu)
On Monday, July 28, 2025 7:43 PM shveta malik wrote: > On Mon, Jul 28, 2025 at 4:38 PM shveta malik > wrote: > > > > On Fri, Jul 25, 2025 at 4:38 PM Zhijie Hou (Fujitsu) > > wrote: > > > > > > > > > The V53-0001 also includes Shveta's comments in [1]. > > > > > > > Thanks, I have not yet complet

Re: Conflict detection for update_deleted in logical replication

2025-07-28 Thread shveta malik
On Mon, Jul 28, 2025 at 4:38 PM shveta malik wrote: > > On Fri, Jul 25, 2025 at 4:38 PM Zhijie Hou (Fujitsu) > wrote: > > > > > > The V53-0001 also includes Shveta's comments in [1]. > > > > Thanks, I have not yet completed the review, but please find a few > comments on 001: > > 1) > IsIndexUsab

Re: Conflict detection for update_deleted in logical replication

2025-07-28 Thread shveta malik
On Fri, Jul 25, 2025 at 4:38 PM Zhijie Hou (Fujitsu) wrote: > > > The V53-0001 also includes Shveta's comments in [1]. > Thanks, I have not yet completed the review, but please find a few comments on 001: 1) IsIndexUsableForFindingDeletedTuple() We first have: + /* + * A frozen transaction ID in

Re: Conflict detection for update_deleted in logical replication

2025-07-28 Thread Amit Kapila
On Fri, Jul 25, 2025 at 4:38 PM Zhijie Hou (Fujitsu) wrote: > > Right, I think it makes sense to do with the index scan when the index's xmin > is > less than the conflict detection xmin, as that can ensure that all the tuples > deleted before the index creation or re-indexing are irrelevant for

Re: Conflict detection for update_deleted in logical replication

2025-07-25 Thread shveta malik
On Fri, Jul 25, 2025 at 2:31 PM Amit Kapila wrote: > > On Fri, Jul 25, 2025 at 12:37 PM shveta malik wrote: > > > > On Thu, Jul 24, 2025 at 9:12 AM shveta malik wrote: > > > > > > > > > 2) > > > + if (MySubscription->retaindeadtuples && > > > + FindMostRecentl

Re: Conflict detection for update_deleted in logical replication

2025-07-25 Thread Nisha Moond
Hi All, We conducted performance testing of a bi-directional logical replication setup, focusing on the primary use case of the update_deleted feature. To simulate a realistic scenario, we used a high workload with limited concurrent updates, and well-distributed writes among servers. Used source

RE: Conflict detection for update_deleted in logical replication

2025-07-25 Thread Zhijie Hou (Fujitsu)
On Friday, July 25, 2025 2:33 PM Amit Kapila wrote: > > On Wed, Jul 23, 2025 at 12:53 PM Zhijie Hou (Fujitsu) > wrote: > > > > Thanks for pushing. I have rebased the remaining patches. > > > > + * This function performs a full table scan instead of using indexes > + because > + * index scans co

RE: Conflict detection for update_deleted in logical replication

2025-07-25 Thread Zhijie Hou (Fujitsu)
On Thursday, July 24, 2025 11:42 AM shveta malik wrote: > > On Wed, Jul 23, 2025 at 12:53 PM Zhijie Hou (Fujitsu) > wrote: > > > > On Wednesday, July 23, 2025 12:08 PM Amit Kapila > wrote: > > > > > > On Wed, Jul 23, 2025 at 3:51 AM Masahiko Sawada > > > wrote: > > > > > > > > I've reviewed th

Re: Conflict detection for update_deleted in logical replication

2025-07-25 Thread Amit Kapila
On Fri, Jul 25, 2025 at 12:37 PM shveta malik wrote: > > On Thu, Jul 24, 2025 at 9:12 AM shveta malik wrote: > > > > > > 2) > > + if (MySubscription->retaindeadtuples && > > + FindMostRecentlyDeletedTupleInfo(localrel, > > remoteslot, > > + > >

Re: Conflict detection for update_deleted in logical replication

2025-07-25 Thread shveta malik
On Thu, Jul 24, 2025 at 9:12 AM shveta malik wrote: > > > 2) > + if (MySubscription->retaindeadtuples && > + FindMostRecentlyDeletedTupleInfo(localrel, remoteslot, > + > &conflicttuple.xmin, > + > &conflicttuple.origin, > + >

Re: Conflict detection for update_deleted in logical replication

2025-07-24 Thread Amit Kapila
On Wed, Jul 23, 2025 at 12:53 PM Zhijie Hou (Fujitsu) wrote: > > Thanks for pushing. I have rebased the remaining patches. > + * This function performs a full table scan instead of using indexes because + * index scans could miss deleted tuples if an index has been re-indexed or + * re-created du

Re: Conflict detection for update_deleted in logical replication

2025-07-23 Thread shveta malik
On Wed, Jul 23, 2025 at 12:53 PM Zhijie Hou (Fujitsu) wrote: > > On Wednesday, July 23, 2025 12:08 PM Amit Kapila > wrote: > > > > On Wed, Jul 23, 2025 at 3:51 AM Masahiko Sawada > > wrote: > > > > > > I've reviewed the 0001 patch and it looks good to me. > > > > > > > Thanks, I have pushed the

RE: Conflict detection for update_deleted in logical replication

2025-07-23 Thread Zhijie Hou (Fujitsu)
On Wednesday, July 23, 2025 12:08 PM Amit Kapila wrote: > > On Wed, Jul 23, 2025 at 3:51 AM Masahiko Sawada > wrote: > > > > I've reviewed the 0001 patch and it looks good to me. > > > > Thanks, I have pushed the 0001 patch. Thanks for pushing. I have rebased the remaining patches. I have re

Re: Conflict detection for update_deleted in logical replication

2025-07-22 Thread Amit Kapila
On Wed, Jul 23, 2025 at 3:51 AM Masahiko Sawada wrote: > > I've reviewed the 0001 patch and it looks good to me. > Thanks, I have pushed the 0001 patch. The patch still > has XXX comments at several places. Do we want to keep all of them > Yes, those are primarily the ideas for future optimiza

Re: Conflict detection for update_deleted in logical replication

2025-07-22 Thread Masahiko Sawada
On Mon, Jul 21, 2025 at 8:49 PM Amit Kapila wrote: > > On Mon, Jul 21, 2025 at 11:27 PM Masahiko Sawada > wrote: > > > > On Sun, Jul 20, 2025 at 9:00 PM Amit Kapila wrote: > > > > > > > > If so, I agree > > > with you, we don't need XIDs of other databases as logical WALSender > > > will anyway

Re: Conflict detection for update_deleted in logical replication

2025-07-21 Thread Amit Kapila
On Mon, Jul 21, 2025 at 11:27 PM Masahiko Sawada wrote: > > On Sun, Jul 20, 2025 at 9:00 PM Amit Kapila wrote: > > > > > If so, I agree > > with you, we don't need XIDs of other databases as logical WALSender > > will anyway won't process transactions in other databases, so we can > > exclude tho

Re: Conflict detection for update_deleted in logical replication

2025-07-21 Thread Masahiko Sawada
On Sun, Jul 20, 2025 at 9:00 PM Amit Kapila wrote: > > On Sat, Jul 19, 2025 at 10:32 AM Amit Kapila wrote: > > > > On Sat, Jul 19, 2025 at 3:01 AM Masahiko Sawada > > wrote: > > > > > > On Fri, Jul 18, 2025 at 5:03 AM Zhijie Hou (Fujitsu) > > > wrote: > > > > > > > > > > Here are some review c

Re: Conflict detection for update_deleted in logical replication

2025-07-20 Thread Amit Kapila
On Sat, Jul 19, 2025 at 10:32 AM Amit Kapila wrote: > > On Sat, Jul 19, 2025 at 3:01 AM Masahiko Sawada wrote: > > > > On Fri, Jul 18, 2025 at 5:03 AM Zhijie Hou (Fujitsu) > > wrote: > > > > > > > Here are some review comments and questions: > > > > > > --- > > + if (inCommitOnly && > > +

Re: Conflict detection for update_deleted in logical replication

2025-07-18 Thread Amit Kapila
On Sat, Jul 19, 2025 at 3:01 AM Masahiko Sawada wrote: > > On Fri, Jul 18, 2025 at 5:03 AM Zhijie Hou (Fujitsu) > wrote: > > > > Here are some review comments and questions: > > > --- > + if (inCommitOnly && > + (proc->delayChkptFlags & DELAY_CHKPT_IN_COMMIT) == 0) > + c

Re: Conflict detection for update_deleted in logical replication

2025-07-18 Thread Masahiko Sawada
On Fri, Jul 18, 2025 at 5:03 AM Zhijie Hou (Fujitsu) wrote: > > On Friday, July 18, 2025 1:25 AM Masahiko Sawada > wrote: > > > > On Fri, Jul 11, 2025 at 3:58 AM Amit Kapila wrote: > > > > > > On Thu, Jul 10, 2025 at 6:46 PM Masahiko Sawada > > wrote: > > > > > > > > On Wed, Jul 9, 2025 at 9:0

Re: Conflict detection for update_deleted in logical replication

2025-07-18 Thread Masahiko Sawada
On Mon, Jul 7, 2025 at 3:31 AM Zhijie Hou (Fujitsu) wrote: > > On Mon, Jul 7, 2025 at 10:13 AM Zhijie Hou (Fujitsu) wrote: > > > > On Sun, Jul 6, 2025 at 10:51 PM Masahiko Sawada wrote: > > > > > > On Fri, Jul 4, 2025 at 8:18 PM Zhijie Hou (Fujitsu) > > > > > > wrote: > > > > > > > > On Wed, Jul

Re: Conflict detection for update_deleted in logical replication

2025-07-18 Thread Nisha Moond
On Thu, Jul 17, 2025 at 4:44 PM shveta malik wrote: > > On Thu, Jul 17, 2025 at 9:56 AM Dilip Kumar wrote: > > > > On Fri, Jul 11, 2025 at 4:28 PM Amit Kapila wrote: > > > > > > On Thu, Jul 10, 2025 at 6:46 PM Masahiko Sawada > > > wrote: > > > > > > > > On Wed, Jul 9, 2025 at 9:09 PM Amit Kap

Re: Conflict detection for update_deleted in logical replication

2025-07-17 Thread Dilip Kumar
On Thu, Jul 17, 2025 at 4:44 PM shveta malik wrote: > > On Thu, Jul 17, 2025 at 9:56 AM Dilip Kumar wrote: > > > > I was just thinking about what are the most practical use cases where > > a user would need multiple active writer nodes. Most applications > > typically function well with a single

Re: Conflict detection for update_deleted in logical replication

2025-07-17 Thread Masahiko Sawada
On Fri, Jul 11, 2025 at 3:58 AM Amit Kapila wrote: > > On Thu, Jul 10, 2025 at 6:46 PM Masahiko Sawada wrote: > > > > On Wed, Jul 9, 2025 at 9:09 PM Amit Kapila wrote: > > > > > > > > > I think that even with retain_conflict_info = off, there is probably a > > > > point at which the subscriber c

Re: Conflict detection for update_deleted in logical replication

2025-07-17 Thread shveta malik
On Thu, Jul 17, 2025 at 9:56 AM Dilip Kumar wrote: > > On Fri, Jul 11, 2025 at 4:28 PM Amit Kapila wrote: > > > > On Thu, Jul 10, 2025 at 6:46 PM Masahiko Sawada > > wrote: > > > > > > On Wed, Jul 9, 2025 at 9:09 PM Amit Kapila > > > wrote: > > > > > > > > > > > > I think that even with retai

Re: Conflict detection for update_deleted in logical replication

2025-07-16 Thread Dilip Kumar
On Fri, Jul 11, 2025 at 4:28 PM Amit Kapila wrote: > > On Thu, Jul 10, 2025 at 6:46 PM Masahiko Sawada wrote: > > > > On Wed, Jul 9, 2025 at 9:09 PM Amit Kapila wrote: > > > > > > > > > I think that even with retain_conflict_info = off, there is probably a > > > > point at which the subscriber c

Re: Conflict detection for update_deleted in logical replication

2025-07-11 Thread Amit Kapila
On Thu, Jul 10, 2025 at 6:46 PM Masahiko Sawada wrote: > > On Wed, Jul 9, 2025 at 9:09 PM Amit Kapila wrote: > > > > > > I think that even with retain_conflict_info = off, there is probably a > > > point at which the subscriber can no longer keep up with the > > > publisher. For example, if with

Re: Conflict detection for update_deleted in logical replication

2025-07-10 Thread Masahiko Sawada
On Wed, Jul 9, 2025 at 9:09 PM Amit Kapila wrote: > > On Tue, Jul 8, 2025 at 12:18 AM Masahiko Sawada wrote: > > > > On Mon, Jul 7, 2025 at 12:03 PM Zhijie Hou (Fujitsu) > > wrote: > > > > I think these performance regressions occur because at some point the > > subscriber can no longer keep up

RE: Conflict detection for update_deleted in logical replication

2025-07-10 Thread Hayato Kuroda (Fujitsu)
Dear hackers, > -- Also, worker and logical replication related parameters were increased >as needed (see attached scripts for details). Sorry, I forgot to attach scripts. Best regards, Hayato Kuroda FUJITSU LIMITED row_filter_measure.sh Description: row_filter_measure.sh row_fi

RE: Conflict detection for update_deleted in logical replication

2025-07-10 Thread Hayato Kuroda (Fujitsu)
Dear hackers, > As per the above observations, it is less of a regression of this > feature but more of a lack of parallel apply or some kind of pre-fetch > for apply, as is recently proposed [1]. I feel there are use cases, as > explained above, for which this feature would work without any > dow

Re: Conflict detection for update_deleted in logical replication

2025-07-09 Thread Nisha Moond
On Wed, Jul 9, 2025 at 5:39 PM Amit Kapila wrote: > > On Tue, Jul 8, 2025 at 12:18 AM Masahiko Sawada wrote: > > > > On Mon, Jul 7, 2025 at 12:03 PM Zhijie Hou (Fujitsu) > > wrote: > > > > I think these performance regressions occur because at some point the > > subscriber can no longer keep up

RE: Conflict detection for update_deleted in logical replication

2025-07-09 Thread Hayato Kuroda (Fujitsu)
Dear hackers, All the tests in [1] are done with autovacuum=off, so I checked how would be in autovacuum=on case. Highlights == * Regression on subscriber-side became bit larger than autovacuum=on case when pgbench was run on both side * Other than that, they were mostly same. Used sou

Re: Conflict detection for update_deleted in logical replication

2025-07-09 Thread Amit Kapila
On Tue, Jul 8, 2025 at 12:18 AM Masahiko Sawada wrote: > > On Mon, Jul 7, 2025 at 12:03 PM Zhijie Hou (Fujitsu) > wrote: > > I think these performance regressions occur because at some point the > subscriber can no longer keep up with the changes occurring on the > publisher. This is because the

Re: Conflict detection for update_deleted in logical replication

2025-07-07 Thread Masahiko Sawada
On Mon, Jul 7, 2025 at 12:29 PM Hayato Kuroda (Fujitsu) wrote: > > Dear Sawada-san, > > > What does each duration mean in these results? Can we interpret the > > test case of max_conflict_retention_duration=120s that when 7 clients > > and 15 clients are working on the publisher and the subscriber

Re: Conflict detection for update_deleted in logical replication

2025-07-07 Thread Masahiko Sawada
On Mon, Jul 7, 2025 at 12:03 PM Zhijie Hou (Fujitsu) wrote: > > On Sun, Jul 6, 2025 at 10:51 PM Masahiko Sawada wrote: > > > > > On Sun, Jul 6, 2025 at 8:03 PM Hayato Kuroda (Fujitsu) > > wrote: > > > > > > Dear hackers, > > > > > > As a confirmation purpose, I did performance testing with four >

RE: Conflict detection for update_deleted in logical replication

2025-07-07 Thread Zhijie Hou (Fujitsu)
On Mon, Jul 7, 2025 at 11:03 AM Zhijie Hou (Fujitsu) wrote: > > On Sun, Jul 6, 2025 at 10:51 PM Masahiko Sawada wrote: > > > == > > > == > > > The workload is mostly same as [4]. > > > > > > Workload: > > > - Initially ran pgbenc

RE: Conflict detection for update_deleted in logical replication

2025-07-06 Thread Hayato Kuroda (Fujitsu)
Dear Sawada-san, > What does each duration mean in these results? Can we interpret the > test case of max_conflict_retention_duration=120s that when 7 clients > and 15 clients are working on the publisher and the subscriber > respectively, the TPS on the subscriber was about one fourth (17835.3 >

RE: Conflict detection for update_deleted in logical replication

2025-07-06 Thread Zhijie Hou (Fujitsu)
On Sun, Jul 6, 2025 at 10:51 PM Masahiko Sawada wrote: > > On Sun, Jul 6, 2025 at 8:03 PM Hayato Kuroda (Fujitsu) > wrote: > > > > Dear hackers, > > > > As a confirmation purpose, I did performance testing with four > > workloads we did before. > > Thank you for doing the performance tests! >

RE: Conflict detection for update_deleted in logical replication

2025-07-06 Thread Zhijie Hou (Fujitsu)
On Sun, Jul 6, 2025 at 10:51 PM Masahiko Sawada wrote: > > On Fri, Jul 4, 2025 at 8:18 PM Zhijie Hou (Fujitsu) > wrote: > > > > On Wed, Jul 2, 2025 at 3:28 PM Hou, Zhijie wrote: > > > Kindly use the latest patch set for performance testing. > > > > During testing, we observed a limitation in casc

Re: Conflict detection for update_deleted in logical replication

2025-07-06 Thread Masahiko Sawada
On Sun, Jul 6, 2025 at 8:03 PM Hayato Kuroda (Fujitsu) wrote: > > Dear hackers, > > As a confirmation purpose, I did performance testing with four workloads > we did before. Thank you for doing the performance tests! > > 03. pgbench on both sides > > The workload is most

Re: Conflict detection for update_deleted in logical replication

2025-07-06 Thread Masahiko Sawada
On Fri, Jul 4, 2025 at 8:18 PM Zhijie Hou (Fujitsu) wrote: > > On Wed, Jul 2, 2025 at 3:28 PM Hou, Zhijie wrote: > > Kindly use the latest patch set for performance testing. > > During testing, we observed a limitation in cascading logical replication > setups, such as (A -> B -> C). When retain_c

RE: Conflict detection for update_deleted in logical replication

2025-07-06 Thread Hayato Kuroda (Fujitsu)
Dear hackers, As a confirmation purpose, I did performance testing with four workloads we did before. Highlights == The retests on the latest patch set v46 show results consistent with previous observations: - There is no performance impact on the publisher side - There is no performanc

Re: Conflict detection for update_deleted in logical replication

2025-07-05 Thread Dilip Kumar
On Sat, Jul 5, 2025 at 2:26 PM Dilip Kumar wrote: > > On Fri, Jul 4, 2025 at 4:48 PM Zhijie Hou (Fujitsu) > wrote: > > > > On Wed, Jul 2, 2025 at 3:28 PM Hou, Zhijie wrote: > > > Kindly use the latest patch set for performance testing. > > > > During testing, we observed a limitation in cascading

Re: Conflict detection for update_deleted in logical replication

2025-07-05 Thread Dilip Kumar
On Fri, Jul 4, 2025 at 4:48 PM Zhijie Hou (Fujitsu) wrote: > > On Wed, Jul 2, 2025 at 3:28 PM Hou, Zhijie wrote: > > Kindly use the latest patch set for performance testing. > > During testing, we observed a limitation in cascading logical replication > setups, such as (A -> B -> C). When retain_c

Re: Conflict detection for update_deleted in logical replication

2025-07-03 Thread Dilip Kumar
On Thu, Jul 3, 2025 at 4:21 PM Amit Kapila wrote: > > On Thu, Jul 3, 2025 at 10:57 AM Dilip Kumar wrote: > > > > On Thu, Jul 3, 2025 at 10:43 AM Amit Kapila wrote: > > > > > > On Thu, Jul 3, 2025 at 10:26 AM Dilip Kumar wrote: > > > > > > > > > > > > You changes related to write barrier LGTM, h

Re: Conflict detection for update_deleted in logical replication

2025-07-03 Thread Amit Kapila
On Thu, Jul 3, 2025 at 10:57 AM Dilip Kumar wrote: > > On Thu, Jul 3, 2025 at 10:43 AM Amit Kapila wrote: > > > > On Thu, Jul 3, 2025 at 10:26 AM Dilip Kumar wrote: > > > > > > > > > You changes related to write barrier LGTM, however I have question > > > regarding below change, IIUC, in logical

Re: Conflict detection for update_deleted in logical replication

2025-07-02 Thread Dilip Kumar
On Thu, Jul 3, 2025 at 10:43 AM Amit Kapila wrote: > > On Thu, Jul 3, 2025 at 10:26 AM Dilip Kumar wrote: > > > > On Wed, Jul 2, 2025 at 12:58 PM Zhijie Hou (Fujitsu) > > wrote: > > > > > > > > During local testing, I discovered a bug caused by my oversight in > > > assigning > > > the new xmin

Re: Conflict detection for update_deleted in logical replication

2025-07-02 Thread Amit Kapila
On Thu, Jul 3, 2025 at 10:26 AM Dilip Kumar wrote: > > On Wed, Jul 2, 2025 at 12:58 PM Zhijie Hou (Fujitsu) > wrote: > > > > > During local testing, I discovered a bug caused by my oversight in assigning > > the new xmin to slot.effective, which resulted in dead tuples remaining > > non-removable

Re: Conflict detection for update_deleted in logical replication

2025-07-02 Thread Dilip Kumar
On Wed, Jul 2, 2025 at 12:58 PM Zhijie Hou (Fujitsu) wrote: > > During local testing, I discovered a bug caused by my oversight in assigning > the new xmin to slot.effective, which resulted in dead tuples remaining > non-removable until restart. I apologize for the error and have provided > corre

RE: Conflict detection for update_deleted in logical replication

2025-07-01 Thread Zhijie Hou (Fujitsu)
On Tue, Jul 1, 2025 at 6:10 PM Zhijie Hou (Fujitsu) wrote: > Here is V45 patch set. With the main patch set now stable, I am summarizing the performance tests conducted before for reference. In earlier tests [1], we confirmed that in a pub-sub cluster with high workload on the publisher (via pgbe

Re: Conflict detection for update_deleted in logical replication

2025-07-01 Thread Dilip Kumar
On Tue, Jul 1, 2025 at 3:39 PM Zhijie Hou (Fujitsu) wrote: > > On Tue, Jul 1, 2025 at 5:07 PM Dilip Kumar wrote: > > > > On Tue, Jul 1, 2025 at 2:24 PM Amit Kapila wrote: > > > > > > On Tue, Jul 1, 2025 at 10:53 AM Dilip Kumar wrote: > > > > > > > > On Tue, Jul 1, 2025 at 10:31 AM Dilip Kumar >

Re: Conflict detection for update_deleted in logical replication

2025-07-01 Thread Dilip Kumar
On Tue, Jul 1, 2025 at 2:24 PM Amit Kapila wrote: > > On Tue, Jul 1, 2025 at 10:53 AM Dilip Kumar wrote: > > > > On Tue, Jul 1, 2025 at 10:31 AM Dilip Kumar wrote: > > > > > > On Mon, Jun 30, 2025 at 6:59 PM Zhijie Hou (Fujitsu) > > > wrote: > > > > > > > > On Mon, Jun 30, 2025 at 7:22 PM Amit

Re: Conflict detection for update_deleted in logical replication

2025-07-01 Thread Amit Kapila
On Tue, Jul 1, 2025 at 10:53 AM Dilip Kumar wrote: > > On Tue, Jul 1, 2025 at 10:31 AM Dilip Kumar wrote: > > > > On Mon, Jun 30, 2025 at 6:59 PM Zhijie Hou (Fujitsu) > > wrote: > > > > > > On Mon, Jun 30, 2025 at 7:22 PM Amit Kapila wrote: > > > > > > > > I was looking at 0001, it mostly looks

Re: Conflict detection for update_deleted in logical replication

2025-06-30 Thread Dilip Kumar
On Tue, Jul 1, 2025 at 10:31 AM Dilip Kumar wrote: > > On Mon, Jun 30, 2025 at 6:59 PM Zhijie Hou (Fujitsu) > wrote: > > > > On Mon, Jun 30, 2025 at 7:22 PM Amit Kapila wrote: > > > > > I was looking at 0001, it mostly looks fine to me except this one > case. So here we need to ensure that commi

Re: Conflict detection for update_deleted in logical replication

2025-06-30 Thread Dilip Kumar
On Mon, Jun 30, 2025 at 6:59 PM Zhijie Hou (Fujitsu) wrote: > > On Mon, Jun 30, 2025 at 7:22 PM Amit Kapila wrote: > > I was looking at 0001, it mostly looks fine to me except this one case. So here we need to ensure that commits must be acquired after marking the flag, don't you think we need t

Re: Conflict detection for update_deleted in logical replication

2025-06-30 Thread Amit Kapila
On Fri, Jun 27, 2025 at 7:58 AM Zhijie Hou (Fujitsu) wrote: > > Here is the V43 patch set which includes the following changes: > Few minor comments: 1. @@ -29645,8 +29651,10 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset Copies an existing logical rep

Re: Conflict detection for update_deleted in logical replication

2025-06-27 Thread shveta malik
On Fri, Jun 27, 2025 at 7:58 AM Zhijie Hou (Fujitsu) wrote: > > On Thu, Jun 26, 2025 at 4:28 PM shveta malik wrote: > > > > On Thu, Jun 26, 2025 at 8:31 AM Zhijie Hou (Fujitsu) > > wrote: > > > > > > Thanks for the comments. All of them look good to me and > > > have been addressed in V42. > > >

Re: Conflict detection for update_deleted in logical replication

2025-06-26 Thread shveta malik
On Thu, Jun 26, 2025 at 8:31 AM Zhijie Hou (Fujitsu) wrote: > > Thanks for the comments. All of them look good to me and > have been addressed in V42. > Thank You for the patches. Few comments. t/035_conflicts.pl: 1) Both the subscriptions subname_BA and subname_AB have rci enabled during CREAT

RE: Conflict detection for update_deleted in logical replication

2025-06-25 Thread Zhijie Hou (Fujitsu)
On Wed, Jun 25, 2025 at 2:57 PM shveta malik wrote: > > > > > Here is the V41 patch set which includes the following changes: > > > > Thanks for the patches. Few trivial things: > > 1) > In ReplicationSlotAcquire(), does it make more sense to move the error after > checking the slot's existence

Re: Conflict detection for update_deleted in logical replication

2025-06-25 Thread shveta malik
On Mon, Jun 23, 2025 at 4:20 PM Zhijie Hou (Fujitsu) wrote: > > > Here is the V40 patch set Thanks for the patches. Few comments: 1) In get_subscription_info(), we are doing COUNT of rci-subscriptions using below query: SELECT count(*) AS nsub, COUNT(CASE WHEN subretainconflictinfo THEN 1 END) A

Re: Conflict detection for update_deleted in logical replication

2025-06-25 Thread Amit Kapila
On Wed, Jun 25, 2025 at 8:38 AM Zhijie Hou (Fujitsu) wrote: > > Here is the V41 patch set which includes the following changes: > Few comments on 0004 === 1. + +# Remember the next transaction ID to be assigned +my $next_xid = $node_A->safe_psql('postgres', "SELECT txid_current()

Re: Conflict detection for update_deleted in logical replication

2025-06-24 Thread shveta malik
> > Here is the V41 patch set which includes the following changes: > Thanks for the patches. Few trivial things: 1) In ReplicationSlotAcquire(), does it make more sense to move the error after checking the slot's existence first? If a user is trying to use a slot which does not exist, he should

Re: Conflict detection for update_deleted in logical replication

2025-06-23 Thread Amit Kapila
On Fri, Jun 20, 2025 at 4:48 PM Zhijie Hou (Fujitsu) wrote: > > Here is the V39 patch set which includes the following changes: > 1. -static void -create_conflict_slot_if_not_exists(void) +void +ApplyLauncherCreateConflictDetectionSlot(void) I am not so sure about adding ApplyLauncher in front o

Re: Conflict detection for update_deleted in logical replication

2025-06-23 Thread shveta malik
> > Here is the V39 patch set which includes the following changes: > Few trivial comments: 1) Currently we have this error and detail: ERROR: Enabling retain_conflict_info requires "wal_level" >= "replica" DETAIL: A replication slot must be created to retain conflict information. Shall we ch

Re: Conflict detection for update_deleted in logical replication

2025-06-20 Thread shveta malik
On Thu, Jun 19, 2025 at 4:34 PM shveta malik wrote: > > > > > Here is the V38 patch set which includes the following changes: > > > > Thank You for the patches. Few comments: > > 1) > + > +Note that commit timestamps and origin data retained by enabling the > + linkend="sql-createsubscr

Re: Conflict detection for update_deleted in logical replication

2025-06-19 Thread shveta malik
> > Here is the V38 patch set which includes the following changes: > Thank You for the patches. Few comments: 1) + +Note that commit timestamps and origin data retained by enabling the +retain_conflict_info +option will not be preserved during the upgrade. As a +result, the up

Re: Conflict detection for update_deleted in logical replication

2025-06-19 Thread Amit Kapila
On Tue, Jun 17, 2025 at 4:26 PM Zhijie Hou (Fujitsu) wrote: > > On Mon, Jun 16, 2025 at 7:37 PM Amit Kapila wrote: > > > > > > 3. Isn't the new check for logical slots in > > check_new_cluster_subscription_configuration() somewhat redundant with > > the previous check done in check_new_cluster_log

Re: Conflict detection for update_deleted in logical replication

2025-06-16 Thread Amit Kapila
On Thu, Jun 12, 2025 at 11:34 AM Zhijie Hou (Fujitsu) wrote: > Few comments on v36 patches: == 1. In advance_conflict_slot_xmin(), we first save the slot to disk, then update its effective xmin, and then do the required xmin computation. Now, if we don't save the slot ever

Re: Conflict detection for update_deleted in logical replication

2025-06-16 Thread shveta malik
On Mon, Jun 16, 2025 at 9:29 AM Zhijie Hou (Fujitsu) wrote: > > 0001: > * Removed the slot acquisition as suggested above. > > 0002: > * Addressed the comments above. > Thanks for the patches. In advance_conflict_slot_xmin(), if new_xmin is same as slot's current xmin, then shall we simply retu

Re: Conflict detection for update_deleted in logical replication

2025-06-12 Thread shveta malik
On Thu, Jun 12, 2025 at 4:22 PM shveta malik wrote: > > On Thu, Jun 12, 2025 at 11:34 AM Zhijie Hou (Fujitsu) > wrote: > > > > > > Here is the V35 patch set which includes the following changes: > > > > Thank You for the patches. Few comments: > > 1) > Since now we create slot for rci enabled sub

Re: Conflict detection for update_deleted in logical replication

2025-06-12 Thread shveta malik
On Thu, Jun 12, 2025 at 11:34 AM Zhijie Hou (Fujitsu) wrote: > > > Here is the V35 patch set which includes the following changes: > Thank You for the patches. Few comments: 1) Since now we create slot for rci enabled subscription, it will require wal_level >= replica even on subscribers. We sha

Re: Conflict detection for update_deleted in logical replication

2025-06-11 Thread Amit Kapila
On Tue, Jun 10, 2025 at 11:55 AM Zhijie Hou (Fujitsu) wrote: > > Here is the V35 patch set which includes the following changes: > Few minor comments: === 1. +Â * Check if the subscriber's configuration is adequate to enable the +Â * retain_conflict_info option. I see some funny

Re: Conflict detection for update_deleted in logical replication

2025-06-11 Thread shveta malik
On Tue, Jun 10, 2025 at 11:55 AM Zhijie Hou (Fujitsu) wrote: > > Here is the V35 patch set which includes the following changes: > Thank You for the patches, few comments: 1) compute_min_nonremovable_xid: + /* + * Stop advancing xmin if an invalid non-removable transaction ID is + * found, othe

Re: Conflict detection for update_deleted in logical replication

2025-06-06 Thread Amit Kapila
On Wed, Jun 4, 2025 at 4:12 PM Zhijie Hou (Fujitsu) wrote: > > Here is the V33 patch set which includes the following changes: > Few comments: 1. + if (sub->enabled) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("cannot set option %s for enabled subscription", +

Re: Conflict detection for update_deleted in logical replication

2025-06-05 Thread shveta malik
On Wed, Jun 4, 2025 at 4:12 PM Zhijie Hou (Fujitsu) wrote: > > Here is the V33 patch set which includes the following changes: > Thank You for the patches, please find few comments for patch003: 1) + /* + * Skip the track_commit_timestamp check by passing it as + * true, since it has already bee

RE: Conflict detection for update_deleted in logical replication

2025-06-04 Thread Zhijie Hou (Fujitsu)
On Wed, Jun 4, 2025 at 6:43 PM Zhijie Hou (Fujitsu) wrote: > > On Mon, Jun 2, 2025 at 2:39 PM Amit Kapila wrote: > > > > On Mon, May 26, 2025 at 12:46 PM Zhijie Hou (Fujitsu) > > wrote: > > > > > > Attaching the V32 patch set which addressed comments in [1]~[5]. > > > > > > > Review comments: >

Re: Conflict detection for update_deleted in logical replication

2025-06-04 Thread shveta malik
On Mon, Jun 2, 2025 at 12:09 PM Amit Kapila wrote: > > On Mon, May 26, 2025 at 12:46 PM Zhijie Hou (Fujitsu) > wrote: > > > > Attaching the V32 patch set which addressed comments in [1]~[5]. > > > > Review comments: > === > * > +advance_conflict_slot_xmin(FullTransactionId new_xmin) >

Re: Conflict detection for update_deleted in logical replication

2025-06-01 Thread Amit Kapila
On Mon, May 26, 2025 at 12:46 PM Zhijie Hou (Fujitsu) wrote: > > Attaching the V32 patch set which addressed comments in [1]~[5]. > Review comments: === * +advance_conflict_slot_xmin(FullTransactionId new_xmin) +{ + FullTransactionId full_xmin; + FullTransactionId next_full_xid; + + A

Re: Conflict detection for update_deleted in logical replication

2025-05-29 Thread shveta malik
On Tue, May 27, 2025 at 3:59 PM shveta malik wrote: > > On Mon, May 26, 2025 at 12:46 PM Zhijie Hou (Fujitsu) > wrote: > > > > Attaching the V32 patch set which addressed comments in [1]~[5]. > > Thanks for the patch, I am still reviewing the patches, please find > few trivial comments for patch0

Re: Conflict detection for update_deleted in logical replication

2025-05-28 Thread Dilip Kumar
On Tue, May 27, 2025 at 11:45 AM Amit Kapila wrote: > > On Mon, May 26, 2025 at 12:46 PM Zhijie Hou (Fujitsu) > wrote: > > > > On Sun, May 25, 2025 at 4:36 PM Dilip Kumar wrote: > > > > > > > > I am thinking can't we make it more deterministic such that when we > > > get the status first time if

Re: Conflict detection for update_deleted in logical replication

2025-05-27 Thread Amit Kapila
On Tue, May 27, 2025 at 3:59 PM shveta malik wrote: > > On Mon, May 26, 2025 at 12:46 PM Zhijie Hou (Fujitsu) > wrote: > > > > Attaching the V32 patch set which addressed comments in [1]~[5]. > > Thanks for the patch, I am still reviewing the patches, please find > few trivial comments for patch0

Re: Conflict detection for update_deleted in logical replication

2025-05-27 Thread shveta malik
On Mon, May 26, 2025 at 12:46 PM Zhijie Hou (Fujitsu) wrote: > > Attaching the V32 patch set which addressed comments in [1]~[5]. Thanks for the patch, I am still reviewing the patches, please find few trivial comments for patch001: 1) + FullTransactionId last_phase_at; /* publisher transaction

Re: Conflict detection for update_deleted in logical replication

2025-05-27 Thread Amit Kapila
On Mon, May 26, 2025 at 12:46 PM Zhijie Hou (Fujitsu) wrote: > > On Sun, May 25, 2025 at 4:36 PM Dilip Kumar wrote: > > > > > I am thinking can't we make it more deterministic such that when we > > get the status first time if we find some transactions that are in > > commit phase then we should j

RE: Conflict detection for update_deleted in logical replication

2025-05-26 Thread Zhijie Hou (Fujitsu)
On Fri, May 23, 2025 at 11:51 PM Xuneng Zhou wrote: > Thanks for the effort on the patches. I did a quick look on them before > diving into the logic and discussion. Below are a few minor typos found in > version 31. Thanks for the comments! I have fixed these typos in latest version. Best Regar

RE: Conflict detection for update_deleted in logical replication

2025-05-26 Thread Zhijie Hou (Fujitsu)
On Fri, May 23, 2025 at 2:45 PM shveta malik wrote: > > Thanks you for v31 patch-set. Please find few comments on patch001: Thanks for the comments. > > 1) > > wait_for_local_flush: > > + if (data->last_recv_time && > + TimestampDifferenceExceeds(data->flushpos_update_time, > +data->last

RE: Conflict detection for update_deleted in logical replication

2025-05-26 Thread Zhijie Hou (Fujitsu)
On Sat, May 24, 2025 at 6:28 PM Dilip Kumar wrote: > > On Sat, May 24, 2025 at 11:00 AM Amit Kapila > wrote: > > > > On Sat, May 24, 2025 at 10:29 AM Dilip Kumar > wrote: > > > > > > On Sat, May 24, 2025 at 10:04 AM Dilip Kumar > wrote: > > > > > > > > On Fri, May 23, 2025 at 9:21 PM Xuneng Zh

RE: Conflict detection for update_deleted in logical replication

2025-05-26 Thread Zhijie Hou (Fujitsu)
On Sun, May 25, 2025 at 4:36 PM Dilip Kumar wrote: > > On Sat, May 24, 2025 at 4:46 PM Amit Kapila > wrote: > > > > This sounds reasonable to me. Let us see what others think. > > > > I was looking into the for getting the transaction status from > publisher, what I would assume this patch shou

Re: Conflict detection for update_deleted in logical replication

2025-05-25 Thread Dilip Kumar
On Sat, May 24, 2025 at 4:46 PM Amit Kapila wrote: > > This sounds reasonable to me. Let us see what others think. > I was looking into the for getting the transaction status from publisher, what I would assume this patch should be doing is request the publisher status first time, and if some tra

Re: Conflict detection for update_deleted in logical replication

2025-05-24 Thread Amit Kapila
On Sat, May 24, 2025 at 3:58 PM Dilip Kumar wrote: > > On Sat, May 24, 2025 at 11:00 AM Amit Kapila wrote: > > > > On Sat, May 24, 2025 at 10:29 AM Dilip Kumar wrote: > > > > > > On Sat, May 24, 2025 at 10:04 AM Dilip Kumar > > > wrote: > > > > > > > > On Fri, May 23, 2025 at 9:21 PM Xuneng Zh

Re: Conflict detection for update_deleted in logical replication

2025-05-24 Thread Dilip Kumar
On Sat, May 24, 2025 at 11:00 AM Amit Kapila wrote: > > On Sat, May 24, 2025 at 10:29 AM Dilip Kumar wrote: > > > > On Sat, May 24, 2025 at 10:04 AM Dilip Kumar wrote: > > > > > > On Fri, May 23, 2025 at 9:21 PM Xuneng Zhou wrote: > > > > > > > Looking at v31-0001 again, most of it looks fine e

Re: Conflict detection for update_deleted in logical replication

2025-05-23 Thread Amit Kapila
On Sat, May 24, 2025 at 10:29 AM Dilip Kumar wrote: > > On Sat, May 24, 2025 at 10:04 AM Dilip Kumar wrote: > > > > On Fri, May 23, 2025 at 9:21 PM Xuneng Zhou wrote: > > > > > Looking at v31-0001 again, most of it looks fine except this logic of > > getting the commit_ts after marking the trans

Re: Conflict detection for update_deleted in logical replication

2025-05-23 Thread Dilip Kumar
On Sat, May 24, 2025 at 10:04 AM Dilip Kumar wrote: > > On Fri, May 23, 2025 at 9:21 PM Xuneng Zhou wrote: > > > Looking at v31-0001 again, most of it looks fine except this logic of > getting the commit_ts after marking the transaction in commit. I see > in RecordTransactionCommit(), we are set

Re: Conflict detection for update_deleted in logical replication

2025-05-23 Thread Dilip Kumar
On Fri, May 23, 2025 at 9:21 PM Xuneng Zhou wrote: > Looking at v31-0001 again, most of it looks fine except this logic of getting the commit_ts after marking the transaction in commit. I see in RecordTransactionCommit(), we are setting this flag (DELAY_CHKPT_IN_COMMIT) to put the transaction in

Re: Conflict detection for update_deleted in logical replication

2025-05-23 Thread Xuneng Zhou
Hi Zhijie, Thanks for the effort on the patches. I did a quick look on them before diving into the logic and discussion. Below are a few minor typos found in version 31. ⸻ 1. Spelling of “non-removable” [PATCH v31 1/7] In docs and code “removeable” vs. “removable” are used alternatively

Re: Conflict detection for update_deleted in logical replication

2025-05-23 Thread Amit Kapila
On Thu, May 22, 2025 at 8:28 AM Zhijie Hou (Fujitsu) wrote: > > Attaching the V31 patch set which addressed comments in [1]~[8]. > Few comments: 1. + The oldest transaction ID along that is currently in the commit + phase on the server, along with its epoch. The first 'a

Re: Conflict detection for update_deleted in logical replication

2025-05-23 Thread Amit Kapila
On Fri, May 23, 2025 at 12:15 PM shveta malik wrote: > > Thanks you for v31 patch-set. Please find few comments on patch001: > > 1) > > wait_for_local_flush: > > + if (data->last_recv_time && > + TimestampDifferenceExceeds(data->flushpos_update_time, > +data->last_recv_time, WalWriterDelay)) >

Re: Conflict detection for update_deleted in logical replication

2025-05-22 Thread shveta malik
Thanks you for v31 patch-set. Please find few comments on patch001: 1) wait_for_local_flush: + if (data->last_recv_time && + TimestampDifferenceExceeds(data->flushpos_update_time, +data->last_recv_time, WalWriterDelay)) + { + XLogRecPtr writepos; + XLogRecPtr flushpos; + bool have_pending_tx

RE: Conflict detection for update_deleted in logical replication

2025-05-21 Thread Zhijie Hou (Fujitsu)
On Fri, May 16, 2025 at 7:31 PM Amit Kapila wrote: > > A few more comments: > = > 3. > maybe_advance_nonremovable_xid(RetainConflictInfoData *data, >bool status_received) > { > /* Exit early if retaining conflict information is not required */ > if (!MySubscription->retainconfl

RE: Conflict detection for update_deleted in logical replication

2025-05-21 Thread Zhijie Hou (Fujitsu)
On Tue, May 20, 2025 at 6:30 PM shveta malik wrote: > > Few more comments mostly for patch001: Thanks for the comments! > > 4) > For this feature, since we are only interested in remote UPDATEs happening > concurrently, so shall we ask primary to provide oldest "UPDATE" > transaction-id in comm

  1   2   3   >