Re: Collect statistics about conflicts in logical replication

2024-09-03 Thread Amit Kapila
On Wed, Sep 4, 2024 at 9:17 AM Peter Smith wrote: > > On Tue, Sep 3, 2024 at 9:23 PM Zhijie Hou (Fujitsu) > wrote: > > > > I confirmed that it only increased the testing time by 1 second on my > > machine. > > > > It seems a pity to throw away perfectly good test cases just because > they increa

Re: Collect statistics about conflicts in logical replication

2024-09-03 Thread Peter Smith
On Tue, Sep 3, 2024 at 9:23 PM Zhijie Hou (Fujitsu) wrote: > > On Tuesday, September 3, 2024 7:12 PM Amit Kapila > wrote: > > > > Testing the stats for all types of conflicts is not required for this patch > > especially because they increase the timings by 3-4s. We can add tests for > > one >

RE: Collect statistics about conflicts in logical replication

2024-09-03 Thread Zhijie Hou (Fujitsu)
On Tuesday, September 3, 2024 7:23 PM Zhijie Hou (Fujitsu) wrote: > > On Tuesday, September 3, 2024 7:12 PM Amit Kapila > wrote: > > > > On Fri, Aug 30, 2024 at 12:15 PM Zhijie Hou (Fujitsu) > > > > wrote: > > > > > > Here is V5 patch which addressed above and Shveta's[1] comments. > > > > > >

RE: Collect statistics about conflicts in logical replication

2024-09-03 Thread Zhijie Hou (Fujitsu)
On Tuesday, September 3, 2024 7:12 PM Amit Kapila wrote: > > On Fri, Aug 30, 2024 at 12:15 PM Zhijie Hou (Fujitsu) > wrote: > > > > Here is V5 patch which addressed above and Shveta's[1] comments. > > > > Testing the stats for all types of conflicts is not required for this patch > especially

Re: Collect statistics about conflicts in logical replication

2024-09-03 Thread Amit Kapila
On Fri, Aug 30, 2024 at 12:15 PM Zhijie Hou (Fujitsu) wrote: > > Here is V5 patch which addressed above and Shveta's[1] comments. > Testing the stats for all types of conflicts is not required for this patch especially because they increase the timings by 3-4s. We can add tests for one or two typ

Re: Collect statistics about conflicts in logical replication

2024-09-02 Thread Peter Smith
On Mon, Sep 2, 2024 at 1:28 PM shveta malik wrote: > > On Mon, Sep 2, 2024 at 4:20 AM Peter Smith wrote: > > > > On Fri, Aug 30, 2024 at 4:24 PM shveta malik wrote: > > > > > > On Fri, Aug 30, 2024 at 10:53 AM Peter Smith > > > wrote: > > > > > > ... > > > > 2. Arrange all the counts into an i

Re: Collect statistics about conflicts in logical replication

2024-09-01 Thread shveta malik
On Mon, Sep 2, 2024 at 4:20 AM Peter Smith wrote: > > On Fri, Aug 30, 2024 at 4:24 PM shveta malik wrote: > > > > On Fri, Aug 30, 2024 at 10:53 AM Peter Smith wrote: > > > > ... > > > 2. Arrange all the counts into an intuitive/natural order > > > > > > There is an intuitive/natural ordering for

Re: Collect statistics about conflicts in logical replication

2024-09-01 Thread Peter Smith
On Fri, Aug 30, 2024 at 4:24 PM shveta malik wrote: > > On Fri, Aug 30, 2024 at 10:53 AM Peter Smith wrote: > > ... > > 2. Arrange all the counts into an intuitive/natural order > > > > There is an intuitive/natural ordering for these counts. For example, > > the 'confl_*' count fields are in the

Re: Collect statistics about conflicts in logical replication

2024-09-01 Thread Peter Smith
On Fri, Aug 30, 2024 at 6:36 PM shveta malik wrote: > > On Fri, Aug 30, 2024 at 12:15 PM Zhijie Hou (Fujitsu) > wrote: > > > > > > Here is V5 patch which addressed above and Shveta's[1] comments. > > > > The patch looks good to me. > Patch v5 LGTM. == Kind Regards, Peter Smith. Fujitsu Aust

Re: Collect statistics about conflicts in logical replication

2024-08-30 Thread shveta malik
On Fri, Aug 30, 2024 at 12:15 PM Zhijie Hou (Fujitsu) wrote: > > > Here is V5 patch which addressed above and Shveta's[1] comments. > The patch looks good to me. thanks Shveta

RE: Collect statistics about conflicts in logical replication

2024-08-29 Thread Zhijie Hou (Fujitsu)
On Friday, August 30, 2024 2:24 PM shveta malik wrote: > > On Fri, Aug 30, 2024 at 10:53 AM Peter Smith > wrote: > > > > Hi Hou-San. Here are my review comments for v4-0001. Thanks Shveta and Peter for giving comments ! > > > > == > > > > 1. Add links in the docs > > > > IMO it would be go

Re: Collect statistics about conflicts in logical replication

2024-08-29 Thread shveta malik
On Fri, Aug 30, 2024 at 10:53 AM Peter Smith wrote: > > Hi Hou-San. Here are my review comments for v4-0001. > > == > > 1. Add links in the docs > > IMO it would be good for all these confl_* descriptions (in > doc/src/sgml/monitoring.sgml) to include links back to where each of > those confli

Re: Collect statistics about conflicts in logical replication

2024-08-29 Thread shveta malik
On Fri, Aug 30, 2024 at 9:40 AM shveta malik wrote: > > On Thu, Aug 29, 2024 at 11:06 AM Zhijie Hou (Fujitsu) > wrote: > > > > > > Agreed. Here is new version patch which change the names as suggested. I > > also > > rebased the patch based on another renaming commit 640178c9. > > > > Thanks for

Re: Collect statistics about conflicts in logical replication

2024-08-29 Thread Peter Smith
Hi Hou-San. Here are my review comments for v4-0001. == 1. Add links in the docs IMO it would be good for all these confl_* descriptions (in doc/src/sgml/monitoring.sgml) to include links back to where each of those conflict types was defined [1]. Indeed, when links are included to the orig

Re: Collect statistics about conflicts in logical replication

2024-08-29 Thread shveta malik
On Thu, Aug 29, 2024 at 11:06 AM Zhijie Hou (Fujitsu) wrote: > > > Agreed. Here is new version patch which change the names as suggested. I also > rebased the patch based on another renaming commit 640178c9. > Thanks for the patch. Few minor things: 1) conflict.h: * This enum is used in statist

RE: Collect statistics about conflicts in logical replication

2024-08-28 Thread Zhijie Hou (Fujitsu)
On Thursday, August 29, 2024 11:18 AM shveta malik wrote: > > On Thu, Aug 29, 2024 at 4:59 AM Peter Smith > wrote: > > > > On Wed, Aug 28, 2024 at 9:19 PM Amit Kapila > wrote: > > > > > > On Wed, Aug 28, 2024 at 11:43 AM shveta malik > wrote: > > > > > > > > Thanks for the patch. Just thinkin

Re: Collect statistics about conflicts in logical replication

2024-08-28 Thread shveta malik
On Thu, Aug 29, 2024 at 4:59 AM Peter Smith wrote: > > On Wed, Aug 28, 2024 at 9:19 PM Amit Kapila wrote: > > > > On Wed, Aug 28, 2024 at 11:43 AM shveta malik > > wrote: > > > > > > Thanks for the patch. Just thinking out loud, since we have names like > > > 'apply_error_count', 'sync_error_co

RE: Collect statistics about conflicts in logical replication

2024-08-28 Thread Zhijie Hou (Fujitsu)
On Thursday, August 29, 2024 8:31 AM Peter Smith wrote: Hi, > I tried an experiment where I deliberately violated a primary key during > initial > table synchronization. > > For example: ... > test_sub=# 2024-08-29 09:53:57.245 AEST [4345] LOG: logical replication > apply worker for subscript

Re: Collect statistics about conflicts in logical replication

2024-08-28 Thread Peter Smith
Hi Hou-San. I tried an experiment where I deliberately violated a primary key during initial table synchronization. For example: test_sub=# create table t1(a int primary key); CREATE TABLE test_sub=# insert into t1 values(1); INSERT 0 1 test_sub=# create subscription sub1 connection 'dbname=te

Re: Collect statistics about conflicts in logical replication

2024-08-28 Thread Peter Smith
On Wed, Aug 28, 2024 at 9:19 PM Amit Kapila wrote: > > On Wed, Aug 28, 2024 at 11:43 AM shveta malik wrote: > > > > Thanks for the patch. Just thinking out loud, since we have names like > > 'apply_error_count', 'sync_error_count' which tells that they are > > actually error-count, will it be bet

Re: Collect statistics about conflicts in logical replication

2024-08-28 Thread Amit Kapila
On Wed, Aug 28, 2024 at 11:43 AM shveta malik wrote: > > Thanks for the patch. Just thinking out loud, since we have names like > 'apply_error_count', 'sync_error_count' which tells that they are > actually error-count, will it be better to have something similar in > conflict-count cases, like 'i

Re: Collect statistics about conflicts in logical replication

2024-08-27 Thread shveta malik
On Tue, Aug 27, 2024 at 3:21 PM Zhijie Hou (Fujitsu) wrote: > > On Tuesday, August 27, 2024 10:59 AM Peter Smith > wrote: > > > > ~~~ > > > > 3. > > +# Enable track_commit_timestamp to detect origin-differ conflicts in > > +logical # replication. Reduce wal_retrieve_retry_interval to 1ms to > >

RE: Collect statistics about conflicts in logical replication

2024-08-27 Thread Zhijie Hou (Fujitsu)
On Tuesday, August 27, 2024 10:59 AM Peter Smith wrote: > > ~~~ > > 3. > +# Enable track_commit_timestamp to detect origin-differ conflicts in > +logical # replication. Reduce wal_retrieve_retry_interval to 1ms to > +accelerate the # restart of the logical replication worker after > encounterin

Re: Collect statistics about conflicts in logical replication

2024-08-26 Thread Peter Smith
On Mon, Aug 26, 2024 at 10:13 PM Zhijie Hou (Fujitsu) wrote: > > On Monday, August 26, 2024 3:30 PM Peter Smith wrote: > > > > == > > src/include/replication/conflict.h > > > > nit - defined 'NUM_CONFLICT_TYPES' inside the enum (I think this way is > > often used in other PG source enums) > >

RE: Collect statistics about conflicts in logical replication

2024-08-26 Thread Zhijie Hou (Fujitsu)
On Monday, August 26, 2024 3:30 PM Peter Smith wrote: > > == > src/include/replication/conflict.h > > nit - defined 'NUM_CONFLICT_TYPES' inside the enum (I think this way is > often used in other PG source enums) I think we have recently tended to avoid doing that, as it has been commented

Re: Collect statistics about conflicts in logical replication

2024-08-26 Thread Peter Smith
Hi Hou-san. Here are some review comments for your patch v1-0001. == doc/src/sgml/logical-replication.sgml nit - added a comma. == doc/src/sgml/monitoring.sgml nit - use for 'apply_error_count'. nit - added a period when there are multiple sentences. nit - adjusted field descriptions u

Collect statistics about conflicts in logical replication

2024-08-22 Thread Zhijie Hou (Fujitsu)
Hi hackers, Cc people involved in the related work. In the original conflict resolution thread[1], we have decided to split the conflict resolution work into multiple patches to facilitate incremental progress towards supporting conflict resolution in logical replication, and one of the work is st