Re: Fix slot synchronization with two_phase decoding enabled

2025-04-26 Thread Amit Kapila
On Fri, Apr 25, 2025 at 9:57 PM Masahiko Sawada wrote: > > On Fri, Apr 25, 2025 at 3:43 AM Amit Kapila wrote: > > > > On Fri, Apr 25, 2025 at 6:02 AM Masahiko Sawada > > wrote: > > > > > > I realized that users who create a logical slot using > > > pg_create_logical_replication_slot() would not

Re: Fix slot synchronization with two_phase decoding enabled

2025-04-25 Thread Masahiko Sawada
On Fri, Apr 25, 2025 at 3:43 AM Amit Kapila wrote: > > On Fri, Apr 25, 2025 at 6:02 AM Masahiko Sawada wrote: > > > > I realized that users who create a logical slot using > > pg_create_logical_replication_slot() would not be able to enable both > > options at slot creation, and there is no easy

Re: Fix slot synchronization with two_phase decoding enabled

2025-04-25 Thread Nisha Moond
On Thu, Apr 24, 2025 at 3:57 PM shveta malik wrote: > > On Thu, Apr 24, 2025 at 2:54 PM Nisha Moond wrote: > > > > Please find the updated patch for Approach 3, which implements the > > idea suggested by Swada-san above. > > > > Thank You for the patch. > > 1) > > CreateDecodingContext: > > if

Re: Fix slot synchronization with two_phase decoding enabled

2025-04-25 Thread Amit Kapila
On Fri, Apr 25, 2025 at 6:02 AM Masahiko Sawada wrote: > > I realized that users who create a logical slot using > pg_create_logical_replication_slot() would not be able to enable both > options at slot creation, and there is no easy way to enable the > failover after two_phase-enabled-slot creati

Re: Fix slot synchronization with two_phase decoding enabled

2025-04-24 Thread Masahiko Sawada
On Thu, Apr 24, 2025 at 10:48 AM Masahiko Sawada wrote: > > On Thu, Apr 24, 2025 at 2:24 AM Nisha Moond wrote: > > > > On Thu, Apr 24, 2025 at 12:28 PM Amit Kapila > > wrote: > > > > > > On Wed, Apr 23, 2025 at 11:04 PM Masahiko Sawada > > > wrote: > > > > > > > > On Tue, Apr 22, 2025 at 3:00

Re: Fix slot synchronization with two_phase decoding enabled

2025-04-24 Thread Masahiko Sawada
On Thu, Apr 24, 2025 at 2:24 AM Nisha Moond wrote: > > On Thu, Apr 24, 2025 at 12:28 PM Amit Kapila wrote: > > > > On Wed, Apr 23, 2025 at 11:04 PM Masahiko Sawada > > wrote: > > > > > > On Tue, Apr 22, 2025 at 3:00 AM Amit Kapila > > > wrote: > > > > > > > > On Mon, Apr 21, 2025 at 8:44 AM Z

Re: Fix slot synchronization with two_phase decoding enabled

2025-04-24 Thread shveta malik
On Thu, Apr 24, 2025 at 2:54 PM Nisha Moond wrote: > > Please find the updated patch for Approach 3, which implements the > idea suggested by Swada-san above. > Thank You for the patch. 1) CreateDecodingContext: if (ctx->twophase && !slot->data.two_phase) { + /* + * Do not allow two-phase

Re: Fix slot synchronization with two_phase decoding enabled

2025-04-24 Thread Nisha Moond
On Thu, Apr 24, 2025 at 12:28 PM Amit Kapila wrote: > > On Wed, Apr 23, 2025 at 11:04 PM Masahiko Sawada > wrote: > > > > On Tue, Apr 22, 2025 at 3:00 AM Amit Kapila wrote: > > > > > > On Mon, Apr 21, 2025 at 8:44 AM Zhijie Hou (Fujitsu) > > > wrote: > > > > > > > > On Sat, Apr 19, 2025 at 2:1

Re: Fix slot synchronization with two_phase decoding enabled

2025-04-23 Thread Amit Kapila
On Wed, Apr 23, 2025 at 11:04 PM Masahiko Sawada wrote: > > On Tue, Apr 22, 2025 at 3:00 AM Amit Kapila wrote: > > > > On Mon, Apr 21, 2025 at 8:44 AM Zhijie Hou (Fujitsu) > > wrote: > > > > > > On Sat, Apr 19, 2025 at 2:19 AM Masahiko Sawada wrote: > > > > > > > > On Tue, Apr 8, 2025 at 10:14 P

Re: Fix slot synchronization with two_phase decoding enabled

2025-04-23 Thread Masahiko Sawada
On Tue, Apr 22, 2025 at 3:00 AM Amit Kapila wrote: > > On Mon, Apr 21, 2025 at 8:44 AM Zhijie Hou (Fujitsu) > wrote: > > > > On Sat, Apr 19, 2025 at 2:19 AM Masahiko Sawada wrote: > > > > > > On Tue, Apr 8, 2025 at 10:14 PM Zhijie Hou (Fujitsu) > > > wrote: > > > > > > > > > > > > -- > >

Re: Fix slot synchronization with two_phase decoding enabled

2025-04-23 Thread Nisha Moond
On Tue, Apr 22, 2025 at 3:23 PM shveta malik wrote: > > On Fri, Apr 11, 2025 at 1:03 PM Nisha Moond wrote: > > > > > > Patch "v5_aprch_3-0001" implements the above Approach 3. > > > > Thanks Hou-san for implementing approach-2 and providing the patch. > > > > I find Approach 2 better than Approac

Re: Fix slot synchronization with two_phase decoding enabled

2025-04-22 Thread Amit Kapila
On Mon, Apr 21, 2025 at 8:44 AM Zhijie Hou (Fujitsu) wrote: > > On Sat, Apr 19, 2025 at 2:19 AM Masahiko Sawada wrote: > > > > On Tue, Apr 8, 2025 at 10:14 PM Zhijie Hou (Fujitsu) > > wrote: > > > > > > > > > -- > > > Approach 2 > > > -- > > > > > > Instead of disallowing the use

Re: Fix slot synchronization with two_phase decoding enabled

2025-04-22 Thread shveta malik
On Fri, Apr 11, 2025 at 1:03 PM Nisha Moond wrote: > > > Patch "v5_aprch_3-0001" implements the above Approach 3. > > Thanks Hou-san for implementing approach-2 and providing the patch. > I find Approach 2 better than Approach1. Yet to review Approach3. Please find my initial comments: Approach

RE: Fix slot synchronization with two_phase decoding enabled

2025-04-21 Thread Zhijie Hou (Fujitsu)
On Tue, Apr 22, 2025 at 11:23 AM Amit Kapila wrote: > > On Fri, Apr 18, 2025 at 9:58 AM Amit Kapila wrote: > > > > On Thu, Apr 17, 2025 at 6:14 PM Zhijie Hou (Fujitsu) > > wrote: > > > > > > - > > > Fix > > > - > > > > > > I think we should keep the confirmed_flush even if the previous

Re: Fix slot synchronization with two_phase decoding enabled

2025-04-21 Thread Amit Kapila
On Fri, Apr 18, 2025 at 9:58 AM Amit Kapila wrote: > > On Thu, Apr 17, 2025 at 6:14 PM Zhijie Hou (Fujitsu) > wrote: > > > > - > > Fix > > - > > > > I think we should keep the confirmed_flush even if the previous synced > > restart_lsn/catalog_xmin is newer. Attachments include a patch fo

RE: Fix slot synchronization with two_phase decoding enabled

2025-04-20 Thread Zhijie Hou (Fujitsu)
On Sat, Apr 19, 2025 at 2:19 AM Masahiko Sawada wrote: > > On Tue, Apr 8, 2025 at 10:14 PM Zhijie Hou (Fujitsu) > wrote: > > > > > > -- > > Approach 2 > > -- > > > > Instead of disallowing the use of two-phase and failover together, a more > > flexible strategy could be only restr

Re: Fix slot synchronization with two_phase decoding enabled

2025-04-18 Thread Masahiko Sawada
On Tue, Apr 8, 2025 at 10:14 PM Zhijie Hou (Fujitsu) wrote: > > > -- > Approach 2 > -- > > Instead of disallowing the use of two-phase and failover together, a more > flexible strategy could be only restrict failover for slots with two-phase > enabled when there's a possibility of

Re: Fix slot synchronization with two_phase decoding enabled

2025-04-17 Thread Amit Kapila
On Thu, Apr 17, 2025 at 6:14 PM Zhijie Hou (Fujitsu) wrote: > > Hi, > > The recently added tests for slotsync on two-phase enabled slots failed[1] due > to a broken assertion triggered while decoding the COMMIT PREPARED record on > the promoted standby. > > - > Analysis > - > > if

RE: Fix slot synchronization with two_phase decoding enabled

2025-04-17 Thread Zhijie Hou (Fujitsu)
On Thu, Apr 17, 2025 at 8:44 PM Zhijie Hou (Fujitsu) wrote: > > Hi, > > The recently added tests for slotsync on two-phase enabled slots > failed[1] due to a broken assertion triggered while decoding the > COMMIT PREPARED record on the promoted standby. Here is the missing link: [1] https:/

RE: Fix slot synchronization with two_phase decoding enabled

2025-04-17 Thread Zhijie Hou (Fujitsu)
Hi, The recently added tests for slotsync on two-phase enabled slots failed[1] due to a broken assertion triggered while decoding the COMMIT PREPARED record on the promoted standby. - Analysis - if ((txn->final_lsn < two_phase_at) && is_commit) { /*

Re: Fix slot synchronization with two_phase decoding enabled

2025-04-11 Thread Nisha Moond
HI, Please find the patches attached for all three approaches. On Wed, Apr 9, 2025 at 10:45 AM Zhijie Hou (Fujitsu) wrote: > > On Thu, Apr 3, 2025 at 3:16 PM Zhijie Hou (Fujitsu) wrote: > > > > On Thu, Apr 3, 2025 at 1:38 PM Masahiko Sawada wrote: > > > > > > > > On Wed, Apr 2, 2025 at 7:58 PM A

RE: Fix slot synchronization with two_phase decoding enabled

2025-04-08 Thread Zhijie Hou (Fujitsu)
On Thu, Apr 3, 2025 at 3:16 PM Zhijie Hou (Fujitsu) wrote: > > On Thu, Apr 3, 2025 at 1:38 PM Masahiko Sawada wrote: > > > > > On Wed, Apr 2, 2025 at 7:58 PM Amit Kapila > > wrote: > > > > > > On Thu, Apr 3, 2025 at 7:50 AM Zhijie Hou (Fujitsu) > > > wrote: > > > > > > > > On Thu, Apr 3, 2025 a

RE: Fix slot synchronization with two_phase decoding enabled

2025-04-05 Thread Zhijie Hou (Fujitsu)
On Thu, Apr 3, 2025 at 1:38 PM Masahiko Sawada wrote: > > On Wed, Apr 2, 2025 at 7:58 PM Amit Kapila > wrote: > > > > On Thu, Apr 3, 2025 at 7:50 AM Zhijie Hou (Fujitsu) > > wrote: > > > > > > On Thu, Apr 3, 2025 at 3:30 AM Masahiko Sawada wrote: > > > > > > > > > > > On Wed, Apr 2, 2025 at 6:3

RE: Fix slot synchronization with two_phase decoding enabled

2025-04-05 Thread Zhijie Hou (Fujitsu)
On Thu, Mar 27, 2025 at 2:29 PM Amit Kapila wrote: > > On Tue, Mar 25, 2025 at 12:1 PM Amit Kapila > wrote: > > > > On Tue, Mar 25, 2025 at 11:05 AM Zhijie Hou (Fujitsu) > > wrote: > > > > > > Hi, > > > > > > When testing the slot synchronization with logical replication slots that > > > enable

Re: Fix slot synchronization with two_phase decoding enabled

2025-04-02 Thread Amit Kapila
On Thu, Apr 3, 2025 at 11:08 AM Masahiko Sawada wrote: > > On Wed, Apr 2, 2025 at 7:58 PM Amit Kapila wrote: > > > > On Thu, Apr 3, 2025 at 7:50 AM Zhijie Hou (Fujitsu) > > wrote: > > > > > > On Thu, Apr 3, 2025 at 3:30 AM Masahiko Sawada wrote: > > > > > > > > > > > On Wed, Apr 2, 2025 at 6:33 

Re: Fix slot synchronization with two_phase decoding enabled

2025-04-02 Thread Masahiko Sawada
On Wed, Apr 2, 2025 at 7:58 PM Amit Kapila wrote: > > On Thu, Apr 3, 2025 at 7:50 AM Zhijie Hou (Fujitsu) > wrote: > > > > On Thu, Apr 3, 2025 at 3:30 AM Masahiko Sawada wrote: > > > > > > > > On Wed, Apr 2, 2025 at 6:33 AM Zhijie Hou (Fujitsu) > > > wrote: > > > > > > Thank you for the explanat

Re: Fix slot synchronization with two_phase decoding enabled

2025-04-02 Thread Amit Kapila
On Thu, Apr 3, 2025 at 7:50 AM Zhijie Hou (Fujitsu) wrote: > > On Thu, Apr 3, 2025 at 3:30 AM Masahiko Sawada wrote: > > > > > On Wed, Apr 2, 2025 at 6:33 AM Zhijie Hou (Fujitsu) > > wrote: > > > > Thank you for the explanation! I agree that the issue happens in these > > cases. > > > > As anoth

RE: Fix slot synchronization with two_phase decoding enabled

2025-04-02 Thread Zhijie Hou (Fujitsu)
On Thu, Apr 3, 2025 at 3:30 AM Masahiko Sawada wrote: > > On Wed, Apr 2, 2025 at 6:33 AM Zhijie Hou (Fujitsu) > wrote: > > > > On Wed, Apr 2, 2025 at 3:45 PM Masahiko Sawada wrote: > > > > Hi, > > > > > > > > On Mon, Mar 31, 2025 at 4:3 AM Zhijie Hou (Fujitsu) > > > wrote: > > > > After further

Re: Fix slot synchronization with two_phase decoding enabled

2025-04-02 Thread Masahiko Sawada
On Wed, Apr 2, 2025 at 6:33 AM Zhijie Hou (Fujitsu) wrote: > > On Wed, Apr 2, 2025 at 3:45 PM Masahiko Sawada wrote: > > Hi, > > > > > On Mon, Mar 31, 2025 at 4:3 AM Zhijie Hou (Fujitsu) > > wrote: > > > > > > On Thu, Mar 27, 2025 at 2:29 PM Amit Kapila wrote: > > > > > > > > > > > On Tue, Mar 25

RE: Fix slot synchronization with two_phase decoding enabled

2025-04-02 Thread Zhijie Hou (Fujitsu)
On Wed, Apr 2, 2025 at 3:45 PM Masahiko Sawada wrote: Hi, > > On Mon, Mar 31, 2025 at 4:3 AM Zhijie Hou (Fujitsu) > wrote: > > > > On Thu, Mar 27, 2025 at 2:29 PM Amit Kapila wrote: > > > > > > > > On Tue, Mar 25, 2025 at 12:1 PM Amit Kapila > > > > > > wrote: > > > > > > > > On Tue, Mar 25, 2

Re: Fix slot synchronization with two_phase decoding enabled

2025-04-02 Thread Masahiko Sawada
On Mon, Mar 31, 2025 at 4:34 AM Zhijie Hou (Fujitsu) wrote: > > On Thu, Mar 27, 2025 at 2:29 PM Amit Kapila wrote: > > > > > On Tue, Mar 25, 2025 at 12:1 PM Amit Kapila > > wrote: > > > > > > On Tue, Mar 25, 2025 at 11:05 AM Zhijie Hou (Fujitsu) > > > wrote: > > > > > > > > Hi, > > > > > > > > W

RE: Fix slot synchronization with two_phase decoding enabled

2025-04-01 Thread Zhijie Hou (Fujitsu)
On Wed, Apr 2, 2025 at 12:41 PM Amit Kapila wrote: > > On Tue, Apr 1, 2025 at 4:28 PM Zhijie Hou (Fujitsu) > > wrote: > > > > Here is the V3 patch set which addressed all the comments. > > > > Comment 0n 0001 > NULL for logical slots where > + two_phase is false and physical slots. > +

Re: Fix slot synchronization with two_phase decoding enabled

2025-04-01 Thread Amit Kapila
On Tue, Apr 1, 2025 at 4:28 PM Zhijie Hou (Fujitsu) wrote: > > Here is the V3 patch set which addressed all the comments. > Comment 0n 0001 NULL for logical slots where + two_phase is false and physical slots. + change above to: NULL for logical slots where two_phase is false and for

Re: Fix slot synchronization with two_phase decoding enabled

2025-04-01 Thread Amit Kapila
On Mon, Mar 31, 2025 at 5:04 PM Zhijie Hou (Fujitsu) wrote: > > Thanks for the comments, they have been addressed in V2. > In the PG17 patch, we only have a check preventing failover and two_phase in CreateSubscription. Don't we need a similar check for AlterSubscription? Apart from the above, I

RE: Fix slot synchronization with two_phase decoding enabled

2025-04-01 Thread Zhijie Hou (Fujitsu)
On Tue, Apr 1, 2025 at 2:09 PM Amit Kapila wrote: > > On Mon, Mar 31, 2025 at 5:0 PM Zhijie Hou (Fujitsu) > wrote: > > > > Thanks for the comments, they have been addressed in V2. > > > > In the PG17 patch, we only have a check preventing failover and > two_phase in CreateSubscription. Don't

Re: Fix slot synchronization with two_phase decoding enabled

2025-03-31 Thread Amit Kapila
On Mon, Mar 31, 2025 at 5:04 PM Zhijie Hou (Fujitsu) wrote: > > On Thu, Mar 27, 2025 at 2:29 PM Amit Kapila wrote: > > > > > I suspect that this can happen in PG17 as well, but I need to think > > more about it to make a reproducible test case. > > After further analysis, I was able to reproduce t

Re: Fix slot synchronization with two_phase decoding enabled

2025-03-26 Thread Amit Kapila
On Tue, Mar 25, 2025 at 12:14 PM Amit Kapila wrote: > > On Tue, Mar 25, 2025 at 11:05 AM Zhijie Hou (Fujitsu) > wrote: > > > > Hi, > > > > When testing the slot synchronization with logical replication slots that > > enabled two_phase decoding, I found that transactions prepared before > > two-p

Re: Fix slot synchronization with two_phase decoding enabled

2025-03-24 Thread Amit Kapila
On Tue, Mar 25, 2025 at 11:05 AM Zhijie Hou (Fujitsu) wrote: > > Hi, > > When testing the slot synchronization with logical replication slots that > enabled two_phase decoding, I found that transactions prepared before > two-phase > decoding is enabled may fail to replicate to the subscriber afte

Fix slot synchronization with two_phase decoding enabled

2025-03-24 Thread Zhijie Hou (Fujitsu)
Hi, When testing the slot synchronization with logical replication slots that enabled two_phase decoding, I found that transactions prepared before two-phase decoding is enabled may fail to replicate to the subscriber after being committed on a promoted standby following a failover. To reproduce