RE: Add support for specifying tables in pg_createsubscriber.

2025-08-01 Thread Zhijie Hou (Fujitsu)
On Saturday, August 2, 2025 12:59 AM Andrew Dunstan wrote: > On 2025-08-01 Fr 11:03 AM, Zhijie Hou (Fujitsu) wrote: > > On Friday, August 1, 2025 8:56 PM Andrew Dunstan mailto:and...@dunslane.net > wrote: > > > > > We have another example to consider: pg_amc

RE: Conflict detection for update_deleted in logical replication

2025-08-01 Thread Zhijie Hou (Fujitsu)
On Friday, August 1, 2025 7:42 PM Dilip Kumar wrote: > > On Fri, Aug 1, 2025 at 5:02 PM Zhijie Hou (Fujitsu) > wrote: > > > > > 2. > > > > > > + If set to true, the detection of > > > + is enabled, and > > >

RE: Add support for specifying tables in pg_createsubscriber.

2025-08-01 Thread Zhijie Hou (Fujitsu)
On Friday, August 1, 2025 8:56 PM Andrew Dunstan wrote: > On 2025-08-01 Fr 4:03 AM, Zhijie Hou (Fujitsu) wrote: > > On Monday, July 28, 2025 1:07 PM Hayato Kuroda (Fujitsu) > > mailto:kuroda.hay...@fujitsu.com wrote: > > > Dear Shubham, > > > > > > Th

RE: Conflict detection for update_deleted in logical replication

2025-08-01 Thread Zhijie Hou (Fujitsu)
t detection for update_deleted in logical replication > > On Thu, Jul 31, 2025 at 3:49 PM Zhijie Hou (Fujitsu) > wrote: > > > > On Thursday, July 31, 2025 5:29 PM Amit Kapila > wrote: > > > > > > On Tue, Jul 29, 2025 at 10:51 AM Zhijie Hou (Fujitsu) >

RE: Add support for specifying tables in pg_createsubscriber.

2025-08-01 Thread Zhijie Hou (Fujitsu)
On Monday, July 28, 2025 1:07 PM Hayato Kuroda (Fujitsu) wrote: > > Dear Shubham, > > > The attached patch introduces a new '--table' option that can be > > specified after each '--database' argument. > > Do we have another example which we consider the ordering of options? I'm > unsure > for

RE: Add support for specifying tables in pg_createsubscriber.

2025-08-01 Thread Zhijie Hou (Fujitsu)
On Monday, July 21, 2025 1:31 PM Shubham Khanna wrote: > > Hi hackers, > > Currently, pg_createsubscriber supports converting streaming > replication to logical replication for selected databases or all > databases. However, there is no provision to replicate only a few > selected tables. For s

RE: Conflict detection for update_deleted in logical replication

2025-07-31 Thread Zhijie Hou (Fujitsu)
On Thursday, July 31, 2025 5:26 PM shveta malik wrote: > > On Tue, Jul 29, 2025 at 10:51 AM Zhijie Hou (Fujitsu) > wrote: > > > > > > This is the V54 patch set, with only patch 0001 updated to address the > > latest comments. > > > > Thanks for

RE: Conflict detection for update_deleted in logical replication

2025-07-31 Thread Zhijie Hou (Fujitsu)
On Thursday, July 31, 2025 5:29 PM Amit Kapila wrote: > > On Tue, Jul 29, 2025 at 10:51 AM Zhijie Hou (Fujitsu) > wrote: > > > > This is the V54 patch set, with only patch 0001 updated to address the > > latest comments. > > > > Few minor comments: Than

RE: Conflict detection for update_deleted in logical replication

2025-07-28 Thread Zhijie Hou (Fujitsu)
On Monday, July 28, 2025 5:54 PM Amit Kapila wrote: > > On Fri, Jul 25, 2025 at 4:38 PM Zhijie Hou (Fujitsu) > wrote: > > > > Right, I think it makes sense to do with the index scan when the > > index's xmin is less than the conflict detection xmin, as that ca

RE: Conflict detection for update_deleted in logical replication

2025-07-28 Thread Zhijie Hou (Fujitsu)
On Monday, July 28, 2025 7:43 PM shveta malik wrote: > On Mon, Jul 28, 2025 at 4:38 PM shveta malik > wrote: > > > > On Fri, Jul 25, 2025 at 4:38 PM Zhijie Hou (Fujitsu) > > wrote: > > > > > > > > > The V53-0001 also includes Shveta'

RE: Conflict detection for update_deleted in logical replication

2025-07-25 Thread Zhijie Hou (Fujitsu)
On Friday, July 25, 2025 2:33 PM Amit Kapila wrote: > > On Wed, Jul 23, 2025 at 12:53 PM Zhijie Hou (Fujitsu) > wrote: > > > > Thanks for pushing. I have rebased the remaining patches. > > > > + * This function performs a full table scan instead of using inde

RE: Conflict detection for update_deleted in logical replication

2025-07-25 Thread Zhijie Hou (Fujitsu)
On Thursday, July 24, 2025 11:42 AM shveta malik wrote: > > On Wed, Jul 23, 2025 at 12:53 PM Zhijie Hou (Fujitsu) > wrote: > > > > On Wednesday, July 23, 2025 12:08 PM Amit Kapila > wrote: > > > > > > On Wed, Jul 23, 2025 at 3:51 AM Masahiko Sawada &

RE: Conflict detection for update_deleted in logical replication

2025-07-23 Thread Zhijie Hou (Fujitsu)
On Wednesday, July 23, 2025 12:08 PM Amit Kapila wrote: > > On Wed, Jul 23, 2025 at 3:51 AM Masahiko Sawada > wrote: > > > > I've reviewed the 0001 patch and it looks good to me. > > > > Thanks, I have pushed the 0001 patch. Thanks for pushing. I have rebased the remaining patches. I have re

RE: Logical replication prefetch

2025-07-14 Thread Zhijie Hou (Fujitsu)
On Monday, July 14, 2025 2:36 PM Konstantin Knizhnik wrote: > On 14/07/2025 4:20 am, Zhijie Hou (Fujitsu) wrote: > > Additionally, I was also exploring ways to improve performance and have > > tried an > > alternative version of prefetch for experimentation. The alter

RE: Logical replication prefetch

2025-07-13 Thread Zhijie Hou (Fujitsu)
On Tuesday, July 8, 2025 2:36 PM Konstantin Knizhnik wrote: > > There is well known Postgres problem that logical replication subscriber can > not caught-up with publisher just because LR changes are applied by single > worker and at publisher changes are made by multiple concurrent backends. > T

RE: Conflict detection for update_deleted in logical replication

2025-07-07 Thread Zhijie Hou (Fujitsu)
On Mon, Jul 7, 2025 at 11:03 AM Zhijie Hou (Fujitsu) wrote: > > On Sun, Jul 6, 2025 at 10:51 PM Masahiko Sawada wrote: > > > == > > > == > > > The workload is mostly same as [4]. > >

RE: Conflict detection for update_deleted in logical replication

2025-07-06 Thread Zhijie Hou (Fujitsu)
On Sun, Jul 6, 2025 at 10:51 PM Masahiko Sawada wrote: > > On Sun, Jul 6, 2025 at 8:03 PM Hayato Kuroda (Fujitsu) > wrote: > > > > Dear hackers, > > > > As a confirmation purpose, I did performance testing with four > > workloads we did before. > > Thank you for doing the performance tests! >

RE: Conflict detection for update_deleted in logical replication

2025-07-06 Thread Zhijie Hou (Fujitsu)
On Sun, Jul 6, 2025 at 10:51 PM Masahiko Sawada wrote: > > On Fri, Jul 4, 2025 at 8:18 PM Zhijie Hou (Fujitsu) > wrote: > > > > On Wed, Jul 2, 2025 at 3:28 PM Hou, Zhijie wrote: > > > Kindly use the latest patch set for performance testing. > > > > D

RE: Report replica identity in pg_publication_tables

2025-07-02 Thread Zhijie Hou (Fujitsu)
On Mon, Jun 30, 2025 at 8:09 PM Ashutosh Bapat wrote: > > On Mon, Jun 30, 2025 at 5:17 PM Amit Kapila > wrote: > > > > On Mon, Jun 30, 2025 at 3:44 PM Ashutosh Bapat > > wrote: > > > > > Given a set of publications that a WAL sender is using, > > > pg_publication_tables can be used to get the

RE: Invalid pointer access in logical decoding after error

2025-07-02 Thread Zhijie Hou (Fujitsu)
On Wed, Jul 2, 2025 at 2:42 PM vignesh C wrote: > > Hi, > > I encountered an invalid pointer access issue. Below are the steps to > reproduce the issue: ... > The error occurs because entry->columns is allocated in the entry > private context (entry->entry_cxt) by pub_collist_to_bitmapset(). T

RE: Conflict detection for update_deleted in logical replication

2025-07-01 Thread Zhijie Hou (Fujitsu)
On Tue, Jul 1, 2025 at 6:10 PM Zhijie Hou (Fujitsu) wrote: > Here is V45 patch set. With the main patch set now stable, I am summarizing the performance tests conducted before for reference. In earlier tests [1], we confirmed that in a pub-sub cluster with high workload on the publisher (

RE: Conflict detection for update_deleted in logical replication

2025-06-25 Thread Zhijie Hou (Fujitsu)
On Wed, Jun 25, 2025 at 2:57 PM shveta malik wrote: > > > > > Here is the V41 patch set which includes the following changes: > > > > Thanks for the patches. Few trivial things: > > 1) > In ReplicationSlotAcquire(), does it make more sense to move the error after > checking the slot's existence

RE: Slot's restart_lsn may point to removed WAL segment after hard restart unexpectedly

2025-06-24 Thread Zhijie Hou (Fujitsu)
Hi, After commit ca307d5, I noticed another crash when testing some other logical replication features. The server with max_replication_slots set to 0 would crash when executing CHECKPOINT. TRAP: failed Assert("ReplicationSlotCtl != NULL"), File: "slot.c", Line: 1162, PID: 577315 postgres: che

RE: Replication slot is not able to sync up

2025-06-15 Thread Zhijie Hou (Fujitsu)
On Sat, Jun 14, 2025 at 11:37 PM Dilip Kumar wrote: > > On Fri, May 30, 2025 at 3:38 PM Zhijie Hou (Fujitsu) > wrote: > > > > On Wed, May 28, 2025 at 2:09 AM Masahiko Sawada wrote: > > > > > > On Fri, May 23, 2025 at 10:07 PM Amit Kapila > > > &g

RE: Logical Replication slot disappeared after promote Standby

2025-06-12 Thread Zhijie Hou (Fujitsu)
On Thu, Jun 12, 2025 at 4:08 PM Perumal Raj wrote: > Hi Community, > > I have installed postgres version 17.5 with following setup, > > Primary >-- Secondary A >-- Secondary B > -- Secondary C > > Config: > wal_level = 'logical' > max_wal_senders = '10' > max

RE: failover logical replication slots

2025-06-10 Thread Zhijie Hou (Fujitsu)
On Tue, Jun 10, 2025 at 11:46 PM Fabrice Chapuis wrote: > I'm working with logical replication in a PostgreSQL 17 setup, and I'm > exploring the new option to make replication slots failover safe in a highly > available environment using physical standby nodes managed by Patroni. > > After a swit

RE: Replication slot is not able to sync up

2025-06-10 Thread Zhijie Hou (Fujitsu)
On Thu, May 29, 2025 at 11:09 AM shveta malik wrote: > > On Wed, May 28, 2025 at 11:56 AM Masahiko Sawada > wrote: > > > > > > I didn't know it was intended for testing and debugging purposes so > > clearilying it in the documentation would be a good idea. > > I have added the suggested docs i

RE: Conflict detection for update_deleted in logical replication

2025-06-04 Thread Zhijie Hou (Fujitsu)
On Wed, Jun 4, 2025 at 6:43 PM Zhijie Hou (Fujitsu) wrote: > > On Mon, Jun 2, 2025 at 2:39 PM Amit Kapila wrote: > > > > On Mon, May 26, 2025 at 12:46 PM Zhijie Hou (Fujitsu) > > wrote: > > > > > > Attaching the V32 patch set which addressed comments

RE: Replication slot is not able to sync up

2025-05-30 Thread Zhijie Hou (Fujitsu)
On Wed, May 28, 2025 at 2:09 AM Masahiko Sawada wrote: > > On Fri, May 23, 2025 at 10:07 PM Amit Kapila > wrote: > > > > In the case presented here, the logical slot is expected to keep > > forwarding, and in the consecutive sync cycle, the sync should be > > successful. Users using logical decod

RE: Replication slot is not able to sync up

2025-05-27 Thread Zhijie Hou (Fujitsu)
On Wed, May 28, 2025 at 2:09 AM Masahiko Sawada wrote: > > On Fri, May 23, 2025 at 10:07 PM Amit Kapila > wrote: > > > > In the case presented here, the logical slot is expected to keep > > forwarding, and in the consecutive sync cycle, the sync should be > > successful. Users using logical decod

RE: Conflict detection for update_deleted in logical replication

2025-05-26 Thread Zhijie Hou (Fujitsu)
On Fri, May 23, 2025 at 11:51 PM Xuneng Zhou wrote: > Thanks for the effort on the patches. I did a quick look on them before > diving into the logic and discussion. Below are a few minor typos found in > version 31. Thanks for the comments! I have fixed these typos in latest version. Best Regar

RE: Conflict detection for update_deleted in logical replication

2025-05-26 Thread Zhijie Hou (Fujitsu)
On Fri, May 23, 2025 at 2:45 PM shveta malik wrote: > > Thanks you for v31 patch-set. Please find few comments on patch001: Thanks for the comments. > > 1) > > wait_for_local_flush: > > + if (data->last_recv_time && > + TimestampDifferenceExceeds(data->flushpos_update_time, > +data->last

RE: Conflict detection for update_deleted in logical replication

2025-05-26 Thread Zhijie Hou (Fujitsu)
On Sat, May 24, 2025 at 6:28 PM Dilip Kumar wrote: > > On Sat, May 24, 2025 at 11:00 AM Amit Kapila > wrote: > > > > On Sat, May 24, 2025 at 10:29 AM Dilip Kumar > wrote: > > > > > > On Sat, May 24, 2025 at 10:04 AM Dilip Kumar > wrote: > > > > > > > > On Fri, May 23, 2025 at 9:21 PM Xuneng Zh

RE: Conflict detection for update_deleted in logical replication

2025-05-26 Thread Zhijie Hou (Fujitsu)
On Sun, May 25, 2025 at 4:36 PM Dilip Kumar wrote: > > On Sat, May 24, 2025 at 4:46 PM Amit Kapila > wrote: > > > > This sounds reasonable to me. Let us see what others think. > > > > I was looking into the for getting the transaction status from > publisher, what I would assume this patch shou

RE: Conflict detection for update_deleted in logical replication

2025-05-21 Thread Zhijie Hou (Fujitsu)
On Fri, May 16, 2025 at 7:31 PM Amit Kapila wrote: > > A few more comments: > = > 3. > maybe_advance_nonremovable_xid(RetainConflictInfoData *data, >bool status_received) > { > /* Exit early if retaining conflict information is not required */ > if (!MySubscription->retainconfl

RE: Conflict detection for update_deleted in logical replication

2025-05-21 Thread Zhijie Hou (Fujitsu)
On Tue, May 20, 2025 at 6:30 PM shveta malik wrote: > > Few more comments mostly for patch001: Thanks for the comments! > > 4) > For this feature, since we are only interested in remote UPDATEs happening > concurrently, so shall we ask primary to provide oldest "UPDATE" > transaction-id in comm

RE: Conflict detection for update_deleted in logical replication

2025-05-21 Thread Zhijie Hou (Fujitsu)
On Tue, May 20, 2025 at 11:08 AM shveta malik wrote: > > Please find few more comments: Thanks for the comments! > > 2) > > -- > send_feedback(last_received, requestReply, requestReply); > > + maybe_advance_nonremovable_xid(&data, false); > + > /* > * Force reporting to ensure l

RE: Fix slot synchronization with two_phase decoding enabled

2025-05-08 Thread Zhijie Hou (Fujitsu)
On Thu, May 8, 2025 at 6:04 PM Zhijie Hou (Fujitsu) wrote: > > On Tue, May 6, 2025 at 7:22 PM Zhijie Hou (Fujitsu) wrote: > > > > > On Mon, May 5, 2025 at 6:59 PM Amit Kapila wrote: > > > > > > On Sun, May 4, 2025 at 2:33 PM Masahiko Sawada > > &

RE: Fix slot synchronization with two_phase decoding enabled

2025-05-08 Thread Zhijie Hou (Fujitsu)
On Tue, May 6, 2025 at 7:22 PM Zhijie Hou (Fujitsu) wrote: > > On Mon, May 5, 2025 at 6:59 PM Amit Kapila wrote: > > > > On Sun, May 4, 2025 at 2:33 PM Masahiko Sawada > > > wrote: > > > > > > While I cannot be entirely certain of my analysis, I beli

RE: Fix slot synchronization with two_phase decoding enabled

2025-05-06 Thread Zhijie Hou (Fujitsu)
On Mon, May 5, 2025 at 6:59 PM Amit Kapila wrote: > > On Sun, May 4, 2025 at 2:33 PM Masahiko Sawada > wrote: > > > > While I cannot be entirely certain of my analysis, I believe the root > > cause might be related to the backward movement of the confirmed_flush > > LSN. The following scenario se

RE: Fix slot synchronization with two_phase decoding enabled

2025-04-30 Thread Zhijie Hou (Fujitsu)
On Tue, Apr 29, 2025 at 6:57 PM Amit Kapila wrote: > > On Tue, Apr 29, 2025 at 3:01 AM Masahiko Sawada > wrote: > > > > Your fix looks good to me. Is it worth considering putting an > > assertion to verify if new two_phase_at is equal to or greater than > > confirmed_lsn (or at least it doesn't g

RE: Fix slot synchronization with two_phase decoding enabled

2025-04-28 Thread Zhijie Hou (Fujitsu)
On Mon, Apr 28, 2025 at 7:33 PM Zhijie Hou (Fujitsu) wrote: > > Thanks for reviewing. Here is V3 patch that addressed it. > > BTW, I also did some tests to confirm the catalog_xmin could still be > ahead in some case, and here is an example: > > 1. Create a failover r

RE: Fix slot synchronization with two_phase decoding enabled

2025-04-28 Thread Zhijie Hou (Fujitsu)
On Mon, Apr 28, 2025 at 5:33 PM shveta malik wrote: > > On Mon, Apr 28, 2025 at 2:27 PM Zhijie Hou (Fujitsu) > wrote: > > > > On Fri, Apr 18, 2025 at 12:29 PM Amit Kapila wrote: > > > > > > On Thu, Apr 17, 2025 at 6:14 PM Zhijie Hou (Fujitsu) &

RE: Fix slot synchronization with two_phase decoding enabled

2025-04-28 Thread Zhijie Hou (Fujitsu)
On Fri, Apr 18, 2025 at 12:29 PM Amit Kapila wrote: > > On Thu, Apr 17, 2025 at 6:14 PM Zhijie Hou (Fujitsu) > > > > - > > Fix > > - > > > > I think we should keep the confirmed_flush even if the previous synced > > restart_lsn/catalog_xm

RE: Fix premature xmin advancement during fast forward decoding

2025-04-27 Thread Zhijie Hou (Fujitsu)
On Sun, Apr 27, 2025 at 2:07 PM Masahiko Sawada wrote: > > On Sat, Apr 26, 2025 at 5:01 AM Amit Kapila > wrote: > > > > On Sat, Apr 26, 2025 at 12:01 AM Masahiko Sawada > wrote: > > > > > > On Fri, Apr 25, 2025 at 4:42 AM Amit Kapila > wrote: > > > > > > > > Can you think of any better ideas? >

RE: Fix premature xmin advancement during fast forward decoding

2025-04-24 Thread Zhijie Hou (Fujitsu)
On Fri, Apr 25, 2025 at 5:44 AM Masahiko Sawada wrote: (Resending this email after compressing the attachment) > On Tue, Apr 22, 2025 at 12:06 AM Zhijie Hou (Fujitsu) > wrote: > > > > Hi, > > > > When analyzing some issues in another thread[1], I found a bug that &

RE: Fix premature xmin advancement during fast forward decoding

2025-04-23 Thread Zhijie Hou (Fujitsu)
On Wed, Apr 23, 2025 at 2:31 PM shveta malik wrote: > > On Tue, Apr 22, 2025 at 12:36 PM Zhijie Hou (Fujitsu) > wrote: > > > > Hi, > > > > To fix this, I think we can allow the base snapshot to be built during fast > > forward decoding, as implemented i

Fix premature xmin advancement during fast forward decoding

2025-04-22 Thread Zhijie Hou (Fujitsu)
Hi, When analyzing some issues in another thread[1], I found a bug that fast forward decoding could lead to premature advancement of catalog_xmin, resulting in required catalog data being removed by vacuum. The issue arises because we do not build a base snapshot when decoding changes during fast

RE: Fix slot synchronization with two_phase decoding enabled

2025-04-21 Thread Zhijie Hou (Fujitsu)
On Tue, Apr 22, 2025 at 11:23 AM Amit Kapila wrote: > > On Fri, Apr 18, 2025 at 9:58 AM Amit Kapila wrote: > > > > On Thu, Apr 17, 2025 at 6:14 PM Zhijie Hou (Fujitsu) > > wrote: > > > > > > - > > > Fix > > > - > > >

RE: Fix slot synchronization with two_phase decoding enabled

2025-04-20 Thread Zhijie Hou (Fujitsu)
On Sat, Apr 19, 2025 at 2:19 AM Masahiko Sawada wrote: > > On Tue, Apr 8, 2025 at 10:14 PM Zhijie Hou (Fujitsu) > wrote: > > > > > > -- > > Approach 2 > > -- > > > > Instead of disallowing the use of two-phase and failover t

RE: Fix slot synchronization with two_phase decoding enabled

2025-04-17 Thread Zhijie Hou (Fujitsu)
On Thu, Apr 17, 2025 at 8:44 PM Zhijie Hou (Fujitsu) wrote: > > Hi, > > The recently added tests for slotsync on two-phase enabled slots > failed[1] due to a broken assertion triggered while decoding the > COMMIT PREPARED record on the promoted standby. Here is the missing

RE: Fix slot synchronization with two_phase decoding enabled

2025-04-17 Thread Zhijie Hou (Fujitsu)
Hi, The recently added tests for slotsync on two-phase enabled slots failed[1] due to a broken assertion triggered while decoding the COMMIT PREPARED record on the promoted standby. - Analysis - if ((txn->final_lsn < two_phase_at) && is_commit) { /*

RE: Skipping schema changes in publication

2025-04-15 Thread Zhijie Hou (Fujitsu)
On Thu, Apr 10, 2025 at 7:25 PM Amit Kapila wrote: > > On Tue, Jan 9, 2024 at 12:02 PM vignesh C wrote: > > > > As I did not see much interest from others, I'm withdrawing this patch > > for now. But if there is any interest others in future, I would be > > more than happy to work on this feature

Fix replica identity checks for MERGE command on published table.

2025-04-10 Thread Zhijie Hou (Fujitsu)
Hi, While testing publication DDLs, I noticed an unexpected behavior where the MERGE command can be executed on tables lacking replica identity keys, regardless of whether they are part of a publication that publishes updates and deletes. Replication and application of the updates and deletes gen

RE: Fix slot synchronization with two_phase decoding enabled

2025-04-08 Thread Zhijie Hou (Fujitsu)
On Thu, Apr 3, 2025 at 3:16 PM Zhijie Hou (Fujitsu) wrote: > > On Thu, Apr 3, 2025 at 1:38 PM Masahiko Sawada wrote: > > > > > On Wed, Apr 2, 2025 at 7:58 PM Amit Kapila > > wrote: > > > > > > On Thu, Apr 3, 2025 at 7:50 AM Zhijie Hou (Fujitsu) >

RE: BUG #18815: Logical replication worker Segmentation fault

2025-04-07 Thread Zhijie Hou (Fujitsu)
On Tue, Apr 8, 2025 at 2:00 PM Amit Kapila wrote: > > On Mon, Apr 7, 2025 at 3:28 PM Zhijie Hou (Fujitsu) > wrote: > > > > > What is the > > > behavior of conflict reporting code in case of exclusion constraints? > > > > Under logical replication cont

RE: BUG #18815: Logical replication worker Segmentation fault

2025-04-07 Thread Zhijie Hou (Fujitsu)
On Mon, Apr 7, 2025 at 5:37 PM Amit Kapila wrote: > > On Wed, Mar 26, 2025 at 10:3 AM Zhijie Hou (Fujitsu) > wrote: > > > > On Wed, Feb 19, 2025 at 4:38 AM Tom Lane wrote: > > > > My colleague Nisha reported off-list about a crash in logical replication

RE: pg_recvlogical cannot create slots with failover=true

2025-04-06 Thread Zhijie Hou (Fujitsu)
On Sat, Apr 5, 2025 at 1:45 AM Masahiko Sawada wrote: Hi, > Thank you for updating the patch! Pushed with small cosmetic changes. Thanks for pushing the feature ! I noticed one typo in the doc and here is a tiny patch to fix it. -The --two-phase and --falover options +The --two

RE: Conflict detection for multiple_unique_conflicts in logical replication

2025-04-05 Thread Zhijie Hou (Fujitsu)
On Fri, Mar 21, 2025 at 12:50 PM Nisha Moond wrote: > Thanks, Hou-san, for the review and fix patches. I’ve incorporated > your suggestions. > Attached are the v7 patches, including patch 002, which implements > stats collection for 'multiple_unique_conflicts' in > pg_stat_subscription_stats.

RE: Fix slot synchronization with two_phase decoding enabled

2025-04-05 Thread Zhijie Hou (Fujitsu)
On Thu, Apr 3, 2025 at 1:38 PM Masahiko Sawada wrote: > > On Wed, Apr 2, 2025 at 7:58 PM Amit Kapila > wrote: > > > > On Thu, Apr 3, 2025 at 7:50 AM Zhijie Hou (Fujitsu) > > wrote: > > > > > > On Thu, Apr 3, 2025 at 3:30 AM Masahiko Sawada wrote: &

RE: Fix slot synchronization with two_phase decoding enabled

2025-04-05 Thread Zhijie Hou (Fujitsu)
On Thu, Mar 27, 2025 at 2:29 PM Amit Kapila wrote: > > On Tue, Mar 25, 2025 at 12:1 PM Amit Kapila > wrote: > > > > On Tue, Mar 25, 2025 at 11:05 AM Zhijie Hou (Fujitsu) > > wrote: > > > > > > Hi, > > > > > > When testi

RE: Fix slot synchronization with two_phase decoding enabled

2025-04-02 Thread Zhijie Hou (Fujitsu)
On Thu, Apr 3, 2025 at 3:30 AM Masahiko Sawada wrote: > > On Wed, Apr 2, 2025 at 6:33 AM Zhijie Hou (Fujitsu) > wrote: > > > > On Wed, Apr 2, 2025 at 3:45 PM Masahiko Sawada wrote: > > > > Hi, > > > > > > > > On Mon, Mar 31, 2025 a

RE: Fix slot synchronization with two_phase decoding enabled

2025-04-02 Thread Zhijie Hou (Fujitsu)
On Wed, Apr 2, 2025 at 3:45 PM Masahiko Sawada wrote: Hi, > > On Mon, Mar 31, 2025 at 4:3 AM Zhijie Hou (Fujitsu) > wrote: > > > > On Thu, Mar 27, 2025 at 2:29 PM Amit Kapila wrote: > > > > > > > > On Tue, Mar 25, 2025 at 12:1 PM Amit Kapila >

RE: Fix slot synchronization with two_phase decoding enabled

2025-04-01 Thread Zhijie Hou (Fujitsu)
On Wed, Apr 2, 2025 at 12:41 PM Amit Kapila wrote: > > On Tue, Apr 1, 2025 at 4:28 PM Zhijie Hou (Fujitsu) > > wrote: > > > > Here is the V3 patch set which addressed all the comments. > > > > Comment 0n 0001 > NULL for logical slots where > +

RE: Fix slot synchronization with two_phase decoding enabled

2025-04-01 Thread Zhijie Hou (Fujitsu)
On Tue, Apr 1, 2025 at 2:09 PM Amit Kapila wrote: > > On Mon, Mar 31, 2025 at 5:0 PM Zhijie Hou (Fujitsu) > wrote: > > > > Thanks for the comments, they have been addressed in V2. > > > > In the PG17 patch, we only have a check preventing failover and > tw

RE: BUG #18815: Logical replication worker Segmentation fault

2025-03-25 Thread Zhijie Hou (Fujitsu)
On Wed, Feb 19, 2025 at 4:38 AM Tom Lane wrote: Hi, > > Here's a completed set of patches for bug #18815 [1] (which, briefly, > is that the logrep worker opens/closes indexes multiple times and > thereby breaks brininsertcleanup). > > 0001 adds a subscriber-side BRIN index to 013_partition.pl

Fix slot synchronization with two_phase decoding enabled

2025-03-24 Thread Zhijie Hou (Fujitsu)
Hi, When testing the slot synchronization with logical replication slots that enabled two_phase decoding, I found that transactions prepared before two-phase decoding is enabled may fail to replicate to the subscriber after being committed on a promoted standby following a failover. To reproduce

RE: Conflict detection for multiple_unique_conflicts in logical replication

2025-03-20 Thread Zhijie Hou (Fujitsu)
On Thu, Mar 20, 2025 at 3:06 PM Nisha Moond wrote: > > Attached is v6 patch with above comments addressed. Thanks updating the patch. I have some comments: 1. The naming style of variables changed in this function seems a bit Inconsistent with existing ones, I feel we'd better use similar style

RE: Adding a '--clean-publisher-objects' option to 'pg_createsubscriber' utility.

2025-03-19 Thread Zhijie Hou (Fujitsu)
On Wed, Mar 19, 2025 at 2:55 PM Shubham Khanna wrote: > > I have merged the changes and prepared the latest patch. The attached > patch contains the suggested changes. Thanks for updating the patch. Here are few comments: 1. pg_log_error("object type \"%s

RE: Proposal: Deferred Replica Filtering for PostgreSQL Logical Replication

2025-03-18 Thread Zhijie Hou (Fujitsu)
On 2025/03/19 10:00:00 wrote: On Wed, Mar 19, 2025 at 8:56 AM Dean wrote: > Unfortunately, neither column lists nor row filters can provide the level of > control I'm proposing. These revised examples might help illustrate the use > case for DRF: > > Alice, Bob, and Eve subscribe to changes on a

RE: Adding a '--clean-publisher-objects' option to 'pg_createsubscriber' utility.

2025-03-14 Thread Zhijie Hou (Fujitsu)
On 2025/03/14 15:25:00 wrote: On Wed, Mar 12, 2025 at 12:13 PM David G. Johnston wrote: > On Tuesday, March 11, 2025, Amit Kapila > wrote: > > On Tue, Mar 11, 2025 at 8:21 PM David G. Johnston > > mailto:david.g.johns...@gmail.com> > wrote: > > > > > > On Tue,

RE: 001_rep_changes.pl succeeds after a long time

2025-03-13 Thread Zhijie Hou (Fujitsu)
On Thursday, March 13, 2025 12:29 AM Álvaro Herrera wrote: Hi, > I noticed that the test file 001_repo_changes.pl finished successfully after > having taken 180s to run. This seems pretty suspicious -- normally that step > takes around one second. > > The problem is seen in this step: > > [1

RE: Selectively invalidate caches in pgoutput module

2025-03-11 Thread Zhijie Hou (Fujitsu)
On Monday, March 10, 2025 9:12 PM Kuroda, Hayato wrote: > > > Currently, only the leaf partition is invalidated when the published > > table is partitioned. However, I think pgoutput could cache both the > > partitioned table and the leaf partition table as relsync entries. > > > > For INSERT/UP

RE: Selectively invalidate caches in pgoutput module

2025-03-10 Thread Zhijie Hou (Fujitsu)
On Monday, March 10, 2025 7:00 PM Kuroda, Hayato wrote: > > I did a self-reviewing and updated a patch. PSA new version. What's new: Thanks for updating the patch. I tested the behavior for partitioned table and have a comment on this. > + relids = GetPublicationRelations(pubform-

RE: Selectively invalidate caches in pgoutput module

2025-03-04 Thread Zhijie Hou (Fujitsu)
On Tuesday, March 4, 2025 7:44 PM Kuroda, Hayato/黒田 隼人 wrote: Hi, > > > > When the ALTER PUBLICATION command is executed, all entries in > > RelationSyncCache > > > will be discarded anyway. This mechanism works well but is sometimes > > > not > > efficient. > > > For example, when the ALTER P

RE: long-standing data loss bug in initial sync of logical replication

2025-03-02 Thread Zhijie Hou (Fujitsu)
On Friday, February 28, 2025 4:28 PM Benoit Lobréau wrote: > > It took me a while but I ran the test on my laptop with 20 runs per test. I > asked > for a dedicated server and will re-run the tests if/when I have it. > > count of partitions | Head (sec) |Fix (sec) |Degradation (%) >

RE: long-standing data loss bug in initial sync of logical replication

2025-02-27 Thread Zhijie Hou (Fujitsu)
On Monday, February 24, 2025 5:50 PM Amit Kapila wrote: > > On Wed, Dec 11, 2024 at 12:37 PM Masahiko Sawada > wrote: > > > > I confirmed that the proposed patch fixes these issues. I have one > > question about the patch: > > > > In the main loop in SnapBuildDistributeSnapshotAndInval(), we ha

RE: pg_copy_logical_replication_slot doesn't copy the failover property

2025-02-24 Thread Zhijie Hou (Fujitsu)
On Monday, February 24, 2025 10:46 AM Amit Kapila wrote: (The previous email was blocked, here is another attempt) > > On Sat, Feb 22, 2025 at 11:26 AM Shlok Kyal > wrote: > > > > Adding Amit to the thread. > > > > Thanks. > > Looking at the original complaint: > > > It says > >

RE: create subscription with (origin = none, copy_data = on)

2025-02-20 Thread Zhijie Hou (Fujitsu)
On Wednesday, January 29, 2025 8:19 PM Shlok Kyal wrote: > > I have addressed the comments. Here is an updated patch. Thanks for updating the patch. The patches look mostly OK to me, I only have one minor comments in 0002. 1. +CREATE PUBLICATION pub_b_c_2 FOR TABLE tab_part2_1; +)); + +($resu

RE: ReplicationSlotRelease() crashes when the instance is in the single user mode

2025-02-20 Thread Zhijie Hou (Fujitsu)
On Thursday, February 20, 2025 10:23 AM Hayato Kuroda (Fujitsu) wrote: > > Dear Michael, > > > I did not check how these call behave individually, just a few > > comments while putting my eyes on the patch. > > > > + if (!IsUnderPostmaster) > > + elog(ERROR, > > +

RE: Restrict copying of invalidated replication slots

2025-02-18 Thread Zhijie Hou (Fujitsu)
On Monday, February 17, 2025 7:31 PM Shlok Kyal wrote: > > On Thu, 13 Feb 2025 at 15:54, vignesh C wrote: > > > > On Tue, 4 Feb 2025 at 15:27, Shlok Kyal wrote: > > > > > > Hi, > > > > > > Currently, we can copy an invalidated slot using the function > > > 'pg_copy_logical_replication_slot'. A

RE: Enhance 'pg_createsubscriber' to retrieve databases automatically when no database is provided.

2025-02-18 Thread Zhijie Hou (Fujitsu)
On Thursday, February 13, 2025 11:28 AM Shubham Khanna wrote: Hi, > The attached patch contains the required changes. Thanks for updating the patch. I think it's a useful feature. Here are few review comments: 1. + if (opt.all_databases) +

RE: Introduce XID age and inactive timeout based replication slot invalidation

2025-02-11 Thread Zhijie Hou (Fujitsu)
On Wednesday, February 12, 2025 11:56 AM Amit Kapila wrote: > > On Tue, Feb 11, 2025 at 9:39 PM Nathan Bossart > wrote: > > > > On Tue, Feb 11, 2025 at 03:22:49PM +0100, Álvaro Herrera wrote: > > > I find this proposed patch a bit strange and I feel it needs more > > > explanation. > > > > > >

RE: Introduce XID age and inactive timeout based replication slot invalidation

2025-02-10 Thread Zhijie Hou (Fujitsu)
On Monday, February 10, 2025 8:03 PM Nisha Moond wrote: > > On Sat, Feb 8, 2025 at 12:28 PM Zhijie Hou (Fujitsu) > wrote: > > > > > 3. > > > > + if (cause & RS_INVAL_HORIZON) > > + { > >

RE: Introduce XID age and inactive timeout based replication slot invalidation

2025-02-10 Thread Zhijie Hou (Fujitsu)
On Monday, February 10, 2025 2:10 PM Amit Kapila wrote: > > On Sat, Feb 8, 2025 at 12:28 PM Zhijie Hou (Fujitsu) > wrote: > > > > 3. > > > > + if (cause & RS_INVAL_HORIZON) > > + { > > +

RE: Introduce XID age and inactive timeout based replication slot invalidation

2025-02-07 Thread Zhijie Hou (Fujitsu)
On Friday, February 7, 2025 9:06 PM Nisha Moond wrote: > > Attached v72 patches, addressed the above comments as well as Vignesh's > comments in [2]. > - There are no new changes in patch-002. Thanks for updating the patch, I have few review comments: 1. > InvalidateObsoleteReplicationSlots(R

RE: Avoid updating inactive_since for invalid replication slots

2025-02-03 Thread Zhijie Hou (Fujitsu)
On Monday, February 3, 2025 8:03 PM Nisha Moond wrote: > > Hi Hackers, > (CC people involved in the earlier discussion) > > Right now, it is possible for the 'inactive_since' value of an invalid > replication > slot to be updated multiple times, which is unexpected behavior. > As suggested in

RE: Proposal: Filter irrelevant change before reassemble transactions during logical decoding

2025-02-03 Thread Zhijie Hou (Fujitsu)
On Friday, January 31, 2025 9:43 PM Kuroda, Hayato/黒田 隼人 > > Dear Ajin, Hou, > > > - Snapshot construction > > I understand the approach that we do not try to filter for all the workloads; > do > just best-effort. > > 3. > Both ReorderBuffer and RelFileLocatorFilterCache have the same lifeti

RE: Proposal: Filter irrelevant change before reassemble transactions during logical decoding

2025-01-23 Thread Zhijie Hou (Fujitsu)
Hi, I think the proposed feature is interesting, and after reviewing the original discussion thread, I'd like to help move it forward. To that end, I've summarized the key points of the discussion so far, including the challenges highlighted in the thread and some issues I identified off-list, alo

RE: create subscription with (origin = none, copy_data = on)

2025-01-23 Thread Zhijie Hou (Fujitsu)
On Thursday, January 23, 2025 4:43 PM Shlok Kyal wrote: > > On Thu, 23 Jan 2025 at 12:35, Shlok Kyal wrote: > > > > On Wed, 22 Jan 2025 at 09:00, Zhijie Hou (Fujitsu) > > wrote: > > > > > > Thanks for the patch. > > > > > > I agre

RE: create subscription with (origin = none, copy_data = on)

2025-01-21 Thread Zhijie Hou (Fujitsu)
On Tuesday, January 21, 2025 1:31 AM vignesh C wrote: Hi, > > On Mon, 20 Jan 2025 at 17:31, Amit Kapila wrote: > > > > On Sat, Jan 18, 2025 at 10:31 AM vignesh C wrote: > > > > > > Attached patch has the fix for this issue which includes the > > > partition tables also for the publication now

RE: Conflict detection for update_deleted in logical replication

2025-01-10 Thread Zhijie Hou (Fujitsu)
On Friday, January 10, 2025 8:43 AM Masahiko Sawada wrote: Hi, > > On Wed, Jan 8, 2025 at 7:26 PM Zhijie Hou (Fujitsu) > wrote: > > > > On Thursday, January 9, 2025 9:48 AM Masahiko Sawada > wrote: > > > > Hi, > > > > > >

RE: Conflict detection for update_deleted in logical replication

2025-01-09 Thread Zhijie Hou (Fujitsu)
On Wednesday, January 8, 2025 3:49 PM Nisha Moond wrote: > > On Tue, Jan 7, 2025 at 6:04 PM Zhijie Hou (Fujitsu) > wrote: > > > > > > Attached the V19 patch which addressed comments in [1][2][3][4][5][6][7]. > > > > Here are a couple of initial review

RE: Conflict detection for update_deleted in logical replication

2025-01-09 Thread Zhijie Hou (Fujitsu)
On Wednesday, January 8, 2025 7:03 PM vignesh C wrote: Hi, > Consider a LR setup with retain_conflict_info=true for a table t1: > Publisher: > insert into t1 values(1); > -- Have a open transaction before delete operation in subscriber begin; > > Subscriber: > -- delete the record that was repl

RE: Conflict detection for update_deleted in logical replication

2025-01-08 Thread Zhijie Hou (Fujitsu)
On Thursday, January 9, 2025 9:48 AM Masahiko Sawada Hi, > > On Wed, Jan 8, 2025 at 3:00 AM Zhijie Hou (Fujitsu) > wrote: > > > > On Wednesday, January 8, 2025 6:33 PM Masahiko Sawada > wrote: > > > > Hi, > > > > > On Wed, Jan 8, 2025 at 1:

RE: Conflict detection for update_deleted in logical replication

2025-01-08 Thread Zhijie Hou (Fujitsu)
On Thursday, January 9, 2025 9:48 AM Masahiko Sawada wrote: Hi, > > On Wed, Jan 8, 2025 at 3:00 AM Zhijie Hou (Fujitsu) > wrote: > > > > On Wednesday, January 8, 2025 6:33 PM Masahiko Sawada > wrote: > > > > Hi, > > > > > On Wed, Jan 8, 2

RE: Conflict detection for update_deleted in logical replication

2025-01-08 Thread Zhijie Hou (Fujitsu)
On Wednesday, January 8, 2025 6:33 PM Masahiko Sawada wrote: Hi, > On Wed, Jan 8, 2025 at 1:53 AM Amit Kapila > wrote: > > On Wed, Jan 8, 2025 at 3:02 PM Masahiko Sawada > wrote: > > > > > > On Thu, Dec 19, 2024 at 11:11 PM Nisha Moond > wrote: > > > > > > > > > > > > [3] Test with pgbench r

RE: Conflict detection for update_deleted in logical replication

2025-01-07 Thread Zhijie Hou (Fujitsu)
On Thursday, January 2, 2025 2:30 PM Amit Kapila wrote: > > Sounds reasonable but OTOH, all other places that create physical > slots (which we are doing here) don't use this trick. So, don't they > need similar reliability? I have not figured the reason for existing physical slots' handling, b

RE: Conflict detection for update_deleted in logical replication

2025-01-07 Thread Zhijie Hou (Fujitsu)
On Thursday, January 2, 2025 6:34 PM vignesh C wrote: > > Few suggestions: > 1) If we have a subscription with detect_update_deleted option and we > try to upgrade it with default settings(in case dba forgot to set > track_commit_timestamp), the upgrade will fail after doing a lot of > steps like

RE: Conflict detection for update_deleted in logical replication

2025-01-07 Thread Zhijie Hou (Fujitsu)
On Tuesday, January 7, 2025 3:05 PM Masahiko Sawada wrote: Hi, > On Mon, Jan 6, 2025 at 10:40 PM Zhijie Hou (Fujitsu) > wrote: > > > > On Tuesday, January 7, 2025 2:00 PM Masahiko Sawada > wrote: > > > > Hi, > > > > > > > > On Mon, J

  1   2   3   4   5   >