Re: Add memory_limit_hits to pg_stat_replication_slots

2025-09-22 Thread shveta malik
On Mon, Sep 22, 2025 at 4:35 PM Amit Kapila wrote: > > On Mon, Sep 22, 2025 at 1:41 PM Bertrand Drouvot > wrote: > > > > > > > > Since other statistics counter names are camel cases I think it's > > > better to follow that for the new counter. > > > > Makes sense, done with memoryLimitHits in v2

Re: Logical Replication of sequences

2025-09-22 Thread shveta malik
6) I tried to test the patch. When sequences are more than the batch-size, but not enough to make a complete batch of max size, I get this error: LOG: logical replication sequence synchronization for subscription "sub1" - total unsynchronized: 118 LOG: logical replication sequence synchronizatio

Re: Logical Replication of sequences

2025-09-21 Thread shveta malik
On Thu, Sep 18, 2025 at 4:07 PM vignesh C wrote: > > Thanks for the comments, these are handled in the attached patch. > Please find a few comments: patch005: 1) GetSubscriptionRelations: + /* Skip sequences if they were not requested */ + if (!get_sequences && (relkind == RELKIND_SEQUENCE)) +

Re: Clear logical slot's 'synced' flag on promotion of standby

2025-09-21 Thread shveta malik
On Fri, Sep 19, 2025 at 7:29 PM Ashutosh Sharma wrote: > > On Fri, Sep 19, 2025 at 3:04 PM shveta malik wrote: > > > > On Thu, Sep 18, 2025 at 5:20 PM Ashutosh Sharma > > wrote: > > > > > > Hi Ajin, > > > > > > On Thu, Sep 18, 2025 at

Re: Clear logical slot's 'synced' flag on promotion of standby

2025-09-19 Thread shveta malik
On Thu, Sep 18, 2025 at 4:16 PM Ajin Cherian wrote: > > On Fri, Sep 12, 2025 at 1:56 PM shveta malik wrote: > > > > The approach seems valid and should work, but introducing a new file > > like promote.inprogress for this purpose might be excessive. We can > >

Re: Report bytes and transactions actually sent downtream

2025-09-19 Thread shveta malik
On Thu, Sep 18, 2025 at 3:54 PM Ashutosh Bapat wrote: > > > > > > Few comments: > > > > 1) > > postgres=# select slot_name, > > total_bytes,plugin_filtered_bytes,plugin_sent_bytes from > > pg_stat_replication_slots order by slot_name; > > slot_name | total_bytes | plugin_filtered_bytes | plugin_

Re: Clear logical slot's 'synced' flag on promotion of standby

2025-09-19 Thread shveta malik
On Thu, Sep 18, 2025 at 5:20 PM Ashutosh Sharma wrote: > > Hi Ajin, > > On Thu, Sep 18, 2025 at 4:16 PM Ajin Cherian wrote: > > > > On Fri, Sep 12, 2025 at 1:56 PM shveta malik wrote: > > > > > > The approach seems valid and should work, but introducin

Re: Report bytes and transactions actually sent downtream

2025-09-17 Thread shveta malik
On Wed, Aug 27, 2025 at 7:14 PM Bertrand Drouvot wrote: > > Hi, > > On Thu, Jul 24, 2025 at 12:24:26PM +0530, Ashutosh Bapat wrote: > > Here's the next patch which considers all the discussion so far. It > > adds four fields to pg_stat_replication_slots. > > - plugin - name of the output plugi

Re: Issue with logical replication slot during switchover

2025-09-17 Thread shveta malik
On Wed, Aug 13, 2025 at 11:34 AM shveta malik wrote: > > On Fri, Aug 8, 2025 at 7:01 PM Fabrice Chapuis > wrote: > > > Another problem as you pointed out is inconsistent behaviour across > switchovers. On the first switchover, we get the error on new standby:

Re: Improve pg_sync_replication_slots() to wait for primary to advance

2025-09-17 Thread shveta malik
On Tue, Sep 16, 2025 at 5:12 PM Ashutosh Sharma wrote: > > > On Tue, Sep 16, 2025 at 11:53 AM shveta malik wrote: > > > > On Mon, Sep 15, 2025 at 6:17 PM Ajin Cherian wrote: > > > > > > On Wed, Sep 10, 2025 at 2:45 PM shveta malik > > > wrote

Re: Logical Replication of sequences

2025-09-16 Thread shveta malik
Few comments: 1) The message of patch001 says: When a sequence is synchronized to the subscriber, the page LSN of the sequence from the publisher is also captured and stored in pg_subscription_rel.srsublsn. This LSN will reflect the state of the sequence at the time of synchronization. By com

Re: Logical Replication of sequences

2025-09-16 Thread shveta malik
On Mon, Sep 15, 2025 at 2:36 PM vignesh C wrote: > > > This is addressed in the attached patch, also rebased the patch > because of recent commits. > One of the patches conflict with recent commit 0d48d393 (launcher.c changes) and thus needs a rebase. thanks Shveta

Re: Improve pg_sync_replication_slots() to wait for primary to advance

2025-09-15 Thread shveta malik
On Mon, Sep 15, 2025 at 6:17 PM Ajin Cherian wrote: > > On Wed, Sep 10, 2025 at 2:45 PM shveta malik wrote: > > > > On Tue, Sep 9, 2025 at 5:37 PM Ajin Cherian wrote: > > > > > > Attached v11 patch addressing the above comments. > > >

Re: Conflict detection for update_deleted in logical replication

2025-09-15 Thread shveta malik
On Tue, Sep 16, 2025 at 10:08 AM Zhijie Hou (Fujitsu) wrote: > > On Tuesday, September 16, 2025 11:54 AM Zhijie Hou (Fujitsu) > wrote: > > > > On Monday, September 15, 2025 8:11 PM Amit Kapila > > wrote: > > > > > > On Mon, Sep 15, 2025 at 1:07 PM Zhijie Hou (Fujitsu) > > > wrote: > > > > > >

Re: Conflict detection for update_deleted in logical replication

2025-09-14 Thread shveta malik
One concern: if (should_stop_conflict_info_retention(rdt_data)) + { + /* + * Stop retention if not yet. Otherwise, reset to the initial phase to + * retry all phases. This is required to recalculate the current wait + * time and resume retention if the time falls within + * max_retention_duratio

Re: Conflict detection for update_deleted in logical replication

2025-09-12 Thread shveta malik
On Fri, Sep 12, 2025 at 8:55 AM Zhijie Hou (Fujitsu) wrote: > > > I agree. Here is a V73 patch that will restart the worker if the retention > resumes. I also addressed other comments posted by Amit[1]. > Thanks for the patch. Few comments: 1) There is a small window where worker can exit while

Re: Clear logical slot's 'synced' flag on promotion of standby

2025-09-11 Thread shveta malik
On Thu, Sep 11, 2025 at 7:29 PM Ashutosh Sharma wrote: > > On Thu, Sep 11, 2025 at 9:17 AM shveta malik wrote: > > > > On Tue, Sep 9, 2025 at 2:19 PM Ashutosh Sharma > > wrote: > > > > > > Hi, > > > > > > > > > + *

Re: Clear logical slot's 'synced' flag on promotion of standby

2025-09-11 Thread shveta malik
On Thu, Sep 11, 2025 at 3:16 PM Ashutosh Sharma wrote: > > Hi, > > > We already read the recovery signal files (standby.signal or > recovery.signal) at the start of StartupXLOG() via InitWalRecovery(), > which sets the StandbyModeRequested flag. Couldn’t we use this to > distinguish whether the se

Re: Conflict detection for update_deleted in logical replication

2025-09-11 Thread shveta malik
On Thu, Sep 11, 2025 at 2:29 PM Zhijie Hou (Fujitsu) wrote: > > On Monday, September 8, 2025 7:21 PM Zhijie Hou (Fujitsu) > wrote: > > > > On Monday, September 8, 2025 3:13 PM Amit Kapila > > wrote: > > > > > > On Fri, Sep 5, 2025 at 5:03 PM Zhijie Hou (Fujitsu) > > > > > > wrote: > > > > > >

Re: Issue with logical replication slot during switchover

2025-09-10 Thread shveta malik
On Sun, Sep 7, 2025 at 1:32 PM Fabrice Chapuis wrote: > > Hi, > Here the version 1 of the patch with the modifications. > I do a git diff --check until there are no more errors. > In a next version of the patch, I think I have make change to call > ReplicationSlotCreate() function with the value

Re: Clear logical slot's 'synced' flag on promotion of standby

2025-09-10 Thread shveta malik
On Tue, Sep 9, 2025 at 2:19 PM Ashutosh Sharma wrote: > > Hi, > > > + * required resources. Clear any leftover 'synced' flags on replication > + * slots when in crash recovery on the primary. The DB_IN_CRASH_RECOVERY > + * state check ensures that this code is only reached when a standby > + * ser

Re: Clear logical slot's 'synced' flag on promotion of standby

2025-09-10 Thread shveta malik
On Wed, Sep 10, 2025 at 5:23 AM Masahiko Sawada wrote: > > On Mon, Sep 8, 2025 at 11:21 PM shveta malik wrote: > > > > Hi, > > > > This is a spin-off thread from [1]. > > > > Currently, in the slot-sync worker, we have an error scenario [2] > > wh

Re: Improve pg_sync_replication_slots() to wait for primary to advance

2025-09-09 Thread shveta malik
On Tue, Sep 9, 2025 at 5:37 PM Ajin Cherian wrote: > > Attached v11 patch addressing the above comments. > Please find a few comments: 1) + Retry is done after 2 + * sec wait. Exits early if promotion is triggered or certain critical We can say: Retry is done after SLOTSYNC_API_NAPTIME_MS wait

Re: Conflict detection for update_deleted in logical replication

2025-09-09 Thread shveta malik
On Tue, Sep 9, 2025 at 11:47 AM Zhijie Hou (Fujitsu) wrote: > > > Here is V71 patch set which addressed above comments and [1]. > Thank You for the patches. Please find a few comments on 001: 1) In compute_min_nonremovable_xid, when 'wait_for_xid' is true, we should have Assert(!worker->oldest_n

Clear logical slot's 'synced' flag on promotion of standby

2025-09-08 Thread shveta malik
Hi, This is a spin-off thread from [1]. Currently, in the slot-sync worker, we have an error scenario [2] where, during slot synchronization, if we detect a slot with the same name and its synced flag is set to false, we emit an error. The rationale is to avoid potentially overwriting a user-crea

Re: Conflict detection for update_deleted in logical replication

2025-09-08 Thread shveta malik
On Mon, Sep 8, 2025 at 3:06 AM Tom Lane wrote: > > Coverity is not happy with commit a850be2fe: > > /srv/coverity/git/pgsql-git/postgresql/src/backend/replication/logical/worker.c: > 3276 in FindDeletedTupleInLocalRel() > 3270 * maybe_advance_nonremovable_xid() for

Re: Improve pg_sync_replication_slots() to wait for primary to advance

2025-09-07 Thread shveta malik
On Fri, Sep 5, 2025 at 6:51 PM Ashutosh Bapat wrote: > > On Wed, Sep 3, 2025 at 11:58 AM Ajin Cherian wrote: > > > > > > On Fri, Aug 29, 2025 at 6:50 PM Ashutosh Bapat > > wrote: > > > > > > On Fri, Aug 29, 2025 at 11:42 AM Ajin Cherian wrote: > > > > > +/* > > > + * Flag used by pg_sync_replic

Re: Improve pg_sync_replication_slots() to wait for primary to advance

2025-09-06 Thread shveta malik
On Wed, Sep 3, 2025 at 3:19 PM Ajin Cherian wrote: > > On Wed, Sep 3, 2025 at 6:47 PM shveta malik wrote: > > > > On Wed, Sep 3, 2025 at 11:58 AM Ajin Cherian wrote: > > > > > > Attaching v10 with the above changes. > > > > > > >

Re: Conflict detection for update_deleted in logical replication

2025-09-04 Thread shveta malik
On Thu, Sep 4, 2025 at 3:30 PM Zhijie Hou (Fujitsu) wrote: > > Hi, > > As reported by Robert[1], it is worth adding a test for the race condition in > the RecordTransactionCommitPrepared() function to reduce the risk of future > code > changes: > > /* > * Note it is important to

Re: Improve pg_sync_replication_slots() to wait for primary to advance

2025-09-03 Thread shveta malik
On Wed, Sep 3, 2025 at 11:58 AM Ajin Cherian wrote: > > Attaching v10 with the above changes. > The patch does not apply on HEAD. Can you please rebase? thanks Shveta

Re: Conflict detection for update_deleted in logical replication

2025-09-02 Thread shveta malik
On Tue, Sep 2, 2025 at 3:30 PM shveta malik wrote: > > > > > > > > > Here is V70 patch set. > > > > > Please find a few comments on v70-003: 1) Doc of dead_tuple_retention_active says: True if retain_dead_tuples is enabled and the retention duration for

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-09-02 Thread shveta malik
Few trivial comments for doc:

Re: Conflict detection for update_deleted in logical replication

2025-09-02 Thread shveta malik
On Mon, Sep 1, 2025 at 5:45 PM shveta malik wrote: > > > > > Here is V70 patch set. > > > > The patch v70-0001 looks good to me. Verified, all the old issues are > resolved. > > Will resume review of v70-0002 now. > Please find a few comments on v70

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-09-01 Thread shveta malik
On Tue, Sep 2, 2025 at 10:24 AM shveta malik wrote: > > Few trivial comments for doc: Sorry, the email got sent without comments. 1) +It is important to note that when wal_level is set to +replica the effective WAL level can automatically change comma after replica missi

Re: Conflict detection for update_deleted in logical replication

2025-09-01 Thread shveta malik
On Mon, Sep 1, 2025 at 5:07 PM Zhijie Hou (Fujitsu) wrote: > > > I reviewed the patch internally and tweaked a small detail of the apply worker > to reduce the waiting time in the main loop when max_retention_duration is > defined (set wait_time = min(wait_time, max_retention_duration)). Also, I

Re: Issue with logical replication slot during switchover

2025-08-31 Thread shveta malik
On Sat, Aug 30, 2025 at 11:43 AM Shlok Kyal wrote: > > Hi Fabrice, > > Thanks for providing the patch. I reviewed your patch and have > following comment: > > 1. I think we should add a commit message in the patch. It would help > to give an understanding of the patch. > > 2. I tried applying patc

Re: Improve pg_sync_replication_slots() to wait for primary to advance

2025-08-31 Thread shveta malik
On Fri, Aug 29, 2025 at 4:14 PM Ashutosh Bapat wrote: > > On Fri, Aug 29, 2025 at 2:37 PM shveta malik wrote: > > > > On Fri, Aug 29, 2025 at 2:20 PM Ashutosh Bapat > > wrote: > > > > > > @@ -1276,7 +1331,7 @@ wait_for_slot_activity(bool some_slot_upd

Re: Conflict detection for update_deleted in logical replication

2025-08-29 Thread shveta malik
On Fri, Aug 29, 2025 at 11:49 AM Zhijie Hou (Fujitsu) wrote: > > Here is the new version patch set which also addressed Shveta's comments[1]. > Thanks for the patch. On 001 alone, I’m observing a behavior where, if sub1 has stopped retention, and I then create a new subscription sub2, the worker

Re: Improve pg_sync_replication_slots() to wait for primary to advance

2025-08-29 Thread shveta malik
On Fri, Aug 29, 2025 at 2:20 PM Ashutosh Bapat wrote: > > On Fri, Aug 29, 2025 at 11:42 AM Ajin Cherian wrote: > > > > On Fri, Aug 29, 2025 at 3:01 PM shveta malik wrote: > > > > > > On Tue, Aug 26, 2025 at 9:58 AM Ajin Cherian wrote: > > >

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-08-29 Thread shveta malik
On Fri, Aug 29, 2025 at 9:59 AM Masahiko Sawada wrote: > > On Thu, Aug 28, 2025 at 4:29 AM shveta malik wrote: > > > > On Wed, Aug 27, 2025 at 12:12 PM Masahiko Sawada > > wrote: > > > > > > On Tue, Aug 26, 2025 at 2:32 AM shveta malik > > >

Re: Improve pg_sync_replication_slots() to wait for primary to advance

2025-08-28 Thread shveta malik
On Tue, Aug 26, 2025 at 9:58 AM Ajin Cherian wrote: > > On Fri, Aug 22, 2025 at 3:44 PM shveta malik wrote: > > > > On Wed, Aug 20, 2025 at 10:53 AM Ajin Cherian wrote: > > > > > > > > > I've removed them. > > > Attaching patch v8

Re: Issue with logical replication slot during switchover

2025-08-28 Thread shveta malik
On Thu, Aug 28, 2025 at 9:11 PM Fabrice Chapuis wrote: > > What is the procedure to create this patch. Thank you for your help. > We use 'git format-patch' to generate formatted patches. I have given a few links ([1],[2],[3)] on know-how. I usually use these steps: git add git add git c

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-08-28 Thread shveta malik
On Wed, Aug 27, 2025 at 12:12 PM Masahiko Sawada wrote: > > On Tue, Aug 26, 2025 at 2:32 AM shveta malik wrote: > > > > On Tue, Aug 26, 2025 at 12:54 AM Masahiko Sawada > > wrote: > > > > > > I've attached the updated patch that incorporated t

Re: Conflict detection for update_deleted in logical replication

2025-08-28 Thread shveta malik
On Thu, Aug 28, 2025 at 8:02 AM Zhijie Hou (Fujitsu) wrote: > > > I noticed that Cfbot failed to compile the document due to a typo after > renaming > the subscription option. Here are the updated V67 patches to fix that, only > the doc > in 0001 is modified. > Please find a few comments: pat

Re: Issue with logical replication slot during switchover

2025-08-28 Thread shveta malik
On Wed, Aug 27, 2025 at 7:55 PM Fabrice Chapuis wrote: > > For the first step (a), the pg_create_logical_replication_slot interface is > extended. > The slot on the new attached standby will be dropped and recreated if the > flag allow_overwrite is set to true. > I tested the modified source, co

Re: Avoid retaining conflict-related data when no tables are subscribed

2025-08-27 Thread shveta malik
On Thu, Aug 28, 2025 at 7:54 AM Zhijie Hou (Fujitsu) wrote: > > Hi, > > My colleague Nisha reported an issue to me off-list: dead tuples can't > be removed when retain_dead_tuples is enabled for a subscription with no > tables. > > This appears to stem from the inability to advance the non-remova

Re: Conflict detection for update_deleted in logical replication

2025-08-26 Thread shveta malik
Please find some more comments: 1) In CheckSubDeadTupleRetention(), shall we have below instead of retain_dead_tuples check in all conditions? if (retain_dead_tuples) guc checks (wal_level and tracl_commit) else max retention check 2) Currently stop and resume messages are: ~~ LOG: logic

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-08-26 Thread shveta malik
On Tue, Aug 26, 2025 at 12:54 AM Masahiko Sawada wrote: > > I've attached the updated patch that incorporated the comments and is > rebased to the current HEAD. > Thanks for the patch, please find a few comments concerning LOG messages: 1) slotsync worker gives LOG: LOG: replication slot synchr

Re: Logical Replication of sequences

2025-08-25 Thread shveta malik
On Thu, Aug 21, 2025 at 10:08 PM vignesh C wrote: > > I have also addressed all the comments from [1] in the attached > v20250823 version patch. > [1] - > https://www.postgresql.org/message-id/CAA4eK1%2BoVQW8oP%3DLo1X8qac6dzg-fgGQ6R_F_psfokUEqe%2Ba6w%40mail.gmail.com > Thank You for the patches

Re: Conflict detection for update_deleted in logical replication

2025-08-25 Thread shveta malik
On Mon, Aug 25, 2025 at 12:09 PM shveta malik wrote: > > On Mon, Aug 25, 2025 at 10:06 AM Zhijie Hou (Fujitsu) > wrote: > > > > Attach the V65 patch set which addressed above and > > Shveta's comments[1]. > > > > Thank You for the patches, please

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-08-24 Thread shveta malik
On Sat, Aug 23, 2025 at 3:51 AM Masahiko Sawada wrote: > > > I found that we don't need to expose LogicalDecodingCtlData in > logicalctl.h header file. I've updated some cosmetic changes including > that point. > > I think the patch is getting pretty good shape Yes, I agree. > and am aiming at >

Re: Conflict detection for update_deleted in logical replication

2025-08-24 Thread shveta malik
On Mon, Aug 25, 2025 at 10:06 AM Zhijie Hou (Fujitsu) wrote: > > Attach the V65 patch set which addressed above and > Shveta's comments[1]. > Thank You for the patches, please find a few comments on v64 itself (I think valid on v65 as well): 1) in resume_conflict_info_retention(), shall we rewri

Re: Conflict detection for update_deleted in logical replication

2025-08-22 Thread shveta malik
On Fri, Aug 22, 2025 at 5:10 PM Amit Kapila wrote: > > On Mon, Aug 4, 2025 at 3:11 PM Amit Kapila wrote: > > > > On Mon, Aug 4, 2025 at 11:46 AM shveta malik wrote: > > > > > > 7) > > > Shall we rename 'max_conflict_retention_duration'

Re: Improve pg_sync_replication_slots() to wait for primary to advance

2025-08-21 Thread shveta malik
On Wed, Aug 20, 2025 at 10:53 AM Ajin Cherian wrote: > > > I've removed them. > Attaching patch v8 addressing the above comments. > Thanks for the patch. Please find a few comments: 1) When the API is in progress, and meanwhile in another session we turn off hot_standby_feedback, the API session

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-08-21 Thread shveta malik
On Thu, Aug 21, 2025 at 10:34 PM Masahiko Sawada wrote: > > On Wed, Aug 20, 2025 at 3:11 AM shveta malik wrote: > > > > Please find a few comments: > > Thank you for reviewing the patch! > > > > > 1) > > ReplicationSlotsDropDBSlots: > > + bool

Re: Conflict detection for update_deleted in logical replication

2025-08-21 Thread shveta malik
On Thu, Aug 21, 2025 at 2:09 PM Zhijie Hou (Fujitsu) wrote: > > On Thursday, August 21, 2025 2:01 PM shveta malik > wrote: > > > > On Wed, Aug 20, 2025 at 12:12 PM Zhijie Hou (Fujitsu) > > wrote: > > > > > > > > > I agree. Here is V63 versi

Re: Logical Replication of sequences

2025-08-20 Thread shveta malik
On Wed, Aug 20, 2025 at 2:25 PM vignesh C wrote: > > On Tue, 19 Aug 2025 at 23:33, Masahiko Sawada wrote: > > > > I imagined something like case 2. For logical replication of tables, > > if we support DDL replication (i.e., CREATE/ALTER/DROP TABLE), all > > changes the apply worker executes are s

Re: Conflict detection for update_deleted in logical replication

2025-08-20 Thread shveta malik
On Wed, Aug 20, 2025 at 12:12 PM Zhijie Hou (Fujitsu) wrote: > > > I agree. Here is V63 version which implements this approach. > Thank You for the patches. > The retention status is recorded in the pg_subscription catalog > (subretentionactive) to prevent unnecessary retention initiation upon s

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-08-20 Thread shveta malik
Please find a few comments: 1) ReplicationSlotsDropDBSlots: + bool dropped = false; We can name 'dropped ' as 'dropped_logical' similar to ReplicationSlotCleanup. 2) ReplicationSlotsDropDBSlots() + + if (dropped && nlogicalslots == 0) + DisableLogicalDecodingIfNecessary(); I could not understan

Re: Improve pg_sync_replication_slots() to wait for primary to advance

2025-08-19 Thread shveta malik
On Tue, Aug 19, 2025 at 10:55 AM Ajin Cherian wrote: > > Attaching patch v7 addressing all the above comments. > Thank You for the patches. Please find a few comments: 1) We are not resetting 'slot_persistence_pending' to false anywhere. So once it hits the flow which sets it to true, it will ne

Re: Conflict detection for update_deleted in logical replication

2025-08-14 Thread shveta malik
Few more comments: 1) src/sgml/monitoring.sgml: + + True if retain_dead_tuples + is enabled and the duration for which information useful for conflict + detection is retained by this apply worker does not exceed + max_conflict_retention_duration; NULL for + par

Re: Improve pg_sync_replication_slots() to wait for primary to advance

2025-08-13 Thread shveta malik
On Thu, Aug 14, 2025 at 7:28 AM Ajin Cherian wrote: > > > Patch v6 attached. > Thanks Ajin. Please find my comments: 1) SyncReplicationSlots: + remote_slots = fetch_or_refresh_remote_slots(wrconn, NIL); + + /* Retry until all slots are sync ready atleast */ + for (;;) + { + bool some_slot_update

Re: Issue with logical replication slot during switchover

2025-08-13 Thread shveta malik
On Wed, Aug 13, 2025 at 9:29 PM Fabrice Chapuis wrote: > > Thanks for sharing this. In fact, I agree, introducing an allow_overwrite > slot property makes seems cleaner than a GUC for this specific use case. > > a) At first, an extension of pg_create_logical_replication_slot() could be > propos

Re: Conflict detection for update_deleted in logical replication

2025-08-13 Thread shveta malik
On Wed, Aug 13, 2025 at 4:15 PM shveta malik wrote: > > On Wed, Aug 13, 2025 at 10:41 AM Zhijie Hou (Fujitsu) > wrote: > > > > > > Here is the V61 patch set which addressed above comments and the comment by > > Nisha[2]. > > > > Thank You for the

Re: Conflict detection for update_deleted in logical replication

2025-08-13 Thread shveta malik
On Wed, Aug 13, 2025 at 5:42 PM Amit Kapila wrote: > > > > > 4) > > For the DETAIL part of resume and stop messages, how about these: > > > > The retention duration for information used in conflict detection has > > exceeded the limit of xx. > > The retention duration for information used in confl

Re: Conflict detection for update_deleted in logical replication

2025-08-13 Thread shveta malik
On Wed, Aug 13, 2025 at 10:41 AM Zhijie Hou (Fujitsu) wrote: > > > Here is the V61 patch set which addressed above comments and the comment by > Nisha[2]. > Thank You for the patch. I tested the patch, please find a few comments: 1) Now when it stops-retention and later resumes it due to the fa

Re: Issue with logical replication slot during switchover

2025-08-12 Thread shveta malik
On Fri, Aug 8, 2025 at 7:01 PM Fabrice Chapuis wrote: > > Thanks Shveta for coming on this point again and fixing the link. > The idea is to check if the slot has same name to try to resynchronize it > with the primary. > ok the check on the failover status for the remote slot is perhaps redundan

Re: Improve pg_sync_replication_slots() to wait for primary to advance

2025-08-12 Thread shveta malik
On Mon, Aug 11, 2025 at 1:37 PM Ajin Cherian wrote: > > On Fri, Aug 8, 2025 at 11:22 PM Ashutosh Bapat > wrote: > > > > > > There's also a minor merge conflict because func.sgml is not split > > into multiple files. > > > > Yes, I fixed this. > Thanks for the patch. Please find a few comments:

Re: Conflict detection for update_deleted in logical replication

2025-08-12 Thread shveta malik
025 at 2:21 PM Amit Kapila > > > wrote: > > > > > > > > On Tue, Aug 12, 2025 at 2:06 PM shveta malik > > > wrote: > > > > > > > > > > 2) > > > > > postgres=# create subscription sub2 connection 'dbname

Re: Conflict detection for update_deleted in logical replication

2025-08-12 Thread shveta malik
On Mon, Aug 11, 2025 at 2:40 PM Zhijie Hou (Fujitsu) wrote: > > I agree. So, following the above points and some off-list discussions, I have > revised the option to be a subscription option in the V60 version. > Thank You for the patches. Tried to test the new sub-level parameter, have few comme

Re: Excessive LOG messages from replication slot sync worker

2025-08-11 Thread shveta malik
On Tue, Aug 12, 2025 at 8:52 AM Fujii Masao wrote: > > Hi, > > While testing the replication slot synchronization feature, I set up > three instances on the same machine: > > - Physical replication primary (also Logical replication publisher) > - Physical replication standby > - Logical replicatio

Re: Proposal: Conflict log history table for Logical Replication

2025-08-08 Thread shveta malik
On Fri, Aug 8, 2025 at 10:01 AM Dilip Kumar wrote: > > On Fri, Aug 8, 2025 at 8:58 AM shveta malik wrote: > > > > On Thu, Aug 7, 2025 at 3:08 PM Dilip Kumar wrote: > > > > > > So logically for PostgreSQL its an > > > user table but yeah

Re: Conflict detection for update_deleted in logical replication

2025-08-07 Thread shveta malik
On Thu, Aug 7, 2025 at 10:10 AM Zhijie Hou (Fujitsu) wrote: > > On Tuesday, August 5, 2025 10:09 AM Zhijie Hou (Fujitsu) > wrote: > > Here is V57 patch set which addressed most of comments. > > > > In this version, I also fixed a bug that the apply worker continued to find > > dead > > tuples e

Re: Issue with logical replication slot during switchover

2025-08-07 Thread shveta malik
On Thu, Aug 7, 2025 at 6:50 PM Fabrice Chapuis wrote: > > Hi, > > An issue occurred during the initial switchover using PostgreSQL version > 17.5. The setup consists of a cluster with two nodes, managed by Patroni > version 4.0.5. > Logical replication is configured on the same instance, and the

Re: Proposal: Conflict log history table for Logical Replication

2025-08-07 Thread shveta malik
On Thu, Aug 7, 2025 at 3:08 PM Dilip Kumar wrote: > > On Thu, Aug 7, 2025 at 1:43 PM shveta malik wrote: > > > > On Thu, Aug 7, 2025 at 12:25 PM shveta malik wrote: > > Thanks Shveta for your opinion on the design. > > > > On Tue, Aug 5

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-08-07 Thread shveta malik
On Fri, Aug 8, 2025 at 3:30 AM Masahiko Sawada wrote: > > On Tue, Aug 5, 2025 at 11:23 PM shveta malik wrote: > > > > Please find a few comments on v6: > > > > 1) > > +/* > > + * Initialize logical decoding status on shmem at server startup. This > &

Re: Conflict detection for update_deleted in logical replication

2025-08-07 Thread shveta malik
On Thu, Aug 7, 2025 at 6:05 PM Dilip Kumar wrote: > > On Mon, Aug 4, 2025 at 3:11 PM Amit Kapila wrote: > > > > On Mon, Aug 4, 2025 at 11:46 AM shveta malik wrote: > > > > > > 7) > > > Shall we rename 'max_conflict_retention_duration'

Re: Proposal: Conflict log history table for Logical Replication

2025-08-07 Thread shveta malik
On Thu, Aug 7, 2025 at 12:25 PM shveta malik wrote: > > On Tue, Aug 5, 2025 at 5:54 PM Dilip Kumar wrote: > > > > Currently we log conflicts to the server's log file and updates, this > > approach has limitations, 1) Difficult to query and analyze, parsing > >

Re: Proposal: Conflict log history table for Logical Replication

2025-08-06 Thread shveta malik
On Tue, Aug 5, 2025 at 5:54 PM Dilip Kumar wrote: > > Currently we log conflicts to the server's log file and updates, this > approach has limitations, 1) Difficult to query and analyze, parsing > plain text log files for conflict details is inefficient. 2) Lack of > structured data, key conflict

Re: Logical Replication of sequences

2025-08-06 Thread shveta malik
On Wed, Aug 6, 2025 at 4:29 PM shveta malik wrote: > > On Wed, Aug 6, 2025 at 2:28 PM vignesh C wrote: > > > > The attached v20250806 version patch has the changes for the same. > > > > Thank You for the patches. Please find a few comments: > > 1) >

Re: Logical Replication of sequences

2025-08-06 Thread shveta malik
On Wed, Aug 6, 2025 at 2:28 PM vignesh C wrote: > > The attached v20250806 version patch has the changes for the same. > Thank You for the patches. Please find a few comments: 1) * If 'resync_all_sequences' is false: * Add or remove tables and sequences that have been added to or removed

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-08-05 Thread shveta malik
Please find a few comments on v6: 1) +/* + * Initialize logical decoding status on shmem at server startup. This + * must be called ONCE during postmaster or standalone-backend startup, + * before initializing replication slots. + */ +void +StartupLogicalDecodingStatus(bool last_status) The comme

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-08-05 Thread shveta malik
On Wed, Aug 6, 2025 at 6:18 AM Masahiko Sawada wrote: > > > I've attached the updated version patch. > Thank You for the patch. The patch does not apply to the latest head due to conflict with slot-sync fix (commit-Id: 4614d53d). thanks Shveta

Re: Improve pg_sync_replication_slots() to wait for primary to advance

2025-08-05 Thread shveta malik
On Wed, Aug 6, 2025 at 7:35 AM Ajin Cherian wrote: > > On Tue, Aug 5, 2025 at 4:22 PM Amit Kapila wrote: > > > > On Tue, Aug 5, 2025 at 9:28 AM shveta malik wrote: > > > > > > On Mon, Aug 4, 2025 at 3:41 PM Amit Kapila > > > wrote: > > >

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-08-05 Thread shveta malik
On Tue, Aug 5, 2025 at 5:14 AM Masahiko Sawada wrote: > > On Mon, Aug 4, 2025 at 3:38 AM shveta malik wrote: > > > > On Sat, Aug 2, 2025 at 4:53 AM Masahiko Sawada > > wrote: > > > > > > On Thu, Jul 31, 2025 at 5:00 AM Hayato Kuroda (Fujitsu) >

Re: Improve pg_sync_replication_slots() to wait for primary to advance

2025-08-04 Thread shveta malik
On Mon, Aug 4, 2025 at 3:41 PM Amit Kapila wrote: > > On Mon, Aug 4, 2025 at 12:19 PM shveta malik wrote: > > > > On Mon, Aug 4, 2025 at 11:31 AM Amit Kapila wrote: > > > > > > On Fri, Aug 1, 2025 at 2:50 PM shveta malik > > > wrote: > > >

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-08-04 Thread shveta malik
On Sat, Aug 2, 2025 at 4:53 AM Masahiko Sawada wrote: > > On Thu, Jul 31, 2025 at 5:00 AM Hayato Kuroda (Fujitsu) > wrote: > > > > Dear Sawada-san, > > > > > I thought we could fix this issue by checking the number of in-use > > > logical slots while holding ReplicationSlotControlLock and > > > L

Re: Improve pg_sync_replication_slots() to wait for primary to advance

2025-08-03 Thread shveta malik
On Mon, Aug 4, 2025 at 11:31 AM Amit Kapila wrote: > > On Fri, Aug 1, 2025 at 2:50 PM shveta malik wrote: > > > > 5) > > I tried a test where there were 4 slots on the publisher, where one > > was getting used while the others were not. Initiated > > pg_sync_

Re: Conflict detection for update_deleted in logical replication

2025-08-03 Thread shveta malik
On Fri, Aug 1, 2025 at 9:16 PM Zhijie Hou (Fujitsu) wrote: > > > Thanks for confirming. Here is V56 patch set which addressed all the > comments including the comments from Amit[1] and Shveta[2]. > > I have merged V55-0002 into 0001 and updated the list of author > and reviewers based on my knowle

Re: Improve pg_sync_replication_slots() to wait for primary to advance

2025-08-01 Thread shveta malik
On Fri, Aug 1, 2025 at 12:02 PM shveta malik wrote: > > On Thu, Jul 31, 2025 at 3:11 PM Ajin Cherian wrote: > > > > > > Patch v3 attached. > > > > Thanks for the patch. I tested it, please find a few comments: > > > 1) > it hit

Re: Improve pg_sync_replication_slots() to wait for primary to advance

2025-07-31 Thread shveta malik
On Thu, Jul 31, 2025 at 3:11 PM Ajin Cherian wrote: > > > Patch v3 attached. > Thanks for the patch. I tested it, please find a few comments: 1) it hits an assert (slotsync_reread_config()-->Assert(sync_replication_slots)) when API is trying to sync and is in wait loop while in another session,

Re: Conflict detection for update_deleted in logical replication

2025-07-31 Thread shveta malik
On Thu, Jul 31, 2025 at 3:49 PM Zhijie Hou (Fujitsu) wrote: > > On Thursday, July 31, 2025 5:26 PM shveta malik > wrote: > > > > On Tue, Jul 29, 2025 at 10:51 AM Zhijie Hou (Fujitsu) > > > > wrote: > > > > > > > > > This is th

Re: Conflict detection for update_deleted in logical replication

2025-07-31 Thread shveta malik
On Tue, Jul 29, 2025 at 10:51 AM Zhijie Hou (Fujitsu) wrote: > > > This is the V54 patch set, with only patch 0001 updated to address the latest > comments. > Thanks for the patch. While performing tests on the latest patch, I found an assert in tablesync worker in FindDeletedTupleInLocalRel (se

Re: Logical Replication of sequences

2025-07-30 Thread shveta malik
On Wed, Jul 30, 2025 at 11:16 AM shveta malik wrote: > > On Mon, Jul 28, 2025 at 3:37 PM vignesh C wrote: > > > Thanks for the comments, the attached v20250728 version patch has the > > changes for the same. > > > Thanks for the patches, please find a few comments:

Re: Logical Replication of sequences

2025-07-29 Thread shveta malik
On Mon, Jul 28, 2025 at 3:37 PM vignesh C wrote: > Thanks for the comments, the attached v20250728 version patch has the > changes for the same. > Thanks for the patches, please find a few comments: 1) WARNING: WITH clause parameters do not affect sequence synchronization a) How about: WITH cl

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-07-28 Thread shveta malik
On Fri, Jul 25, 2025 at 11:45 AM Masahiko Sawada wrote: > > Thank you for testing the patch! > > I've reworked the locking part in the patch. The attached v4 patch > should address all review comments including your previous > comments[1]. > Thank You for the patch. I have not reviewed fully, but

Re: Conflict detection for update_deleted in logical replication

2025-07-28 Thread shveta malik
On Mon, Jul 28, 2025 at 4:38 PM shveta malik wrote: > > On Fri, Jul 25, 2025 at 4:38 PM Zhijie Hou (Fujitsu) > wrote: > > > > > > The V53-0001 also includes Shveta's comments in [1]. > > > > Thanks, I have not yet completed the review, bu

Re: Conflict detection for update_deleted in logical replication

2025-07-28 Thread shveta malik
On Fri, Jul 25, 2025 at 4:38 PM Zhijie Hou (Fujitsu) wrote: > > > The V53-0001 also includes Shveta's comments in [1]. > Thanks, I have not yet completed the review, but please find a few comments on 001: 1) IsIndexUsableForFindingDeletedTuple() We first have: + /* + * A frozen transaction ID in

Re: Conflict detection for update_deleted in logical replication

2025-07-25 Thread shveta malik
On Fri, Jul 25, 2025 at 2:31 PM Amit Kapila wrote: > > On Fri, Jul 25, 2025 at 12:37 PM shveta malik wrote: > > > > On Thu, Jul 24, 2025 at 9:12 AM shveta malik wrote: > > > > > > > > > 2) > > >

Re: Logical replication launcher did not automatically restart when got SIGKILL

2025-07-25 Thread shveta malik
On Fri, Jul 25, 2025 at 7:17 AM Fujii Masao wrote: > > On Thu, Jul 24, 2025 at 6:46 PM shveta malik wrote: > > Sounds reasonable. > > Thinking out loud, when cleaning up after a backend or background > > worker crash, process_pm_child_exit() is invoked, which sub

  1   2   3   4   5   6   7   >