Re: How can end users know the cause of LR slot sync delays?

2025-09-22 Thread Amit Kapila
ORMAT_ARGS(remote_slot->confirmed_lsn), >remote_slot->name, >LSN_FORMAT_ARGS(latestFlushPtr))); > > return false; > } > > Slot sync skip can happen even for persistent slots. So why should we > avoid displaying the skip reason in such cases? > We should display the skip reason even for persistent slots and clear the same after a successful sync. -- With Regards, Amit Kapila.

Re: Add memory_limit_hits to pg_stat_replication_slots

2025-09-22 Thread Amit Kapila
ter slot_name in the view? Keeping it in the end after total_bytes seems out of place. -- With Regards, Amit Kapila.

Re: pgsql: Preserve conflict-relevant data during logical replication.

2025-09-21 Thread Amit Kapila
On Wed, Sep 3, 2025 at 5:10 PM Zhijie Hou (Fujitsu) wrote: > > On Tuesday, September 2, 2025 11:03 PM Robert Haas > wrote: > > > > On Wed, Jul 23, 2025 at 11:44 PM Amit Kapila > > wrote: > > > > If it is in fact important to acquire the commit timestamp a

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-09-20 Thread Amit Kapila
On Mon, Sep 8, 2025 at 11:14 PM Masahiko Sawada wrote: > > On Fri, Sep 5, 2025 at 8:50 PM Amit Kapila wrote: > > > > On Thu, Sep 4, 2025 at 1:24 AM Masahiko Sawada > > wrote: > > > > > > > > 2. > > > > - /* > > > &g

Re: Proposal: Conflict log history table for Logical Replication

2025-09-20 Thread Amit Kapila
On Thu, Sep 18, 2025 at 11:46 PM Masahiko Sawada wrote: > > On Thu, Sep 18, 2025 at 1:33 AM Amit Kapila wrote: > > > > If we compare conflict_history_table with the slot that gets created > > with subscription, one can say the same thing about slots. Users can >

Re: Parallel Apply

2025-09-19 Thread Amit Kapila
mply wait for a third transaction to be distributed or just apply it and process another change. If we follow the earlier then it is quite possible that the sender fills the network queue to send data and simply timed out. -- With Regards, Amit Kapila.

Re: Reword messages using "as" instead of "because"

2025-09-18 Thread Amit Kapila
On Fri, Sep 19, 2025 at 9:13 AM Tom Lane wrote: > > Amit Kapila writes: > > We have a similar message for stop retention. I feel it would be good > > to mention that as a reason, so users can increase it. I could think > > of two alternatives for stop message b

Re: Reword messages using "as" instead of "because"

2025-09-18 Thread Amit Kapila
On Thu, Sep 18, 2025 at 5:41 PM Tom Lane wrote: > > Amit Kapila writes: > > Yeah, this sounds clear but shall we consider using > > max_retention_duration like: "Retention is re-enabled because the > > apply process has caught up with the pu

Re: Proposal: Conflict log history table for Logical Replication

2025-09-18 Thread Amit Kapila
same thing about slots. Users can drop the slots and whole replication will stop. I think this table will be created with the same privileges as the owner of a subscription which can be either a superuser or a user with the privileges of the pg_create_subscription role, so we can rely on such users. -- With Regards, Amit Kapila.

Re: [Patch] add new parameter to pg_replication_origin_session_setup

2025-09-18 Thread Amit Kapila
w about checking this case before setting current_state as shown in attached. If we do that, we shouldn't even need new variables like current_state and initialized. Additionally, as shown in attached, it is better to make this a user-facing error by using ereport. 3. Merge all patches a

Re: How can end users know the cause of LR slot sync delays?

2025-09-17 Thread Amit Kapila
On Wed, Sep 17, 2025 at 8:19 PM Ashutosh Sharma wrote: > > On Wed, Sep 17, 2025 at 5:14 PM Amit Kapila wrote: > > > > On Wed, Sep 17, 2025 at 4:24 PM Hayato Kuroda (Fujitsu) > > wrote: > > > > > > Dear Shlok, > > > > > > Thanks for

Re: Allow logical replication in the same cluster

2025-09-17 Thread Amit Kapila
On Mon, Sep 8, 2025 at 8:20 AM Chao Li wrote: > > Hi Amit, > > Thanks for your info. > > On Sep 6, 2025, at 12:32, Amit Kapila wrote: > > You can avoid this problem by creating a slot first on publisher with > something like: > postgres=# select pg_create_logical_re

Re: Reword messages using "as" instead of "because"

2025-09-17 Thread Amit Kapila
On Thu, Sep 18, 2025 at 10:22 AM Tom Lane wrote: > > Amit Kapila writes: > > On Thu, Sep 18, 2025 at 8:56 AM Tom Lane wrote: > >> +1 for the first change, but for this: > >> > >> - ? errdetail("Retention is re-enabled as the appl

Re: Reword messages using "as" instead of "because"

2025-09-17 Thread Amit Kapila
On Thu, Sep 18, 2025 at 8:56 AM Tom Lane wrote: > > Amit Kapila writes: > > LGTM. Horiguchi-San, do let me know if you have suggestions here. I am > > planning to push this tomorrow. > > +1 for the first change, but for this: > > - ? errdeta

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-09-17 Thread Amit Kapila
On Wed, Sep 17, 2025 at 10:24 PM Masahiko Sawada wrote: > > On Wed, Sep 17, 2025 at 4:19 AM Amit Kapila wrote: > > > > On Tue, Sep 16, 2025 at 11:49 PM Masahiko Sawada > > wrote: > > > > > > On Tue, Sep 16, 2025 at 1:30 AM Amit Kapila > >

Re: Reword messages using "as" instead of "because"

2025-09-17 Thread Amit Kapila
me DETAIL > message and this message has now been updated. > LGTM. Horiguchi-San, do let me know if you have suggestions here. I am planning to push this tomorrow. -- With Regards, Amit Kapila.

Re: How can end users know the cause of LR slot sync delays?

2025-09-17 Thread Amit Kapila
ion than approach1. > I also think so but Ashutosh thought that it would be hacky. Ashutosh, did you have an opinion on this matter after seeing the patches? -- With Regards, Amit Kapila.

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-09-17 Thread Amit Kapila
On Tue, Sep 16, 2025 at 11:49 PM Masahiko Sawada wrote: > > On Tue, Sep 16, 2025 at 1:30 AM Amit Kapila wrote: > > > > When user is dropping a temporary slot, we should disable the > > decoding. The lazy behaviour should be for ERROR or session_exit > > cases.

Re: [Patch] add new parameter to pg_replication_origin_session_setup

2025-09-16 Thread Amit Kapila
On Tue, Sep 16, 2025 at 10:07 AM Hayato Kuroda (Fujitsu) wrote: > > Dear Doruk, > > Thanks for updating the patch and sorry for being late. > The new patch looks good to me. > Can we think of writing a few tests for this newly exposed functionality? -- With Regards, Amit Kapila.

Re: Parallel Apply

2025-09-16 Thread Amit Kapila
cases where we could let some pa complete its current ongoing transaction if it is independent of other transactions and has received all its changes. -- With Regards, Amit Kapila.

Re: Parallel Apply

2025-09-16 Thread Amit Kapila
> > • Stage 3: Execute changes > > • Stage 4: Commit and track progress > > > This creates a pipeline where Transaction A executes changes while > Transaction B analyzes dependencies > I don't know how to make this work in the current framework of apply. But feel free to propose this with some more details as to how it will work? -- With Regards, Amit Kapila.

Re: Conflict detection for update_deleted in logical replication

2025-09-16 Thread Amit Kapila
Thanks, I have modified the patch based on what you suggested and pushed. -- With Regards, Amit Kapila.

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-09-16 Thread Amit Kapila
On Mon, Sep 15, 2025 at 10:15 PM Masahiko Sawada wrote: > > On Sun, Sep 14, 2025 at 7:55 PM Amit Kapila wrote: > > > > On Fri, Sep 12, 2025 at 11:18 PM Masahiko Sawada > > wrote: > > > > > > On Thu, Sep 11, 2025 at 9:08 PM Amit Kapila > >

Re: Reword messages using "as" instead of "because"

2025-09-15 Thread Amit Kapila
ation of %u ms."? Similarly for the first message, how about: "Retention is stopped because the apply process could not advance its xmin within the configured max_retention_duration of %u ms."? -- With Regards, Amit Kapila.

Re: Conflict detection for update_deleted in logical replication

2025-09-15 Thread Amit Kapila
re if I have any new findings. [1]: https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=kestrel&dt=2025-09-15%2009%3A08%3A07&stg=subscription-check -- With Regards, Amit Kapila.

Re: Conflict detection for update_deleted in logical replication

2025-09-14 Thread Amit Kapila
t advancing its xmin within the configured max_retention_duration of %u ms." Apart from these, I have made some cosmetic changes in the attached. -- With Regards, Amit Kapila. diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c index 1d42c

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-09-14 Thread Amit Kapila
On Fri, Sep 12, 2025 at 11:18 PM Masahiko Sawada wrote: > > On Thu, Sep 11, 2025 at 9:08 PM Amit Kapila wrote: > > > > > For the shutdown sequence, can't we think of resetting effective_wal > > after a restart? > > Does it mean that effective_wal_level k

Re: How can end users know the cause of LR slot sync delays?

2025-09-12 Thread Amit Kapila
ng all the sync information together has merit but don't you think in this case the sync_skip_reason doesn't seem to be matching with the existing columns in pg_stat_replication_slots? -- With Regards, Amit Kapila.

Re: Conflict detection for update_deleted in logical replication

2025-09-12 Thread Amit Kapila
n't think it is worth optimizing for such a rare case at the cost of making code difficult to understand. Apart from this, I have changed a few comments in the attached. -- With Regards, Amit Kapila. diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/lo

Re: Proposal: Conflict log history table for Logical Replication

2025-09-11 Thread Amit Kapila
ut, I would like to hear opinions from others as > well here. > But OTOH, there could be users who want such a table to be dropped. One possibility is that if we user provided us a pre-created table then we leave it to user to remove the table, otherwise, we can remove with drop subscription. BTW, did we decide that we want a conflict-table-per-subscription or one table for all subscriptions, if later, then I guess the problem would be that it has to be a shared table across databases. -- With Regards, Amit Kapila.

Re: issue with synchronized_standby_slots

2025-09-11 Thread Amit Kapila
we should have the same solution for other similar parameters as well. As for the synchronized_standby_slots, we can follow the behavior similar to check_synchronous_standby_names and just give parsing ERRORs. Any non-existent slot related errors can be given when that parameter is later used. -- With Regards, Amit Kapila.

Re: Add support for specifying tables in pg_createsubscriber.

2025-09-11 Thread Amit Kapila
er than > (b). > Yeah, I am also not sure that it is worth adding a new option to save backward compatibility for this option. It seems intuitive to use existing publications when they exist, though we should clearly document it. -- With Regards, Amit Kapila.

Re: Conflict detection for update_deleted in logical replication

2025-09-11 Thread Amit Kapila
On Fri, Sep 5, 2025 at 5:03 PM Zhijie Hou (Fujitsu) wrote: > > Here are v2 patches which addressed above comments. > I have pushed the first patch. I find that the test can't reliably fail without a fix. Can you please investigate it? -- With Regards, Amit Kapila.

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-09-11 Thread Amit Kapila
On Thu, Sep 11, 2025 at 11:16 PM Masahiko Sawada wrote: > > On Wed, Sep 10, 2025 at 11:32 PM Amit Kapila wrote: > > > > On Sat, Sep 6, 2025 at 3:46 AM Masahiko Sawada > > wrote: > > > > > > I've attached the updated patch tha

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-09-10 Thread Amit Kapila
le/disable decoding for temporary logical slots? The temporary slots are released during ERROR or at session end, is that a good time to do the disable processing that even requires WAL writing. -- With Regards, Amit Kapila.

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-09-10 Thread Amit Kapila
then the patch's behavior will match with HEAD. But I think that may not be as straight-forward because standby may not have that information readily available. Anyway, if there is no simple way to change HEAD's behavior then we can leave that as it is. -- With Regards, Amit Kapila.

Re: issue with synchronized_standby_slots

2025-09-10 Thread Amit Kapila
On Thu, Sep 11, 2025 at 9:02 AM Amit Kapila wrote: > > On Wed, Sep 10, 2025 at 5:23 PM Alexander Kukushkin > wrote: > > > > On Wed, 10 Sept 2025 at 13:34, Shlok Kyal wrote: > >> > >> I think we should also add a parsing check for slot names specified in

Re: issue with synchronized_standby_slots

2025-09-10 Thread Amit Kapila
for this parameter? I am asking because if we change the current behavior, tomorrow, we can get complaints that one expects the old behaviour as that was similar to other GUCs like default_tablespace and default_table_access_method. -- With Regards, Amit Kapila.

Re: Proposal: Conflict log history table for Logical Replication

2025-09-10 Thread Amit Kapila
d but an insertion fails. > > I am currently working on a POC patch for the same, but will post that > > once we have some thoughts on design choices. > > How about streaming the conflicts in fixed format to a separate log > file other than regular postgres server log file? > I would prefer this info to be stored in tables as it would be easy to query them. If we use separate LOGs then we should provide some views to query the LOG. -- With Regards, Amit Kapila.

Re: Conflict detection for update_deleted in logical replication

2025-09-10 Thread Amit Kapila
On Wed, Sep 10, 2025 at 9:08 AM Zhijie Hou (Fujitsu) wrote: > > On Tuesday, September 9, 2025 7:01 PM Amit Kapila > wrote: > > On Tue, Sep 9, 2025 at 11:47 AM Zhijie Hou (Fujitsu) > > > > wrote: > > > > Few other comments: > > 1. > > + /*

Re: Conflict detection for update_deleted in logical replication

2025-09-09 Thread Amit Kapila
ume the rentention. + */ + if (!MySubscription->retentionactive) + { + rdt_data->phase = RDT_RESUME_CONFLICT_INFO_RETENTION; + process_rdt_phase_transition(rdt_data, false); + return; + } In this check, where do we check the time has come in acceptable range? Can you update comments to make it clear? -- With Regards, Amit Kapila.

Re: [PATCH] Accept connections post recovery without waiting for RemoveOldXlogFiles

2025-09-08 Thread Amit Kapila
dXlogFiles(). > Isn't it better to fix the reasons for WAL accumulation? Because even without recovery, this can fill up the disk. For example, one can use idle_replication_slot_timeout for inactive slots. Similarly, we can see what leads to slow PITR and try to avoid that. -- With Regards, Amit Kapila.

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-09-08 Thread Amit Kapila
On Mon, Sep 8, 2025 at 11:22 PM Masahiko Sawada wrote: > > On Fri, Sep 5, 2025 at 9:12 PM Amit Kapila wrote: > > > > On Sat, Sep 6, 2025 at 3:58 AM Masahiko Sawada > > wrote: > > > > > > On Tue, Sep 2, 2025 at 5:12 AM Shlok Kyal > > > wr

Re: How can end users know the cause of LR slot sync delays?

2025-09-06 Thread Amit Kapila
a patch for it. >> >> On Wed, 3 Sept 2025 at 17:52, Ashutosh Sharma wrote: >> > >> > Hi Amit, >> > >> > On Thu, Aug 28, 2025 at 3:26 PM Amit Kapila >> > wrote: >> >> >> >> On Thu, Aug 28, 2025 at 11:07 AM A

Re: [Patch] add new parameter to pg_replication_origin_session_setup

2025-09-05 Thread Amit Kapila
gt; attributes for clarity as it is the code declaration. > Then why didn't you specified PARALLEL UNSAFE as well? BTW, yesterday a new thread started with the same requirement [1]. It uses a slightly different way to define the new function. do you have any opinion on it? [1] - https://www.postgresql.org/message-id/CAE2gYzyTSNvHY1%2BiWUwykaLETSuAZsCWyryokjP6rG46ZvRgQA%40mail.gmail.com -- With Regards, Amit Kapila.

Re: Allow logical replication in the same cluster

2025-09-05 Thread Amit Kapila
--- (s1,0/01BFF178) (1 row) Then while creating subscription you can use the above created slot as follows: db1=# create subscription sub1 connection 'dbname=postgres' publication pub1 WITH(create_slot=false, slot_name='s1'); CREATE SUBSCRIPTION -- With Regards, Amit Kapila.

Re: Parallel Apply

2025-09-05 Thread Amit Kapila
On Fri, Sep 5, 2025 at 2:59 PM Dilip Kumar wrote: > > On Mon, Aug 11, 2025 at 10:16 AM Amit Kapila wrote: > > > > +1 for the idea. So I see we already have the parallel apply workers > for the large streaming transaction so I am trying to think what > additional probl

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-09-05 Thread Amit Kapila
e primary server with 'wal_level = > > replica', the slot sync worker can restart and connect to the primary > > but with patch it cannot start after restart due to the check in > > ValidateSlotSyncParams. > > But the slotsync worker is launched again once logical decoding is > enabled, no? I'm not sure that we want to launch the slotsync worker > also when we know logical decoding is not enabled. > Why in the first place the logical_decoding enabled check has failed because IIUC, the wal_level on standby is still 'logical'? -- With Regards, Amit Kapila.

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-09-05 Thread Amit Kapila
On Thu, Sep 4, 2025 at 1:24 AM Masahiko Sawada wrote: > > On Tue, Sep 2, 2025 at 4:35 AM Amit Kapila wrote: > > > > On Fri, Aug 29, 2025 at 9:38 AM Masahiko Sawada > > wrote: > > > > > > I've attached the updated patch. > > &

Re: Improve pg_sync_replication_slots() to wait for primary to advance

2025-09-05 Thread Amit Kapila
rstanding; please let me know if you see it differently. > Such a case can be addressed by having additional timeout parameters. We can do that as an additional patch if the use case is important enough to address. -- With Regards, Amit Kapila.

Re: Parallel Apply

2025-09-05 Thread Amit Kapila
On Fri, Sep 5, 2025 at 5:15 PM Mihail Nikalayeu wrote: > > Hello, Amit! > > Amit Kapila : > > So, in such cases as we won't be able to detect > > transaction dependencies, it would be better to allow out-of-order > > commits optionally. > > I think it is b

Re: Allow using replication origins in SQL level parallel sessions

2025-09-05 Thread Amit Kapila
ady discussing the same feature in an email thread [1]. Can you check that and share your inputs there? [1] - https://www.postgresql.org/message-id/CAMPB6wfe4zLjJL8jiZV5kjjpwBM2%3DrTRme0UCL7Ra4L8MTVdOg%40mail.gmail.com -- With Regards, Amit Kapila.

Re: Logical Replication of sequences

2025-09-05 Thread Amit Kapila
On Fri, Sep 5, 2025 at 2:46 PM vignesh C wrote: > > On Fri, 5 Sept 2025 at 04:01, Masahiko Sawada wrote: > > > > On Wed, Aug 20, 2025 at 4:57 AM Amit Kapila wrote: > > > > > > On Mon, Aug 18, 2025 at 3:36 PM vignesh C wrote: > > > > > > &g

Re: Avoid retaining conflict-related data when no tables are subscribed

2025-09-04 Thread Amit Kapila
On Tue, Sep 2, 2025 at 10:51 AM Zhijie Hou (Fujitsu) wrote: > > On Friday, August 29, 2025 12:05 PM Amit Kapila > wrote: > > > > bool > > -AllTablesyncsReady(void) > > +AllTablesyncsReady(bool ready_if_no_tables) > > > > This change serves the p

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-09-02 Thread Amit Kapila
e comments as to why we didn't support wal_level to be changed from minimal to logical? It will be helpful for future readers/authors to understand what it would require to further extend this functionality. -- With Regards, Amit Kapila.

Re: Fix use of variable after pfree

2025-09-02 Thread Amit Kapila
t and accepting TCP/IP connections? > > I have moved the pfree(app_data.name) after its usage. > > This change was introduced in PG_18. > The patch applies in the HEAD and REL_18_STABLE branches. > Thanks for the patch. It looks good to me. I'll take care of it. -- With Regards, Amit Kapila.

Re: How can end users know the cause of LR slot sync delays?

2025-08-28 Thread Amit Kapila
On Thu, Aug 28, 2025 at 3:29 PM Kirill Reshke wrote: > > On Thu, 28 Aug 2025 at 14:56, Amit Kapila wrote: > > > > On Thu, Aug 28, 2025 at 11:07 AM Ashutosh Sharma > > wrote: > > > > > > We have seen cases where slot synchronization gets delayed, for

Re: Avoid retaining conflict-related data when no tables are subscribed

2025-08-28 Thread Amit Kapila
e serves the purpose but I find it makes the API complex to understand because now it needs to make decisions based on different states depending on the boolean parameter passed. Can we introduce a new API for the empty subscription case? -- With Regards, Amit Kapila.

Re: Conflict detection for update_deleted in logical replication

2025-08-28 Thread Amit Kapila
On Thu, Aug 28, 2025 at 4:39 PM Amit Kapila wrote: > > On Thu, Aug 28, 2025 at 8:02 AM Zhijie Hou (Fujitsu) > wrote: > > > > I noticed that Cfbot failed to compile the document due to a typo after > > renaming > > the subscription option. Here are the upda

Re: Conflict detection for update_deleted in logical replication

2025-08-28 Thread Amit Kapila
a number of cosmetic and comment changes in the attached atop 0001 patch. Kindly include in next version, if you think they are good to include. -- With Regards, Amit Kapila. diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 29d0c9a6e45..3df1828

Re: How can end users know the cause of LR slot sync delays?

2025-08-28 Thread Amit Kapila
ad_remote_LSN. I think ideally users can compare standby's slot LSN/XMIN with remote_slot being synced. Do you have any better ideas? -- With Regards, Amit Kapila.

Re: [BUG?] check_exclusion_or_unique_constraint false negative

2025-08-26 Thread Amit Kapila
On Mon, Aug 25, 2025 at 7:02 PM Mihail Nikalayeu wrote: > > Amit Kapila : > > > > What if the new insert happens in a page prior to the current page? I > > mean that the scan won't encounter the page where Insert happens. > > Hmm Yes - if the TID la

Re: Conflict detection for update_deleted in logical replication

2025-08-25 Thread Amit Kapila
On Mon, Aug 25, 2025 at 5:05 PM Amit Kapila wrote: > > A few comments on 0001: > Some more comments: 1. + /* + * Return false if the leader apply worker has stopped retaining + * information for detecting conflicts. This implies that update_deleted + * can no longer be reliably

Re: [BUG?] check_exclusion_or_unique_constraint false negative

2025-08-25 Thread Amit Kapila
earby code unless I am missing something. > > The case of logical replication giving wrong results > > [0] is the behavior from the beginning of logical replication. > > Logical replication was mainly focused on replication without any > concurrent updates on the subscriber side. So, I think this is why the > issue was overlooked. > The other possibility is that as this is a rare scenario so we didn't consider it. -- With Regards, Amit Kapila.

Re: Conflict detection for update_deleted in logical replication

2025-08-25 Thread Amit Kapila
_info_retention). I don't see resume_conflict_info_retention in 0001, so I couldn't make sense of this part of the comment. -- With Regards, Amit Kapila.

Re: [BUG?] check_exclusion_or_unique_constraint false negative

2025-08-24 Thread Amit Kapila
s - this can occur during a single UPDATE, as well as a > DELETE followed by an INSERT of the same key within the same > transaction (which is effectively equivalent to an UPDATE). > BTW, then isn't it possible that INSERT happens on a different page? -- With Regards, Amit Kapila.

Re: [BUG?] check_exclusion_or_unique_constraint false negative

2025-08-24 Thread Amit Kapila
because of an unrelated title) then I suggest you start a separate email thread to discuss just that case and see what others think. [0]: https://www.postgresql.org/message-id/flat/CADzfLwWC49oanFSGPTf%3D6FJoTw-kAnpPZV8nVqAyR5KL68LrHQ%40mail.gmail.com#5f6b3be849f8d95c166decfae541df09 -- With Regards, Amit Kapila.

Re: Conflict detection for update_deleted in logical replication

2025-08-22 Thread Amit Kapila
On Mon, Aug 4, 2025 at 3:11 PM Amit Kapila wrote: > > On Mon, Aug 4, 2025 at 11:46 AM shveta malik wrote: > > > > 7) > > Shall we rename 'max_conflict_retention_duration' to > > 'max_conflict_info_retention_duration' as the latter one is more >

Re: Conflict detection for update_deleted in logical replication

2025-08-22 Thread Amit Kapila
nflict_retention(true); Can't we combine these checks by passing both parameters to CheckSubDeadTupleRetention() and let that function handle all inappropriate value cases? BTW, even for other places, see if you can reduce the length of the function name notify_ineffective_max_conflict_retention. -- With Regards, Amit Kapila.

Re: [BUG?] check_exclusion_or_unique_constraint false negative

2025-08-22 Thread Amit Kapila
strategy implemented, it is the subscriber operation that will win. So, the current behavior shouldn't cause any problem. -- With Regards, Amit Kapila.

Re: Logical Replication of sequences

2025-08-21 Thread Amit Kapila
On Thu, Aug 21, 2025 at 10:52 PM Masahiko Sawada wrote: > > On Wed, Aug 20, 2025 at 9:04 PM Amit Kapila wrote: > > > > On Wed, Aug 20, 2025 at 11:00 PM Masahiko Sawada > > wrote: > > > > > > On Tue, Aug 19, 2025 at 9:14 PM Amit Kapila > > >

Re: memory leak in logical WAL sender with pgoutput's cachectx

2025-08-21 Thread Amit Kapila
ehow we can decode the DROP TABLE WAL record, say delete of relid from pg_class then we can use that to remove the corresponding entry from RelationSyncCache. -- With Regards, Amit Kapila.

Re: memory leak in logical WAL sender with pgoutput's cachectx

2025-08-21 Thread Amit Kapila
considering after we have some tests to show the high memory usage. BTW, I think it would be better to run some algorithm to evict entries when we reach the threshold limit as we do for shared buffers, see StrategyGetBuffer. Otherwise, we may be paying the cost to maintain such a list when in practice it may be required only very few times. -- With Regards, Amit Kapila.

Re: doc patch: correct function name for slot synchronization.

2025-08-21 Thread Amit Kapila
ddress the issue. The typo > exists > till PG17. > Thanks for noticing this and providing a patch. I'll take care of this. -- With Regards, Amit Kapila.

Re: Logical Replication of sequences

2025-08-20 Thread Amit Kapila
On Wed, Aug 20, 2025 at 11:00 PM Masahiko Sawada wrote: > > On Tue, Aug 19, 2025 at 9:14 PM Amit Kapila wrote: > > > > If so, I don't think we can do much with the design > > choice we made. During DDL replication of sequences, we need to > > consider it as a c

Re: Proposal: Conflict log history table for Logical Replication

2025-08-20 Thread Amit Kapila
On Wed, Aug 20, 2025 at 11:47 AM Dilip Kumar wrote: > > On Mon, Aug 18, 2025 at 12:25 PM Amit Kapila wrote: > > > > > One idea to keep things simple for the first version is that we allow > > users to specify the table_name for storing conflicts but the table > >

Re: Logical Replication of sequences

2025-08-20 Thread Amit Kapila
E_INIT also doesn't sound intuitive, even though it serves the purpose. I feel it is better to use SUBREL_STATE_DATASYNC state as that indicates data is being synchronized, and let the LSN value be the same as the previous. [1]: https://www.postgresql.org/message-id/CAA4eK1LcBoPBCKa9yFOQnvpBv3a2ejf_EWC%3DZKksGcvqW7e0Zg%40mail.gmail.com -- With Regards, Amit Kapila.

Re: Logical Replication of sequences

2025-08-19 Thread Amit Kapila
EFRESH before the ALTER SEQUENCE command; otherwise, the ALTER SEQUENCE won't be replicated, right? If so, I don't think we can do much with the design choice we made. During DDL replication of sequences, we need to consider it as a conflict. BTW, note that the same situation can happen even when the user manually changed the sequence value on the subscriber in some way. So, we can't prevent that. -- With Regards, Amit Kapila.

Re: Parallel Apply

2025-08-18 Thread Amit Kapila
ests are attached. > > > > Next, I plan to extend the testing to larger workloads by running > > pgbench for 20–30 minutes. > > We will also benchmark performance across different workload types to > > evaluate the improvements once the patch has matured further. > > > > -- > > Thanks, > > Nisha > > > I also did some benchmarking of the proposed parallel apply patch and > compare it with my prewarming approach. > And parallel apply is significantly more efficient than prefetch (it is > expected). > Thanks to you and Nisha for doing some preliminary performance testing, the results are really encouraging (more than 3 to 4 times improvement in multiple workloads). I hope we keep making progress on this patch and make it ready for the next release. -- With Regards, Amit Kapila.

Re: Conflict detection for update_deleted in logical replication

2025-08-18 Thread Amit Kapila
On Mon, Aug 18, 2025 at 5:05 PM Zhijie Hou (Fujitsu) wrote: > > On Monday, August 18, 2025 2:32 PM Dilip Kumar wrote: > > > > On Mon, Aug 18, 2025 at 10:36 AM Amit Kapila > > wrote: > > > > > > > --- > > > > Even

Re: Proposal: Conflict log history table for Logical Replication

2025-08-17 Thread Amit Kapila
ow storing conflicts in pre-created tables with more checks about its schema. -- With Regards, Amit Kapila.

Re: Conflict detection for update_deleted in logical replication

2025-08-17 Thread Amit Kapila
ans that it is possible that before restart one would deduce that the update_deleted conflict won't be reliably detected for a particular subscription but after restart it could lead to the opposite conclusion. But note that to make it behave similarly we need to store this value persistently in pg_subscription unless you have better ideas for this. Theoretically, there are two places where we can persist this information, one is with pg_subscription, and other in origin. I find it is closer to pg_subscription. -- With Regards, Amit Kapila.

Re: Proposal: Conflict log history table for Logical Replication

2025-08-14 Thread Amit Kapila
On Thu, Aug 14, 2025 at 4:26 PM Alastair Turner wrote: > > On Wed, 13 Aug 2025 at 11:09, Amit Kapila wrote: >> >> On Fri, Aug 8, 2025 at 10:01 AM Dilip Kumar wrote: >> > >> > On Fri, Aug 8, 2025 at 8:58 AM shveta malik wrote: >> > > >>

Re: Parallel Apply

2025-08-14 Thread Amit Kapila
On Wed, Aug 13, 2025 at 8:57 PM Bruce Momjian wrote: > > On Wed, Aug 13, 2025 at 09:50:27AM +0530, Amit Kapila wrote: > > On Tue, Aug 12, 2025 at 10:40 PM Bruce Momjian wrote: > > > > Currently, PostgreSQL supports parallel apply only for large streaming > > > &

Re: Conflict detection for update_deleted in logical replication

2025-08-14 Thread Amit Kapila
On Thu, Aug 14, 2025 at 9:02 AM shveta malik wrote: > > On Wed, Aug 13, 2025 at 5:42 PM Amit Kapila wrote: > > > > > > > > 4) > > > For the DETAIL part of resume and stop messages, how about these: > > > > > > The retention duration for

Re: Excessive LOG messages from replication slot sync worker

2025-08-14 Thread Amit Kapila
On Thu, Aug 14, 2025 at 9:43 AM Fujii Masao wrote: > > On Wed, Aug 13, 2025 at 1:00 PM Amit Kapila wrote: > > > > On Tue, Aug 12, 2025 at 4:28 PM Fujii Masao wrote: > > > > > > I'm not sure these messages are useful for end users, and LOG might not be &

Re: Conflict detection for update_deleted in logical replication

2025-08-13 Thread Amit Kapila
now indefinite. > Similar to the previous point, will it be better to keep it short by using "conflict detection info", for example, it will lead to message like "The retention duration for conflict detection info is now indefinite."? -- With Regards, Amit Kapila.

Re: Proposal: Conflict log history table for Logical Replication

2025-08-13 Thread Amit Kapila
qjxnA2mGt5O%3DDht7sw%40mail.gmail.com [5] - https://www.postgresql.org/message-id/CANhcyEW%2BuJB_bvQLEaZCgoRTc1%3Di%2BQnrPPHxZ2%3D0SBSCyj9pkg%40mail.gmail.com -- With Regards, Amit Kapila.

Re: PG 18 release notes draft committed

2025-08-12 Thread Amit Kapila
xplicitly. -- With Regards, Amit Kapila.

Re: Parallel Apply

2025-08-12 Thread Amit Kapila
> This is a point to investigate if we observe so. But till now in our internal testing parallel apply gives good improvement in pgbench kind of workload. -- With Regards, Amit Kapila.

Re: Parallel Apply

2025-08-12 Thread Amit Kapila
On Tue, Aug 12, 2025 at 10:40 PM Bruce Momjian wrote: > > On Mon, Aug 11, 2025 at 10:15:41AM +0530, Amit Kapila wrote: > > Hi, > > > > Background and Motivation > > - > > In high-throughput systems, where hundreds of session

Re: Excessive LOG messages from replication slot sync worker

2025-08-12 Thread Amit Kapila
ce, I can tell that these messages have been helpful in finding BF failures and debugging bugs from user reports, so there is value in keeping them at LOG level. -- With Regards, Amit Kapila.

Re: Excessive LOG messages from replication slot sync worker

2025-08-12 Thread Amit Kapila
imply applying Shveta's patch to revert the incorrect format change > to fix the problem, and then discussing LSN-based filename standardization > in a separate thread. > +1. -- With Regards, Amit Kapila.

Re: [Patch] add new parameter to pg_replication_origin_session_setup

2025-08-12 Thread Amit Kapila
On Mon, Aug 11, 2025 at 10:41 PM Doruk Yilmaz wrote: > > On Mon, Aug 11, 2025 at 9:44 AM Amit Kapila wrote: > > How do you advance the origin? Did you use > > > pg_replication_origin_advance()? If so, you should be aware that it > > can be used for initial setup; s

Re: Parallel Apply

2025-08-12 Thread Amit Kapila
On Mon, Aug 11, 2025 at 3:00 PM Kirill Reshke wrote: > > On Mon, 11 Aug 2025 at 13:45, Amit Kapila wrote: > > > > I am not sure if that is directly applicable because this work > > proposes to track dependencies based on logical WAL contents. However, > > if you

Re: Conflict detection for update_deleted in logical replication

2025-08-12 Thread Amit Kapila
E: created replication slot "sub2" on publisher > CREATE SUBSCRIPTION > > Shall we give notice that max_conflict_retention_duration is ignored > as retain_dead_tuples is false. > How about disallowing this combination? -- With Regards, Amit Kapila.

Re: Parallel Apply

2025-08-11 Thread Amit Kapila
On Tue, Aug 12, 2025 at 12:04 PM Andrei Lepikhov wrote: > > On 11/8/2025 06:45, Amit Kapila wrote: > > The core idea is that the leader apply worker ensures the following: > > a. Identifies dependencies between transactions. b. Coordinates > > parallel workers to apply

Re: Parallel Apply

2025-08-11 Thread Amit Kapila
al replication speedup projects like[0] > I am not sure if that is directly applicable because this work proposes to track dependencies based on logical WAL contents. However, if you can point me to README on the overall design of the work you are pointing to then I can check it once. -- With Regards, Amit Kapila.

Re: [Patch] add new parameter to pg_replication_origin_session_setup

2025-08-10 Thread Amit Kapila
On Wed, Jul 30, 2025 at 12:00 AM Doruk Yilmaz wrote: > > On Mon, Jul 29, 2025 at 8:13 AM Amit Kapila wrote: > > That is true but I still feel there has to be some mechanism where we > > can catch and give an ERROR to the user, if it doesn't follow th

Parallel Apply

2025-08-10 Thread Amit Kapila
nt to maintain commit order because they don't explicitly annotate FK, PK for columns but maintain the integrity via application. So, in such cases as we won't be able to detect transaction dependencies, it would be better to allow out-of-order commits optionally. Thoughts? -- With Regards, Amit Kapila.

  1   2   3   4   5   6   7   8   9   10   >