Re: Fix slot synchronization with two_phase decoding enabled

2025-04-26 Thread Amit Kapila
On Fri, Apr 25, 2025 at 9:57 PM Masahiko Sawada wrote: > > On Fri, Apr 25, 2025 at 3:43 AM Amit Kapila wrote: > > > > On Fri, Apr 25, 2025 at 6:02 AM Masahiko Sawada > > wrote: > > > > > > I realized that users who create a logical slot using > &

Re: Fix premature xmin advancement during fast forward decoding

2025-04-26 Thread Amit Kapila
On Sat, Apr 26, 2025 at 12:01 AM Masahiko Sawada wrote: > > On Fri, Apr 25, 2025 at 4:42 AM Amit Kapila wrote: > > > > On Fri, Apr 25, 2025 at 10:46 AM Masahiko Sawada > > wrote: > > > > > > What I'm concerned about is the back branches. With

Re: Fix premature xmin advancement during fast forward decoding

2025-04-25 Thread Amit Kapila
any better ideas? -- With Regards, Amit Kapila.

Re: Fix slot synchronization with two_phase decoding enabled

2025-04-25 Thread Amit Kapila
er in the form of code or in giving the information to the user. -- With Regards, Amit Kapila.

Re: long-standing data loss bug in initial sync of logical replication

2025-04-25 Thread Amit Kapila
. > Thanks for spotting the problem. I've pushed your patch. -- With Regards, Amit Kapila.

Re: Fix premature xmin advancement during fast forward decoding

2025-04-24 Thread Amit Kapila
apBuildProcessChange, but not sure how much it will change. However, the bigger question is, is it really worth optimizing this code path? If we really want to optimize this code path for the test Hou-San did, I see that SnapBuildCommitTxn() has a bigger overhead, so we should investigate whether we really need it for the fast-forward mode, where we won't process changes. I am of the opinion that we need to pay this additional cost of setting base_snapshot for the correctness of the fast-forward mode. -- With Regards, Amit Kapila.

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-04-24 Thread Amit Kapila
On Wed, Apr 23, 2025 at 9:35 PM Masahiko Sawada wrote: > > On Wed, Apr 23, 2025 at 5:46 AM Amit Kapila wrote: > > > > BTW, did we consider the idea to automatically transition to 'logical' > > when the first logical slot is created and transition back to > &g

Re: Fix premature xmin advancement during fast forward decoding

2025-04-24 Thread Amit Kapila
ForSlot(lsn, xmin); Is my understanding correct? If so, then I think it is a miss in the commit f49a80c4 to consider fast_forward mode. Please refer commit in the commit message. The code changes look correct to me. I have some suggestions related to comments in the code. See attached. Please pr

Re: long-standing data loss bug in initial sync of logical replication

2025-04-24 Thread Amit Kapila
aches for some > reasons. > > With only 0001, the walsender sends 'd4' insertion or later. > > WIth both 0001 and 0002, the wansender sends 'd3' insertion or later. > > ISTM the difference between without both 0001 and 0002 and with 0001 > is significant. So I think it's worth applying 0001 for v13. > Pushed to v13 as well, thanks for sharing the feedback. -- With Regards, Amit Kapila.

Re: Fix slot synchronization with two_phase decoding enabled

2025-04-23 Thread Amit Kapila
On Wed, Apr 23, 2025 at 11:04 PM Masahiko Sawada wrote: > > On Tue, Apr 22, 2025 at 3:00 AM Amit Kapila wrote: > > > > On Mon, Apr 21, 2025 at 8:44 AM Zhijie Hou (Fujitsu) > > wrote: > > > > > > On Sat, Apr 19, 2025 at 2:19 AM Masahiko Sawada wrote:

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-04-23 Thread Amit Kapila
> with wal_level set to 'replica'. However, this would necessitate > invalidating all logical replication slots during promotion, > potentially extending downtime during failover. > BTW, did we consider the idea to automatically transition to 'logical' when the first logical slot is created and transition back to 'replica' when last logical slot gets dropped? I see some ideas around this last time we discussed this topic. [1] - https://www.postgresql.org/message-id/CAA4eK1J0we5qsZ-ZOwXPbZyvwdWbnT43knO2Cxidia2aHxZSJw%40mail.gmail.com -- With Regards, Amit Kapila.

Re: long-standing data loss bug in initial sync of logical replication

2025-04-22 Thread Amit Kapila
On Tue, Apr 22, 2025 at 10:57 PM Masahiko Sawada wrote: > > On Tue, Mar 18, 2025 at 2:55 AM Amit Kapila wrote: > > > > On Mon, Mar 17, 2025 at 4:56 PM Hayato Kuroda (Fujitsu) > > wrote: > > > > > > > Regarding the PG13, it cannot be > > > &g

Re: Fix slot synchronization with two_phase decoding enabled

2025-04-22 Thread Amit Kapila
ase is later modified to true after > tablesync. Although the simpler check is more user-visible, it may offer less > flexibility. > I agree with your point, but OTOH, I am also afraid of adding too many smart checks in the back-branch. If we follow what you say here, then users have the

Re: Fix slot synchronization with two_phase decoding enabled

2025-04-21 Thread Amit Kapila
On Fri, Apr 18, 2025 at 9:58 AM Amit Kapila wrote: > > On Thu, Apr 17, 2025 at 6:14 PM Zhijie Hou (Fujitsu) > wrote: > > > > - > > Fix > > - > > > > I think we should keep the confirmed_flush even if the previous synced > > restart_lsn

Re: Fix slot synchronization with two_phase decoding enabled

2025-04-17 Thread Amit Kapila
confirmed_flush even if the previous synced > restart_lsn/catalog_xmin is newer. Attachments include a patch for the same. > This will fix the case we are facing but adds a new rule for slot synchronization. Can we think of a simpler way to fix this by avoiding updating other slot fields (like two_phase, two_phase_at) if restart_lsn or catalog_xmin of the local slot is ahead of the remote slot? -- With Regards, Amit Kapila.

Re: Skipping schema changes in publication

2025-04-16 Thread Amit Kapila
On Wed, Apr 16, 2025 at 8:22 AM Zhijie Hou (Fujitsu) wrote: > > On Thu, Apr 10, 2025 at 7:25 PM Amit Kapila wrote: > > > > On Tue, Jan 9, 2024 at 12:02 PM vignesh C wrote: > > > > > > As I did not see much interest from others, I'm withdrawing this

Re: Feature Recommendations for Logical Subscriptions

2025-04-10 Thread Amit Kapila
data except 2 columns, (b) adding new columns to table would require users to again run the DDL to change the column list. These are primarily the two pain points YeXiu wants us to solve. YeXiu, if I misunderstood your intention, feel free to add. -- With Regards, Amit Kapila.

Re: Feature Recommendations for Logical Subscriptions

2025-04-10 Thread Amit Kapila
gt; not a C developer and cannot implement them myself. > > Okay, we previously discussed this feature, but due to a lack of interest, it has been dropped. You can join that thread [1] and help with testing and feature specifications. [1] - https://www.postgresql.org/message-id/CALDaNm3dWZCYDih55qTNAYsjCvYXMFv=46usdwmfcnxmt3k...@mail.gmail.com -- With Regards, Amit Kapila.

Re: Skipping schema changes in publication

2025-04-10 Thread Amit Kapila
resql.org/message-id/tencent_DCDF626FCD4A556C51BE270FDC3047540208%40qq.com -- With Regards, Amit Kapila.

Re: long-standing data loss bug in initial sync of logical replication

2025-04-10 Thread Amit Kapila
On Tue, Apr 8, 2025 at 3:40 PM Amit Kapila wrote: > > On Tue, Mar 18, 2025 at 3:25 PM Amit Kapila wrote: > > > > Sawada-San, and others involved here, do you have any suggestions on > > this matter? > > > > Seeing no responses for a long time, I am plannin

Re: Enhance 'pg_createsubscriber' to retrieve databases automatically when no database is provided.

2025-04-10 Thread Amit Kapila
On Thu, Apr 10, 2025 at 5:22 AM Peter Smith wrote: > > OK. Here is a single patch which combines previous patches 0001 and 0003. > Pushed. -- With Regards, Amit Kapila.

Re: Fix 035_standby_logical_decoding.pl race conditions

2025-04-09 Thread Amit Kapila
y to debug this. > I can't think of a good reason to have this DEBUG1 as there is no predictability of it getting generated even with tests using an injection point. OTOH, I don't have any objections to it if you would like to proceed with this. -- With Regards, Amit Kapila.

Re: Feature Recommendations for Logical Subscriptions

2025-04-09 Thread Amit Kapila
atabase while decoding changes, so won't some interaction with database privileges still be required? BTW, are you planning to work on a patch on these proposals or you expect someone else in the community to work on these proposals? -- With Regards, Amit Kapila.

Re: Enhance 'pg_createsubscriber' to retrieve databases automatically when no database is provided.

2025-04-09 Thread Amit Kapila
t when --all option is used. For other cases, see the description in the -d option. So, this change is not required. If you agree, then you can combine the other two patches, and we can proceed with those. -- With Regards, Amit Kapila.

Re: BUG #18815: Logical replication worker Segmentation fault

2025-04-09 Thread Amit Kapila
this patch yesterday. -- With Regards, Amit Kapila.

Re: long-standing data loss bug in initial sync of logical replication

2025-04-08 Thread Amit Kapila
On Tue, Mar 18, 2025 at 3:25 PM Amit Kapila wrote: > > On Mon, Mar 17, 2025 at 4:56 PM Hayato Kuroda (Fujitsu) > wrote: > > > > > Regarding the PG13, it cannot be > > > applied > > > as-is thus some adjustments are needed. I will share it in upcoming p

Re: Large expressions in indexes can't be stored (non-TOASTable)

2025-04-08 Thread Amit Kapila
On Fri, Apr 4, 2025 at 7:58 PM Nathan Bossart wrote: > > On Fri, Apr 04, 2025 at 05:16:43PM +0530, Amit Kapila wrote: > > Can we dodge adding this push call if we restrict the length of the > > origin name to some reasonable limit like 256 or 512 and avoid the > > need of

Re: BUG #18815: Logical replication worker Segmentation fault

2025-04-08 Thread Amit Kapila
simply report the original constraint violation ERROR. > Fair enough. On considering it again, I find your idea of building conflict-related information when it is actually required sounds better, as it may also save us performance in some corner cases. -- With Regards, Amit Kapila.

Re: Fix 035_standby_logical_decoding.pl race conditions

2025-04-07 Thread Amit Kapila
ith Regards, Amit Kapila. v6-0001-Stabilize-035_standby_logical_decoding.pl.patch Description: Binary data

Re: Fix 035_standby_logical_decoding.pl race conditions

2025-04-07 Thread Amit Kapila
On Thu, Apr 3, 2025 at 3:15 PM Amit Kapila wrote: > > On Thu, Apr 3, 2025 at 12:29 PM Bertrand Drouvot > wrote: > > > > On Thu, Apr 03, 2025 at 05:34:10AM +, Hayato Kuroda (Fujitsu) wrote: > > > Dear Bertrand, Amit, > > > > > > > > I do

Re: BUG #18815: Logical replication worker Segmentation fault

2025-04-07 Thread Amit Kapila
e actual update or insert. E.g., do that in > InitConflictIndexes(). > Right, this is also worth considering. But let us first conclude on the existing approach to build the required information in ExecOpenIndices(). -- With Regards, Amit Kapila.

Re: Fix 035_standby_logical_decoding.pl race conditions

2025-04-05 Thread Amit Kapila
lusion of these discussions, slots are created but > it seldomly be activated. > > Naming of patches are bit different, but please ignore... > Isn't patch 0001-Fix-invalid-referring-of-hash-ref-for-replication-sl unrelated to this thread? Or am, I missing something? -- With Regards, Amit Kapila.

Re: Enhance 'pg_createsubscriber' to retrieve databases automatically when no database is provided.

2025-04-05 Thread Amit Kapila
a + --all + + + -D + --pgdata + +datadir + + -P + --publisher-server + +connstr Most of this is unrelated to this patch. I suggest making a top-up patch, we can commit it separately. -- With Regards, Amit Kapila.

Re: Separate GUC for replication origins

2025-04-05 Thread Amit Kapila
x_replication_slots, the other line: "Most options are relevant only on one side of the replication." doesn't make sense because there is no other option that applies to both sides and if there is one then we should mention about that. > The patch looks good to me. > Other than the above, the patch looks good to me as well. -- With Regards, Amit Kapila.

Re: long-standing data loss bug in initial sync of logical replication

2025-04-05 Thread Amit Kapila
t invasive, so we can go with it as well. My preference would be to go with just 0001 for v13 to minimize the risk of adding new bugs or breaking something unintentionally. Sawada-San, and others involved here, do you have any suggestions on this matter? -- With Regards, Amit Kapila.

Re: Adding a '--clean-publisher-objects' option to 'pg_createsubscriber' utility.

2025-04-05 Thread Amit Kapila
On Thu, Mar 20, 2025 at 9:45 AM Shubham Khanna wrote: > The patch had a bug in dry-run mode such that it was not emitting the drop-related command for publications created by the tool with the new option. I fixed that and pushed the patch. Thanks! -- With Regards, Amit Kapila.

Re: Fix 035_standby_logical_decoding.pl race conditions

2025-04-05 Thread Amit Kapila
ing-xacts")) It is better to name the injection point as skip-log-running-xacts as that will be appropriate based on its usage. -- With Regards, Amit Kapila. diff --git a/src/backend/storage/ipc/standby.c b/src/backend/storage/ipc/standby.c index 0e621e9996a..7fa8d9247e0 100644 --

Re: Large expressions in indexes can't be stored (non-TOASTable)

2025-04-05 Thread Amit Kapila
> Can we dodge adding this push call if we restrict the length of the origin name to some reasonable limit like 256 or 512 and avoid the need of toast altogether? -- With Regards, Amit Kapila.

Re: Improve error reporting for few options in pg_createsubscriber

2025-04-04 Thread Amit Kapila
On Fri, Apr 4, 2025 at 10:11 AM vignesh C wrote: > > On Fri, 4 Apr 2025 at 09:36, Amit Kapila wrote: > > > > The attached v2 version patch has the changes for the same. > Pushed. -- With Regards, Amit Kapila.

Re: rename pg_log_standby_snapshot

2025-04-04 Thread Amit Kapila
t; pg_standby_snapshot_xip_to_wal? > The other option could be pg_create_standby_snapshot(), which would be similar to the existing function pg_create_restore_point(). I think we need to think about backward compatibility if we agree with moving in this direction. -- With Regards, Amit Kapila.

Re: Conflict detection for multiple_unique_conflicts in logical replication

2025-04-03 Thread Amit Kapila
com/docs/pgd/4/bdr/conflicts/#insert-operations-that-violate-multiple-unique-constraints [2] - https://www.enterprisedb.com/docs/pgd/4/bdr/conflicts/#update-operations-that-violate-multiple-unique-constraints -- With Regards, Amit Kapila.

Re: Improve error reporting for few options in pg_createsubscriber

2025-04-03 Thread Amit Kapila
orm where applicable. * "invalid object type \"%s\" specified for --remove". Isn't it better to use the short form as well in this error message? -- With Regards, Amit Kapila.

Re: Adding a '--clean-publisher-objects' option to 'pg_createsubscriber' utility.

2025-04-03 Thread Amit Kapila
gt; It seems the final SGML docs are mostly from this [1] suggestion, but > > I think there are one or two more improvements that can be made to it: > > > > Thanks for the patch, the changes look good to me. > Pushed. -- With Regards, Amit Kapila.

Re: Fix 035_standby_logical_decoding.pl race conditions

2025-04-03 Thread Amit Kapila
qq[SELECT pg_drop_replication_slot('$inactive_slot')]); > " > > You'd see the slot being active and the "$node_standby->psql" not reporting > any error. > Hmm, but adding some additional smarts also makes this test less easy to backpatch. I see your points related to the benefits, but I still mildly prefer to go with the lesser changes approach for backbranches patch. Normally, we don't enhance backbranches code without making equivalent changes in HEAD, so adding some new bugs only in backbranches has a lesser chance. -- With Regards, Amit Kapila.

Re: Fix slot synchronization with two_phase decoding enabled

2025-04-02 Thread Amit Kapila
On Thu, Apr 3, 2025 at 11:08 AM Masahiko Sawada wrote: > > On Wed, Apr 2, 2025 at 7:58 PM Amit Kapila wrote: > > > > On Thu, Apr 3, 2025 at 7:50 AM Zhijie Hou (Fujitsu) > > wrote: > > > > > > On Thu, Apr 3, 2025 at 3:30 AM Masahiko Sawada wrote: >

Re: Fix 035_standby_logical_decoding.pl race conditions

2025-04-02 Thread Amit Kapila
ot; to HEAD. That said, I think that we > should > keep the slots active and only avoid doing the checks for them (they are > invalidated > that's fine, they are not that's fine too). > I don't mind doing that, but there is no benefit in making slots active unless we can validate them. And we will end up adding some more checks, as in function check_slots_conflict_reason without any advantage. I feel Kuroda-San's second patch is simple, and we have fewer chances to make mistakes and easy to maintain in the future as well. -- With Regards, Amit Kapila.

Re: Fix slot synchronization with two_phase decoding enabled

2025-04-02 Thread Amit Kapila
roposed by Sawada-san. Note that we enable two_phase once all the tables are in ready state (See run_apply_worker() and comments atop worker.c (TWO_PHASE TRANSACTIONS)). -- With Regards, Amit Kapila.

Re: Fix 035_standby_logical_decoding.pl race conditions

2025-04-02 Thread Amit Kapila
less because after that, we don't need to make changes related to drop and create. Sure, in some cases, we will test two inactive slots instead of one, but I guess that would be the price to keep the tests simple and more like HEAD. -- With Regards, Amit Kapila.

Re: Fix slot synchronization with two_phase decoding enabled

2025-04-01 Thread Amit Kapila
se enabled subscription/, + "Enabling failover is not allowed for a two_phase enabled subscription"); Is there a need for this test to be in .pl file? Can't we add it in .sql file? Apart from the above, I have made minor modifications to the PG17 patch in the attached. -- With R

Re: Fix 035_standby_logical_decoding.pl race conditions

2025-04-01 Thread Amit Kapila
ut using > LogLogicalMessage()? > We are using injection points for testing purposes, which means the caller is aware of skipping the running_xacts record during the test run. So, there doesn't seem to be any reason to do anything extra. -- With Regards, Amit Kapila.

Re: Fix 035_standby_logical_decoding.pl race conditions

2025-04-01 Thread Amit Kapila
dded any injection point for the above case. Isn't it possible that if running_xact record is logged concurrently to the pruning record, it should move the active slot on standby, and the same failure should occur in this case as well? -- With Regards, Amit Kapila.

Re: Fix slot synchronization with two_phase decoding enabled

2025-04-01 Thread Amit Kapila
from the above, I have a few suggestions for changes in docs, error messages, and comments for both versions. See attached. -- With Regards, Amit Kapila. diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml index 141c140331d..bff0d143ac5 100644 --- a/doc/src/sgml/system-vi

Re: Fix slot synchronization with two_phase decoding enabled

2025-03-31 Thread Amit Kapila
On Mon, Mar 31, 2025 at 5:04 PM Zhijie Hou (Fujitsu) wrote: > > On Thu, Mar 27, 2025 at 2:29 PM Amit Kapila wrote: > > > > > I suspect that this can happen in PG17 as well, but I need to think > > more about it to make a reproducible test case. > > After further

Re: Enhance 'pg_createsubscriber' to retrieve databases automatically when no database is provided.

2025-03-28 Thread Amit Kapila
On Thu, Mar 27, 2025 at 1:35 PM Shubham Khanna wrote: > > The attached patches contain the suggested changes. They also address > the comments provided by Kuroda-san (at [1]). > Pushed after some cosmetic changes. -- With Regards, Amit Kapila.

Re: Selectively invalidate caches in pgoutput module

2025-03-28 Thread Amit Kapila
A the patch to fix the issue. > Pushed after slight modification. -- With Regards, Amit Kapila.

Re: Selectively invalidate caches in pgoutput module

2025-03-27 Thread Amit Kapila
nsure the initial replication is setup between publisher and subscriber. This is how we use these commands at other places. -- With Regards, Amit Kapila.

Re: Fix 035_standby_logical_decoding.pl race conditions

2025-03-27 Thread Amit Kapila
ort for PG17 as-is. > We can use 'wait' mode API in PG17 as used in one of the tests (injection_points_attach('heap_update-before-pin', 'wait');) but I think it may be better to just leave testing active slots in backbranches because anyway the new development happens on HEAD and we want to ensure that no breakage happens there. -- With Regards, Amit Kapila.

Re: Enhance 'pg_createsubscriber' to retrieve databases automatically when no database is provided.

2025-03-26 Thread Amit Kapila
ns." Please use this one at the above and other places where required. -- With Regards, Amit Kapila.

Re: Fix slot synchronization with two_phase decoding enabled

2025-03-26 Thread Amit Kapila
On Tue, Mar 25, 2025 at 12:14 PM Amit Kapila wrote: > > On Tue, Mar 25, 2025 at 11:05 AM Zhijie Hou (Fujitsu) > wrote: > > > > Hi, > > > > When testing the slot synchronization with logical replication slots that > > enabled two_phase decoding, I found tha

Re: Enhance 'pg_createsubscriber' to retrieve databases automatically when no database is provided.

2025-03-25 Thread Amit Kapila
On Tue, Mar 25, 2025 at 5:24 PM Ashutosh Bapat wrote: > > On Tue, Mar 25, 2025 at 5:15 PM Amit Kapila wrote: > > > > On Tue, Mar 25, 2025 at 5:08 PM Ashutosh Bapat > > wrote: > > > > > > This looks mostly ready except the test changes. I believe when &g

Re: Enhance 'pg_createsubscriber' to retrieve databases automatically when no database is provided.

2025-03-25 Thread Amit Kapila
On Tue, Mar 25, 2025 at 5:30 PM Ashutosh Bapat wrote: > > On Thu, Mar 20, 2025 at 5:54 PM Amit Kapila wrote: > > > > * > > + > > + pg_createsubscriber > > + option > > + > > + > > + -a > > +

Re: Enhance 'pg_createsubscriber' to retrieve databases automatically when no database is provided.

2025-03-25 Thread Amit Kapila
complexity introduced > into vacuumdb (per table, per schema) seems to justify multiple synopsis too. > However, the same logic doesn't apply here. IMO 0002 shouldn't be applied > because the additional option (--all) doesn't justify multiple synopsis for > syntax clarification. > Yeah, also, vacuumdb doesn't have a separate line for --all in synopsis. So, I agree with you about not adding '--all' option in the synopsis. -- With Regards, Amit Kapila.

Re: Fix 035_standby_logical_decoding.pl race conditions

2025-03-25 Thread Amit Kapila
and similar cases, or can't we have an injection point to control logging this WAL record? I have seen the need to control logging running_xact record in other cases as well. [1] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=skink&dt=2025-03-19%2007%3A08%3A16 -- With Regards, Amit Kapila.

Re: Enhance 'pg_createsubscriber' to retrieve databases automatically when no database is provided.

2025-03-25 Thread Amit Kapila
of test +# run pg_createsubscriber with '--all' option without '--dry-run'. I am not so sure whether it is really worth adding test cycles for this option except for dry-run mode tests but we can discuss after committing the core patch. -- With Regards, Amit Kapila.

Re: Fix slot synchronization with two_phase decoding enabled

2025-03-24 Thread Amit Kapila
o send at that time. > > This problem arises after the support for altering the two-phase option > (1462aad). > Thanks for the report and patch. I'll look into it. -- With Regards, Amit Kapila.

Re: pgoutput: comment atop rel_sync_cache_publication_cb() has an inconsistency

2025-03-24 Thread Amit Kapila
g_publication and pg_namespace. > */ > static void > rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue) > ``` > > But this won't be called for invalidating pg_publication. It should have been > updated by 3abe9dc1, but it was an oversight. > Agreed and thanks for the patch. -- With Regards, Amit Kapila.

Re: Conflict detection for multiple_unique_conflicts in logical replication

2025-03-24 Thread Amit Kapila
On Tue, Mar 25, 2025 at 8:51 AM vignesh C wrote: > > On Fri, 21 Mar 2025 at 16:44, Nisha Moond wrote: > > > > On Fri, Mar 21, 2025 at 3:38 PM Amit Kapila wrote: > > > > > > On Fri, Mar 21, 2025 at 1:48 PM Zhijie Hou (Fujitsu) > > > wrote: > >

Re: Conflict detection for multiple_unique_conflicts in logical replication

2025-03-24 Thread Amit Kapila
On Fri, Mar 21, 2025 at 4:43 PM Nisha Moond wrote: > > Attached are the v8 patches. > Thanks, the patches look good to me, so pushed. -- With Regards, Amit Kapila.

Re: Parallel safety for extern params

2025-03-23 Thread Amit Kapila
On Sat, Mar 22, 2025 at 1:25 AM Tom Lane wrote: > > Robert Haas writes: > > I'm happy to have you tidy up here in whatever way seems best to you. > > Cool, done. > Thanks for taking care of this, and sorry for not digging deeper to find the appropriate fix. -- With Regards, Amit Kapila.

Re: Conflict detection for multiple_unique_conflicts in logical replication

2025-03-21 Thread Amit Kapila
the > updates made to the input parameters. > Right, I also noticed this and updated it in the attached. Apart from this, I have made a number of cosmetic changes in the attached. Please review and include it in the next version unless you see any problem with it. -- With Regards, Amit Kapi

Re: Separate GUC for replication origins

2025-03-20 Thread Amit Kapila
On Thu, Mar 20, 2025 at 10:37 PM Masahiko Sawada wrote: > > On Wed, Mar 19, 2025 at 8:15 PM Amit Kapila wrote: > > > > On Wed, Mar 19, 2025 at 10:43 AM Masahiko Sawada > > wrote: > > > > > > On Mon, Mar 17, 2025 at 6:05 PM Euler Taveira wrote: > &

Re: Conflict detection for multiple_unique_conflicts in logical replication

2025-03-19 Thread Amit Kapila
e tests spread over existing tests where such conflict-related tests were present. Shall we name it as t/035_conflicts.pl? That will allow us to add more tests related to other conflicts like update_delete in the same file? -- With Regards, Amit Kapila.

Re: Adding a '--clean-publisher-objects' option to 'pg_createsubscriber' utility.

2025-03-19 Thread Amit Kapila
e pg_upgrade to have LOG and or SQL files in which case we may need to have an option for --retain (-r) similar to pg_upgrade. So, using -R for this option sounds reasonable to me. -- With Regards, Amit Kapila.

Re: Fix 035_standby_logical_decoding.pl race conditions

2025-03-18 Thread Amit Kapila
in = running->oldestRunningXid; elog(DEBUG3, "xmin: %u, xmax: %u, oldest running: %u, oldest xmin: %u", builder->xmin, builder->xmax, running->oldestRunningXid, xmin); LogicalIncreaseXminForSlot(lsn, xmin); ... Note that I have not tested this case, so I could be wrong. But if possible, we should try to find some solution which could be backpatched to 16 as well. -- With Regards, Amit Kapila.

Re: Adding a '--clean-publisher-objects' option to 'pg_createsubscriber' utility.

2025-03-18 Thread Amit Kapila
On Wed, Mar 19, 2025 at 2:11 AM David G. Johnston wrote: > > On Tue, Mar 18, 2025 at 4:47 AM Amit Kapila wrote: >> >> I don't understand --dry-run part of conversation here. As per >> existing code (or with the patch), we seem to be already printing the >> pu

Re: Adding a '--clean-publisher-objects' option to 'pg_createsubscriber' utility.

2025-03-18 Thread Amit Kapila
function check_and_drop_existing_subscriptions(). The new function should check if the user requested to remove the publication then it should query the publisher, otherwise, just remove the one specified by dbinfo. Also, the core drop_publication() function should take the required parameters instead of dbinfo after this patch. That would simplify the code. -- With Regards, Amit Kapila.

Re: Adding a '--clean-publisher-objects' option to 'pg_createsubscriber' utility.

2025-03-17 Thread Amit Kapila
m, I think at this stage, we don't need to add the provision for 'all'. We can add it when more than one object 'publication' will be allowed to be removed. With Regards, Amit Kapila.

Re: pgsql: Avoid invalidating all RelationSyncCache entries on publication

2025-03-17 Thread Amit Kapila
On Mon, Mar 17, 2025 at 9:35 PM Robert Haas wrote: > > On Thu, Mar 13, 2025 at 12:00 AM Amit Kapila wrote: > > Avoid invalidating all RelationSyncCache entries on publication rename. > > > > On Publication rename, we need to only invalidate the RelationSyncCache >

Re: Fix couple of typos

2025-03-17 Thread Amit Kapila
it/MultiXactCutoff are frozen in the common case. > Should have been" > Only tuples with XIDs/MXIDs older than the FreezeLimit/MultiXactCutoff > are frozen in the common case. > > The attached patch has the changes for the same. > The changes look good to me. -- With Regards, Amit Kapila.

Re: Proposal: Deferred Replica Filtering for PostgreSQL Logical Replication

2025-03-17 Thread Amit Kapila
ication filters, enhancing security and data privacy. > We provide column lists [1] and row filters [2]. Doesn't that suffice the need, if not, kindly let us know what exactly you need with some examples. [1] - https://www.postgresql.org/docs/devel/logical-replication-col-lists.html [2] - https://www.postgresql.org/docs/devel/logical-replication-row-filter.html -- With Regards, Amit Kapila.

Re: Conflict detection for multiple_unique_conflicts in logical replication

2025-03-17 Thread Amit Kapila
flictSlots. 2. From the commit message: "Also, the patch adds a new column 'confl_multiple_unique_conflicts' in view pg_stat_subscription_stats to support stats collection for this conflict type.". This part can be split into a second patch. Let's try to get the core patch first. -- With Regards, Amit Kapila.

Re: Restrict copying of invalidated replication slots

2025-03-16 Thread Amit Kapila
lready past slot_cp location in which case it doesn't need to wait for the backend. I have changed the comments for the master branch patch. Do let me know what you think of the attached change. If you agree with this, then please prepare the patches for back-branches that reflect this change.

Re: long-standing data loss bug in initial sync of logical replication

2025-03-16 Thread Amit Kapila
e simple test using SQL API based on what Andres presented in an email [1]? 3. I have made minor changes in the comments in the attached. [1] - https://www.postgresql.org/message-id/20231119021830.d6t6aaxtrkpn743y%40awork3.anarazel.de -- With Regards, Amit Kapila. diff --git a/src/back

Re: Adding a '--clean-publisher-objects' option to 'pg_createsubscriber' utility.

2025-03-16 Thread Amit Kapila
On Sat, Mar 15, 2025 at 8:03 PM David G. Johnston wrote: > > On Friday, March 14, 2025, Amit Kapila wrote: >> >> >> Style-1 sounds reasonable to me, but how exactly we want to do. One >> idea is to have short and long switches like -r and >> --remove_exiting_o

Re: Add an option to skip loading missing publication to avoid logical replication failure

2025-03-15 Thread Amit Kapila
On Thu, Mar 13, 2025 at 7:26 PM Dilip Kumar wrote: > > On Thu, Mar 13, 2025 at 3:20 PM Amit Kapila wrote: > > > > > > I think that is too much related to pub-sub model, and ideally, > > pgoutput should not care about it. I have written a comment > > consider

Re: Adding a '--clean-publisher-objects' option to 'pg_createsubscriber' utility.

2025-03-15 Thread Amit Kapila
cations,Slots,Subscriptions,Databases,Tablespaces > > David, I understand you are saying that you prefer style #1 because of > the potentially cumbersome length of the csv value part of style #2 if > there are many future object kinds (e.g. > "Publications,Slots,Subscriptions,Databases,Tablespaces" in the above > examples). > Style-1 sounds reasonable to me, but how exactly we want to do. One idea is to have short and long switches like -r and --remove_exiting_object=publication. The user would be allowed to give multiple options like -r publications -r slots, etc. -- With Regards, Amit Kapila.

Re: long-standing data loss bug in initial sync of logical replication

2025-03-14 Thread Amit Kapila
hose as scenarios C and D won't be that common, and we should go ahead with the fix as it is. In the future, if we get any way to avoid regression due to scenario-D, then we can do that for the HEAD branch. Thoughts? -- With Regards, Amit Kapila.

Re: Adding a '--clean-publisher-objects' option to 'pg_createsubscriber' utility.

2025-03-13 Thread Amit Kapila
On Fri, Mar 14, 2025 at 3:06 AM Euler Taveira wrote: > > On Wed, Mar 12, 2025, at 3:41 AM, Amit Kapila wrote: > > This could be another option, but I feel an option with a tool is more > meaningful than allowing users to do afterward steps. > > > I wasn't paying much

Re: Selectively invalidate caches in pgoutput module

2025-03-13 Thread Amit Kapila
ed new patch which adds a test code. > Thanks. I have pushed the patch after minor changes in the test. -- With Regards, Amit Kapila.

Re: Add an option to skip loading missing publication to avoid logical replication failure

2025-03-13 Thread Amit Kapila
ror hint like: > > "If the publication is missing, create and refresh it. Otherwise, wait > > for the slot to reach the WAL record for the created publication, then > > refresh" > I have tried to split this information into errdetail and errhint in the attached. See and let me know what you think of the same. -- With Regards, Amit Kapila. v9-0001-Fix-ALTER-SUBSCRIPTION-.-SET-PUBLICATION-.-comman.patch Description: Binary data

Re: Parallel heap vacuum

2025-03-12 Thread Amit Kapila
On Wed, Mar 12, 2025 at 11:24 AM Dilip Kumar wrote: > > On Wed, Mar 12, 2025 at 3:17 AM Masahiko Sawada wrote: > > > > On Tue, Mar 11, 2025 at 5:51 AM Amit Kapila wrote: > > > > > Some thoughts/questions on the idea > > I notice that we are always consider

Re: Add an option to skip loading missing publication to avoid logical replication failure

2025-03-12 Thread Amit Kapila
On Tue, Mar 11, 2025 at 9:48 AM Dilip Kumar wrote: > > On Mon, Mar 10, 2025 at 10:54 AM Amit Kapila wrote: > > > > > >> BTW, I am planning to commit this only on HEAD as this is a behavior > > >> change. Please let me know if you guys think otherwise. >

Re: Parallel heap vacuum

2025-03-12 Thread Amit Kapila
On Wed, Mar 12, 2025 at 3:12 AM Masahiko Sawada wrote: > > On Tue, Mar 11, 2025 at 6:00 AM Amit Kapila wrote: > > > > On Mon, Mar 10, 2025 at 11:57 PM Masahiko Sawada > > wrote: > > > > > > On Sun, Mar 9, 2025 at 11:12 PM Amit Kapila > > &g

Re: Selectively invalidate caches in pgoutput module

2025-03-12 Thread Amit Kapila
er Publication ... Rename tests don't test invalidations arising from that command. As this patch changes that code path, it would be good to add a few tests for the same. We can add one for individual relations and another for ALL Tables publication. -- With Regards, Amit Kapila. v11-0001-Avo

Re: Adding a '--clean-publisher-objects' option to 'pg_createsubscriber' utility.

2025-03-11 Thread Amit Kapila
On Wed, Mar 12, 2025 at 10:42 AM David G. Johnston wrote: > > On Tuesday, March 11, 2025, Amit Kapila wrote: >> >> On Wed, Mar 12, 2025 at 9:43 AM David G. Johnston >> wrote: >> > >> > I’m honestly question how much value this provides - no improve

Re: Adding a '--clean-publisher-objects' option to 'pg_createsubscriber' utility.

2025-03-11 Thread Amit Kapila
a file is a way, but as a user, I would prefer to get the things done automatically via a tool instead of me doing it afterward unless there is no other way.. -- With Regards, Amit Kapila.

Re: Adding a '--clean-publisher-objects' option to 'pg_createsubscriber' utility.

2025-03-11 Thread Amit Kapila
ion to remove publications that are pre-existing (aka not created by this tool). -- With Regards, Amit Kapila.

Re: Adding a '--clean-publisher-objects' option to 'pg_createsubscriber' utility.

2025-03-11 Thread Amit Kapila
on this in the original thread where we developed this tool but left it as a future improvement. We should definitely go in this direction in the future, but let's improve this tool incrementally, otherwise, there is a risk of making no improvements. -- With Regards, Amit Kapila.

Re: Add an option to skip loading missing publication to avoid logical replication failure

2025-03-11 Thread Amit Kapila
On Mon, Mar 10, 2025 at 10:15 AM Dilip Kumar wrote: > > On Mon, Mar 10, 2025 at 9:33 AM Amit Kapila wrote: >> >> On Tue, Mar 4, 2025 at 6:54 PM vignesh C wrote: >> > >> > On further thinking, I felt the use of publications_updated variable >> >

Re: Parallel heap vacuum

2025-03-11 Thread Amit Kapila
On Mon, Mar 10, 2025 at 11:57 PM Masahiko Sawada wrote: > > On Sun, Mar 9, 2025 at 11:12 PM Amit Kapila wrote: > > > > > > > However, in the heap vacuum phase, the leader process needed > > > to process all blocks, resulting in soft page faults while cre

  1   2   3   4   5   6   7   8   9   10   >