Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-20 Thread Amit Kapila
eplication_error, or pg_subscription_lrep_error? We can use similar naming at another place (view) if required. -- With Regards, Amit Kapila.

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-20 Thread Amit Kapila
ker. The error during that phase will be no different than the apply worker. One idea to make the separation more obvious is to introduce 'worker_type' column similar to backend_type in pg_stat_activity which will tell whether it is an apply worker or a table sync worker. -- With Regards, Amit Kapila.

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-20 Thread Amit Kapila
ions explicitly rolled back) * stats_reset (Time at which these statistics were last reset) The view name could be pg_stat_subscription_lrep, pg_stat_logical_replication, or something on those lines. -- With Regards, Amit Kapila.

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-21 Thread Amit Kapila
On Mon, Feb 21, 2022 at 1:18 PM Andres Freund wrote: > > On 2022-02-21 12:39:31 +0530, Amit Kapila wrote: > > Fair enough. Then, how about the following keeping the following > > information: > > Mostly sounds good. > > > > * subid (subscription id) > >

Re: pg_stat_get_replication_slot and pg_stat_get_subscription_worker incorrectly marked as proretset

2022-02-21 Thread Amit Kapila
; > > Thoughts? > > Agreed. > +1. How about attached? -- With Regards, Amit Kapila. fix_pgproc_get_repl_slot_1.patch Description: Binary data

Re: pg_stat_get_replication_slot and pg_stat_get_subscription_worker incorrectly marked as proretset

2022-02-21 Thread Amit Kapila
On Mon, Feb 21, 2022 at 4:56 PM Michael Paquier wrote: > > On Mon, Feb 21, 2022 at 04:19:59PM +0530, Amit Kapila wrote: > > On Mon, Feb 21, 2022 at 11:21 AM Masahiko Sawada > > wrote: > >> Agreed. > >> > > > > +1. How about attached? > > Th

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-21 Thread Amit Kapila
On Mon, Feb 21, 2022 at 9:37 PM David G. Johnston wrote: > > On Mon, Feb 21, 2022 at 2:19 AM Amit Kapila wrote: >> >> On Mon, Feb 21, 2022 at 1:18 PM Andres Freund wrote: >> >> > > The view name could be pg_stat_subscription_lrep, >> > > pg_stat_

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-22 Thread Amit Kapila
apply_error_callback_arg and then use that to send the appropriate message. I think this will obviate the need for try..catch. -- With Regards, Amit Kapila.

Re: row filtering for logical replication

2022-02-22 Thread Amit Kapila
On Tue, Feb 22, 2022 at 4:47 AM Euler Taveira wrote: > > On Thu, Feb 17, 2022, at 3:36 AM, Amit Kapila wrote: > > As there is a new version, I would like to wait for a few more days > before committing. I am planning to commit this early next week (by > Tuesday) unless others

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-22 Thread Amit Kapila
On Tue, Feb 22, 2022 at 8:02 PM Masahiko Sawada wrote: > > On Tue, Feb 22, 2022 at 6:53 PM Amit Kapila wrote: > > > > > 3. Can we send error stats pgstat_report_stat() as that will be called > > via proc_exit() path. We can set the phase (apply/sync) in > > appl

Re: pg_stat_get_replication_slot and pg_stat_get_subscription_worker incorrectly marked as proretset

2022-02-22 Thread Amit Kapila
On Tue, Feb 22, 2022 at 8:15 AM Michael Paquier wrote: > > On Mon, Feb 21, 2022 at 05:00:43PM +0530, Amit Kapila wrote: > > On Mon, Feb 21, 2022 at 4:56 PM Michael Paquier wrote: > > > > It is still under discussion. I'll take the necessary action along > > wit

Re: Failed transaction statistics to measure the logical replication progress

2022-02-22 Thread Amit Kapila
ending stats instead of using pgstat_report_stat. I think the one thing which we could improve is to avoid trying to send it each time before receiving each message by walrcv_receive and rather try to send it before we try to wait (WaitLatchOrSocket). Trying after each message doesn't seem to be required and could lead to some overhead as well. What do you think? -- With Regards, Amit Kapila.

Re: Logical replication timeout problem

2022-02-23 Thread Amit Kapila
+ */ + skipped_changes_count = 0 When the skipped_changes_count is reset, the sendTime should also be reset. Can we reset it whenever the UpdateProgress function is called with send_keep_alive as false? [1] - https://www.postgresql.org/message-id/TYAPR01MB5866BD2248EF82FF432FE599F52D9%40TYAPR01MB5866.jpnprd01.prod.outlook.com -- With Regards, Amit Kapila.

Re: logical decoding and replication of sequences

2022-02-23 Thread Amit Kapila
;FOR ALL TABLES" (and same for sequences) is > hard-coded in the grammar, not defined as PublicationObjSpec. This also > means you can't do > > ALTER PUBLICATION p ADD ALL TABLES; > > AFAICS there are two ways to fix this - adding the combinations into the > definition of CreatePublicationStmt, or adding FOR ALL TABLES (and > sequences) to PublicationObjSpec. > I can imagine that adding to PublicationObjSpec will look compatible with existing code but maybe another way will also be okay. [1] - https://www.postgresql.org/message-id/877603.1629120678%40sss.pgh.pa.us -- With Regards, Amit Kapila.

Re: row filtering for logical replication

2022-02-23 Thread Amit Kapila
PUBLICATION pub1 FOR TABLE data1,data2 WHERE (c1 < 1000); > CREATE PUBLICATION postgres=# CREATE PUBLICATION pub_data_1 FOR TABLE data1 WHERE (c1 > 10), data2 WHERE (c1 < 1000); CREATE PUBLICATION -- With Regards, Amit Kapila.

Re: row filtering for logical replication

2022-02-23 Thread Amit Kapila
ted row > might not be found on the subscriber side. > Your analysis seems correct to me and it is evident from the result as well. Reviewing the test, it seems other similar places already have the catchup but it is missed after this update test. > I will post a patch to fix this shortly. > Thanks. -- With Regards, Amit Kapila.

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-23 Thread Amit Kapila
y with the description: "Number of times the error occurred during the initial table synchronization". -- With Regards, Amit Kapila.

Re: speed up a logical replica setup

2022-02-23 Thread Amit Kapila
tabases (cluster) to see how it impacts the time for small to large size databases as compared to the traditional method? This might help giving users advice on when to use this tool? -- With Regards, Amit Kapila.

Re: Optionally automatically disable logical replication subscriptions on error

2022-02-24 Thread Amit Kapila
nternal > +* error occurs in the following code. > +*/ > + EmitErrorReport(); > + AbortOutOfAnyTransaction(); > + FlushErrorState(); > > Do we need to hold interrupts during cleanup here? > I think so. We do prevent interrupts via HOLD_INTER

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-24 Thread Amit Kapila
ut that at end. It is better to add it near to similar RESET enums. > > 13. src/test/subscription/t/026_worker_stats.pl - missing test? > > Shouldn't there also be some test to reset the counters to confirm > that they really do get reset to zero? > > ~~~ > I think we avoid writing tests for stats for each and every case as it is not reliable in nature (the message can be lost). If we can find a reliable way then it is okay. -- With Regards, Amit Kapila.

Re: why do hash index builds use smgrextend() for new splitpoint pages

2022-02-24 Thread Amit Kapila
apping? > We allocate the chunk of pages (power-of-2 groups) at the time of split which allows them to appear consecutively in an index. This helps us to compute the physical block number from bucket number easily (BUCKET_TO_BLKNO mapping) with some minimal control information. -- With Regards, Amit Kapila.

Re: pg_stat_get_replication_slot and pg_stat_get_subscription_worker incorrectly marked as proretset

2022-02-24 Thread Amit Kapila
On Thu, Feb 24, 2022 at 12:03 PM Michael Paquier wrote: > > On Wed, Feb 23, 2022 at 09:52:02AM +0530, Amit Kapila wrote: > > Thanks, so you are okay with me pushing that patch just to HEAD. > > Yes, I am fine with that. I am wondering about patching the second > function

Re: why do hash index builds use smgrextend() for new splitpoint pages

2022-02-24 Thread Amit Kapila
On Fri, Feb 25, 2022 at 8:54 AM Amit Kapila wrote: > > On Fri, Feb 25, 2022 at 4:41 AM Melanie Plageman > wrote: > > > > I'm trying to understand why hash indexes are built primarily in shared > > buffers except when allocating a new splitpoint's worth of b

Re: Optionally automatically disable logical replication subscriptions on error

2022-02-24 Thread Amit Kapila
On Thu, Feb 24, 2022 at 6:30 PM Masahiko Sawada wrote: > > On Thu, Feb 24, 2022 at 8:08 PM Amit Kapila wrote: > > > > On Thu, Feb 24, 2022 at 1:20 PM Masahiko Sawada > > wrote: > > > > > > Here are some comments: > > > > > >

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-24 Thread Amit Kapila
oid > > +pgstat_reset_subscription_counter(Oid subid) > > > > SUGGESTION (function name) > > "pgstat_reset_subscription_counter" --> > > "pgstat_reset_subscription_counters" (plural) > > Fixed. > We don't use the plural form in other similar cases like pgstat_reset_replslot_counter, pgstat_reset_slru_counter, so why do it differently here? -- With Regards, Amit Kapila.

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-25 Thread Amit Kapila
was not clear. Do let me know what you think of the attached? -- With Regards, Amit Kapila. v4-0001-Reconsider-pg_stat_subscription_workers-view.patch Description: Binary data

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-25 Thread Amit Kapila
avoid that word altogether? > > SUGGESTIONS > > e.g.1. "One row per subscription, showing statistics about > subscription activity." --> "One row per subscription, showing > statistics about errors." > I preferred this one and made another change suggested by you in the latest version posted by me. Thanks! -- With Regards, Amit Kapila.

Re: why do hash index builds use smgrextend() for new splitpoint pages

2022-02-25 Thread Amit Kapila
est for it via sgmrextend before meta page update. -- With Regards, Amit Kapila.

Re: Fix typo in logicalfuncs.c - :%s/private date/Private data

2022-02-25 Thread Amit Kapila
On Fri, Feb 25, 2022 at 8:39 PM Bharath Rupireddy wrote: > > Hi, > > Here's a tiny patch to do $subject. > LGTM. I'll push this in some time. -- With Regards, Amit Kapila.

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-27 Thread Amit Kapila
On Sat, Feb 26, 2022 at 1:35 PM osumi.takami...@fujitsu.com wrote: > > On Saturday, February 26, 2022 11:51 AM Amit Kapila > wrote: > > I have reviewed the latest version and made a few changes along with fixing > > some of the pending comments by Peter Smith. The changes a

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-27 Thread Amit Kapila
On Mon, Feb 28, 2022 at 8:17 AM Masahiko Sawada wrote: > > On Mon, Feb 28, 2022 at 11:33 AM Amit Kapila wrote: > > > > > > > > (2) doc/src/sgml/monitoring.sgml > > > > > > +Resets statistics for a single subscription shown in the &

Re: Commitfest manager for 2022-03

2022-02-27 Thread Amit Kapila
ink Amit forgot to mark it as committed. Done. > I was waiting for some build farm cycles to complete but it is fine to mark it now. -- With Regards, Amit Kapila.

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-27 Thread Amit Kapila
ter to keep the view name in this description important as we have another view (pg_stat_susbcription) as well. So, I am planning to retain the current wording. -- With Regards, Amit Kapila.

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-27 Thread Amit Kapila
On Mon, Feb 28, 2022 at 8:49 AM osumi.takami...@fujitsu.com wrote: > > On Monday, February 28, 2022 11:34 AM Amit Kapila > wrote: > > On Sat, Feb 26, 2022 at 1:35 PM osumi.takami...@fujitsu.com > > wrote: > > > > > > On Saturday, February 26, 2022 11:

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-27 Thread Amit Kapila
On Mon, Feb 28, 2022 at 8:17 AM Masahiko Sawada wrote: > > On Mon, Feb 28, 2022 at 11:33 AM Amit Kapila wrote: > > > > > > > > (2) doc/src/sgml/monitoring.sgml > > > > > > +Resets statistics for a single subscription shown in the &

Re: why do hash index builds use smgrextend() for new splitpoint pages

2022-02-27 Thread Amit Kapila
On Sat, Feb 26, 2022 at 9:17 PM Melanie Plageman wrote: > > On Fri, Feb 25, 2022 at 11:17 PM Amit Kapila wrote: > > > > On Sat, Feb 26, 2022 at 3:01 AM Melanie Plageman > > wrote: > > > > > > Since _hash_alloc_buckets() WAL-logs the last page of th

Re: logical decoding and replication of sequences

2022-02-28 Thread Amit Kapila
34,7 @@ table public.replication_example: INSERT: id[integer]:7 somedata[integer]:6 text[character varying]:null testcolumn1[integer]:4 testcolumn2[integer]:null table public.replication_example: INSERT: id[integer]:8 somedata[integer]:7 text[character varying]:null testcolumn1[integer]:5 testcolumn2[integer]:null testcolumn3[integer]:1 COMMIT - (15 rows) + (16 rows) -- With Regards, Amit Kapila.

Re: Add the replication origin name and commit-LSN to logical replication worker errcontext

2022-02-28 Thread Amit Kapila
ll be helpful for users to set the origin name and LSN to > pg_replication_origin_advance(). > +1. This will make the use of pg_replication_origin_advance() relatively easy. -- With Regards, Amit Kapila.

Re: logical decoding and replication of sequences

2022-03-01 Thread Amit Kapila
On Mon, Feb 28, 2022 at 5:16 PM Amit Kapila wrote: > > On Sat, Feb 12, 2022 at 6:04 AM Tomas Vondra > wrote: > > > > On 2/10/22 19:17, Tomas Vondra wrote: > > > I've polished & pushed the first part adding sequence decoding > > > infrastr

Re: Design of pg_stat_subscription_workers vs pgstats

2022-03-01 Thread Amit Kapila
On Mon, Feb 28, 2022 at 1:15 PM tanghy.f...@fujitsu.com wrote: > > On Mon, Feb 28, 2022 12:32 PM Amit Kapila wrote: > > > > I decided to keep this part of the docs as it is and fixed a few other > > minor comments raised by you and Peter. Additionally, I have bumped >

Re: pg_stat_get_replication_slot and pg_stat_get_subscription_worker incorrectly marked as proretset

2022-03-01 Thread Amit Kapila
On Fri, Feb 25, 2022 at 8:57 AM Amit Kapila wrote: > > On Thu, Feb 24, 2022 at 12:03 PM Michael Paquier wrote: > > > > On Wed, Feb 23, 2022 at 09:52:02AM +0530, Amit Kapila wrote: > > > Thanks, so you are okay with me pushing that patch just to HEAD. > > &g

Re: Add the replication origin name and commit-LSN to logical replication worker errcontext

2022-03-01 Thread Amit Kapila
quot; : "removed"); The word order within the sentence might be different in other languages. -- With Regards, Amit Kapila.

Re: Add the replication origin name and commit-LSN to logical replication worker errcontext

2022-03-01 Thread Amit Kapila
44] DETAIL: Key (c1)=(1) already exists. 2022-02-26 07:45:25.092 IST [17644] CONTEXT: processing remote data during "INSERT" for replication target relation "public.t1" in transaction 724 at 2022-02-26 07:45:09.083848+05:30 Now, here, won't the time at the starting of CONTEXT serves our purpose. If we can remove the timestamp, I think the message won't appear too long. What do you think? -- With Regards, Amit Kapila.

Re: Design of pg_stat_subscription_workers vs pgstats

2022-03-01 Thread Amit Kapila
with slru/slots, so we providied it via pg_stat_reset_subscription_stats. [1] - https://www.postgresql.org/message-id/TYWPR01MB8362B30A904274A911C0B1CCED039%40TYWPR01MB8362.jpnprd01.prod.outlook.com -- With Regards, Amit Kapila.

Re: Add the replication origin name and commit-LSN to logical replication worker errcontext

2022-03-01 Thread Amit Kapila
On Wed, Mar 2, 2022 at 9:33 AM Masahiko Sawada wrote: > > On Wed, Mar 2, 2022 at 12:21 PM Amit Kapila wrote: > > > > On Wed, Mar 2, 2022 at 8:25 AM Peter Smith wrote: > > > > > > On Mon, Feb 28, 2022 at 11:16 PM Masahiko Sawada > > > wrote: > &

Re: PG DOCS - logical replication filtering

2022-03-02 Thread Amit Kapila
#x27;s not, I believe the > commented part should be removed before committing. > I think we can remove any Column Filters related information (placeholders), if that patch gets committed, we can always extend the existing docs. -- With Regards, Amit Kapila.

Re: Handle infinite recursion in logical replication setup

2022-03-02 Thread Amit Kapila
above case for tablesync phase, we make both nodes to start a transaction, create a slot on another node (with USE_SNAPSHOT option), and then allow copy from another node. I think it will be important to allow copy on each node once the slots are created and the initial snapshot is established. For more than two nodes, I think we can suggest having either of the option-1 or 2 for setup. But, there could be other ways as well depending on how the user wants to do the setup. Thoughts? -- With Regards, Amit Kapila.

Re: Design of pg_stat_subscription_workers vs pgstats

2022-03-02 Thread Amit Kapila
ld prefer to use the same function to reset all stats of that subsytem. Now, for WAL, bgwriter, etc., we don't want to reset any specific counter, so doing it via a shared function is okay but not for others. [1] - https://www.postgresql.org/docs/devel/monitoring-stats.html#MONITORING-STATS-FUNCTIONS -- With Regards, Amit Kapila.

Re: row filtering for logical replication

2022-03-02 Thread Amit Kapila
Rp+ testpub5 > > IIRC this test was written before adding the row filter information into the > psql. We could add \d+ testpub_rf_tbl3 before and after the modification. > Agreed. We can use \d instead of \d+ as row filter is available with \d. > 2) There are no pg_dump tests. > > WFM. > This is a miss. I feel we can add a few more. -- With Regards, Amit Kapila.

Re: PG DOCS - logical replication filtering

2022-03-02 Thread Amit Kapila
true. I know that that patch later specifies that "Row filters have no effect for TRUNCATE commands." but the above modification is not very clear. [1] - https://www.postgresql.org/docs/devel/logical-replication.html -- With Regards, Amit Kapila.

Re: Add the replication origin name and commit-LSN to logical replication worker errcontext

2022-03-02 Thread Amit Kapila
On Wed, Mar 2, 2022 at 1:05 PM Masahiko Sawada wrote: > > On Wed, Mar 2, 2022 at 4:14 PM Amit Kapila wrote: > > I've attached updated patches. > The first patch LGTM. Some comments on the second patch: 1. @@ -3499,6 +3503,17 @@ ApplyWorkerMain(Datum main_arg) myslotname =

Re: Skipping logical replication transactions on subscriber side

2022-03-03 Thread Amit Kapila
to apply changes. IOW it's cleared after > either skipping the whole transaction or successfully committing a > non-empty transaction, preventing the skip-LSN to remain in the > catalog. Also, since the latter case means that the user set the wrong > skip-LSN we clear it with a warning. > As this will be displayed only in server logs and by background apply worker, should it be LOG or WARNING? -- With Regards, Amit Kapila.

Re: row filtering for logical replication

2022-03-03 Thread Amit Kapila
On Thu, Mar 3, 2022 at 11:18 AM shiy.f...@fujitsu.com wrote: > > On Thu, Mar 3, 2022 10:40 AM Amit Kapila wrote: > > > > On Wed, Mar 2, 2022 at 5:42 PM Euler Taveira wrote: > > > > > > On Wed, Mar 2, 2022, at 8:45 AM, Tomas Vondra wrote: > > > >

Re: Add the replication origin name and commit-LSN to logical replication worker errcontext

2022-03-03 Thread Amit Kapila
BLE first. Then, the transaction can be skipped by calling the pg_replication_origin_advance() function with the node_name (i.e., pg_16395) and the next LSN of the commit LSN (i.e., LSN 0/14C0379). -- With Regards, Amit Kapila.

Re: Add the replication origin name and commit-LSN to logical replication worker errcontext

2022-03-03 Thread Amit Kapila
appropriate way. But if others also think it should do that too > along with this update, I'm happy to add tests. > I also don't see a reason why this patch should add any tests related to origin. -- With Regards, Amit Kapila.

Re: Add the replication origin name and commit-LSN to logical replication worker errcontext

2022-03-03 Thread Amit Kapila
the pg_replication_origin_advance(), right ? > If this is true, I wasn't sure if the name 'commit_lsn' is the > most accurate name for this variable. Should we adjust the name a bit ? > If we want to change this variable name, the other options could be 'end_lsn', or 'finish_lsn'. -- With Regards, Amit Kapila.

Re: Column Filtering in Logical Replication

2022-03-04 Thread Amit Kapila
, and the column "a" is part of replica identity, but it is filtered, which caused an error on the subscriber side. ERROR: publisher did not send replica identity column expected by the logical replication target relation "public.tbl" CONTEXT: processing remote data during "UPDATE" for replication target relation "public.tbl" in transaction 723 at 2022-03-04 11:56:33.905843+08 -- With Regards, Amit Kapila.

Re: Add the replication origin name and commit-LSN to logical replication worker errcontext

2022-03-04 Thread Amit Kapila
On Fri, Mar 4, 2022 at 6:02 PM Euler Taveira wrote: > > On Fri, Mar 4, 2022, at 2:54 AM, Amit Kapila wrote: > > The LSN of the transaction that contains the change violating the > constraint and the replication origin name can be found from the > server log (LSN 0/14C0378 and

Re: row filtering for logical replication

2022-03-06 Thread Amit Kapila
On Mon, Mar 7, 2022 at 12:50 AM Tomas Vondra wrote: > > On 3/3/22 21:07, Euler Taveira wrote: > > On Thu, Mar 3, 2022, at 7:47 AM, Amit Kapila wrote: > >> LGTM. I'll push this tomorrow unless Tomas or Euler feels otherwise. > > Sounds good to me. >

Re: Handle infinite recursion in logical replication setup

2022-03-06 Thread Amit Kapila
gainst this case? > In the above topology if you want local changes from both node3 and node2 then I think the way to get that would be you have to create two subscriptions on node1. The first one points to node2 (with only_local=off) and the second one points to node3 (with only_local =off). Will that address your case or am I missing something? -- With Regards, Amit Kapila.

Re: Add the replication origin name and commit-LSN to logical replication worker errcontext

2022-03-06 Thread Amit Kapila
his tomorrow unless there are more comments. -- With Regards, Amit Kapila. v6-0001-Add-the-additional-information-to-the-logical-rep.patch Description: Binary data

Re: Handle infinite recursion in logical replication setup

2022-03-06 Thread Amit Kapila
On Mon, Mar 7, 2022 at 10:05 AM Amit Kapila wrote: > > On Mon, Mar 7, 2022 at 9:26 AM Dilip Kumar wrote: > > > > On Wed, Feb 23, 2022 at 11:59 AM vignesh C wrote: > > > Here there are two problems for the user: a) incremental > > > synchronization of table se

Re: Optionally automatically disable logical replication subscriptions on error

2022-03-07 Thread Amit Kapila
the > existing PG_RE_THROW logic, which in turn was only possible using > special knowledge that this just happens to be the last try/catch > block for the apply worker. > I think we should re_throw the error in case we have not handled it by disabling the subscription (in which case we can exit with success code (0)). -- With Regards, Amit Kapila.

Re: Handle infinite recursion in logical replication setup

2022-03-07 Thread Amit Kapila
ion as an > option while creating publication or providing only_local option as an > option while creating subscription as Peter has pointed out at [1]. > option 1: > CREATE PUBLICATION p1 FOR TABLE t1; > CREATE SUBSCRITION s1 ... FOR PUBLICATION p1 WITH (only_local = true); > > option 2: > CREATE PUBLICATION p1 FOR TABLE t1 WITH (publish = 'only_local'); > CREATE SUBSCRITION s1 ... FOR PUBLICATION p1; > > Shall we get a few opinions on this and take it in that direction? > Allowing multiple publications will lead to a lot of duplicate data in catalogs like pg_publication_rel, so, I don't see how option-2 can be beneficial? -- With Regards, Amit Kapila.

Re: Handle infinite recursion in logical replication setup

2022-03-07 Thread Amit Kapila
On Mon, Mar 7, 2022 at 1:11 PM Dilip Kumar wrote: > > On Mon, Mar 7, 2022 at 10:15 AM Amit Kapila wrote: > > > > I haven't yet gone through the patch, but I have a question about the > > > > idea. Suppose I want to set up a logical replication like, > >

Re: Handle infinite recursion in logical replication setup

2022-03-07 Thread Amit Kapila
for the reasons specified by Vignesh in his previous email [1]. Now, if we think those all are ignorable things and it is more important to allow this option first by publication or we must allow it via both publication and subscription then it makes sense to change it. [1] - https://www.postgresql.org/message-id/CALDaNm3jkotRhKfCqu5CXOf36_yiiW_cYE5%3DbG%3Dj6N3gOWJkqw%40mail.gmail.com -- With Regards, Amit Kapila.

Re: Handle infinite recursion in logical replication setup

2022-03-07 Thread Amit Kapila
consider is whether exposing the boolean option is okay or shall we do something different so that we can extend it later to specific origin ids? -- With Regards, Amit Kapila.

Re: Optionally automatically disable logical replication subscriptions on error

2022-03-07 Thread Amit Kapila
the transaction, the subscription needs to be disabled temporarily by ALTER SUBSCRIPTION ... DISABLE first or alternatively, the subscription can be used with the disable_on_error option. Feel free to use something on the above lines, if you agree. -- With Regards, Amit Kapila.

Re: Handle infinite recursion in logical replication setup

2022-03-07 Thread Amit Kapila
On Tue, Mar 8, 2022 at 12:17 PM Peter Smith wrote: > > On Tue, Mar 8, 2022 at 4:21 PM Amit Kapila wrote: > > > > On Tue, Mar 8, 2022 at 10:31 AM Peter Smith wrote: > > > > > > IIUC the new option may be implemented subscriber-side and/or > > > publis

Re: Add the replication origin name and commit-LSN to logical replication worker errcontext

2022-03-08 Thread Amit Kapila
On Mon, Mar 7, 2022 at 10:06 AM Amit Kapila wrote: > > On Mon, Mar 7, 2022 at 6:36 AM Masahiko Sawada wrote: > > > > Thank you for the comment. +1. > > > > I've attached updated patches. > > > > Pushed the first patch. Fixed one typo in the second

Re: Optionally automatically disable logical replication subscriptions on error

2022-03-08 Thread Amit Kapila
_subscription_rel sr +WHERE sr.srsubstate IN ('s', 'r') AND sr.srrelid = 'tbl'::regclass)); See other scripts like t/015_stream.pl and wait for data replication in the same way. I think there are two things to change: (a) In the above query, we use NOT IN at other places (b) use $node_publisher->wait_for_catchup before this query. -- With Regards, Amit Kapila.

Re: Optionally automatically disable logical replication subscriptions on error

2022-03-08 Thread Amit Kapila
PG_ENSURE_ERROR_CLEANUP construct. What do you think? -- With Regards, Amit Kapila.

Re: Optionally automatically disable logical replication subscriptions on error

2022-03-08 Thread Amit Kapila
On Tue, Mar 8, 2022 at 6:53 PM Amit Kapila wrote: > > On Tue, Mar 8, 2022 at 1:37 PM osumi.takami...@fujitsu.com > wrote: > > > > Kindly have a look at v30. > > > > Review comments: > === > Few comments on test script: === 1.

Re: Optionally automatically disable logical replication subscriptions on error

2022-03-08 Thread Amit Kapila
On Wed, Mar 9, 2022 at 11:22 AM Masahiko Sawada wrote: > > On Wed, Mar 9, 2022 at 12:37 PM Amit Kapila wrote: > > > > On Wed, Mar 9, 2022 at 6:29 AM Masahiko Sawada > > wrote: > > > > > > --- > > > It might have already been discussed but the

Re: Column Filtering in Logical Replication

2022-03-09 Thread Amit Kapila
On Mon, Mar 7, 2022 at 8:48 PM Tomas Vondra wrote: > > On 3/4/22 11:42, Amit Kapila wrote: > > On Wed, Mar 2, 2022 at 5:43 PM Tomas Vondra > > wrote: > >> > >> Attached is an updated patch, addressing most of the issues reported so > >> far. There a

Re: Optionally automatically disable logical replication subscriptions on error

2022-03-09 Thread Amit Kapila
On Wed, Mar 9, 2022 at 2:21 PM Masahiko Sawada wrote: > > On Wed, Mar 9, 2022 at 4:33 PM osumi.takami...@fujitsu.com > wrote: > > > > On Tuesday, March 8, 2022 10:23 PM Amit Kapila > > wrote: > > > On Tue, Mar 8, 2022 at 1:37 PM osumi.takami...@fujitsu.com &

Re: logical decoding and replication of sequences

2022-03-09 Thread Amit Kapila
fter you pushed a fix? I don't think so or am I missing something? I feel even if doesn't occur again it would have been better if we had some theory on how it occurred in the first place because that would make us feel more confident that we won't have any related problem left. -- With Regards, Amit Kapila.

Re: Column Filtering in Logical Replication

2022-03-09 Thread Amit Kapila
On Wed, Mar 9, 2022 at 3:33 PM Amit Kapila wrote: > > On Mon, Mar 7, 2022 at 8:48 PM Tomas Vondra > wrote: > > > OK, I reworked this to do the same thing as the row filtering patch. > > > > Thanks, I'll check this. > Some assorted comments: ==

Re: Optionally automatically disable logical replication subscriptions on error

2022-03-09 Thread Amit Kapila
there are any other major comments. -- With Regards, Amit Kapila. v33-0001-Optionally-disable-subscriptions-on-error.patch Description: Binary data

Re: Skipping logical replication transactions on subscriber side

2022-03-10 Thread Amit Kapila
+ 'postgresql.conf', + qq[ +wal_retrieve_retry_interval = 2s +]); Can we change this test to use disable_on_error feature? I am thinking if the disable_on_error feature got committed first, maybe we can have one test file for this and disable_on_error feature (something like conflicts.pl). -- With Regards, Amit Kapila.

Re: Column Filtering in Logical Replication

2022-03-10 Thread Amit Kapila
On Fri, Mar 11, 2022 at 12:44 AM Tomas Vondra wrote: > > On 3/10/22 04:09, Amit Kapila wrote: > > On Wed, Mar 9, 2022 at 3:33 PM Amit Kapila wrote: > >> > >> On Mon, Mar 7, 2022 at 8:48 PM Tomas Vondra > >> wrote: > >> > >>> OK, I r

Re: Column Filtering in Logical Replication

2022-03-10 Thread Amit Kapila
x it separately than to fix it along with row filter or column filter work. > In any case, this sync issue looks > entirely unrelated to the column filtering patch. > Right. [1] - https://www.postgresql.org/message-id/OS0PR01MB5716DC2982CC735FDE388804940B9%40OS0PR01MB5716.jpnprd01.prod.outlook.com -- With Regards, Amit Kapila.

Re: Column Filtering in Logical Replication

2022-03-11 Thread Amit Kapila
to ensure that publish_as_relid exists after ancestor in ancestors list before assigning the ancestor to publish_as_relid? This only needs to be done in case of (if (!publish)). I have not tried this so I could be wrong. -- With Regards, Amit Kapila.

Re: logical decoding and replication of sequences

2022-03-11 Thread Amit Kapila
in some way we can increase checkpoint timeout or other parameters for these tests then it would reduce the chances of such failures. The other idea could be to perform checkpoint before the start of tests to reduce the possibility of another checkpoint. -- With Regards, Amit Kapila.

Re: logical decoding and replication of sequences

2022-03-11 Thread Amit Kapila
On Fri, Mar 11, 2022 at 5:04 PM Amit Kapila wrote: > > On Tue, Mar 8, 2022 at 11:59 PM Tomas Vondra > wrote: > > > > On 3/7/22 22:25, Tomas Vondra wrote: > > >> > > >> Interesting. I can think of one reason that might cause this - we log > > >

Re: Issue with pg_stat_subscription_stats

2022-03-11 Thread Amit Kapila
156971+05:30 (1 row) The stats_reset time seems to be populated. Similarly, I have tried by passing NULL to pg_stat_reset_subscription_stats and it works. I think I am missing something here, can you please explain the exact scenario/steps where you observed that this API is not working. -- With Regards, Amit Kapila.

Re: Column Filtering in Logical Replication

2022-03-11 Thread Amit Kapila
On Fri, Mar 11, 2022 at 6:20 PM Tomas Vondra wrote: > > On 3/11/22 10:52, Amit Kapila wrote: > > On Fri, Mar 11, 2022 at 7:26 AM Tomas Vondra > > wrote: > >> > >> On 3/10/22 20:10, Tomas Vondra wrote: > >>> > >>> > >>>

Re: Column Filtering in Logical Replication

2022-03-14 Thread Amit Kapila
On Mon, Mar 14, 2022 at 2:37 AM Tomas Vondra wrote: > > On 3/12/22 05:30, Amit Kapila wrote: > >> ... > > > > Okay, please find attached. I have done basic testing of this, if we > > agree with this approach then this will require some more testing. > > >

Re: Optionally automatically disable logical replication subscriptions on error

2022-03-14 Thread Amit Kapila
On Thu, Mar 10, 2022 at 12:04 PM Amit Kapila wrote: > > On Wed, Mar 9, 2022 at 7:57 PM osumi.takami...@fujitsu.com > wrote: > > > > Hi, attached v32 removed my additional code for maybe_reread_subscription. > > > > Thanks, the patch looks good to me. I have made

Re: Column Filtering in Logical Replication

2022-03-14 Thread Amit Kapila
.org/message-id/os0pr01mb5716dc2982cc735fde38880494...@os0pr01mb5716.jpnprd01.prod.outlook.com -- With Regards, Amit Kapila.

Re: Issue with pg_stat_subscription_stats

2022-03-14 Thread Amit Kapila
On Sun, Mar 13, 2022 at 1:45 AM Andres Freund wrote: > > On 2022-03-12 08:28:35 +0530, Amit Kapila wrote: > > On Sat, Mar 12, 2022 at 2:14 AM Melanie Plageman > > wrote: > > > > > > So, I noticed that pg_stat_reset_subscription_stats() wasn't

Re: Column Filtering in Logical Replication

2022-03-14 Thread Amit Kapila
On Mon, Mar 14, 2022 at 4:42 PM houzj.f...@fujitsu.com wrote: > > On Monday, March 14, 2022 5:08 AM Tomas Vondra > wrote: > > > > On 3/12/22 05:30, Amit Kapila wrote: > > >> ... > > > > > > Okay, please find attached. I have done basic testing

Re: Column Filtering in Logical Replication

2022-03-14 Thread Amit Kapila
On Mon, Mar 14, 2022 at 7:02 PM Tomas Vondra wrote: > > On 3/14/22 13:47, Amit Kapila wrote: > > On Mon, Mar 14, 2022 at 5:42 PM Tomas Vondra > > wrote: > >> > >> On 3/14/22 12:12, houzj.f...@fujitsu.com wrote: > >>> On Monday, March 14, 2022 5:08

Re: Skipping logical replication transactions on subscriber side

2022-03-15 Thread Amit Kapila
* occurs */ + XLogRecPtr skiplsn; /* All changes which finished at this LSN are + * skipped */ No need for 'which' in the above comments. 9. Can we merge 029_disable_on_error in 030_skip_xact and name it as 029_on_error (or 029_on_error_skip_disable or some variant of it)? Both seem

Re: Skipping logical replication transactions on subscriber side

2022-03-15 Thread Amit Kapila
On Wed, Mar 16, 2022 at 6:03 AM Masahiko Sawada wrote: > > On Tue, Mar 15, 2022 at 7:18 PM Amit Kapila wrote: > > > > On Tue, Mar 15, 2022 at 11:43 AM Masahiko Sawada > > wrote: > > > > > > > 6. > > @@ -1583,7 +1649,8 @@ apply_handle_in

Re: Skipping logical replication transactions on subscriber side

2022-03-15 Thread Amit Kapila
) is not interested. > I think when we skip non-streamed transactions we don't start a transaction. So, if we do what you are suggesting, we will miss to clear the skip_lsn after skipping the transaction. -- With Regards, Amit Kapila.

Re: Skipping logical replication transactions on subscriber side

2022-03-15 Thread Amit Kapila
On Wed, Mar 16, 2022 at 7:58 AM Amit Kapila wrote: > > On Wed, Mar 16, 2022 at 6:03 AM Masahiko Sawada wrote: > > > > On Tue, Mar 15, 2022 at 7:18 PM Amit Kapila wrote: > > > > > > On Tue, Mar 15, 2022 at 11:43 AM Masahiko Sawada > > > wrote: &g

Re: Skipping logical replication transactions on subscriber side

2022-03-15 Thread Amit Kapila
On Wed, Mar 16, 2022 at 7:58 AM Amit Kapila wrote: > > On Wed, Mar 16, 2022 at 6:03 AM Masahiko Sawada wrote: > > > > On Tue, Mar 15, 2022 at 7:18 PM Amit Kapila wrote: > > > > > > On Tue, Mar 15, 2022 at 11:43 AM Masahiko Sawada > > > wrote: &g

Re: Tablesync early exit

2022-03-15 Thread Amit Kapila
inning of function LogicalRepApplyLoop(), last_received is the LSN where the copy has finished in the case of tablesync worker. I think we need to receive the message before trying to ensure whether we have synced with the apply worker or not. -- With Regards, Amit Kapila.

Re: Tablesync early exit

2022-03-15 Thread Amit Kapila
king closely at this, I am not sure whether this is a good idea or not. Responded accordingly. -- With Regards, Amit Kapila.

  1   2   3   4   5   6   7   8   9   10   >