RE: Optionally automatically disable logical replication subscriptions on error

2022-02-20 Thread osumi.takami...@fujitsu.com
On Friday, February 18, 2022 3:27 PM Peter Smith wrote: > Hi. Below are my code review comments for v18. Thank you for your review ! > == > > 1. Commit Message - wording > > BEFORE > To partially remedy the situation, adding a new subscription_parameter named > 'disable_on_error'. > >

RE: Failed transaction statistics to measure the logical replication progress

2022-02-20 Thread osumi.takami...@fujitsu.com
On Saturday, February 19, 2022 12:00 AM osumi.takami...@fujitsu.com wrote: > On Friday, February 18, 2022 3:34 PM Tang, Haiying/唐 海英 > wrote: > > On Wed, Jan 12, 2022 8:35 PM osumi.takami...@fujitsu.com > > wrote: > > 4) I noticed that the abort_count doesn

RE: Failed transaction statistics to measure the logical replication progress

2022-02-21 Thread osumi.takami...@fujitsu.com
On Monday, February 21, 2022 6:06 PM Monday, February 21, 2022 6:06 PM wrote: > On Mon, Feb 21, 2022 11:46 AM Osumi, Takamichi/大墨 昂道 > wrote: > >I've addressed this point in a new v23 patch, since there was no opinion on > this so far. > >Kindly have a look at the attached one. > Thanks for updati

RE: Optionally automatically disable logical replication subscriptions on error

2022-02-21 Thread osumi.takami...@fujitsu.com
On Monday, February 21, 2022 2:56 PM Peter Smith wrote: > Thanks for addressing my previous comments. Now I have looked at v19. > > On Mon, Feb 21, 2022 at 11:25 AM osumi.takami...@fujitsu.com > wrote: > > > > On Friday, February 18, 2022 3:27 PM Peter Smith > wr

RE: Failed transaction statistics to measure the logical replication progress

2022-02-21 Thread osumi.takami...@fujitsu.com
On Tuesday, February 22, 2022 10:15 AM Tang, Haiying/唐 海英 wrote: > On Mon, Feb 21, 2022 11:46 AM osumi.takami...@fujitsu.com > wrote: > > > > On Saturday, February 19, 2022 12:00 AM osumi.takami...@fujitsu.com > > wrote: > > > On Friday, February 18

RE: Optionally automatically disable logical replication subscriptions on error

2022-02-21 Thread osumi.takami...@fujitsu.com
On Tuesday, February 22, 2022 7:53 AM Peter Smith wrote: > On Mon, Feb 21, 2022 at 11:44 PM osumi.takami...@fujitsu.com > wrote: > > > > On Monday, February 21, 2022 2:56 PM Peter Smith > wrote: > > > Thanks for addressing my previous comments. Now I have looked a

RE: Design of pg_stat_subscription_workers vs pgstats

2022-02-22 Thread osumi.takami...@fujitsu.com
On Tuesday, February 22, 2022 2:45 PM Masahiko Sawada wrote: > I've attached a patch that changes pg_stat_subscription_workers view. > It removes non-cumulative values such as error details such as error-XID and > the error message from the view, and consequently the view now has only > cumulativ

RE: Design of pg_stat_subscription_workers vs pgstats

2022-02-22 Thread osumi.takami...@fujitsu.com
On Tuesday, February 22, 2022 11:47 PM Masahiko Sawada wrote: > On Tue, Feb 22, 2022 at 9:22 PM osumi.takami...@fujitsu.com > wrote: > > (4) doc/src/sgml/monitoring.sgml > > > > > > role="column_definition"> > > -

RE: Optionally automatically disable logical replication subscriptions on error

2022-02-23 Thread osumi.takami...@fujitsu.com
On Wednesday, February 23, 2022 6:52 PM Tang, Haiying/唐 海英 wrote: > I have a comment on v21 patch. > > I wonder if we really need subscription s2 in 028_disable_on_error.pl. I > think for > subscription s2, we only tested some normal cases(which could be tested with > s1), and didn't test any e

RE: Failed transaction statistics to measure the logical replication progress

2022-02-24 Thread osumi.takami...@fujitsu.com
On Thursday, February 24, 2022 11:07 AM Masahiko Sawada wrote: > I have some comments on v23 patch: > > @@ -66,6 +66,12 @@ typedef struct LogicalRepWorker > TimestampTz last_recv_time; > XLogRecPtr reply_lsn; > TimestampTz reply_time; > + > + /* > +* Transaction statistics of

RE: Failed transaction statistics to measure the logical replication progress

2022-02-24 Thread osumi.takami...@fujitsu.com
On Wednesday, February 23, 2022 3:30 PM Amit Kapila > On Tue, Feb 22, 2022 at 6:45 AM tanghy.f...@fujitsu.com > wrote: > > > > I found a problem when using it. When a replication workers exits, the > > transaction stats should be sent to stats collector if they were not > > sent before because it

RE: Optionally automatically disable logical replication subscriptions on error

2022-02-25 Thread osumi.takami...@fujitsu.com
On Friday, February 25, 2022 12:57 PM Amit Kapila wrote: > On Thu, Feb 24, 2022 at 6:30 PM Masahiko Sawada > wrote: > > > > On Thu, Feb 24, 2022 at 8:08 PM Amit Kapila > wrote: > > > > > > On Thu, Feb 24, 2022 at 1:20 PM Masahiko Sawada > wrote: > > > > > > > > Here are some comments: > > > >

RE: Optionally automatically disable logical replication subscriptions on error

2022-02-25 Thread osumi.takami...@fujitsu.com
On Thursday, February 24, 2022 8:09 PM Amit Kapila > On Thu, Feb 24, 2022 at 1:20 PM Masahiko Sawada > wrote: > > + /* > > +* Log the error that caused DisableSubscriptionOnError to be > called. We > > +* do this immediately so that it won't be lost if some other > > intern

RE: Optionally automatically disable logical replication subscriptions on error

2022-02-25 Thread osumi.takami...@fujitsu.com
On Thursday, February 24, 2022 4:50 PM Masahiko Sawada wrote: > On Tue, Feb 22, 2022 at 3:03 PM Peter Smith > wrote: > > > > ~~~ > > > > 1. worker.c - comment > > > > + subform = (Form_pg_subscription) GETSTRUCT(tup); > > + > > + /* > > + * We would not be here unless this subscription's disable

RE: Optionally automatically disable logical replication subscriptions on error

2022-02-25 Thread osumi.takami...@fujitsu.com
On Wednesday, February 23, 2022 6:52 PM Tang, Haiying/唐 海英 wrote: > I have a comment on v21 patch. > > I wonder if we really need subscription s2 in 028_disable_on_error.pl. I > think for > subscription s2, we only tested some normal cases(which could be tested with > s1), and didn't test any e

RE: Optionally automatically disable logical replication subscriptions on error

2022-02-25 Thread osumi.takami...@fujitsu.com
On Tuesday, February 22, 2022 3:03 PM Peter Smith wrote: > Here are a couple more review comments for v21. > > ~~~ > > 1. worker.c - comment > > + subform = (Form_pg_subscription) GETSTRUCT(tup); > + > + /* > + * We would not be here unless this subscription's disableonerr field > + was > + * t

RE: Design of pg_stat_subscription_workers vs pgstats

2022-02-26 Thread osumi.takami...@fujitsu.com
On Saturday, February 26, 2022 11:51 AM Amit Kapila wrote: > I have reviewed the latest version and made a few changes along with fixing > some of the pending comments by Peter Smith. The changes are as > follows: (a) Removed m_databaseid in PgStat_MsgSubscriptionError as that is > not required n

RE: Design of pg_stat_subscription_workers vs pgstats

2022-02-27 Thread osumi.takami...@fujitsu.com
On Monday, February 28, 2022 11:34 AM Amit Kapila wrote: > On Sat, Feb 26, 2022 at 1:35 PM osumi.takami...@fujitsu.com > wrote: > > > > On Saturday, February 26, 2022 11:51 AM Amit Kapila > wrote: > > > I have reviewed the latest version and made a few changes a

RE: Design of pg_stat_subscription_workers vs pgstats

2022-02-27 Thread osumi.takami...@fujitsu.com
On Monday, February 28, 2022 12:57 PM Amit Kapila > On Mon, Feb 28, 2022 at 8:49 AM osumi.takami...@fujitsu.com > wrote: > > > > On Monday, February 28, 2022 11:34 AM Amit Kapila > wrote: > > > On Sat, Feb 26, 2022 at 1:35 PM osumi.takami...@fujitsu.com > > &

RE: Failed transaction statistics to measure the logical replication progress

2022-02-28 Thread osumi.takami...@fujitsu.com
On Friday, February 25, 2022 7:58 AM osumi.takami...@fujitsu.com wrote: > Kindly have a look at v24. Hi. The recent commit(7a85073) has redesigned the view pg_stat_subscription_workers and now we have pg_stat_subscription_stats. Therefore, I rebased my patch so that my statistics patch can

RE: Optionally automatically disable logical replication subscriptions on error

2022-02-28 Thread osumi.takami...@fujitsu.com
On Friday, February 25, 2022 9:45 PM osumi.takami...@fujitsu.com wrote: > Kindly have a look at attached the v22. > It has incorporated other improvements of TAP test, refinement of the > DisableSubscriptionOnError function and so on. The recent commit(7a85073) has changed the sub

RE: Optionally automatically disable logical replication subscriptions on error

2022-02-28 Thread osumi.takami...@fujitsu.com
On Tuesday, March 1, 2022 9:49 AM Peter Smith wrote: > Please see below my review comments for v22. > > == > > 1. Commit message > > "table sync worker" -> "tablesync worker" Fixed. > ~~~ > > 2. doc/src/sgml/catalogs.sgml > > + > + If true, the subscription will be disabled w

RE: logical replication restrictions

2022-02-28 Thread osumi.takami...@fujitsu.com
On Tuesday, March 1, 2022 9:19 AM Euler Taveira wrote: > Long time, no patch. Here it is. I will provide documentation in the next > > version. I would appreciate some feedback. Hi, thank you for posting the patch ! $ git am v1-0001-Time-delayed-logical-replication-subscriber.patch Applying: T

RE: Failed transaction statistics to measure the logical replication progress

2022-03-01 Thread osumi.takami...@fujitsu.com
On Tuesday, March 1, 2022 4:12 PM Peter Smith wrote: > Please see below my review comments for v25. > > == > > 1. Commit message > > Introduce cumulative columns of transactions of logical replication subscriber > to the pg_stat_subscription_stats view. > > "cumulative columns of transacti

RE: logical replication restrictions

2022-03-01 Thread osumi.takami...@fujitsu.com
On Wednesday, March 2, 2022 8:54 AM Euler Taveira wrote: > On Tue, Mar 1, 2022, at 3:27 AM, osumi.takami...@fujitsu.com > <mailto:osumi.takami...@fujitsu.com> wrote: > > > $ git am v1-0001-Time-delayed-logical-replication-subscriber.patch > > > I generall

RE: Optionally automatically disable logical replication subscriptions on error

2022-03-01 Thread osumi.takami...@fujitsu.com
On Wednesday, March 2, 2022 9:34 AM Peter Smith wrote: > Please see below my review comments for v24. Thank you for checking my patch ! > == > > 1. src/backend/replication/logical/worker.c - start_table_sync > > + /* Report the worker failed during table synchronization */ > + pgstat_repo

RE: Optionally automatically disable logical replication subscriptions on error

2022-03-02 Thread osumi.takami...@fujitsu.com
On Wednesday, March 2, 2022 12:47 PM Masahiko Sawada wrote: > After more thoughts, should we do both AbortOutOfAnyTransaction() and error > message handling while holding interrupts? That is, > > HOLD_INTERRUPTS(); > EmitErrorReport(); > FlushErrorState(); > AbortOutOfAny Transaction(); > RESUME

RE: Failed transaction statistics to measure the logical replication progress

2022-03-02 Thread osumi.takami...@fujitsu.com
On Wednesday, March 2, 2022 2:18 PM Masahiko Sawada wrote: > On Wed, Mar 2, 2022 at 10:21 AM osumi.takami...@fujitsu.com > wrote: > > Also, I quickly checked other similar views(pg_stat_slru, > > pg_stat_wal_receiver) commit logs, especially when they introduce columns. > &

RE: Failed transaction statistics to measure the logical replication progress

2022-03-02 Thread osumi.takami...@fujitsu.com
On Wednesday, March 2, 2022 5:29 PM Shi, Yu/侍 雨 wrote: > A comments on the v26 patch. Thank you for checking the patch ! > > The following document about pg_stat_subscription_stats view only says that > "showing statistics about errors", should we add something about transactions > here? > >

RE: Add the replication origin name and commit-LSN to logical replication worker errcontext

2022-03-03 Thread osumi.takami...@fujitsu.com
On Friday, March 4, 2022 10:09 AM Masahiko Sawada wrote: > On Thu, Mar 3, 2022 at 10:02 PM Masahiko Sawada > wrote: > > > > > > I'm updating the patches and will submit them. > > Attached updated version patches. Thank you for sharing the patch v3. Few minor comments. (1) v03-0001, apply_erro

RE: Add the replication origin name and commit-LSN to logical replication worker errcontext

2022-03-03 Thread osumi.takami...@fujitsu.com
On Friday, March 4, 2022 2:23 PM Masahiko Sawada wrote: > I've attached updated patches. Hi, thank you for updating the patch. One comment on v4. In v4-0002, we introduce 'commit_lsn' in the ApplyErrorCallbackArg. This member is set for prepare, rollback prepared and stream_abort as well. The ne

RE: Optionally automatically disable logical replication subscriptions on error

2022-03-06 Thread osumi.takami...@fujitsu.com
On Friday, March 4, 2022 3:55 PM Masahiko Sawada wrote: > Thank you for updating the patch. > > Here are some comments on v26 patch: Thank you for your review ! > +/* > + * Disable the current subscription. > + */ > +static void > +DisableSubscriptionOnError(void) > > This function now just u

RE: Optionally automatically disable logical replication subscriptions on error

2022-03-06 Thread osumi.takami...@fujitsu.com
On Monday, March 7, 2022 12:01 PM Shi, Yu/侍 雨 wrote: > On Wed, Mar 2, 2022 5:39 PM osumi.takami...@fujitsu.com > wrote: > > > > Attached an updated patch v26. > > > > Thanks for your patch. A comment on the document. Hi, thank you for checking my patch ! > @@

RE: Optionally automatically disable logical replication subscriptions on error

2022-03-07 Thread osumi.takami...@fujitsu.com
On Monday, March 7, 2022 5:45 PM Amit Kaila wrote: > On Mon, Mar 7, 2022 at 4:55 AM Peter Smith > wrote: > > > > On Fri, Mar 4, 2022 at 5:55 PM Masahiko Sawada > wrote: > > > > > > --- > > > +/* > > > + * First, ensure that we log the error message so > > > that i

RE: Optionally automatically disable logical replication subscriptions on error

2022-03-08 Thread osumi.takami...@fujitsu.com
On Tuesday, March 8, 2022 2:52 PM Amit Kapila wrote: > On Tue, Mar 8, 2022 at 9:37 AM Peter Smith wrote: > > > > Please find below some review comments for v29. > > > > == > > > > 1. src/backend/replication/logical/worker.c - > > worker_post_error_processing > > > > +/* > > + * Abort and clea

RE: Optionally automatically disable logical replication subscriptions on error

2022-03-08 Thread osumi.takami...@fujitsu.com
On Tuesday, March 8, 2022 1:07 PM Peter Smith wrote: > Please find below some review comments for v29. Thank you for your comments ! > == > > 1. src/backend/replication/logical/worker.c - worker_post_error_processing > > +/* > + * Abort and cleanup the current transaction, then do post-e

RE: Optionally automatically disable logical replication subscriptions on error

2022-03-08 Thread osumi.takami...@fujitsu.com
On Wednesday, March 9, 2022 3:02 PM Amit Kapila wrote: > On Wed, Mar 9, 2022 at 11:22 AM Masahiko Sawada > wrote: > > > > On Wed, Mar 9, 2022 at 12:37 PM Amit Kapila > wrote: > > > > > > On Wed, Mar 9, 2022 at 6:29 AM Masahiko Sawada > wrote: > > > > > > > > --- > > > > It might have already be

RE: Optionally automatically disable logical replication subscriptions on error

2022-03-08 Thread osumi.takami...@fujitsu.com
On Wednesday, March 9, 2022 1:29 PM Amit Kapila wrote: > On Tue, Mar 8, 2022 at 6:53 PM Amit Kapila wrote: > > > > On Tue, Mar 8, 2022 at 1:37 PM osumi.takami...@fujitsu.com > > wrote: > > > > > > Kindly have a look at v30. > > > > >

RE: Optionally automatically disable logical replication subscriptions on error

2022-03-08 Thread osumi.takami...@fujitsu.com
On Wednesday, March 9, 2022 9:58 AM Masahiko Sawada wrote: > On Tue, Mar 8, 2022 at 5:07 PM osumi.takami...@fujitsu.com > wrote: > > > > Kindly have a look at v30. > > Thank you for updating the patch. Here are some comments: Hi, thank you for your review ! > +

RE: Optionally automatically disable logical replication subscriptions on error

2022-03-08 Thread osumi.takami...@fujitsu.com
On Tuesday, March 8, 2022 10:23 PM Amit Kapila wrote: > On Tue, Mar 8, 2022 at 1:37 PM osumi.takami...@fujitsu.com > wrote: > > > > Kindly have a look at v30. > > > > Review comments: Thank you for checking ! > === > 1. > + ereport(LOG, >

RE: Optionally automatically disable logical replication subscriptions on error

2022-03-09 Thread osumi.takami...@fujitsu.com
On Wednesday, March 9, 2022 8:22 PM Amit Kapila wrote: > On Wed, Mar 9, 2022 at 2:21 PM Masahiko Sawada > wrote: > > > > On Wed, Mar 9, 2022 at 4:33 PM osumi.takami...@fujitsu.com > > wrote: > > > > > > On Tuesday, March 8, 2022 10:23 PM Amit Kapila >

RE: Skipping logical replication transactions on subscriber side

2022-03-09 Thread osumi.takami...@fujitsu.com
On Wednesday, March 2, 2022 12:01 AM Masahiko Sawada wrote: > I've attached an updated patch along with two patches for cfbot tests since > the > main patch (0003) depends on the other two patches. Both > 0001 and 0002 patches are the same ones I attached on another thread[2]. Hi, few comments o

RE: Skipping logical replication transactions on subscriber side

2022-03-11 Thread osumi.takami...@fujitsu.com
On Friday, March 11, 2022 5:20 PM Masahiko Sawada wrote: > I've attached an updated version patch. This patch can be applied on top of > the > latest disable_on_error patch[1]. Hi, thank you for the patch. I'll share my review comments on v13. (a) src/backend/commands/subscriptioncmds.c @@ -84

RE: Optionally automatically disable logical replication subscriptions on error

2022-03-14 Thread osumi.takami...@fujitsu.com
On Monday, March 14, 2022 7:49 PM Amit Kapila wrote: > On Thu, Mar 10, 2022 at 12:04 PM Amit Kapila > wrote: > > > > On Wed, Mar 9, 2022 at 7:57 PM osumi.takami...@fujitsu.com > > wrote: > > > > > > Hi, attached v32 removed my addit

RE: Skipping logical replication transactions on subscriber side

2022-03-14 Thread osumi.takami...@fujitsu.com
On Friday, March 11, 2022 5:20 PM Masahiko Sawada wrote: > I've attached an updated version patch. This patch can be applied on top of > the > latest disable_on_error patch[1]. Hi, few extra comments on v13. (1) src/backend/replication/logical/worker.c With regard to clear_subscription_skip_l

RE: Optionally automatically disable logical replication subscriptions on error

2022-03-14 Thread osumi.takami...@fujitsu.com
On Tuesday, March 15, 2022 8:04 AM Nathan Bossart wrote: > My compiler is worried that syncslotname may be used uninitialized in > start_table_sync(). The attached patch seems to silence this warning. Thank you for your reporting ! Your fix looks good to me. Best Regards, Takamichi Os

RE: Skipping logical replication transactions on subscriber side

2022-03-15 Thread osumi.takami...@fujitsu.com
On Tuesday, March 15, 2022 3:13 PM Masahiko Sawada wrote: > I've attached an updated version patch. A couple of minor comments on v14. (1) apply_handle_commit_internal + if (is_skipping_changes()) + { + stop_skipping_changes(); + + /* +*

RE: Skipping logical replication transactions on subscriber side

2022-03-15 Thread osumi.takami...@fujitsu.com
On Wednesday, March 16, 2022 11:33 AM Amit Kapila wrote: > On Tue, Mar 15, 2022 at 7:30 PM osumi.takami...@fujitsu.com > wrote: > > > > On Tuesday, March 15, 2022 3:13 PM Masahiko Sawada > wrote: > > > I've attached an updated version patch. > &

RE: Skipping logical replication transactions on subscriber side

2022-03-15 Thread osumi.takami...@fujitsu.com
On Wednesday, March 16, 2022 3:37 PM I wrote: > On Wednesday, March 16, 2022 11:33 AM Amit Kapila > wrote: > > On Tue, Mar 15, 2022 at 7:30 PM osumi.takami...@fujitsu.com > > wrote: > > > > > > On Tuesday, March 15, 2022 3:13 PM Masahiko Sawada > >

RE: Skipping logical replication transactions on subscriber side

2022-03-17 Thread osumi.takami...@fujitsu.com
On Thursday, March 17, 2022 3:04 PM Amit Kapila wrote: > On Wed, Mar 16, 2022 at 1:53 PM Masahiko Sawada > wrote: > > > > I've attached an updated version patch. > > > > The patch LGTM. I have made minor changes in comments and docs in the > attached patch. Kindly let me know what you think of t

RE: Skipping logical replication transactions on subscriber side

2022-03-17 Thread osumi.takami...@fujitsu.com
On Thursday, March 17, 2022 7:56 PM Masahiko Sawada wrote: On Thu, Mar 17, 2022 at 5:52 PM Amit Kapila > wrote: > > > > On Thu, Mar 17, 2022 at 12:39 PM osumi.takami...@fujitsu.com > > wrote: > > > > > > On Thursday, March 17, 2022 3:04 PM Amit Kapila &g

RE: Disable WAL logging to speed up data loading

2020-12-01 Thread osumi.takami...@fujitsu.com
Hi Sorry for being late. On Tuesday, December 1, 2020 10:42 AM Tsunakawa, Takayuki wrote: > From: Kyotaro Horiguchi > > Yeah, although it's enough only to restrict non-harmful records > > practically, if we find that only a few kinds of records are needed, > > maybe it's cleaner to allow only

RE: Disable WAL logging to speed up data loading

2020-12-02 Thread osumi.takami...@fujitsu.com
Hello I've made a new patch v05 that took in comments to filter out WALs more strictly and addressed some minor fixes that were discussed within past few days. Also, I changed the documentations, considering those modifications. Best, Takamichi Osumi disable_WAL_logging_v05.patch Des

RE: Stronger safeguard for archive recovery not to miss data

2020-12-07 Thread osumi.takami...@fujitsu.com
Hi On Thursday, November 26, 2020 4:29 PM Kyotaro Horiguchi wrote: > At Thu, 26 Nov 2020 07:18:39 +, "osumi.takami...@fujitsu.com" > wrote in > > The attached patch is intended to prevent a scenario that archive > > recovery hits WALs which come from wal_

how to use valgrind for TAP tests

2020-12-18 Thread osumi.takami...@fujitsu.com
Hello, hackers I have a question about how to execute valgrind with TAP tests in order to check some patches in the community. My main interest is testing src/test/subscription now but is there any general way to do it ? The documentation [1] says "It's important to realize that the TAP tests wi

RE: how to use valgrind for TAP tests

2020-12-23 Thread osumi.takami...@fujitsu.com
Hello On Saturday, December 19, 2020 1:03 AM Tom Lane wrote: > "osumi.takami...@fujitsu.com" writes: > > I have a question about how to execute valgrind with TAP tests in > > order to check some patches in the community. > > My main interest is testing src/test/su

RE: how to use valgrind for TAP tests

2020-12-23 Thread osumi.takami...@fujitsu.com
Hi, Alexander On Sunday, December 20, 2020 5:00 PM Alexander Lakhin wrote: > > "osumi.takami...@fujitsu.com" writes: > >> I have a question about how to execute valgrind with TAP tests in > >> order to check some patches in the community. > >> My mai

RE: Disable WAL logging to speed up data loading

2020-12-25 Thread osumi.takami...@fujitsu.com
Hi, Michael Thank you for your attention to this thread. On Friday, December 25, 2020 4:09 PM Michael Paquier wrote: > On Thu, Dec 03, 2020 at 03:52:47AM +, tsunakawa.ta...@fujitsu.com > wrote: > > The code looks good, and the performance seems to be nice, so I marked > > this ready for co

RE: Disable WAL logging to speed up data loading

2020-12-27 Thread osumi.takami...@fujitsu.com
Hello Sawada-San On Monday, December 28, 2020 2:29 PM Masahiko Sawada wrote: > On Thu, Dec 3, 2020 at 12:14 PM osumi.takami...@fujitsu.com > wrote: > > > > I've made a new patch v05 that took in comments to filter out WALs > > more strictly and addressed some mi

RE: [HACKERS] logical decoding of two-phase transactions

2020-12-28 Thread osumi.takami...@fujitsu.com
Hi, Amit-San On Thursday, Dec 24, 2020 2:35 PM Amit Kapila wrote: > On Wed, Dec 23, 2020 at 3:08 PM Ajin Cherian wrote: > > > > > Can you please update the patch for the points we agreed upon? > > > > Changed and attached. > > > > Thanks, I have looked at these patches again and it seems patch

RE: Disable WAL logging to speed up data loading

2021-01-04 Thread osumi.takami...@fujitsu.com
Hi, Sawada-San On Monday, December 28, 2020 7:12 PM Masahiko Sawada wrote: > On Mon, Dec 28, 2020 at 4:29 PM osumi.takami...@fujitsu.com > wrote: > > On Monday, December 28, 2020 2:29 PM Masahiko Sawada > wrote: > > > On Thu, Dec 3, 2020 at 12:14 PM osumi.takami...@

RE: Disable WAL logging to speed up data loading

2021-01-06 Thread osumi.takami...@fujitsu.com
Hi, On Tuesday, January 5, 2021 5:45 PM Masahiko Sawada wrote: > On Tue, Jan 5, 2021 at 10:54 AM osumi.takami...@fujitsu.com > wrote: > > On Monday, December 28, 2020 7:12 PM Masahiko Sawada > wrote: > > > On Mon, Dec 28, 2020 at 4:29 PM osumi.takami...@fujitsu.com

Enhance traceability of wal_level changes for backup management

2021-01-06 Thread osumi.takami...@fujitsu.com
Hi, This thread came from another thread about wal_level [1]. Mainly from backup management tools perspective such as pgBackRest, EDB's BART and pg_probackup, it seems worth talking about a way comprehensively to trace and recognize wal_level changes for various purposes and values like necessit

RE: Enhance traceability of wal_level changes for backup management

2021-01-06 Thread osumi.takami...@fujitsu.com
Hello, Sawada-san I'll continue the discussion of [2]. We talked about how to recognize the time or LSN when/where wal_level is changed to 'none' there. You said > The use case I imagined is that the user temporarily > changes wal_level to 'none' from 'replica' or 'logical' to speed up loading

RE: Enhance traceability of wal_level changes for backup management

2021-01-07 Thread osumi.takami...@fujitsu.com
Hi Stephen Thank you so much for replying ! On Thursday, January 7, 2021 2:40 AM Stephen Frost wrote: > * osumi.takami...@fujitsu.com (osumi.takami...@fujitsu.com) wrote: > > You said > > > The use case I imagined is that the user temporarily changes > > > wal_level

RE: [HACKERS] logical decoding of two-phase transactions

2021-02-11 Thread osumi.takami...@fujitsu.com
Hi On Thursday, February 11, 2021 5:10 PM Peter Smith wrote: > Please find attached the new 2PC patch set v39* I started to review the patchset so, let me give some comments I have at this moment. (1) File : v39-0007-Support-2PC-txn-tests-for-concurrent-aborts.patch Modification : @@ -620,6

RE: [HACKERS] logical decoding of two-phase transactions

2021-02-15 Thread osumi.takami...@fujitsu.com
Hi On Tuesday, February 16, 2021 8:33 AM Peter Smith > On Fri, Feb 12, 2021 at 5:59 PM osumi.takami...@fujitsu.com > wrote: > > (2) > > > > File : v39-0006-Support-2PC-txn-Subscription-option.patch > > > > @@ -213,6 +219,15 @@ pars

RE: [HACKERS] logical decoding of two-phase transactions

2021-02-26 Thread osumi.takami...@fujitsu.com
Hi On Thursday, February 25, 2021 4:02 PM Peter Smith > Please find attached the latest patch set v43* > > - Added new patch 0007 "Fix apply worker prepare" as discussed here [2] > > [2] > https://www.postgresql.org/message-id/CAA4eK1L%3DdhuCRvyDvrXX5wZ > gc7s1hLRD29CKCK6oaHtVCPgiFA%40mail.gmai

RE: [HACKERS] logical decoding of two-phase transactions

2021-03-05 Thread osumi.takami...@fujitsu.com
Hi On Saturday, March 6, 2021 10:49 AM Peter Smith wrote: > Please find attached the latest patch set v50* When I read throught the patch set, I found there is a wierd errmsg in apply_handle_begin_prepare(), which seems a mistake. File : v50-0003-Add-support-for-apply-at-prepare-time-to-built-

RE: Enhance traceability of wal_level changes for backup management

2021-03-07 Thread osumi.takami...@fujitsu.com
On Sun, Mar 7, 2021 3:48 AM Peter Eisentraut wrote: > On 28.01.21 01:44, osumi.takami...@fujitsu.com wrote: > >> (1) writing the time or LSN in the control file to indicate > >> when/where wal_level is changed to 'minimal' > >> from upper level to inva

RE: Enhance traceability of wal_level changes for backup management

2021-03-10 Thread osumi.takami...@fujitsu.com
On Tuesday, March 9, 2021 11:13 PM David Steele > On 3/7/21 9:45 PM, osumi.takami...@fujitsu.com wrote: > > On Sun, Mar 7, 2021 3:48 AM Peter Eisentraut > wrote: > >> On 28.01.21 01:44, osumi.takami...@fujitsu.com wrote: > >>>> (1) writing the time or

RE: [HACKERS] logical decoding of two-phase transactions

2021-03-13 Thread osumi.takami...@fujitsu.com
Hi On Friday, March 12, 2021 5:40 PM Peter Smith > Please find attached the latest patch set v58* Thank you for updating those. I'm testing the patchset and I think it's preferable that you add simple two types of more tests in 020_twophase.pl because those aren't checked by v58. (1) execute s

RE: Enhance traceability of wal_level changes for backup management

2021-03-15 Thread osumi.takami...@fujitsu.com
On Friday, March 12, 2021 5:04 PM Peter Eisentraut wrote: > On 08.03.21 03:45, osumi.takami...@fujitsu.com wrote: > > OK. The basic idea is to enable backup management tools to recognize > > wal_level drop between*snapshots*. > > When you have a snapshot of the cluster at

RE: subscriptionCheck failures

2021-03-15 Thread osumi.takami...@fujitsu.com
Hello On Tuesday, March 16, 2021 12:31 PM Amit Kapila wrote: > On Mon, Mar 15, 2021 at 6:00 PM Thomas Munro > wrote: > > > > Hi, > > > > This seems to be a new low frequency failure, I didn't see it mentioned > already: Oh, this is the test I wrote and included as part of the commit ce0fdbfe #

RE: subscriptionCheck failures

2021-03-16 Thread osumi.takami...@fujitsu.com
Hi On Tuesday, March 16, 2021 4:15 PM vignesh C wrote: > On Tue, Mar 16, 2021 at 12:29 PM Amit Kapila > wrote: > > > > On Tue, Mar 16, 2021 at 9:00 AM Amit Kapila > wrote: > > > > > > On Mon, Mar 15, 2021 at 6:00 PM Thomas Munro > wrote: > > > > > > > > Hi, > > > > > > > > This seems to be a

RE: Disable WAL logging to speed up data loading

2020-10-27 Thread osumi.takami...@fujitsu.com
Hi, I wrote and attached the first patch to disable WAL logging. This patch passes the regression test of check-world already and is formatted by pgindent. Also, I conducted 3 types of performance tests to clarify the rough benefits of the patch. I compared two wal_levels both 'minimal' and 'no

RE: Disable WAL logging to speed up data loading

2020-10-28 Thread osumi.takami...@fujitsu.com
Hi, Laurenz > > I wrote and attached the first patch to disable WAL logging. > > This patch passes the regression test of check-world already and is > > formatted by pgindent. > > Without reading the code, I have my doubts about that feature. > > While it clearly will improve performance, it op

RE: Disable WAL logging to speed up data loading

2020-10-28 Thread osumi.takami...@fujitsu.com
Hello > > But what if someone sets wal_level=none, performs some data > > modifications, sets wal_level=archive and after dome more processing > > decides to restore from a backup that was taken before the cluster was set > to wal_level=none? > > Then they would end up with a corrupted database,

RE: extension patch of CREATE OR REPLACE TRIGGER

2020-10-29 Thread osumi.takami...@fujitsu.com
Hi, From: Tsunakawa, Takayuki < tsunakawa.ta...@fujitsu.com> > From: osumi.takami...@fujitsu.com > > > > * I don't think that you've fully thought through the implications > > > > of replacing a trigger for a table that the current transaction

RE: Disable WAL logging to speed up data loading

2020-11-01 Thread osumi.takami...@fujitsu.com
Hello On Friday, October 30, 2020 1:32 PM Masahiko Sawada wrote: > > On 2020/10/29 19:21, Laurenz Albe wrote: > > > On Thu, 2020-10-29 at 11:42 +0900, Fujii Masao wrote: > > >>> But what if someone sets wal_level=none, performs some data > > >>> modifications, sets wal_level=archive and after dom

RE: extension patch of CREATE OR REPLACE TRIGGER

2020-11-03 Thread osumi.takami...@fujitsu.com
Hi Peter-San, thanks for your support. On Monday, November 2, 2020 2:39 PM Peter Smith wrote: > === > > (1) COMMENT > File: NA > Maybe next time consider using format-patch to make the patch. Then you > can include a comment to give the background/motivation for this change. OK. How about v15 ?

RE: extension patch of CREATE OR REPLACE TRIGGER

2020-11-05 Thread osumi.takami...@fujitsu.com
Hi, On Thursday, November 5, 2020 1:22 PM Peter Smith wrote: > On Wed, Nov 4, 2020 at 2:53 PM osumi.takami...@fujitsu.com > wrote: > > > (1) COMMENT > > > File: NA > > > Maybe next time consider using format-patch to make the patch. Then > > > you ca

RE: extension patch of CREATE OR REPLACE TRIGGER

2020-11-06 Thread osumi.takami...@fujitsu.com
Hello On Friday, November 6, 2020 2:25 PM Peter Smith wrote: > > > Yes, OK. But it might be simpler still just to it like: > > > "CREATE OR REPLACE TRIGGER works only for replacing a regular (not > > > constraint) trigger with another regular trigger." > > Yeah, this kind of supplementary words

RE: extension patch of CREATE OR REPLACE TRIGGER

2020-11-08 Thread osumi.takami...@fujitsu.com
Hi, On Saturday, Nov 7, 2020 2:06 PM Dilip Kumar wrote: > The patch looks fine to me however I feel that in the test case there are a > lot > of duplicate statement which can be reduced e.g. > +-- 1. Overwrite existing regular trigger with regular trigger > (without OR REPLACE) > +create trigge

RE: Disable WAL logging to speed up data loading

2020-11-08 Thread osumi.takami...@fujitsu.com
Hello, Stephen On Tuesday, Nov 3, 2020 3:02 AM Stephen Frost wrote: > * Magnus Hagander (mag...@hagander.net) wrote: > > On Mon, Nov 2, 2020 at 4:28 PM Robert Haas > wrote: > > > On Thu, Oct 29, 2020 at 4:00 PM Fujii Masao > wrote: > > > > Yes. What I meant was such a safe guard needs to be im

RE: Disable WAL logging to speed up data loading

2020-11-10 Thread osumi.takami...@fujitsu.com
Horiguchi-San On Tuesday, November 10, 2020 10:00 AM Kyotaro Horiguchi wrote: > FWIW, the following is that, I think it works not only when wal_level=minimal > for SET UNLOGGED, and only works when minimal for SET LOGGED. > > https://www.postgresql.org/message-id/20201002.100621.1668918756520

RE: In-placre persistance change of a relation

2020-11-12 Thread osumi.takami...@fujitsu.com
Hello, Tsunakawa-San > Do you know the reason why data copy was done before? And, it may be > odd for me to ask this, but I think I saw someone referred to the past > discussion that eliminating data copy is difficult due to some processing at > commit. I can't find it. I can share 2 sources wh

RE: Disable WAL logging to speed up data loading

2020-11-15 Thread osumi.takami...@fujitsu.com
Hi On Thursday, October 29, 2020 11:42 AM Fujii Masao wrote: > BTW, with the patch, I observed that PREPARE TRANSACTION and COMMIT > PREPARED caused assertion failure in my env, as I pointed upthread. > > How does the patch handle other feature depending on the existence of WAL, > e.g., pg_log

RE: Disable WAL logging to speed up data loading

2020-11-18 Thread osumi.takami...@fujitsu.com
Hello In the past discussion of wal_level=none, there were some ideas to trace the change of wal_level, in other words, *stronger mechanism* to check wal_level. I agree with the idea to have a new monitoring item and would like to implement those kind of, or one of those ideas for the next patch

RE: Disable WAL logging to speed up data loading

2020-11-18 Thread osumi.takami...@fujitsu.com
Hello On Thursday, Nov 19, 2020 12:45 PM Tsunakawa, Takayuki/綱川 貴之 > > On Tuesday, Nov 3, 2020 3:02 AM Stephen Frost > > wrote: > > > Checking the WAL level certainly seems critical to anything that's > > > reading the WAL. We certainly do this already when running as a > > > replica: > > > >

RE: Disable WAL logging to speed up data loading

2020-11-19 Thread osumi.takami...@fujitsu.com
Hello, Laurenz On Thursday, Nov19, 2020 4:50 PM Laurenz Albe wrote > On Thu, 2020-11-19 at 05:24 +0000, osumi.takami...@fujitsu.com wrote: > > > > > ereport(WARNING, > > > > > (errmsg("WAL was generated with wal_level=

RE: Disable WAL logging to speed up data loading

2020-11-19 Thread osumi.takami...@fujitsu.com
Hello On Friday, Nov 20, 2020 9:33 AM Tsunakawa, Takayuki wrote: > From: Kyotaro Horiguchi > > At Thu, 19 Nov 2020 11:04:17 -0500, Stephen Frost > > > * Laurenz Albe (laurenz.a...@cybertec.at) wrote: > > > > I missed that this is only a warning when I looked at it before. > > > > Yes, it shou

RE: Disable WAL logging to speed up data loading

2020-11-20 Thread osumi.takami...@fujitsu.com
Hi This time, I updated my patch to address comments below only. Please have a look at updated one. On Thursday, Nov 19, 2020 4:51 PM Tsunakawa, Takayuki > (1) > #define RelationNeedsWAL(relation) > \ > + (wal_level != WAL_LEVEL_NONE && >

RE: Disable WAL logging to speed up data loading

2020-11-23 Thread osumi.takami...@fujitsu.com
Hi On Monday, Nov 23, 2020 12:17 PM Tsunakawa, Takayuki wrote: > PREPARE TRANSACTION is the same as COMMIT in that it persists > transaction updates. A crash during wal_level = none loses both of them. > So, I don't think PREPARE TRANSACTION needs special treatment. Yeah, I got it. That makes s

Stronger safeguard for archive recovery not to miss data

2020-11-25 Thread osumi.takami...@fujitsu.com
Hello The attached patch is intended to prevent a scenario that archive recovery hits WALs which come from wal_level=minimal and the server continues to work, which was discussed in the thread of [1]. The motivation is to protect that user ends up with both getting replica that could miss data an

RE: Stronger safeguard for archive recovery not to miss data

2020-11-25 Thread osumi.takami...@fujitsu.com
Hello, Horiguchi-San On Thursday, Nov 26, 2020 4:29 PM Kyotaro Horiguchi > At Thu, 26 Nov 2020 07:18:39 +, "osumi.takami...@fujitsu.com" > wrote in > > In order to test this fix, what I did is > > 1 - get a base backup during wal_level is replica > >

RE: Stronger safeguard for archive recovery not to miss data

2020-11-25 Thread osumi.takami...@fujitsu.com
Hello, Sergei > It is possible only with manually configured hot_standby = off, right? > We have ERROR in hot standby mode just below in > CheckRequiredParameterValues. Yes, you are right. I turned off the hot_standby in the test in the previous e-mail. I'm sorry that I missed writing this tip of

RE: Disable WAL logging to speed up data loading

2020-11-27 Thread osumi.takami...@fujitsu.com
Hello, Sawada-San On Friday, November 27, 2020 3:08 PM Masahiko Sawada wrote: > - (errmsg("WAL was generated with wal_level=minimal, > data may be missing"), > + (errmsg("WAL was generated with wal_level<=minimal, > data may be missing"), > errhint("T

RE: Disable WAL logging to speed up data loading

2020-11-27 Thread osumi.takami...@fujitsu.com
Hi, Tsunakawa-San > I'm afraid "none" doesn't represent the behavior because RM_XLOG_ID and > RM_XACT_ID WAL records, except for XLOG_FPI_*, are emitted. What's the > good name? IIUC, "minimal" is named after the fact that the minimal > amount of WAL necessary for crash recovery is generated.

RE: Disable WAL logging to speed up data loading

2020-11-27 Thread osumi.takami...@fujitsu.com
Thank you, Horiguchi-San > I haven't seen a criteria of whether a record is emitted or not for > wal_leve=none. > > We're emitting only redo logs. So I think theoretically we don't need > anything > other than the shutdown checkpoint record because we don't perform > recovery and checkpoint rec

  1   2   3   4   >