Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-20 Thread Masahiko Sawada
On Sun, Feb 20, 2022 at 1:02 AM Andres Freund wrote: > > Hi, > > On 2022-02-19 22:38:19 +0900, Masahiko Sawada wrote: > > On Sat, Feb 19, 2022 at 5:32 AM Andres Freund wrote: > > > On 2022-02-18 17:26:04 +0900, Masahiko Sawada wrote: > > > > With this chang

Re: pg_stat_get_replication_slot and pg_stat_get_subscription_worker incorrectly marked as proretset

2022-02-20 Thread Masahiko Sawada
t it does not change the fact that we'd > better address the second case on HEAD IMO. > > Thoughts? Agreed. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-21 Thread Masahiko Sawada
nters: apply_error_count and sync_error_count. Since the new view name is under discussion I temporarily chose pg_stat_subscription_activity. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/ v1-0001-Reconsider-pg_stat_subscription_workers-view.patch Description: Binary data

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-22 Thread Masahiko Sawada
On Tue, Feb 22, 2022 at 6:53 PM Amit Kapila wrote: > > On Tue, Feb 22, 2022 at 11:15 AM Masahiko Sawada > wrote: > > > > I've attached a patch that changes pg_stat_subscription_workers view. > > It removes non-cumulative values such as error details such as >

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-22 Thread Masahiko Sawada
On Tue, Feb 22, 2022 at 9:22 PM osumi.takami...@fujitsu.com wrote: > > On Tuesday, February 22, 2022 2:45 PM Masahiko Sawada > wrote: > > I've attached a patch that changes pg_stat_subscription_workers view. > > It removes non-cumulative values such as error det

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-22 Thread Masahiko Sawada
On Wed, Feb 23, 2022 at 11:14 AM Andres Freund wrote: > > Hi, > > On 2022-02-22 14:45:19 +0900, Masahiko Sawada wrote: > > I've attached a patch that changes pg_stat_subscription_workers view. > > Thanks for working on this! > > Why are the stats stored in

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-23 Thread Masahiko Sawada
an continue. > +$node_subscriber->safe_psql('postgres', "TRUNCATE test_tab1;"); > + > # Wait for initial table sync for test_tab1 to finish. > > IMO all these "table sync" should be changed to "tablesync", because a > table "sync

Re: Failed transaction statistics to measure the logical replication progress

2022-02-23 Thread Masahiko Sawada
d(bool is_commit) +{ +if (is_commit) +MyLogicalRepWorker->commit_count++; +else +MyLogicalRepWorker->abort_count++; +} It's slightly odd and it seems unnecessary to me that we modify fields of MyLogicalRepWorker in pgstat.c. Although this function has “

Re: Optionally automatically disable logical replication subscriptions on error

2022-02-23 Thread Masahiko Sawada
do this immediately so that it won't be lost if some other internal +* error occurs in the following code. +*/ + EmitErrorReport(); + AbortOutOfAnyTransaction(); + FlushErrorState(); Do we need to hold interrupts during cleanup here? Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-24 Thread Masahiko Sawada
On Thu, Feb 24, 2022 at 6:53 PM Peter Eisentraut wrote: > > On 24.02.22 02:32, Masahiko Sawada wrote: > > On Wed, Feb 23, 2022 at 12:08 PM Peter Smith wrote: > >> > >> Hi. Below are my review comments for the v1 patch. > > > > Thank you for the commen

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-24 Thread Masahiko Sawada
we avoid writing tests for stats for each and every case as it > is not reliable in nature (the message can be lost). If we can find a > reliable way then it is okay. Yeah, the messages can even be out-of-order. Particularly, in this test, the apply worker and table sync worker keep reporting the messages, it's quite possible that the test becomes unstable. I remember we removed unstable tests of resetting statistics before (e.g., see fc6950913). Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-24 Thread Masahiko Sawada
On Thu, Feb 24, 2022 at 9:23 PM Peter Eisentraut wrote: > > > On 24.02.22 12:46, Masahiko Sawada wrote: > >> We have a view called pg_stat_activity, which is very well known. From > >> that perspective, "activity" means what is happening right now or what

Re: Optionally automatically disable logical replication subscriptions on error

2022-02-24 Thread Masahiko Sawada
On Thu, Feb 24, 2022 at 8:08 PM Amit Kapila wrote: > > On Thu, Feb 24, 2022 at 1:20 PM Masahiko Sawada wrote: > > > > Here are some comments: > > > > Why do we need SyncTableStartWrapper() and ApplyLoopWrapper()? > > > > I have given this comment to mov

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-24 Thread Masahiko Sawada
Hi, Thank you for the comments! On Thu, Feb 24, 2022 at 4:20 PM tanghy.f...@fujitsu.com wrote: > > On Thu, Feb 24, 2022 9:33 AM Masahiko Sawada wrote: > > > > Thank you for the comments! I've attached the latest version patch > > that incorporated all comments I g

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-27 Thread Masahiko Sawada
On Sat, Feb 26, 2022 at 11:51 AM Amit Kapila wrote: > > On Thu, Feb 24, 2022 at 9:20 PM Masahiko Sawada wrote: > > > > I have reviewed the latest version and made a few changes along with > fixing some of the pending comments by Peter Smith. Thank you for updating the patc

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-27 Thread Masahiko Sawada
gt; > +the argument is NULL, reset statistics for all > > +subscriptions. > > > > > > I felt we could improve the first sentence. > > > > From: > > Resets statistics for a single subscription shown in the.. > > > > To(idea1): > > Resets statistics for a single subscription defined by the argument to zero. > > > > Okay, I can use this one. Are you going to remove the part "shown in the pg_stat_subsctiption_stats view"? I think it's better to keep it in order to make it clear which statistics the function resets as we have pg_stat_subscription and pg_stat_subscription_stats. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-27 Thread Masahiko Sawada
On Mon, Feb 28, 2022 at 11:52 AM Amit Kapila wrote: > > On Mon, Feb 28, 2022 at 8:17 AM Masahiko Sawada wrote: > > > > On Mon, Feb 28, 2022 at 11:33 AM Amit Kapila > > wrote: > > > > > > > > > > > (2) doc/src/sgml/monitoring.sgm

Add the replication origin name and commit-LSN to logical replication worker errcontext

2022-02-28 Thread Masahiko Sawada
es with if-else blocks in order to have a translation work, the second patch adds the origin name and commit-LSN to the errcontext message. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/ From dc6d97c71394c7c216920b9aa1d55bf33c5ac472 Mon Sep 17 00:00:00 2001 From: Masahiko Sawada Date

Re: Showing I/O timings spent reading/writing temp buffers in EXPLAIN

2022-02-28 Thread Masahiko Sawada
Hi, Sorry for the late reply. On Fri, Nov 19, 2021 at 7:24 AM Melanie Plageman wrote: > > On Sun, Aug 22, 2021 at 9:47 PM Masahiko Sawada wrote: > > > > On Thu, Aug 19, 2021 at 10:52 PM Ranier Vilela wrote: > > > > > > Em qui., 19 de ago. de 2021 às 09:

Re: Showing I/O timings spent reading/writing temp buffers in EXPLAIN

2022-02-28 Thread Masahiko Sawada
On Wed, Jan 19, 2022 at 5:52 PM Julien Rouhaud wrote: > > Hi, > > On Tue, Nov 16, 2021 at 04:37:44PM +0900, Masahiko Sawada wrote: > > > > I've attached an updated patch. Please review it. > > It seems that the regression tests aren't entirely stable, p

Re: Skipping logical replication transactions on subscriber side

2022-03-01 Thread Masahiko Sawada
nes I attached on another thread[2]. Regards, [1] https://www.postgresql.org/message-id/20220125063131.4cmvsxbz2tdg6g65%40alap3.anarazel.de [2] https://www.postgresql.org/message-id/CAD21AoBarBf2oTF71ig2g_o%3D3Z_Dt6_sOpMQma1kFgbnA5OZ_w%40mail.gmail.com -- Masahiko Sawada EDB: https://www.enterpri

Re: Optionally automatically disable logical replication subscriptions on error

2022-03-01 Thread Masahiko Sawada
query result instead of using poll_query_until() so that we can check the query result in case where the test fails. --- +$cmd = qq(DROP INDEX tbl_unique); +$node_subscriber->safe_psql('postgres', $cmd); In the newly added tap tests, all queries are consistently assigned to $cmd and executed even when the query is used only once. It seems a different style from the one in other tap tests. Is there any reason why we do this style for all queries in the tap tests? Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Add the replication origin name and commit-LSN to logical replication worker errcontext

2022-03-01 Thread Masahiko Sawada
On Wed, Mar 2, 2022 at 12:21 PM Amit Kapila wrote: > > On Wed, Mar 2, 2022 at 8:25 AM Peter Smith wrote: > > > > On Mon, Feb 28, 2022 at 11:16 PM Masahiko Sawada > > wrote: > > > > > > The errcontext message would become like follows: > > &

Re: Failed transaction statistics to measure the logical replication progress

2022-03-01 Thread Masahiko Sawada
it instead of worker_internal.h? worker_internal.h is the header file shared by logical replication workers such as apply worker, tablesync worker, and launcher. So it might not be advisable to include it in pgstat.c. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Add the replication origin name and commit-LSN to logical replication worker errcontext

2022-03-01 Thread Masahiko Sawada
On Wed, Mar 2, 2022 at 11:55 AM Amit Kapila wrote: > > On Mon, Feb 28, 2022 at 5:46 PM Masahiko Sawada wrote: > > > > I've attached two patches: the first one changes > > apply_error_callback() so that it uses complete sentences with if-else > > blocks

Re: Add the replication origin name and commit-LSN to logical replication worker errcontext

2022-03-01 Thread Masahiko Sawada
On Wed, Mar 2, 2022 at 4:14 PM Amit Kapila wrote: > > On Wed, Mar 2, 2022 at 9:33 AM Masahiko Sawada wrote: > > > > On Wed, Mar 2, 2022 at 12:21 PM Amit Kapila wrote: > > > > > > On Wed, Mar 2, 2022 at 8:25 AM Peter Smith wrote: > > > > > >

Re: Add the replication origin name and commit-LSN to logical replication worker errcontext

2022-03-02 Thread Masahiko Sawada
On Wed, Mar 2, 2022 at 4:07 PM Kyotaro Horiguchi wrote: > > At Wed, 2 Mar 2022 14:39:54 +0900, Masahiko Sawada > wrote in > > On Wed, Mar 2, 2022 at 11:55 AM Amit Kapila wrote: > > > > > > On Mon, Feb 28, 2022 at 5:46 PM Masahiko Sawada > > > wrote:

Re: Add the replication origin name and commit-LSN to logical replication worker errcontext

2022-03-03 Thread Masahiko Sawada
On Thu, Mar 3, 2022 at 3:37 PM Amit Kapila wrote: > > On Wed, Mar 2, 2022 at 1:05 PM Masahiko Sawada wrote: > > > > On Wed, Mar 2, 2022 at 4:14 PM Amit Kapila wrote: > > > > I've attached updated patches. > > > > The first patch LGTM. Some co

Re: Add the replication origin name and commit-LSN to logical replication worker errcontext

2022-03-03 Thread Masahiko Sawada
On Thu, Mar 3, 2022 at 10:02 PM Masahiko Sawada wrote: > > > I'm updating the patches and will submit them. Attached updated version patches. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/ v3-0002-Add-the-origin-name-and-remote-commit-LSN-to-logi.patch Descr

Re: Add the replication origin name and commit-LSN to logical replication worker errcontext

2022-03-03 Thread Masahiko Sawada
On Fri, Mar 4, 2022 at 11:27 AM osumi.takami...@fujitsu.com wrote: > > On Friday, March 4, 2022 10:09 AM Masahiko Sawada > wrote: > > On Thu, Mar 3, 2022 at 10:02 PM Masahiko Sawada > > wrote: > > > > > > > > > I'm updating the patches a

Re: Optionally automatically disable logical replication subscriptions on error

2022-03-03 Thread Masahiko Sawada
On Wed, Mar 2, 2022 at 6:38 PM osumi.takami...@fujitsu.com wrote: > > On Wednesday, March 2, 2022 12:47 PM Masahiko Sawada > wrote: > > After more thoughts, should we do both AbortOutOfAnyTransaction() and error > > message handling while holding interrupts? That is, &g

Re: Add the replication origin name and commit-LSN to logical replication worker errcontext

2022-03-04 Thread Masahiko Sawada
On Fri, Mar 4, 2022 at 2:55 PM Amit Kapila wrote: > > On Fri, Mar 4, 2022 at 6:40 AM Masahiko Sawada wrote: > > > > Attached updated version patches. > > > > The patch looks mostly good to me. Few minor comments: Thank you for the comments! > 1. I think

Re: Add the replication origin name and commit-LSN to logical replication worker errcontext

2022-03-06 Thread Masahiko Sawada
lete instructions. Thank you for the comment. +1. I've attached updated patches. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/ v5-0002-Add-the-origin-name-and-remote-commit-LSN-to-logi.patch Description: Binary data v5-0001-Use-complete-sentences-in-logical-replication-wor.patch Description: Binary data

Re: Add index scan progress to pg_stat_progress_vacuum

2022-03-07 Thread Masahiko Sawada
HASH_ELEM | HASH_BLOBS); +} It seems to me that creating a shmem hash with max_table_size entries for parallel vacuum process tracking is too much. IIRC an old patch had parallel vacuum workers advertise its progress and changed the pg_stat_progress_vacuum view so that it aggregates the results including workers' stats. I think it’s better than the current one. Why did you change that? Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Logical replication timeout problem

2022-03-07 Thread Masahiko Sawada
skipped_changes_count = 0; +} +} Since we send a keepalive after continuously skipping 1 changes, the originally reported issue can still occur if skipping 1 changes took more than the timeout and the walsender didn't send any change while that, is that right? Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Add the replication origin name and commit-LSN to logical replication worker errcontext

2022-03-08 Thread Masahiko Sawada
On Tue, Mar 8, 2022 at 7:56 PM Amit Kapila wrote: > > On Mon, Mar 7, 2022 at 10:06 AM Amit Kapila wrote: > > > > On Mon, Mar 7, 2022 at 6:36 AM Masahiko Sawada > > wrote: > > > > > > Thank you for the comment. +1. > > > > > > I'

Re: Optionally automatically disable logical replication subscriptions on error

2022-03-08 Thread Masahiko Sawada
on on an error but doesn't work for a fatal. Is that expected or should we handle that too? Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Add index scan progress to pg_stat_progress_vacuum

2022-03-08 Thread Masahiko Sawada
ew shmem hash. Another idea I come up with is that the parallel vacuum leader checks PVIndStats.status and updates how many indexes are processed to its progress information. The leader can check it and update the progress information before and after index vacuuming. And possibly

Re: Add index scan progress to pg_stat_progress_vacuum

2022-03-08 Thread Masahiko Sawada
ave not considered the callback in the index AM's bulkdelete and > vacuumcleanup, but I can imagine this is not possible since a leader could be > busy vacuuming rather than updating counters, but I may be misunderstanding > the suggestion. Checking PVIndStats.status values is cheap. Probably the leader can check it every 1GB index block, for example. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Optionally automatically disable logical replication subscriptions on error

2022-03-08 Thread Masahiko Sawada
On Wed, Mar 9, 2022 at 12:37 PM Amit Kapila wrote: > > On Wed, Mar 9, 2022 at 6:29 AM Masahiko Sawada wrote: > > > > --- > > It might have already been discussed but the worker disables the > > subscription on an error but doesn't work for a fatal. Is that &g

Re: Logical replication timeout problem

2022-03-08 Thread Masahiko Sawada
On Wed, Mar 9, 2022 at 11:26 AM wangw.f...@fujitsu.com wrote: > > On Tue, Mar 8, 2022 at 3:52 PM Masahiko Sawada wrote: > > I've looked at the patch and have a question: > Thanks for your review and comments. > > > +void > > +SendKeepaliveIfNecessary(Logica

Re: Optionally automatically disable logical replication subscriptions on error

2022-03-09 Thread Masahiko Sawada
r when disable_on_error is changed? It doesn't affect the remote connection at all. I think it can be changed without restarting like synchronous_commit option. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Skipping logical replication transactions on subscriber side

2022-03-10 Thread Masahiko Sawada
On Thu, Mar 10, 2022 at 2:10 PM osumi.takami...@fujitsu.com wrote: > > On Wednesday, March 2, 2022 12:01 AM Masahiko Sawada > wrote: > > I've attached an updated patch along with two patches for cfbot tests since > > the > > main patch (0003) depends on the othe

Re: Skipping logical replication transactions on subscriber side

2022-03-11 Thread Masahiko Sawada
On Thu, Mar 10, 2022 at 9:02 PM Amit Kapila wrote: > > On Tue, Mar 1, 2022 at 8:31 PM Masahiko Sawada wrote: > > > > I've attached an updated patch along with two patches for cfbot tests > > since the main patch (0003) depends on the other two patches. Both > >

Re: Add index scan progress to pg_stat_progress_vacuum

2022-03-13 Thread Masahiko Sawada
allocate shared memory for vacuum worker progress"); Since we raise an error in case of out of memory, I think we can use HASH_ENTER instead of HASH_ENTER_NULL. Or do we want to emit a detailed error message here? --- + VacuumWorkerProgressHash = NULL; This line is not necessary. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Issue with pg_stat_subscription_stats

2022-03-14 Thread Masahiko Sawada
d be to send the message for creating the subscription at the end of CRAETE SUBSCRIPTION which basically resolves them. A caveat is that if CREATE SUBSCRIPTION (that doesn't involve replication slot creation) is rolled back, the first problem still occurs. But it should not practically matter as

Re: Skipping logical replication transactions on subscriber side

2022-03-14 Thread Masahiko Sawada
On Mon, Mar 14, 2022 at 6:50 PM shiy.f...@fujitsu.com wrote: > > On Fri, Mar 11, 2022 4:20 PM Masahiko Sawada wrote: > > > > I've attached an updated version patch. This patch can be applied on > > top of the latest disable_on_error patch[1]. > > > > Than

Re: Issue with pg_stat_subscription_stats

2022-03-14 Thread Masahiko Sawada
On Tue, Mar 15, 2022 at 3:34 AM Melanie Plageman wrote: > > On Mon, Mar 14, 2022 at 4:02 AM Masahiko Sawada wrote: > > > > On Mon, Mar 14, 2022 at 2:05 AM Melanie Plageman > > wrote: > > > > > > On Sat, Mar 12, 2022 at 3:15 PM Andres Freund wrote: >

Re: Skipping logical replication transactions on subscriber side

2022-03-14 Thread Masahiko Sawada
Hi, On Fri, Mar 11, 2022 at 8:37 PM osumi.takami...@fujitsu.com wrote: > > On Friday, March 11, 2022 5:20 PM Masahiko Sawada > wrote: > > I've attached an updated version patch. This patch can be applied on top of > > the > > latest disable_on_error patch[1

Re: Skipping logical replication transactions on subscriber side

2022-03-15 Thread Masahiko Sawada
On Tue, Mar 15, 2022 at 7:18 PM Amit Kapila wrote: > > On Tue, Mar 15, 2022 at 11:43 AM Masahiko Sawada > wrote: > > > > 6. > @@ -1583,7 +1649,8 @@ apply_handle_insert(StringInfo s) > TupleTableSlot *remoteslot; > MemoryContext oldctx; > &g

Re: Skipping logical replication transactions on subscriber side

2022-03-15 Thread Masahiko Sawada
On Tue, Mar 15, 2022 at 7:18 PM Amit Kapila wrote: > > On Tue, Mar 15, 2022 at 11:43 AM Masahiko Sawada > wrote: > > > > I've attached an updated version patch. > > > > Review: > === Thank you for the comments. > 1. > +++ b/doc/src/sgm

Re: Skipping logical replication transactions on subscriber side

2022-03-16 Thread Masahiko Sawada
On Wed, Mar 16, 2022 at 11:28 AM Amit Kapila wrote: > > On Wed, Mar 16, 2022 at 6:03 AM Masahiko Sawada wrote: > > > > On Tue, Mar 15, 2022 at 7:18 PM Amit Kapila wrote: > > > > > > On Tue, Mar 15, 2022 at 11:43 AM Masahiko Sawada > > > wrote: &g

Re: Skipping logical replication transactions on subscriber side

2022-03-16 Thread Masahiko Sawada
On Wed, Mar 16, 2022 at 1:20 PM Amit Kapila wrote: > > On Wed, Mar 16, 2022 at 7:58 AM Amit Kapila wrote: > > > > On Wed, Mar 16, 2022 at 6:03 AM Masahiko Sawada > > wrote: > > > > > > On Tue, Mar 15, 2022 at 7:18 PM Amit Kapila > > > wrot

Re: Logical replication timeout problem

2022-03-16 Thread Masahiko Sawada
On Wed, Mar 16, 2022 at 11:57 AM wangw.f...@fujitsu.com wrote: > > On Wed, Mar 9, 2022 at 2:45 PM Masahiko Sawada wrote: > > > Thanks for your comments. > > > On Wed, Mar 9, 2022 at 10:26 AM I wrote: > > > On Tue, Mar 8, 2022 at 3:52 PM Masahiko Sawada >

Re: Issue with pg_stat_subscription_stats

2022-03-16 Thread Masahiko Sawada
On Wed, Mar 16, 2022 at 8:51 PM Amit Kapila wrote: > > On Tue, Mar 15, 2022 at 10:09 AM Masahiko Sawada > wrote: > > > > On Tue, Mar 15, 2022 at 3:34 AM Melanie Plageman > > wrote: > > > > > > On Mon, Mar 14, 2022 at 4:02 AM Masahiko Sawada >

Re: Skipping logical replication transactions on subscriber side

2022-03-17 Thread Masahiko Sawada
On Thu, Mar 17, 2022 at 5:52 PM Amit Kapila wrote: > > On Thu, Mar 17, 2022 at 12:39 PM osumi.takami...@fujitsu.com > wrote: > > > > On Thursday, March 17, 2022 3:04 PM Amit Kapila > > wrote: > > > On Wed, Mar 16, 2022 at 1:53 PM Masahiko Sawada > >

Re: Logical replication timeout problem

2022-03-17 Thread Masahiko Sawada
On Thu, Mar 17, 2022 at 7:14 PM Amit Kapila wrote: > > On Thu, Mar 17, 2022 at 12:27 PM Amit Kapila wrote: > > > > On Wed, Mar 16, 2022 at 7:38 PM Masahiko Sawada > > wrote: > > > > > > After more thought, can we check only wal_sender_tim

Re: Skipping logical replication transactions on subscriber side

2022-03-17 Thread Masahiko Sawada
On Thu, Mar 17, 2022 at 3:03 PM Amit Kapila wrote: > > On Wed, Mar 16, 2022 at 1:53 PM Masahiko Sawada wrote: > > > > I've attached an updated version patch. > > > > The patch LGTM. I have made minor changes in comments and docs in the > attached patch. Ki

Re: Add index scan progress to pg_stat_progress_vacuum

2022-03-21 Thread Masahiko Sawada
llel worker exiting and the leader deallocating the hash table entry. BTW have we discussed another idea I mentioned before that we have the leader process periodically check the number of completed indexes and advertise it in its progress information? I'm not sure which one is better but this idea would require only changes of vacuum code and probably simpler than the current idea. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: A test for replay of regression tests

2022-03-21 Thread Masahiko Sawada
ecovery/tmp_check/results/vacuum.out 2022-03-22 03:28:09.813377179 +0100 @@ -181,7 +181,7 @@ SELECT pg_relation_size('vac_truncate_test') = 0; ?column? -- - t + f (1 row) VACUUM (TRUNCATE FALSE, FULL TRUE) vac_truncate_test; === EOF === not ok 2 - regression tests pass Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Add index scan progress to pg_stat_progress_vacuum

2022-03-22 Thread Masahiko Sawada
parallel vacuums. > Can the leader pass a callback that checks PVIndStats to ambulkdelete an amvacuumcleanup callbacks? I think that in the passed callback, the leader checks if the number of processed indexes and updates its progress information if the current progress needs to be updated. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: 回复: [PATCH] BUG FIX: Core dump could happen when VACUUM FULL in standalone mode

2020-11-30 Thread Masahiko Sawada
e_index_workers, and thereby create a model for future > uses of max_parallel_maintenance_workers to follow. +1 Regards, -- Masahiko Sawada EnterpriseDB: https://www.enterprisedb.com/

Re: Add Information during standby recovery conflicts

2020-11-30 Thread Masahiko Sawada
errdetail. (You could > test buf.data or buf.len instead, but that seems uglier to me.) +1 Maybe we can also improve the comment of this function from: + * This function also reports the details about the conflicting + * process ids if *wait_list is not NULL. to " This function also reports the details about the conflicting process ids if exist" or something. Regards, -- Masahiko Sawada EnterpriseDB: https://www.enterprisedb.com/

Re: VACUUM (DISABLE_PAGE_SKIPPING on)

2020-11-30 Thread Masahiko Sawada
On Fri, Nov 20, 2020 at 8:47 PM Simon Riggs wrote: > > On Fri, 20 Nov 2020 at 10:15, Simon Riggs wrote: > > > > On Fri, 20 Nov 2020 at 01:40, Masahiko Sawada wrote: > > > > > > On Thu, Nov 19, 2020 at 8:02 PM Simon Riggs wrote: > > > > > >

Re: autovac issue with large number of tables

2020-11-30 Thread Masahiko Sawada
On Tue, Dec 1, 2020 at 1:48 PM Kasahara Tatsuhito wrote: > > Hi, > > On Mon, Nov 30, 2020 at 8:59 PM Fujii Masao > wrote: > > > > > > > > On 2020/11/30 10:43, Masahiko Sawada wrote: > > > On Sun, Nov 29, 2020 at 10:34 PM Kasahara Tatsuhito > &g

Re: autovac issue with large number of tables

2020-12-01 Thread Masahiko Sawada
On Tue, Dec 1, 2020 at 4:32 PM Fujii Masao wrote: > > > > On 2020/12/01 16:23, Masahiko Sawada wrote: > > On Tue, Dec 1, 2020 at 1:48 PM Kasahara Tatsuhito > > wrote: > >> > >> Hi, > >> > >> On Mon, Nov 30, 2020 at 8:59 PM Fujii Mas

Re: autovac issue with large number of tables

2020-12-01 Thread Masahiko Sawada
On Tue, Dec 1, 2020 at 5:31 PM Masahiko Sawada wrote: > > On Tue, Dec 1, 2020 at 4:32 PM Fujii Masao > wrote: > > > > > > > > On 2020/12/01 16:23, Masahiko Sawada wrote: > > > On Tue, Dec 1, 2020 at 1:48 PM Kasahara Tatsuhito > > > wrote: >

Re: autovac issue with large number of tables

2020-12-02 Thread Masahiko Sawada
On Wed, Dec 2, 2020 at 3:33 PM Fujii Masao wrote: > > > > On 2020/12/02 12:53, Masahiko Sawada wrote: > > On Tue, Dec 1, 2020 at 5:31 PM Masahiko Sawada > > wrote: > >> > >> On Tue, Dec 1, 2020 at 4:32 PM Fujii Masao > >> wrote: > >

Re: [HACKERS] logical decoding of two-phase transactions

2020-12-02 Thread Masahiko Sawada
66 JST [44771] tap_sub STATEMENT: START_REPLICATION SLOT "tap_sub" LOGICAL 0/0 (proto_version '2', two_phase 'on', publication_names '"tap_pub","tap_pub_ins_only"') In v29-0009 patch "two_phase" option is added on the subscripti

Re: Add Information during standby recovery conflicts

2020-12-03 Thread Masahiko Sawada
On Fri, Dec 4, 2020 at 2:54 AM Fujii Masao wrote: > > > > On 2020/12/01 17:29, Drouvot, Bertrand wrote: > > Hi, > > > > On 12/1/20 12:35 AM, Masahiko Sawada wrote: > >> CAUTION: This email originated from outside of the organization. Do not > >>

Re: Add Information during standby recovery conflicts

2020-12-04 Thread Masahiko Sawada
On Fri, Dec 4, 2020 at 7:22 PM Drouvot, Bertrand wrote: > > Hi, > > On 12/4/20 2:21 AM, Fujii Masao wrote: > > > > On 2020/12/04 9:28, Masahiko Sawada wrote: > >> On Fri, Dec 4, 2020 at 2:54 AM Fujii Masao > >> wrote: > >>> > >

Re: Proposed patch for key managment

2020-12-05 Thread Masahiko Sawada
4 --- a/src/common/cryptohash_openssl.c +++ b/src/common/cryptohash_openssl.c @@ -81,6 +81,8 @@ pg_cryptohash_create(pg_cryptohash_type type) return NULL; } + memset(ctx, 0, sizeof(pg_cryptohash_ctx)); + memset(state, 0, sizeof(pg_cryptohash_state)); ctx->data = state; ctx->type = type; Regards, -- Masahiko Sawadahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Proposed patch for key managment

2020-12-06 Thread Masahiko Sawada
'd. > > I think that is now fixed too. Updated patch at the same URL: > > > https://github.com/postgres/postgres/compare/master...bmomjian:key.diff Thank you for updating the patch! I think we need explicit_bzero() also in freeing the keywrap context. BTW, when we need -R option pg_ctl command to start the server, how can we start it in the single-user mode? Regards, -- Masahiko Sawadahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: xid wraparound danger due to INDEX_CLEANUP false

2020-12-15 Thread Masahiko Sawada
on (like bulkdelete() version of vacuum_cleanup_index_scale_factor). In the case where the amount of dead tuples is slightly larger than maitenance_work_mem the second time calling to bulkdelete will be called with a small number of dead tuples, which is inefficient. This problem is also solved by this proposal by allowing a subset of indexes to skip bulkdelete() if the number of dead tuple doesn't exceed the threshold. Any thoughts? I'm writing a PoC patch so will share it. Regards, [1] https://www.postgresql.org/message-id/CAH2-Wzm%2BmaE3apHB8NOtmM%3Dp-DO65j2V5GzAWCOEEuy3JZgb2g%40mail.gmail.com [2] https://www.postgresql.org/message-id/425db134-8bba-005c-b59d-56e50de3b41e%40postgrespro.ru -- Masahiko Sawada EnterpriseDB: https://www.enterprisedb.com/

Re: Add Information during standby recovery conflicts

2020-12-15 Thread Masahiko Sawada
On Mon, Dec 14, 2020 at 9:31 PM Fujii Masao wrote: > > > > On 2020/12/05 12:38, Masahiko Sawada wrote: > > On Fri, Dec 4, 2020 at 7:22 PM Drouvot, Bertrand > > wrote: > >> > >> Hi, > >> > >> On 12/4/20 2:21 AM, Fujii Masao wrote: >

Re: Add Information during standby recovery conflicts

2020-12-15 Thread Masahiko Sawada
ckHoldersNum, lock_holders_sbuf.data, lock_waiters_sbuf.data; > > Conflicting processes are 41171, 41194. > > Conflicting processes are: 41171, 41194. If we use the above message we might want to change other similar messages I exemplified as well. Regards, -- Masahiko Sawada EnterpriseDB: https://www.enterprisedb.com/

Re: [HACKERS] logical decoding of two-phase transactions

2020-12-15 Thread Masahiko Sawada
from the publisher would be ignored without any messages. On the other hand, if applying other operations such as HEAP_INSERT conflicts (such as when violating the unique constraint) the apply worker raises an ERROR and stops logical replication until the conflict is resolved. IIUC since we can know that the prepared transaction came from the same publisher again by checking origin_lsn in TwoPhaseFileHeader I guess we can skip the PREPARE message only when the existing prepared transaction has the same LSN and the same identifier. To be exact, it’s still possible that the subscriber gets two PREPARE messages having the same LSN and name from two different publishers but it’s unlikely happen in practice. Regards, -- Masahiko Sawada EnterpriseDB: https://www.enterprisedb.com/

Re: [HACKERS] logical decoding of two-phase transactions

2020-12-16 Thread Masahiko Sawada
On Wed, Dec 16, 2020 at 6:22 PM Amit Kapila wrote: > > On Wed, Dec 16, 2020 at 1:04 PM Masahiko Sawada wrote: > > > > Thank you for updating the patch. I have two questions: > > > > - > > @@ -239,6 +239,19 @@ CREATE SUBSCRIPTION &

Commit fest manager for 2021-01

2020-12-18 Thread Masahiko Sawada
Hi all, The next commit fest is going to begin in two weeks. I would like to volunteer as commit fest manager for 2021-01 if the role is not filled and there are no objections. Regards, -- Masahiko Sawada EnterpriseDB: https://www.enterprisedb.com/

Re: Commit fest manager for 2021-01

2020-12-20 Thread Masahiko Sawada
permissions- Thanks! My usename is masahikosawada. Regards, -- Masahiko Sawada EnterpriseDB: https://www.enterprisedb.com/

Dependency isn't created between extension and schema

2020-12-20 Thread Masahiko Sawada
ted schema obviously depends on the extension when we created the schema specified in the schema option, I think we might want to create the dependency so that DROP EXTENSION drops the schema as well. I’ve attached the draft patch so that CREATE EXTENSION creates the dependency if it newly creates

Re: Deadlock between backend and recovery may not be detected

2020-12-21 Thread Masahiko Sawada
be detected because the deadlock involving a buffer pin isn't detected by CheckDeadLock(). I thought we can replace PROCSIG_RECOVERY_CONFLICT_STARTUP_DEADLOCK with PROCSIG_RECOVERY_CONFLICT_BUFFERPIN but it’s not good because the backend who has a buffer pin blocking the startup process and not waiting on a lock is also canceled after deadlock_timeout. We can have the backend return from RecoveryConflictInterrupt() when it received PROCSIG_RECOVERY_CONFLICT_BUFFERPIN and is not waiting on any lock, but it’s also not good because we cannot cancel the backend after max_standby_streaming_delay that has a buffer pin blocking the startup process. So I wonder if we can have a new signal. When the backend received this signal it returns from RecoveryConflictInterrupt() without deadlock check either if it’s not waiting on any lock or if it doesn’t have a buffer pin blocking the startup process. Otherwise it's cancelled. Regards, -- Masahiko Sawada EnterpriseDB: https://www.enterprisedb.com/

New IndexAM API controlling index vacuum strategies

2020-12-22 Thread Masahiko Sawada
rue in the currene patch). But for amvacuumcleanup() there is no the neutral state, lazy vacuum treats the default as true. Comment and feedback are very welcome. Regards, [1] https://www.postgresql.org/message-id/20200415233848.saqp72pcjv2y6ryi%40alap3.anarazel.de [2] https://www.postgresql.org/message-id/CAH2-

Re: Deadlock between backend and recovery may not be detected

2020-12-23 Thread Masahiko Sawada
On Tue, Dec 22, 2020 at 11:58 PM Fujii Masao wrote: > > > > On 2020/12/22 20:42, Fujii Masao wrote: > > > > > > On 2020/12/22 10:25, Masahiko Sawada wrote: > >> On Fri, Dec 18, 2020 at 6:36 PM Fujii Masao > >> wrote: > >>

Re: Commit fest manager for 2021-01

2020-12-24 Thread Masahiko Sawada
On Mon, Dec 21, 2020 at 5:21 PM Magnus Hagander wrote: > > On Sun, Dec 20, 2020 at 10:57 PM Masahiko Sawada > wrote: > > > > On Sun, Dec 20, 2020 at 10:27 PM Magnus Hagander > > wrote: > > > > > > On Sat, Dec 19, 2020 at 6:00 AM Michael Paquier

Re: Deadlock between backend and recovery may not be detected

2020-12-24 Thread Masahiko Sawada
On Wed, Dec 23, 2020 at 9:42 PM Fujii Masao wrote: > > > > On 2020/12/23 19:28, Masahiko Sawada wrote: > > On Tue, Dec 22, 2020 at 11:58 PM Fujii Masao > > wrote: > >> > >> > >> > >> On 2020/12/22 20:42, Fujii Masao wrote: > >&g

Commitfest 2021-01

2020-12-24 Thread Masahiko Sawada
s and that the tests pass. Check the state at http://cfbot.cputube.org/ If there is a long-running stale discussion, please send a short summary update about its current state, open questions, and TODOs. I hope it will encourage reviewers to pay more attention to the thread. Merry Christmas, --

Re: Commit fest manager for 2021-01

2020-12-24 Thread Masahiko Sawada
On Fri, Dec 25, 2020 at 2:57 PM Michael Paquier wrote: > > On Thu, Dec 24, 2020 at 07:29:37PM +0900, Masahiko Sawada wrote: > > Thank you. After re-logging in it looks the same as before but > > something will change on the CommitFest page? > > There should be a l

Re: Commit fest manager for 2021-01

2020-12-25 Thread Masahiko Sawada
On Fri, Dec 25, 2020 at 8:20 PM Magnus Hagander wrote: > > > On Fri, Dec 25, 2020 at 8:48 AM Michael Paquier wrote: >> >> On Fri, Dec 25, 2020 at 04:35:30PM +0900, Masahiko Sawada wrote: >> > Hmm, on the left of the logout button, I can see only the 'edit >

Re: Disable WAL logging to speed up data loading

2020-12-27 Thread Masahiko Sawada
vel = none’. I think we might need to provide a way to detect the changes other than reading XLOG_PARAMETER_CHANGE. Regards, -- Masahiko Sawada EnterpriseDB: https://www.enterprisedb.com/

Re: New IndexAM API controlling index vacuum strategies

2020-12-27 Thread Masahiko Sawada
On Thu, Dec 24, 2020 at 12:59 PM Peter Geoghegan wrote: > > On Tue, Dec 22, 2020 at 2:54 PM Masahiko Sawada wrote: > > I've started this separate thread from [1] for discussing the general > > API design of index vacuum. > > This is a very difficult and very import

Re: Add table AM 'tid_visible'

2020-12-28 Thread Masahiko Sawada
ions. Regards, [1] https://commitfest.postgresql.org/31/ [2] https://en.wikipedia.org/wiki/Anywhere_on_Earth -- Masahiko Sawada EnterpriseDB: https://www.enterprisedb.com/

Re: Parallel Full Hash Join

2020-12-28 Thread Masahiko Sawada
o the current HEAD may be required as almost two months passed since when this patch is submitted. Thanks for your contributions. Regards, [1] https://commitfest.postgresql.org/31/ [2] https://en.wikipedia.org/wiki/Anywhere_on_Earth Regards, -- Masahiko Sawada EnterpriseDB: https://www.enterprisedb.com/

Re: [PATCH] TAP test showing that pg_replication_slot_advance() works on standby

2020-12-28 Thread Masahiko Sawada
Thanks for your contributions. Regards, [1] https://commitfest.postgresql.org/31/ [2] https://en.wikipedia.org/wiki/Anywhere_on_Earth -- Masahiko Sawada EnterpriseDB: https://www.enterprisedb.com/

Re: Disable WAL logging to speed up data loading

2020-12-28 Thread Masahiko Sawada
On Mon, Dec 28, 2020 at 4:29 PM osumi.takami...@fujitsu.com wrote: > > Hello Sawada-San > > > On Monday, December 28, 2020 2:29 PM Masahiko Sawada > wrote: > > On Thu, Dec 3, 2020 at 12:14 PM osumi.takami...@fujitsu.com > > wrote: > > > > > > I&

Re: PATCH: Report libpq version and configuration

2020-12-28 Thread Masahiko Sawada
l, then you need to take no action. However, if you want your patch to be reviewed as part of the upcoming CommitFest, then you need to add it yourself before 2021-01-01 AOE[2]. Thanks for your contributions. Regards, [1] https://commitfest.postgresql.org/31/ [2] https://en.wikipedia.org/wiki/Anywhere_on_Earth -- Masahiko Sawada EnterpriseDB: https://www.enterprisedb.com/

Re: Table AM modifications to accept column projection lists

2020-12-28 Thread Masahiko Sawada
IndexScan and SeqScan can. We can possibly pass down scanCols > in ExecInitBitmapHeapScan(). We would have to modify the BitmapHeapScan > table AM calls to take a column projection bitmap. > > - There may be more callsites where we can pass a column projection list. > You sent in your patch to pgsql-hackers on Nov 14, but you did not post it to the next CommitFest[1]. If this was intentional, then you need to take no action. However, if you want your patch to be reviewed as part of the upcoming CommitFest, then you need to add it yourself and may need to rebase the patch to the current HEAD before 2021-01-01 AOE[2]. Thanks for your contributions. Regards, [1] https://commitfest.postgresql.org/31/ [2] https://en.wikipedia.org/wiki/Anywhere_on_Earth -- Masahiko Sawada EnterpriseDB: https://www.enterprisedb.com/

Re: [doc] adding way to examine the plan type of prepared statements

2020-12-28 Thread Masahiko Sawada
this was intentional, then you need to take no action. However, if you want your patch to be reviewed as part of the upcoming CommitFest, then you need to add it yourself before 2021-01-01 AoE[2]. Thanks for your contributions. Regards, [1] https://commitfest.postgresql.org/31/

Re: Rename of triggers for partitioned tables

2020-12-28 Thread Masahiko Sawada
/en.wikipedia.org/wiki/Anywhere_on_Earth Regards, -- Masahiko Sawada EnterpriseDB: https://www.enterprisedb.com/

Re: [Doc Patch] Clarify that CREATEROLE roles can GRANT default roles

2020-12-28 Thread Masahiko Sawada
fore 2021-01-01 AoE[2]. Thanks for your contributions. Regards, [1] https://commitfest.postgresql.org/31/ [2] https://en.wikipedia.org/wiki/Anywhere_on_Earth -- Masahiko Sawada EnterpriseDB: https://www.enterprisedb.com/

  1   2   3   4   5   6   7   8   9   10   >