Re: [HACKERS] Support for N synchronous standby servers - take 2

2016-02-27 Thread Masahiko Sawada
> At Fri, 26 Feb 2016 08:52:54 +0900, Masahiko Sawada >> wrote in >> > Previous patch could not parse one character standby name correctly. >> > Attached latest patch. >> >> I haven't looked it in detail but it won't work as you >> expected. flex

Re: [HACKERS] Support for N synchronous standby servers - take 2

2016-03-03 Thread Masahiko Sawada
se 'volatile WalSnd*' (only pointer type) > outside of walsender. > Or it should return the list of index number of > *WalSndCtl->walsnds*. SyncRepGetSyncStandbysUsingPriority() already returns the list of index number of "WalSndCtl->walsnd" as sync_list, no? As I mentioned a

Re: [HACKERS] Support for N synchronous standby servers - take 2

2016-03-03 Thread Masahiko Sawada
On Thu, Mar 3, 2016 at 11:30 PM, Masahiko Sawada wrote: > Hi, > > Thank you so much for reviewing this patch! > > All review comments regarding document and comment are fixed. > Attached latest v14 patch. > >> This accepts 'abc^Id' as a name, which is wrong b

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-05 Thread Masahiko Sawada
isibility map and return counts of the total number of bits of > each type that are set, and similarly for the page level bits. > Presumably that would be much faster than +1. > > I am also tempted to change the API to be a bit more friendly, > although I am not sure exactly how. Thi

Re: [HACKERS] Support for N synchronous standby servers - take 2

2016-03-06 Thread Masahiko Sawada
| main group1 | priority | {b,c}|1 | quorum | main b | | | | sync | group1 c | | | | potential |

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-07 Thread Masahiko Sawada
On Sat, Mar 5, 2016 at 11:25 PM, Masahiko Sawada wrote: > On Sat, Mar 5, 2016 at 1:25 AM, Robert Haas wrote: >> On Wed, Mar 2, 2016 at 6:41 PM, Tom Lane wrote: >>> Jim Nasby writes: >>>> On 3/2/16 4:21 PM, Peter Geoghegan wrote: >>>>> I think you sh

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-08 Thread Masahiko Sawada
On Tue, Mar 8, 2016 at 1:20 PM, Kyotaro HORIGUCHI wrote: > Hello, thank you for updating this tool. > > At Mon, 7 Mar 2016 14:03:08 -0500, Robert Haas wrote > in >> On Mon, Mar 7, 2016 at 12:41 PM, Masahiko Sawada >> wrote: >> > Attached latest version o

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-08 Thread Masahiko Sawada
hed patches are, - Change heap2 rmgr description - Add condition to pg_visibility - Fix typo in pgvisibility.sgml (Sorry for the late notice..) Regards, -- Masahiko Sawada diff --git a/doc/src/sgml/pgvisibility.sgml b/doc/src/sgml/pgvisibility.sgml index 6a98b55..cdd6a6f 100644 --- a/doc/src/sgm

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-09 Thread Masahiko Sawada
On Wed, Mar 9, 2016 at 3:38 AM, Robert Haas wrote: > On Tue, Mar 8, 2016 at 12:59 PM, Masahiko Sawada > wrote: >>> How about instead changing things so that we specifically reject >>> indexes? And maybe some kind of a check that will reject anything >>> tha

[HACKERS] Suspicious behaviour on applying XLOG_HEAP2_VISIBLE.

2016-03-09 Thread Masahiko Sawada
end, but If standby server receives FPI then relation cache would not be cleared. For example, after I applied attached patch to HEAD, (it might not be right way but) this problem seems to be resolved. Is this a bug? or not? Regards, -- Masahiko Sawada heap_xlog_visible_invalidate_cache.patch Desc

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-09 Thread Masahiko Sawada
Thank you for reviewing! Attached updated patch. On Thu, Mar 10, 2016 at 3:37 AM, Robert Haas wrote: > On Wed, Mar 9, 2016 at 9:09 AM, Masahiko Sawada > wrote: Attached latest 2 patches. >> * 000 patch : Incorporated the review comments and made rewriting >> logic more c

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-10 Thread Masahiko Sawada
On Thu, Mar 10, 2016 at 3:27 PM, Masahiko Sawada wrote: > Thank you for reviewing! > Attached updated patch. > > > On Thu, Mar 10, 2016 at 3:37 AM, Robert Haas wrote: >> On Wed, Mar 9, 2016 at 9:09 AM, Masahiko Sawada >> wrote: Attached latest 2 patches. >>

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-10 Thread Masahiko Sawada
dating patch. To deal with this problem, I've changed it so that pg_upgrade checks file size before conversion. And if fork file does not exist or size is 0 (empty), ignore. Attached latest patch. Regards, -- Masahiko Sawada 001_optimize_vacuum_by_frozen_bit_v40.patch Description: Binary da

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-10 Thread Masahiko Sawada
On Fri, Mar 11, 2016 at 6:16 AM, Robert Haas wrote: > On Thu, Mar 10, 2016 at 1:41 PM, Masahiko Sawada > wrote: >> On Fri, Mar 11, 2016 at 1:03 AM, Robert Haas wrote: >>> This 001 patch looks so little like what I was expecting that I >>> decided to start over fr

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-11 Thread Masahiko Sawada
On Sat, Mar 12, 2016 at 2:37 AM, Robert Haas wrote: > On Thu, Mar 10, 2016 at 10:47 PM, Masahiko Sawada > wrote: >>> Thanks. I adopted some of your suggested, rejected others, fixed a >>> few minor things that I missed previously, and committed this. If you >>>

Re: [HACKERS] Support for N synchronous standby servers - take 2

2016-03-22 Thread Masahiko Sawada
sertion error when master server is set up with empty s_s_names. Because current patch always tries to parse s_s_names and use it regardless value of parameter. Attached patch incorporates above comments. Please find it. Regards, -- Masahiko Sawada multi_sync_replication_v17.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Support for N synchronous standby servers - take 2

2016-03-23 Thread Masahiko Sawada
the sort of white-space character that isspace() returns true with. But current s_s_names allows us to specify such a node name. I guess that changing such behaviour is enough for fixing this issue. Thoughts? > >> By the way, the following documentation fix mentioned by Thomas, >> >> -to as 2-safe replication in computer science theory. >> +to as group-safe replication in computer science theory. >> >> should be restored if the discussion in the following message is >> true. And some supplemental description would be needed. >> >> http://www.postgresql.org/message-id/20160316.164833.188624159.horiguchi.kyot...@lab.ntt.co.jp > > Yeah, the document needs to be updated. I will do that. Regards, -- Masahiko Sawada -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Support for N synchronous standby servers - take 2

2016-03-24 Thread Masahiko Sawada
On Thu, Mar 24, 2016 at 2:26 PM, Kyotaro HORIGUCHI wrote: > Hello, > > At Thu, 24 Mar 2016 13:04:49 +0900, Masahiko Sawada > wrote in >> On Thu, Mar 24, 2016 at 11:34 AM, Fujii Masao wrote: >> > On Wed, Mar 23, 2016 at 5:32 PM, Kyotaro HORIGUCHI >> >

Re: [HACKERS] Support for N synchronous standby servers - take 2

2016-03-28 Thread Masahiko Sawada
On Fri, Mar 25, 2016 at 9:20 PM, Robert Haas wrote: > On Thu, Mar 24, 2016 at 9:29 AM, Masahiko Sawada > wrote: >> Also I felt a sense of discomfort regarding using [ and ] as a special >> character for priority method. >> Because (, ) and [, ] are a little similar

Re: [HACKERS] Support for N synchronous standby servers - take 2

2016-03-28 Thread Masahiko Sawada
syncrep.c:601:13: warning: variable ‘next’ set but not used > [-Wunused-but-set-variable] > |ListCell *next; > > > At Thu, 24 Mar 2016 22:29:01 +0900, Masahiko Sawada > wrote in >> >> > SyncRepInitConfig()->SyncRepFreeConfig() has already pfree'd th

Re: [HACKERS] Support for N synchronous standby servers - take 2

2016-03-30 Thread Masahiko Sawada
Tue, 29 Mar 2016 16:51:02 +0900, Fujii Masao wrote > in >> On Tue, Mar 29, 2016 at 4:23 PM, Kyotaro HORIGUCHI >> wrote: >> > Hello, >> > >> > At Mon, 28 Mar 2016 18:38:22 +0900, Masahiko Sawada >> > wrote in >> > >> > sawada

Re: [HACKERS] Support for N synchronous standby servers - take 2

2016-03-30 Thread Masahiko Sawada
On Wed, Mar 30, 2016 at 11:43 PM, Masahiko Sawada wrote: > On Tue, Mar 29, 2016 at 5:36 PM, Kyotaro HORIGUCHI > wrote: >> I personally don't think it needs such a survive measure. It is >> very small syntax and the parser reads very short text. If the >> parser f

Re: [HACKERS] Freeze avoidance of very large table.

2016-01-12 Thread Masahiko Sawada
On Mon, Dec 28, 2015 at 6:38 PM, Masahiko Sawada wrote: > On Mon, Dec 21, 2015 at 11:54 PM, Robert Haas wrote: >> On Mon, Dec 21, 2015 at 3:27 AM, Kyotaro HORIGUCHI >> wrote: >>> Hello, >>> >>> At Fri, 18 Dec 2015 12:09:43 -0500, Robert Haas >>

Re: [HACKERS] Freeze avoidance of very large table.

2016-01-17 Thread Masahiko Sawada
On Wed, Jan 13, 2016 at 12:16 AM, Masahiko Sawada wrote: > On Mon, Dec 28, 2015 at 6:38 PM, Masahiko Sawada > wrote: >> On Mon, Dec 21, 2015 at 11:54 PM, Robert Haas wrote: >>> On Mon, Dec 21, 2015 at 3:27 AM, Kyotaro HORIGUCHI >>> wrote: >>>> Hel

Re: [HACKERS] Support for N synchronous standby servers - take 2

2016-01-17 Thread Masahiko Sawada
er (s_s_name = '2, standby1, standby2) 2. Set up two standby servers 3. Standby1 is down 4. Create some contents on master (But transaction is not committed) 5. Cancel the #4 query. (Also confirm that the flush location of only standby2 makes progress) Regards, -- Masahiko Sawada -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Support for N synchronous standby servers - take 2

2016-01-18 Thread Masahiko Sawada
On Mon, Jan 18, 2016 at 1:20 PM, Michael Paquier wrote: > On Sun, Jan 17, 2016 at 11:09 PM, Masahiko Sawada > wrote: >> On Wed, Jan 13, 2016 at 1:54 AM, Alvaro Herrera wrote: >> * Confirm value of pg_stat_replication.sync_state (sync, async or potential) >> *

Re: [HACKERS] Support for N synchronous standby servers - take 2

2016-01-19 Thread Masahiko Sawada
On Tue, Jan 19, 2016 at 1:52 AM, Thom Brown wrote: > On 3 January 2016 at 13:26, Masahiko Sawada wrote: >> On Fri, Dec 25, 2015 at 7:21 AM, Thomas Munro >> wrote: >>> On Fri, Dec 25, 2015 at 8:50 AM, Masahiko Sawada >>> wrote: >>>> On Wed, Dec

Re: [HACKERS] Support for N synchronous standby servers - take 2

2016-01-19 Thread Masahiko Sawada
On Tue, Jan 19, 2016 at 2:55 PM, Michael Paquier wrote: > On Tue, Jan 19, 2016 at 1:40 AM, Masahiko Sawada > wrote: >> On Mon, Jan 18, 2016 at 1:20 PM, Michael Paquier >> wrote: >>> On Sun, Jan 17, 2016 at 11:09 PM, Masahiko Sawada >>> wrote: >>

[HACKERS] Existence check for suitable index in advance when concurrently refreshing.

2016-01-26 Thread Masahiko Sawada
that table doesn't has suitable index and table is large. it wastes time. I think we should check whether that materialized view can use concurrently refreshing or not in advance. The patch is attached. Please give me feedbacks. -- Regards, -- Masahiko S

[HACKERS] Improve tab completion for REFRESH MATERIALIZED VIEW

2016-01-26 Thread Masahiko Sawada
. Regards, -- Masahiko Sawada improve_tab_completion_for_refresh.patch Description: binary/octet-stream -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Existence check for suitable index in advance when concurrently refreshing.

2016-01-27 Thread Masahiko Sawada
On Wed, Jan 27, 2016 at 4:42 PM, Fujii Masao wrote: > On Tue, Jan 26, 2016 at 9:33 PM, Masahiko Sawada > wrote: >> Hi all, >> >> In concurrently refreshing materialized view, we check whether that >> materialized view has suitable index(unique and not having WHER

Re: [HACKERS] Support for N synchronous standby servers - take 2

2016-01-28 Thread Masahiko Sawada
On Thu, Jan 28, 2016 at 8:05 PM, Fujii Masao wrote: > On Wed, Jan 20, 2016 at 2:35 PM, Masahiko Sawada > wrote: >> On Tue, Jan 19, 2016 at 1:52 AM, Thom Brown wrote: >>> On 3 January 2016 at 13:26, Masahiko Sawada wrote: >>>> On Fri, Dec 25, 2015 at 7:21 AM, T

Re: [HACKERS] Several problems in tab-completions for SET/RESET

2016-01-28 Thread Masahiko Sawada
!ends_with(prev_wd, '=')) > COMPLETE_WITH_CONST("TO"); This change breaks tab completion for ALTER TABLE ... SET [WITH/LOGGED/UNLOGGED]. It think it should be > + else if (Matches2("SET", MatchAny) && Related

Re: [HACKERS] Support for N synchronous standby servers - take 2

2016-01-31 Thread Masahiko Sawada
On Sun, Jan 31, 2016 at 1:17 PM, Michael Paquier wrote: > On Thu, Jan 28, 2016 at 10:10 PM, Masahiko Sawada wrote: >> By the discussions so far, I'm planning to have several replication >> methods such as 'quorum', 'complex' in the feature, and the each &g

Re: [HACKERS] Support for N synchronous standby servers - take 2

2016-01-31 Thread Masahiko Sawada
On Sun, Jan 31, 2016 at 5:18 PM, Michael Paquier wrote: > On Sun, Jan 31, 2016 at 5:08 PM, Masahiko Sawada > wrote: >> On Sun, Jan 31, 2016 at 1:17 PM, Michael Paquier >> wrote: >>> On Thu, Jan 28, 2016 at 10:10 PM, Masahiko Sawada wrote: >>>> By the d

Re: [HACKERS] Support for N synchronous standby servers - take 2

2016-02-01 Thread Masahiko Sawada
On Sun, Jan 31, 2016 at 8:58 PM, Michael Paquier wrote: > On Sun, Jan 31, 2016 at 5:28 PM, Masahiko Sawada > wrote: >> On Sun, Jan 31, 2016 at 5:18 PM, Michael Paquier >> wrote: >>> On Sun, Jan 31, 2016 at 5:08 PM, Masahiko Sawada >>> wrote: >>&

Re: [HACKERS] Freeze avoidance of very large table.

2016-02-01 Thread Masahiko Sawada
On Tue, Feb 2, 2016 at 10:15 AM, Jim Nasby wrote: > On 2/1/16 4:59 PM, Alvaro Herrera wrote: >> >> Masahiko Sawada wrote: >> >>> Attached updated version patch. >>> Please review it. >> >> >> In pg_upgrade, the "page convert" f

Re: [HACKERS] Freeze avoidance of very large table.

2016-02-02 Thread Masahiko Sawada
On Tue, Feb 2, 2016 at 11:42 AM, Masahiko Sawada wrote: > On Tue, Feb 2, 2016 at 10:15 AM, Jim Nasby wrote: >> On 2/1/16 4:59 PM, Alvaro Herrera wrote: >>> >>> Masahiko Sawada wrote: >>> >>>> Attached updated version patch. >>>> Pl

Re: [HACKERS] Freeze avoidance of very large table.

2016-02-02 Thread Masahiko Sawada
On Tue, Feb 2, 2016 at 7:22 PM, Alvaro Herrera wrote: > Masahiko Sawada wrote: > >> I misunderstood. Sorry for noise. >> I agree with adding conversion method as a pageConverter routine. > > \o/ > >> This patch doesn't change page layout actually, but pageCon

Re: [HACKERS] Support for N synchronous standby servers - take 2

2016-02-02 Thread Masahiko Sawada
On Mon, Feb 1, 2016 at 11:28 PM, Fujii Masao wrote: > On Mon, Feb 1, 2016 at 5:36 PM, Masahiko Sawada wrote: >> On Sun, Jan 31, 2016 at 8:58 PM, Michael Paquier >> wrote: >>> On Sun, Jan 31, 2016 at 5:28 PM, Masahiko Sawada >>> wrote: >>>> On

Re: [HACKERS] Freeze avoidance of very large table.

2016-02-02 Thread Masahiko Sawada
y load)? The destination version is not fixed, it depends on new cluster version. I'm planning that convertLayoutVM_add_frozenbit.so is dynamically loaded and used only when rewriting of VM is required. If layout of VM will be changed again in the future, we could add other libraries for convert

Re: [HACKERS] Freeze avoidance of very large table.

2016-02-03 Thread Masahiko Sawada
On Tue, Feb 2, 2016 at 7:22 PM, Alvaro Herrera wrote: > Masahiko Sawada wrote: > >> I misunderstood. Sorry for noise. >> I agree with adding conversion method as a pageConverter routine. > > \o/ > >> This patch doesn't change page layout actually, but pageCon

Re: [HACKERS] Support for N synchronous standby servers - take 2

2016-02-05 Thread Masahiko Sawada
|0 | 2 | potential || 2 | | | london3 ||0 | 3 | potential || 2 | | | tokyo| quorum|1 | 3 | potential | {tokyo1,tokyo2} | 1 | | | tokyo1 ||0 | 1 | quorum | | 2 | | | tokyo2 ||0 | 1 | quorum | | 2 | | | (9 rows) Thought? Regards, -- Masahiko Sawada -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Support for N synchronous standby servers - take 2

2016-02-09 Thread Masahiko Sawada
s standbys enough to decide LSNs then this function return false. * SyncRepGetSyncStandbysFn : This function obtains array of walsnd positions of its standby members that are considered as synchronous. This implementation might not good in some reason, so please give me feedbacks. And I will creat

Re: [HACKERS] Support for N synchronous standby servers - take 2

2016-02-09 Thread Masahiko Sawada
On Wed, Feb 10, 2016 at 9:18 AM, Michael Paquier wrote: > On Wed, Feb 10, 2016 at 2:57 AM, Fujii Masao wrote: >> On Wed, Feb 10, 2016 at 1:36 AM, Masahiko Sawada >> wrote: >>> Attached first version dedicated language patch (document patch is not yet.) >> >&g

Re: [HACKERS] Freeze avoidance of very large table.

2016-02-11 Thread Masahiko Sawada
Thank you for reviewing this patch! On Wed, Feb 10, 2016 at 4:39 PM, Kyotaro HORIGUCHI wrote: > Hello, > > At Thu, 4 Feb 2016 02:32:29 +0900, Masahiko Sawada > wrote in >> On Tue, Feb 2, 2016 at 7:22 PM, Alvaro Herrera >> wrote: >> > Masahiko Sawada wrote:

Re: [HACKERS] Support for N synchronous standby servers - take 2

2016-02-15 Thread Masahiko Sawada
accurately same as what SplitIdentifierString does. We can not use special 4 characters (\,\ \[ \]) without using double-quoted string. Also if we use "(double-quote) character in double-quoted string, we should use ""(double double-quotes). For example, if application_name = 'hoge " bar&

Re: [HACKERS] Freeze avoidance of very large table.

2016-02-16 Thread Masahiko Sawada
ing things more > complex. I see _no_ reason for community Postgres to use a plugin > converter because we are going to need that code for every upgrade from > pre-9.6 to 9.6+, so why not just hard-code in the functions we need. We > can remove it once 9.5 is end-of-life. > Hm,

Re: [HACKERS] Freeze avoidance of very large table.

2016-02-16 Thread Masahiko Sawada
On Wed, Feb 17, 2016 at 12:02 AM, Bruce Momjian wrote: > On Tue, Feb 16, 2016 at 11:56:25PM +0900, Masahiko Sawada wrote: >> > I agreed on ripping out the converter plugin ability of pg_upgrade. >> > Remember pg_upgrade was originally written by EnterpriseDB staff, and I >

Re: [HACKERS] Freeze avoidance of very large table.

2016-02-16 Thread Masahiko Sawada
On Wed, Feb 17, 2016 at 4:08 AM, Bruce Momjian wrote: > On Tue, Feb 16, 2016 at 03:57:01PM -0300, Alvaro Herrera wrote: >> Masahiko Sawada wrote: >> > On Wed, Feb 17, 2016 at 12:02 AM, Bruce Momjian wrote: >> > > On Tue, Feb 16, 2016 at 11:56:25PM +0900, Masahiko Sa

Re: [HACKERS] Freeze avoidance of very large table.

2016-02-16 Thread Masahiko Sawada
On Wed, Feb 17, 2016 at 4:29 PM, Masahiko Sawada wrote: > On Wed, Feb 17, 2016 at 4:08 AM, Bruce Momjian wrote: >> On Tue, Feb 16, 2016 at 03:57:01PM -0300, Alvaro Herrera wrote: >>> Masahiko Sawada wrote: >>> > On Wed, Feb 17, 2016 at 12:02 AM, Bruce Momjian wrote

Re: [HACKERS] ALTER ROLE SET/RESET for multiple options

2016-02-17 Thread Masahiko Sawada
or user who wants to set several GUC parameter for each user. Especially the case where changing logging parameter for each user. But it might not bring us fantastic usability. Regards, -- Masahiko Sawada -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make chang

Re: [HACKERS] ALTER ROLE SET/RESET for multiple options

2016-02-17 Thread Masahiko Sawada
On Wed, Feb 17, 2016 at 7:14 PM, Robert Haas wrote: > On Wed, Feb 17, 2016 at 3:22 PM, Masahiko Sawada > wrote: >> On Sat, Feb 13, 2016 at 2:45 PM, Robert Haas wrote: >>> On Fri, Feb 12, 2016 at 1:35 PM, Payal Singh wrote: >>>> The feature seems to work as

Re: [HACKERS] Support for N synchronous standby servers - take 2

2016-02-24 Thread Masahiko Sawada
result structure. > > It can take old s_s_name format and new list format. We haven't > discussed how to add gruop names but I added it as "" > just before the # of syncronous standbys of [] and {} lists. > > Is this usable for further discussions? Thank you for

Re: [HACKERS] Support for N synchronous standby servers - take 2

2016-02-25 Thread Masahiko Sawada
lexer can pass both INT and NAME as char* to parser, and then parser regards them as integer or char*. It would be more simple. Thoughts? Thank you for giving lexer and parser example but I'm not sure that it makes thing more easier. It seems to make thing more complex. Attached patch h

Re: [HACKERS] Support for N synchronous standby servers - take 2

2016-02-25 Thread Masahiko Sawada
On Fri, Feb 26, 2016 at 1:23 AM, Masahiko Sawada wrote: > Attached latest patch includes document patch. > >> When I changed s_s_names to 'hoge*' and reloaded the configuration file, >> the server crashed unexpectedly with the following error message. >>

Re: [HACKERS] pg_basebackup wish list

2016-08-18 Thread Masahiko Sawada
On Fri, Aug 19, 2016 at 7:06 AM, Peter Eisentraut wrote: > On 7/12/16 9:55 PM, Masahiko Sawada wrote: >> And what I think is pg_baseback never remove the directory specified >> by -D option even if execution is failed. initdb command behaves so. >> I think it's he

Re: [HACKERS] synchronous_commit = remote_flush

2016-08-19 Thread Masahiko Sawada
ems. Is there any sensible way to do that, or is it >> not broken and I should pipe down, or is it just far too entrenched >> and never going to change? > > I don't see why we can't add "remote_flush" as a synonym for "on". Do > you have somethi

[HACKERS] Block level parallel vacuum WIP

2016-08-23 Thread Masahiko Sawada
processes index vacuum. Still lots of work to do but attached PoC patch. Feedback and suggestion are very welcome. Regards, -- Masahiko Sawada From b25d491a05a43fb7adf014b2580c71ec7adb75a2 Mon Sep 17 00:00:00 2001 From: Masahiko Sawada Date: Mon, 8 Aug 2016 16:43:35 -0700 Subject: [PATCH 1/2] All

Re: [HACKERS] Block level parallel vacuum WIP

2016-08-23 Thread Masahiko Sawada
bitmap. It can reduce the number of index whole scan during vacuum and make comparing the index item to the dead tuples faster. This is a listed on Todo list and I've implemented it. Regards, -- Masahiko Sawada -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Block level parallel vacuum WIP

2016-08-23 Thread Masahiko Sawada
On Tue, Aug 23, 2016 at 10:50 PM, Robert Haas wrote: > On Tue, Aug 23, 2016 at 7:02 AM, Masahiko Sawada > wrote: >> I'd like to propose block level parallel VACUUM. >> This feature makes VACUUM possible to use multiple CPU cores. > > Great. This is something t

Re: [HACKERS] Optimization for lazy_scan_heap

2016-08-24 Thread Masahiko Sawada
visible or all-frozen pages. For example, 1. Create very large table. 2. Execute VACUUM FREEZE to freeze all pages of table. 3. Measure the execution time of VACUUM FREEZE. I hope that the second VACUUM FREEZE become fast a little. I modified the comment, and attached v2 patch. Regards, -- Masah

Re: [HACKERS] Transactions involving multiple postgres foreign servers

2016-08-25 Thread Masahiko Sawada
On Fri, Aug 26, 2016 at 3:03 PM, Ashutosh Bapat wrote: > > > On Fri, Aug 26, 2016 at 11:22 AM, Masahiko Sawada > wrote: >> >> On Fri, Aug 26, 2016 at 1:32 PM, Vinayak Pokale >> wrote: >> > Hi All, >> > >> > Ashutosh proposed the fea

Re: [HACKERS] Transactions involving multiple postgres foreign servers

2016-08-26 Thread Masahiko Sawada
On Fri, Aug 26, 2016 at 3:13 PM, Ashutosh Bapat wrote: > > > On Fri, Aug 26, 2016 at 11:37 AM, Masahiko Sawada > wrote: >> >> On Fri, Aug 26, 2016 at 3:03 PM, Ashutosh Bapat >> wrote: >> > >> > >> > On Fri, Aug 26, 2016 at 11:22 AM, Mas

[HACKERS] VACUUM's ancillary tasks

2016-08-28 Thread Masahiko Sawada
n out of steam. And nobody picked it up again ... :( > > It may be worth to look at https://www.postgresql.org/message-id/CAMkU%3D1w3osJJ2FneELhhNRLxfZitDgp9FPHee08NT2FQFmz_pQ%40mail.gmail.com I've updated this patch to apply to current HEAD, can propose it to pg10. Regards, -- Masahiko Sawada -- Regards, -- Masahiko Sawada

Re: [HACKERS] Optimization for lazy_scan_heap

2016-09-05 Thread Masahiko Sawada
On Mon, Sep 5, 2016 at 6:27 PM, Simon Riggs wrote: > On 4 August 2016 at 05:57, Masahiko Sawada wrote: > >> While reviewing freeze map code, Andres pointed out[1] that >> lazy_scan_heap could accesses visibility map twice and its logic is >> seems a bit tricky. >>

Re: [HACKERS] Quorum commit for multiple synchronous replication.

2016-09-06 Thread Masahiko Sawada
On Tue, Sep 6, 2016 at 11:08 PM, Simon Riggs wrote: > On 29 August 2016 at 14:52, Fujii Masao wrote: >> On Sat, Aug 6, 2016 at 6:36 PM, Petr Jelinek wrote: >>> On 04/08/16 06:40, Masahiko Sawada wrote: >>>> >>>> On Wed, Aug 3, 2016 at 3:05 PM, Micha

Re: [HACKERS] Optimization for lazy_scan_heap

2016-09-06 Thread Masahiko Sawada
On Wed, Sep 7, 2016 at 1:47 AM, Robert Haas wrote: > On Mon, Sep 5, 2016 at 8:57 PM, Masahiko Sawada wrote: >>> What performance difference does this make, in a realistic use case? >> >> I have yet to measure performance effect but it would be effect for >> very l

Re: [HACKERS] Optimization for lazy_scan_heap

2016-09-08 Thread Masahiko Sawada
On Wed, Sep 7, 2016 at 4:11 PM, Simon Riggs wrote: > On 7 September 2016 at 04:13, Masahiko Sawada wrote: > >> Since current HEAD could scan visibility map twice, the execution time >> of Patched is approximately half of HEAD. > > Sounds good. > > To ensure we ar

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2016-09-08 Thread Masahiko Sawada
hunks are, in some sense, sorted, as you say, but you don't need > to do qsort() or anything like that. You're just replacing a single > flat array with a data structure that can be grown incrementally in > fixed-size chunks. > If we replaced dead_tuples with an array-of-array, isn't there negative performance impact for lazy_tid_reap()? As chunk is added, that performance would be decrease. Regards, -- Masahiko Sawada NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Quorum commit for multiple synchronous replication.

2016-09-08 Thread Masahiko Sawada
On Wed, Sep 7, 2016 at 12:47 AM, Masahiko Sawada wrote: > On Tue, Sep 6, 2016 at 11:08 PM, Simon Riggs wrote: >> On 29 August 2016 at 14:52, Fujii Masao wrote: >>> On Sat, Aug 6, 2016 at 6:36 PM, Petr Jelinek wrote: >>>> On 04/08/16 06:40, Masahiko Sawada wrote: &

Re: [HACKERS] Quorum commit for multiple synchronous replication.

2016-09-08 Thread Masahiko Sawada
uot;k (n1, n2, n3)" == "first k (n1, n2, n3)" doesn't break backward compatibility but most users would think "k(n1, n2, n3)" as quorum after introduced quorum. I wish we can change the s_s_names syntax of 9.6 to "first k(n1, n2, n3)" style before 9.6 releasing if

Re: [HACKERS] Optimization for lazy_scan_heap

2016-09-08 Thread Masahiko Sawada
On Thu, Sep 8, 2016 at 11:27 PM, Jim Nasby wrote: > On 9/8/16 3:03 AM, Masahiko Sawada wrote: >> >> Current manual vacuum doesn't output how may all_frozen pages we >> skipped according to visibility map. >> That's why I attached 0001 patch which makes the m

[HACKERS] DISABLE_PAGE_SKIPPING option for vacuumdb

2016-09-08 Thread Masahiko Sawada
case, I think that this option for vacuumdb would be convenient. Please give me feedback. Regards, -- Masahiko Sawada NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center disable_page_skipping_option_for_vacuumdb_v1.patch Description: Binary data -- Sent via pgsql-hackers

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2016-09-08 Thread Masahiko Sawada
the start > looking at various table stats. We can also be smart and change from bitmap > to traditional representation as we scan the table and see many more tuples > in the overflow region than we provisioned for. There will be some > challenges in converting representation mid

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2016-09-09 Thread Masahiko Sawada
On Fri, Sep 9, 2016 at 12:33 PM, Pavan Deolasee wrote: > > > On Thu, Sep 8, 2016 at 11:40 PM, Masahiko Sawada > wrote: >> >> >> >> Making the vacuum possible to choose between two data representations >> sounds good. >> I implemented the patch

Re: [HACKERS] DISABLE_PAGE_SKIPPING option for vacuumdb

2016-09-13 Thread Masahiko Sawada
On Wed, Sep 14, 2016 at 2:17 AM, Robert Haas wrote: > On Thu, Sep 8, 2016 at 1:32 PM, Masahiko Sawada wrote: >> Attached patch add --disable-page-skipping option to vacuumed command for >> 9.6. >> If by any chance the freeze map causes the serious data corruption bug >&

Re: [HACKERS] Block level parallel vacuum WIP

2016-09-15 Thread Masahiko Sawada
e the exclusive lock for extension on same relation. Since acquiring the exclusive lock for extension by multiple workers is regarded as locking from same locking group, multiple workers extend fsm or vm at the same time and end up with error. I thought that it might be involved with parallel update operation, so I'd like to discuss about this in advance. Regards, -- Masahiko Sawada NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2016-09-15 Thread Masahiko Sawada
suggested seems like a good idea but considering implementing parallel vacuum, it would be more complicated IMO. I think It's better the multiple process simply allocate memory space for its process than that the single process allocate huge memory space using complicated way. Regards,

Re: [HACKERS] Block level parallel vacuum WIP

2016-09-16 Thread Masahiko Sawada
On Thu, Sep 15, 2016 at 11:44 PM, Robert Haas wrote: > On Thu, Sep 15, 2016 at 7:21 AM, Masahiko Sawada > wrote: >> I'm implementing this patch but I need to resolve the problem >> regarding lock for extension by multiple parallel workers. >> In parallel vacuum,

Re: [HACKERS] Quorum commit for multiple synchronous replication.

2016-09-16 Thread Masahiko Sawada
On Fri, Sep 9, 2016 at 6:23 PM, Simon Riggs wrote: > On 8 September 2016 at 10:26, Masahiko Sawada wrote: > >> "k (n1, n2, n3)" == "first k (n1, n2, n3)" doesn't break backward >> compatibility but most users would think "k(n1, n2, n3)" a

Re: [HACKERS] pg_ctl promote wait

2016-09-22 Thread Masahiko Sawada
> + if (wait_seconds > 0) >> + { >> + sleep(1); >> + wait_seconds--; >> + continue; >> + } >> This may be better this pg_usleep() instead of sleep(). > > Committed with that adjustment. > Commit e7010ce seems t

Re: [HACKERS] Quorum commit for multiple synchronous replication.

2016-09-24 Thread Masahiko Sawada
syntax 'k (n1, n2)' to quorum commit. That is, 1. 'First k (n1, n2, n3)' means that the master server waits for ACKs from k standby servers whose name appear earlier in the list. 2. 'Any k (n1, n2, n3)' means that the master server waits for ACKs from any k listed

Re: [HACKERS] Transactions involving multiple postgres foreign servers

2016-09-26 Thread Masahiko Sawada
approach is for transaction involving foreign servers. Regards, -- Masahiko Sawada NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Transactions involving multiple postgres foreign servers

2016-09-27 Thread Masahiko Sawada
On Mon, Sep 26, 2016 at 9:07 PM, Ashutosh Bapat wrote: > On Mon, Sep 26, 2016 at 5:25 PM, Masahiko Sawada > wrote: >> On Mon, Sep 26, 2016 at 7:28 PM, Ashutosh Bapat >> wrote: >>> My original patch added code to manage the files for 2 phase >>> transactio

Re: [HACKERS] Transactions involving multiple postgres foreign servers

2016-09-27 Thread Masahiko Sawada
On Tue, Sep 27, 2016 at 9:06 PM, Ashutosh Bapat wrote: > On Tue, Sep 27, 2016 at 2:54 PM, Masahiko Sawada > wrote: >> On Mon, Sep 26, 2016 at 9:07 PM, Ashutosh Bapat >> wrote: >>> On Mon, Sep 26, 2016 at 5:25 PM, Masahiko Sawada >>> wrote: >>>>

Re: [HACKERS] Issues with logical replication

2017-10-03 Thread Masahiko Sawada
;> We have locked the same tuple in RelationFindReplTupleByIndex() just >> before this gets called and didn't get the same error. I guess we do >> something wrong with snapshots. Will need to investigate more. >> > > Okay, so it's not snapshot. It's the fact tha

Re: [HACKERS] Block level parallel vacuum WIP

2017-10-04 Thread Masahiko Sawada
On Tue, Sep 19, 2017 at 4:31 PM, Masahiko Sawada wrote: > On Tue, Sep 19, 2017 at 3:33 PM, Thomas Munro > wrote: >> On Fri, Sep 8, 2017 at 10:37 PM, Masahiko Sawada >> wrote: >>> Since v4 patch conflicts with current HEAD I attached the latest version >

[HACKERS] Re: [BUGS] 10.0: Logical replication doesn't execute BEFORE UPDATE OF trigger

2017-10-09 Thread Masahiko Sawada
` > k | v > +--- > k1 | v1;v4 > ``` > > This is what would happen if all insert/update queries would have been > executed on one instance. > > Actual result > - > > Replication fails, log contains: > > ``` > [3227] ERROR: duplic

[HACKERS] Fix a typo in execReplication.c

2017-10-09 Thread Masahiko Sawada
Hi, Attached a patch for $subject. ISTM these are mistakes of copy-and-paste. Regards, -- Masahiko Sawada NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center fix_comment_in_execReplication_c.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql

[HACKERS] Re: [BUGS] 10.0: Logical replication doesn't execute BEFORE UPDATE OF trigger

2017-10-10 Thread Masahiko Sawada
On Tue, Oct 10, 2017 at 11:29 AM, Masahiko Sawada wrote: > On Mon, Oct 9, 2017 at 11:13 PM, Aleksander Alekseev > wrote: >> Hi hackers, >> >> I've found something that looks like a bug. >> >> Steps to reproduce >> -- >> >&

Re: [HACKERS] Slow synchronous logical replication

2017-10-10 Thread Masahiko Sawada
send at least the begin and commit data to all subscriptions and then wait for the reply from them but can we skip to wait them, for example, when the walsender actually didn't send any data modified by the transaction? Regards, -- Masahiko Sawada NIPPON TELEGRAPH AND TELEPHONE CORPORATI

Re: [HACKERS] Fix a typo in execReplication.c

2017-10-11 Thread Masahiko Sawada
On Thu, Oct 12, 2017 at 5:02 AM, Robert Haas wrote: > On Mon, Oct 9, 2017 at 10:59 PM, Masahiko Sawada > wrote: >> Attached a patch for $subject. ISTM these are mistakes of copy-and-paste. > > Committed, but isn't the code itself wrong as well in the DELETE case? >

Re: [HACKERS] Fix a typo in execReplication.c

2017-10-13 Thread Masahiko Sawada
On Thu, Oct 12, 2017 at 11:30 PM, Robert Haas wrote: > On Thu, Oct 12, 2017 at 6:55 AM, Petr Jelinek > wrote: >> Thanks for the patch, looks correct to me. > > Committed and back-patched to v10. > Thank you! Regards, -- Masahiko Sawada NIPPON TELEGRAPH AND TELEPHONE

Re: [HACKERS] GUC for cleanup indexes threshold.

2017-10-16 Thread Masahiko Sawada
that I really wanted to skip. Regards, -- Masahiko Sawada NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [PATCH] pageinspect function to decode infomasks

2017-10-17 Thread Masahiko Sawada
ke FROZEN, and hide COMMITTED|INVALID. Similar for other > combos. > FWIW, I agree with this direction. ISTM the showing the raw flags by default and having an option to show combined flags would be a right way. I sometimes wanted to have the same mechanism for lp_flags but maybe it should be

Re: [HACKERS] pgbench: Skipping the creating primary keys after initialization

2017-10-18 Thread Masahiko Sawada
gt; > The patch needs a rebase in the documentation because of the xml-ilization > of the sgml doc. > Thank you for the notification! Attached rebased patch. Regards, -- Masahiko Sawada NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center pgbench_custom_initializati

Re: [HACKERS] show "aggressive" or not in autovacuum logs

2017-10-19 Thread Masahiko Sawada
On Tue, Sep 5, 2017 at 3:41 PM, Kyotaro HORIGUCHI wrote: > Thank you for the opinions. > > At Tue, 29 Aug 2017 15:00:57 +0900, Masahiko Sawada > wrote in >> On Tue, Aug 29, 2017 at 10:16 AM, Robert Haas wrote: >> > On Mon, Aug 28, 2017 at 5:26 AM, Ky

[HACKERS] Fix a typo in libpq/auth.c

2017-10-19 Thread Masahiko Sawada
Hi, Attached a patch for $subject. s/RAIDUS/RADIUS/ Regards, -- Masahiko Sawada NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center fix_typo_in_auth_c.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes

Re: [HACKERS] More stats about skipped vacuums

2017-10-20 Thread Masahiko Sawada
e of a table bloat. The same > discussion could be applied to analyze but it might be the > another issue. > > There may be a better way to indicate the vacuum soundness. Any > opinions and suggestions are welcome. > > I'm going to make a patch to do the 'forma

  1   2   3   4   5   6   7   8   >