Re: Add const qualifiers to internal range type APIs

2019-10-30 Thread Peter Eisentraut
On 2019-10-29 21:11, Andres Freund wrote: On 2019-10-29 16:48:24 +0100, Peter Eisentraut wrote: On 2019-10-28 14:05, Robert Haas wrote: Just out of curiosity, what is the motivation for this? I don't remember. :-) I had this code lying around from earlier "adventures in const", probably rela

Re: [HACKERS] Block level parallel vacuum

2019-10-30 Thread Dilip Kumar
On Thu, Oct 31, 2019 at 11:33 AM Dilip Kumar wrote: > > On Tue, Oct 29, 2019 at 1:59 PM Masahiko Sawada wrote: > > Actually after increased shared_buffer I got expected results: > > > > * Test1 (after increased shared_buffers) > > normal : 2807 ms (hit 56295, miss 2, dirty 3, total 56300) >

Re: v12.0: ERROR: could not find pathkey item to sort

2019-10-30 Thread Amit Langote
Thanks for checking. On Thu, Oct 31, 2019 at 1:09 AM Tom Lane wrote: > Also, the existing logic around eclass_indexes is that it's only > set for baserels and we know it is valid after we've finished > EC merging. I don't much like modifying add_child_rel_equivalences > to have some different op

Re: [HACKERS] Block level parallel vacuum

2019-10-30 Thread Dilip Kumar
On Tue, Oct 29, 2019 at 1:59 PM Masahiko Sawada wrote: > Actually after increased shared_buffer I got expected results: > > * Test1 (after increased shared_buffers) > normal : 2807 ms (hit 56295, miss 2, dirty 3, total 56300) > 2 workers : 2840 ms (hit 56295, miss 2, dirty 3, total 56300) > 1

Re: Creating foreign key on partitioned table is too slow

2019-10-30 Thread Tom Lane
David Rowley writes: > In Ottawa this year, Andres and I briefly talked about the possibility > of making a series of changes to how equalfuncs.c works. The idea was > to make it easy by using some pre-processor magic to allow us to > create another version of equalfuncs which would let us have an

Re: tableam vs. TOAST

2019-10-30 Thread Prabhat Sahu
On Wed, Oct 30, 2019 at 9:46 PM Robert Haas wrote: > On Wed, Oct 30, 2019 at 3:49 AM Prabhat Sahu < > prabhat.s...@enterprisedb.com> wrote: > >> While testing the Toast patch(PG+v7 patch) I found below server crash. >> System configuration: >> VCPUs: 4, RAM: 8GB, Storage: 320GB >> >> This issue i

Re: [BUG] Partition creation fails after dropping a column and adding a partial index

2019-10-30 Thread Michael Paquier
On Tue, Oct 29, 2019 at 01:16:58PM +0900, Michael Paquier wrote: > Yes, something looks wrong with that. I have not looked at it in > details yet though. I'll see about that tomorrow. So.. When building the attribute map for a cloned index (with either LIKE during the transformation or for part

Re: Allow CREATE OR REPLACE VIEW to rename the columns

2019-10-30 Thread Tom Lane
Fujii Masao writes: > Currently CREATE OR REPLACE VIEW command fails if the column names > are changed. That is, I believe, intentional. It's an effective aid to catching mistakes in view redefinitions, such as misaligning the new set of columns relative to the old. That's particularly importan

Re: Allow cluster_name in log_line_prefix

2019-10-30 Thread Tatsuo Ishii
> Hi folks > > I was recently surprised to notice that log_line_prefix doesn't support a > cluster_name placeholder. I suggest adding one. If I don't hear objections > I'll send a patch. I think it'd be a good thing for users. Best regards, -- Tatsuo Ishii SRA OSS, Inc. Japan English: http://www

Re: Add SQL function to show total block numbers in the relation

2019-10-30 Thread Michael Paquier
On Wed, Oct 30, 2019 at 10:09:47AM -0400, Tom Lane wrote: > btkimurayuzk writes: >> I propose new simple sql query, which shows total block numbers in the >> relation. >> ... >> Of cource, we can know this value such as >> select (pg_relation_size('t') / >> current_setting('block_size')::bigint)

Re: Allow cluster_name in log_line_prefix

2019-10-30 Thread Thomas Munro
On Mon, Oct 28, 2019 at 3:33 PM Craig Ringer wrote: > I was recently surprised to notice that log_line_prefix doesn't support a > cluster_name placeholder. I suggest adding one. If I don't hear objections > I'll send a patch. +1

Allow CREATE OR REPLACE VIEW to rename the columns

2019-10-30 Thread Fujii Masao
Hi, Currently CREATE OR REPLACE VIEW command fails if the column names are changed. For example, =# CREATE VIEW test AS SELECT 0 AS a; =# CREATE OR REPLACE VIEW test AS SELECT 0 AS x; ERROR: cannot change name of view column "a" to "x" I'd like to propose the attached patch that all

Re: Problem with synchronous replication

2019-10-30 Thread Michael Paquier
On Wed, Oct 30, 2019 at 05:43:04PM +0900, Kyotaro Horiguchi wrote: > At Wed, 30 Oct 2019 17:21:17 +0900, Fujii Masao wrote > in >> This change causes every ending backends to always take the exclusive lock >> even when it's not in SyncRep queue. This may be problematic, for example, >> when term

Re: Problem with synchronous replication

2019-10-30 Thread Michael Paquier
On Wed, Oct 30, 2019 at 12:34:28PM +0900, Kyotaro Horiguchi wrote: > If we do that strictly, other functions like > SyncRepGetOldestSyncRecPtr need the same Assert()s. I think static > functions don't need Assert() and caution in their comments would be > enough. Perhaps. I'd rather be careful th

Re: Creating foreign key on partitioned table is too slow

2019-10-30 Thread Andres Freund
Hi, On 2019-10-31 11:19:05 +1300, David Rowley wrote: > In Ottawa this year, Andres and I briefly talked about the possibility > of making a series of changes to how equalfuncs.c works. The idea was > to make it easy by using some pre-processor magic to allow us to > create another version of equa

Re: Zedstore - compressed in-core columnar storage

2019-10-30 Thread Taylor Vesely
Alex Wang and I have been doing some performance analysis of the most recent version of the zedstore branch, and have some interesting statistics to share. We specifically focused on TPC-DS query 2, because it plays to what should be the strength of zedstore- namely it does a full table scan of on

Re: Creating foreign key on partitioned table is too slow

2019-10-30 Thread David Rowley
On Thu, 31 Oct 2019 at 07:30, Tomas Vondra wrote: > > On Thu, Oct 24, 2019 at 04:28:38PM -0700, Andres Freund wrote: > >Hi, > > > >On 2019-10-23 05:59:01 +, kato-...@fujitsu.com wrote: > >> To benchmark with tpcb model, I tried to create a foreign key in the > >> partitioned history table, bu

Re: Thoughts on nbtree with logical/varwidth table identifiers, v12 on-disk representation

2019-10-30 Thread Peter Geoghegan
On Wed, Oct 30, 2019 at 1:25 PM Andres Freund wrote: > I assume you mean that the index would dynamically recognize when it > needs the wider tids ("for the higher portion")? If so, yea, that makes > sense to me. Would that need to encode the 6/8byteness of a tid on a > per-element basis? Or are y

Re: v12.0: ERROR: could not find pathkey item to sort

2019-10-30 Thread David Rowley
On Thu, 31 Oct 2019 at 05:09, Tom Lane wrote: > David --- much of the complexity here comes from the addition of > the eclass_indexes infrastructure, so do you have any thoughts? Hindsight makes me think I should have mentioned in the comment for eclass_indexes that it's only used for simple rels

Re: Thoughts on nbtree with logical/varwidth table identifiers, v12 on-disk representation

2019-10-30 Thread Andres Freund
Hi, On 2019-10-30 12:37:50 -0700, Peter Geoghegan wrote: > On Wed, Oct 30, 2019 at 12:03 PM Andres Freund wrote: > > I'd much rather not entrench this further, even leaving global indexes > > aside. The 4 byte block number is a significant limitation for heap > > tables too, and we should lift th

pgstat.c has brittle response to transient problems

2019-10-30 Thread Tom Lane
While fooling with the NetBSD-vs-libpython issue noted in a nearby thread, I observed that the core regression tests sometimes hang up in the "stats" test on this platform (NetBSD 8.1/amd64). Investigation found that the stats collector process was sometimes exiting like this: 2019-10-29 19:38:14

Re: MarkBufferDirtyHint() and LSN update

2019-10-30 Thread Tomas Vondra
On Wed, Oct 30, 2019 at 02:44:18PM +0100, Antonin Houska wrote: Please consider this scenario (race conditions): 1. FlushBuffer() has written the buffer but hasn't yet managed to clear the BM_DIRTY flag (however BM_JUST_DIRTIED could be cleared by now). 2. Another backend modified a hint bit an

Re: Thoughts on nbtree with logical/varwidth table identifiers, v12 on-disk representation

2019-10-30 Thread Peter Geoghegan
On Wed, Oct 30, 2019 at 12:03 PM Andres Freund wrote: > I'd much rather not entrench this further, even leaving global indexes > aside. The 4 byte block number is a significant limitation for heap > tables too, and we should lift that at some point not too far away. > Then there's also other AMs t

Re: Parallel leader process info in EXPLAIN

2019-10-30 Thread Tomas Vondra
On Wed, Oct 30, 2019 at 10:39:04AM -0700, Peter Geoghegan wrote: On Wed, Oct 30, 2019 at 9:24 AM Melanie Plageman wrote: Checked out the patches a bit and noticed that the tuplesort instrumentation uses spaceUsed and I saw this comment in tuplesort_get_stats() might it be worth trying out th

Re: Thoughts on nbtree with logical/varwidth table identifiers, v12 on-disk representation

2019-10-30 Thread Andres Freund
Hi, On 2019-10-30 11:33:21 -0700, Peter Geoghegan wrote: > On Mon, Apr 22, 2019 at 9:35 AM Andres Freund wrote: > > On 2019-04-21 17:46:09 -0700, Peter Geoghegan wrote: > > > Andres has suggested that I work on teaching nbtree to accommodate > > > variable-width, logical table identifiers, such a

Re: Thoughts on nbtree with logical/varwidth table identifiers, v12 on-disk representation

2019-10-30 Thread Peter Geoghegan
On Mon, Apr 22, 2019 at 9:35 AM Andres Freund wrote: > On 2019-04-21 17:46:09 -0700, Peter Geoghegan wrote: > > Andres has suggested that I work on teaching nbtree to accommodate > > variable-width, logical table identifiers, such as those required for > > indirect indexes, or clustered indexes, w

Re: Creating foreign key on partitioned table is too slow

2019-10-30 Thread Tomas Vondra
On Thu, Oct 24, 2019 at 04:28:38PM -0700, Andres Freund wrote: Hi, On 2019-10-23 05:59:01 +, kato-...@fujitsu.com wrote: To benchmark with tpcb model, I tried to create a foreign key in the partitioned history table, but backend process killed by OOM. the number of partitions is 8192. I tr

Re: Parallel leader process info in EXPLAIN

2019-10-30 Thread Peter Geoghegan
On Wed, Oct 30, 2019 at 9:24 AM Melanie Plageman wrote: > Checked out the patches a bit and noticed that the tuplesort > instrumentation uses spaceUsed and I saw this comment in > tuplesort_get_stats() > might it be worth trying out the memory accounting API > 5dd7fc1519461548eebf26c33eac6878ea3e

Re: PL/Python fails on new NetBSD/PPC 8.0 install

2019-10-30 Thread Tom Lane
I wrote: > Thomas Munro writes: >> From a quick look at the relevant trees, isn't the problem here that >> cpython thinks it can reserve pthread_t value -1 (or rather, that >> number cast to unsigned long, which is the type it uses for its own >> thread IDs): > Yeah, this. I shall now go rant at

Re: Proposal: Global Index

2019-10-30 Thread Andres Freund
Hi, On 2019-10-30 13:05:57 -0400, Tom Lane wrote: > Peter Geoghegan writes: > > On Wed, Oct 30, 2019 at 9:23 AM Tom Lane wrote: > >> Well, the *effects* of the feature seem desirable, but that doesn't > >> mean that we want an implementation that actually has a shared index. > >> As soon as you

Re: [Proposal] Add accumulated statistics

2019-10-30 Thread Pavel Stehule
út 15. 1. 2019 v 2:14 odesílatel Tsunakawa, Takayuki < tsunakawa.ta...@jp.fujitsu.com> napsal: > From: Pavel Stehule [mailto:pavel.steh...@gmail.com] > > the cumulated lock statistics maybe doesn't help with debugging - but it > > is very good indicator of database (in production usage) health. >

Re: Proposal: Global Index

2019-10-30 Thread Tom Lane
Peter Geoghegan writes: > On Wed, Oct 30, 2019 at 9:23 AM Tom Lane wrote: >> Well, the *effects* of the feature seem desirable, but that doesn't >> mean that we want an implementation that actually has a shared index. >> As soon as you do that, you've thrown away most of the benefits of >> having

Re: Proposal: Global Index

2019-10-30 Thread Peter Geoghegan
On Wed, Oct 30, 2019 at 9:23 AM Tom Lane wrote: > Well, the *effects* of the feature seem desirable, but that doesn't > mean that we want an implementation that actually has a shared index. > As soon as you do that, you've thrown away most of the benefits of > having a partitioned data structure i

Re: Parallel leader process info in EXPLAIN

2019-10-30 Thread Melanie Plageman
On Wed, Oct 23, 2019 at 12:30 AM Thomas Munro wrote: > > While working on some slides explaining EXPLAIN, I couldn't resist the > urge to add the missing $SUBJECT. The attached 0001 patch gives the > following: > > Gather ... time=0.146..33.077 rows=1 loops=1) > Workers Planned: 2 > Workers

Re: Proposal: Global Index

2019-10-30 Thread Tom Lane
Robert Haas writes: > On Wed, Oct 30, 2019 at 10:13 AM Tom Lane wrote: >> I believe that the current design of partitioning is explicitly intended >> to avoid the need for such a construct. It'd be absolutely disastrous >> to have such a thing from many standpoints, including the breadth of >> l

Re: tableam vs. TOAST

2019-10-30 Thread Robert Haas
On Wed, Oct 30, 2019 at 3:49 AM Prabhat Sahu wrote: > While testing the Toast patch(PG+v7 patch) I found below server crash. > System configuration: > VCPUs: 4, RAM: 8GB, Storage: 320GB > > This issue is not frequently reproducible, we need to repeat the same > testcase multiple times. > I wonde

Re: Proposal: Global Index

2019-10-30 Thread Robert Haas
On Wed, Oct 30, 2019 at 10:13 AM Tom Lane wrote: > I believe that the current design of partitioning is explicitly intended > to avoid the need for such a construct. It'd be absolutely disastrous > to have such a thing from many standpoints, including the breadth of > locking needed to work with

Re: v12.0: ERROR: could not find pathkey item to sort

2019-10-30 Thread Tom Lane
Amit Langote writes: > Attached updated patches. [ looks at that... ] I seriously, seriously dislike what you did in build_join_rel, ie adding the new joinrel to the global data structures before it's fully filled in. That's inevitably going to bite us on the ass someday, and you couldn't even

Re: PL/Python fails on new NetBSD/PPC 8.0 install

2019-10-30 Thread Tom Lane
Thomas Munro writes: > On Wed, Oct 30, 2019 at 9:25 AM Tom Lane wrote: >> What I'm inclined to do is go file a bug report saying that this >> behavior contradicts both POSIX and NetBSD's own man page, and >> see what they say about that. So I went and filed that bug, http://gnats.netbsd.org/cgi

Re: WIP/PoC for parallel backup

2019-10-30 Thread Asif Rehman
On Mon, Oct 28, 2019 at 8:29 PM Robert Haas wrote: > On Mon, Oct 28, 2019 at 10:03 AM Asif Rehman > wrote: > > I have updated the patch to include the changes suggested by Jeevan. > This patch also implements the thread workers instead of > > processes and fetches a single file at a time. The ta

Re: Proposal: Global Index

2019-10-30 Thread Tom Lane
Ibrar Ahmed writes: > A global index by very definition is a single index on the parent table > that maps to many > underlying table partitions. I believe that the current design of partitioning is explicitly intended to avoid the need for such a construct. It'd be absolutely disastrous to have

Re: Add SQL function to show total block numbers in the relation

2019-10-30 Thread Tom Lane
btkimurayuzk writes: > I propose new simple sql query, which shows total block numbers in the > relation. > ... > Of cource, we can know this value such as > select (pg_relation_size('t') / > current_setting('block_size')::bigint)::int; I don't really see why the existing solution isn't suffici

Re: Binary support for pgoutput plugin

2019-10-30 Thread Dave Cramer
On Sun, 27 Oct 2019 at 11:00, Dmitry Dolgov <9erthali...@gmail.com> wrote: > > On Mon, Jun 17, 2019 at 10:29:26AM -0400, Dave Cramer wrote: > > > Which is what I have done. Thanks > > > > > > I've attached both patches for comments. > > > I still have to add documentation. > > > > Additional patch

Remove HAVE_LONG_LONG_INT

2019-10-30 Thread Peter Eisentraut
HAVE_LONG_LONG_INT is now implied by the requirement for C99, so the separate Autoconf check can be removed. The uses are almost all in ecpg code, and AFAICT the check was originally added specifically for ecpg. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Developmen

MarkBufferDirtyHint() and LSN update

2019-10-30 Thread Antonin Houska
Please consider this scenario (race conditions): 1. FlushBuffer() has written the buffer but hasn't yet managed to clear the BM_DIRTY flag (however BM_JUST_DIRTIED could be cleared by now). 2. Another backend modified a hint bit and called MarkBufferDirtyHint(). 3. In MarkBufferDirtyHint(), if X

Re: Problem with synchronous replication

2019-10-30 Thread Michael Paquier
On Wed, Oct 30, 2019 at 05:21:17PM +0900, Fujii Masao wrote: > This change causes every ending backends to always take the exclusive lock > even when it's not in SyncRep queue. This may be problematic, for example, > when terminating multiple backends at the same time? If yes, > it might be better

Re: Unix-domain socket support on Windows

2019-10-30 Thread Peter Eisentraut
To move this topic a long, I'll submit some preparatory patches in a committable order. First is the patch to deal with getpeereid() that was already included in the previous patch series. This is just some refactoring that reduces the difference between Windows and other platforms and prepar

Re: Remove one use of IDENT_USERNAME_MAX

2019-10-30 Thread Peter Eisentraut
On 2019-10-29 15:34, Tom Lane wrote: Peter Eisentraut writes: On 2019-10-28 14:45, Tom Lane wrote: Kyotaro Horiguchi writes: In think one of the reasons for the coding is the fact that *pw is described to be placed in the static area, which can be overwritten by succeeding calls to getpw*()

Re: pgbench - extend initialization phase control

2019-10-30 Thread Fujii Masao
On Mon, Oct 28, 2019 at 10:36 PM Fabien COELHO wrote: > > > Hello Masao-san, > > >> Maybe. If you cannot check, you can only guess. Probably it should be > >> small, but the current version does not allow to check whether it is so. > > > > Could you elaborate what you actually want to measure the

Re: v12.0: ERROR: could not find pathkey item to sort

2019-10-30 Thread Amit Langote
Thanks for taking a look and sorry about the delay in replying. On Fri, Oct 25, 2019 at 1:51 AM Tom Lane wrote: > Amit Langote writes: > > On Mon, Oct 14, 2019 at 11:54 PM Tom Lane wrote: > >> In view of the proposed patches being dependent on some other > >> 13-only changes, I wonder if we sho

Re: [HACKERS] Block level parallel vacuum

2019-10-30 Thread Masahiko Sawada
On Mon, Oct 28, 2019 at 3:50 PM Amit Kapila wrote: > > On Sun, Oct 27, 2019 at 12:52 PM Dilip Kumar wrote: > > > > On Fri, Oct 25, 2019 at 9:19 PM Masahiko Sawada > > wrote: > > > > > > > > I haven't yet read the new set of the patch. But, I have noticed one > > thing. That we are getting the

Re: Problem with synchronous replication

2019-10-30 Thread Kyotaro Horiguchi
Hello. At Wed, 30 Oct 2019 17:21:17 +0900, Fujii Masao wrote in > This change causes every ending backends to always take the exclusive lock > even when it's not in SyncRep queue. This may be problematic, for example, > when terminating multiple backends at the same time? If yes, > it might be

Proposal: Global Index

2019-10-30 Thread Ibrar Ahmed
A global index by very definition is a single index on the parent table that maps to many underlying table partitions. The parent table itself does not have any underlying storage, so it must, therefore, retrieve the data satisfying index constraints from the underlying tables. In very crude terms,

Re: Problem with synchronous replication

2019-10-30 Thread Fujii Masao
On Wed, Oct 30, 2019 at 4:16 PM lingce.ldm wrote: > > On Oct 29, 2019, at 18:50, Kyotaro Horiguchi wrote: > > > Hello. > > At Fri, 25 Oct 2019 15:18:34 +0800, "Dongming Liu" > wrote in > > > Hi, > > I recently discovered two possible bugs about synchronous replication. > > 1. SyncRepCleanupAtPr

Re: Join Correlation Name

2019-10-30 Thread Fabien COELHO
Bonjour Vik, Is quoting the spec good enough? SQL:2016 Part 2 Foundation Section 7.10 : Ah, this is the one information I did not have when reviewing Peter's patch. ::=     USING[ AS ] ::=     I think possibly what the spec says (and that neither my patch nor Peter's implements)

Re: tableam vs. TOAST

2019-10-30 Thread Prabhat Sahu
Hi All, While testing the Toast patch(PG+v7 patch) I found below server crash. System configuration: VCPUs: 4, RAM: 8GB, Storage: 320GB This issue is not frequently reproducible, we need to repeat the same testcase multiple times. CREATE OR REPLACE FUNCTION toast_chunks_cnt_func(p1 IN text) RE

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2019-10-30 Thread Dilip Kumar
On Wed, Oct 30, 2019 at 9:38 AM vignesh C wrote: > I have noticed one more problem in the logic of setting the logical decoding work mem from the create subscription command. Suppose in subscription command we don't give the work mem then it sends the garbage value to the walsender and the walsen

Add SQL function to show total block numbers in the relation

2019-10-30 Thread btkimurayuzk
Hello, I propose new simple sql query, which shows total block numbers in the relation. I now reviewing this patch (https://commitfest.postgresql.org/25/2211/) and I think, it is usefull for knowing how many blocks there are in the relation to determine whether we use VACUUM RESUME or not.

Re: RFC: split OBJS lines to one object per line

2019-10-30 Thread Michael Paquier
On Tue, Oct 29, 2019 at 11:32:09PM -0700, Andres Freund wrote: > Cool. Any opinion on whether to got for > > OBJS = \ > dest.o \ > fastpath.o \ > ... > > or > > OBJS = dest.o \ > fastpath.o \ > ... > > I'm mildly inclined to go for the former. FWIW, I am more used to the latt

Re: Problem with synchronous replication

2019-10-30 Thread lingce . ldm
On Oct 30, 2019, at 09:45, Michael Paquier mailto:mich...@paquier.xyz>> wrote: > > On Tue, Oct 29, 2019 at 07:50:01PM +0900, Kyotaro Horiguchi wrote: >> At Fri, 25 Oct 2019 15:18:34 +0800, "Dongming Liu" >> wrote in >>> I recently discovered two possible bugs about synchronous replication. >>>

Re: Problem with synchronous replication

2019-10-30 Thread lingce . ldm
On Oct 29, 2019, at 18:50, Kyotaro Horiguchi mailto:horikyota@gmail.com>> wrote: > > Hello. > > At Fri, 25 Oct 2019 15:18:34 +0800, "Dongming Liu" > mailto:lingce@alibaba-inc.com>> wrote in >> >> Hi, >> >> I recently discovered two possible bugs about synchronous replication. >> >> 1