Re: Would it be possible to have parallel archiving?

2018-08-27 Thread Alexander Kukushkin
Hi, There is the archive_status directory in pg_wal, and if there are files with suffixes ".ready", you can archive not only the file which was requested, but quite a few more if there are ".ready" files available. After that you have to rename ".ready" to ".done". Postgres will not call archive_

Would it be possible to have parallel archiving?

2018-08-27 Thread hubert depesz lubaczewski
Hi, I'm in a situation where we quite often generate more WAL than we can archive. The thing is - archiving takes long(ish) time but it's multi-step process and includes talking to remote servers over network. I tested that simply by running archiving in parallel I can easily get 2-3 times higher

Re: First steps to being a contributer

2018-08-27 Thread Heikki Linnakangas
On 28/08/18 03:39, Daniel Wood wrote: Having quit Amazon, where I was doing Postgres development, I've started looking at various things I might work on for fun. One thought is to start with something easy like the scalability of GetSnapshotData(). :-) Cool! :-) I recently found it interestin

Re: Flexible configuration for full-text search

2018-08-27 Thread Aleksandr Parfenov
On Fri, 24 Aug 2018 18:50:38 +0300 Alexander Korotkov wrote: >Agreed, backward compatibility is important here. Probably we should >leave old dictionaries for that. But I just meant that if we >introduce new (better) way of stop words handling and encourage users >to use it, then it would look s

Re: typcache.c typos

2018-08-27 Thread Amit Langote
Hi, On 2018/08/28 13:42, David Rowley wrote: > On 28 August 2018 at 16:40, David Rowley wrote: >> I've attached a patch to fix a typo in typcache.c. I ended up also >> rephrasing the sentence since "information about data types that is", >> almost made me also want to change "is" to "are", since

Re: typcache.c typos

2018-08-27 Thread David Rowley
On 28 August 2018 at 16:40, David Rowley wrote: > I've attached a patch to fix a typo in typcache.c. I ended up also > rephrasing the sentence since "information about data types that is", > almost made me also want to change "is" to "are", since "types" is > plural. That would have been a mistake

typcache.c typos

2018-08-27 Thread David Rowley
I've attached a patch to fix a typo in typcache.c. I ended up also rephrasing the sentence since "information about data types that is", almost made me also want to change "is" to "are", since "types" is plural. That would have been a mistake since it's talking about the information and not the typ

RE: [HACKERS] Proposal to add work_mem option to postgres_fdw module

2018-08-27 Thread Shinoda, Noriyoshi (PN Japan GCS Delivery)
Hi Everyone, thank you for your comment. >> I like the direction of your thinking, but it seems to me that this >> would cause a problem if you want to set search_path=foo,bar. >... OPTIONS ( host 'remhost1', port '5433', dbname 'demodb', >option='option1=foo', option='option2=bar' ); I do not

Re: Pluggable Storage - Andres's take

2018-08-27 Thread Haribabu Kommi
On Fri, Aug 24, 2018 at 12:50 PM Andres Freund wrote: > Hi, > > On 2018-08-24 11:55:41 +1000, Haribabu Kommi wrote: > > On Tue, Aug 21, 2018 at 6:59 PM Andres Freund > wrote: > > > > > On 2018-08-21 16:55:47 +1000, Haribabu Kommi wrote: > > > > On Sun, Aug 5, 2018 at 7:48 PM Andres Freund > wro

Re: Why hash OIDs?

2018-08-27 Thread Andres Freund
On 2018-08-28 14:45:49 +1200, Thomas Munro wrote: > On Tue, Aug 28, 2018 at 2:26 PM Tom Lane wrote: > > Andres Freund writes: > > > On 2018-08-28 13:50:43 +1200, Thomas Munro wrote: > > >> What bad thing would happen if we used OIDs directly as hash values in > > >> internal hash tables (that is,

Re: Refactor textToQualifiedNameList()

2018-08-27 Thread Kyotaro HORIGUCHI
Hello. At Fri, 24 Aug 2018 20:44:12 +0900, Yugo Nagata wrote in <20180824204412.150979ae6b283ddb639f9...@sraoss.co.jp> > When working on other patch[1], I found there are almost same > functions, texttoQualifiedNameList() and stringToQualifiedNameList(). > The only difference is the argument typ

Re: Why hash OIDs?

2018-08-27 Thread Thomas Munro
On Tue, Aug 28, 2018 at 2:26 PM Tom Lane wrote: > Andres Freund writes: > > On 2018-08-28 13:50:43 +1200, Thomas Munro wrote: > >> What bad thing would happen if we used OIDs directly as hash values in > >> internal hash tables (that is, instead of uint32_hash() we'd use > >> uint32_identity(), o

Re: [HACKERS] Proposal to add work_mem option to postgres_fdw module

2018-08-27 Thread Michael Paquier
On Mon, Aug 27, 2018 at 02:34:31PM -0400, Robert Haas wrote: > I like the direction of your thinking, but it seems to me that this > would cause a problem if you want to set search_path=foo,bar. proconfig already handles such cases, that's what I was coming at. We could reuse most of what it does

Re: Postmaster doesn't send SIGTERM to bgworker during fast shutdown when pmState == PM_STARTUP

2018-08-27 Thread Michael Paquier
On Sun, Aug 26, 2018 at 06:00:23PM +0200, Alexander Kukushkin wrote: > it is possible to start bgworker with bgw_start_time = > BgWorkerStart_PostmasterStart, which will be started immediately after > postmaster. Right. > But if you try to do a fast shutdown while postmaster still in the > pmStat

Re: Why hash OIDs?

2018-08-27 Thread Tom Lane
Andres Freund writes: > On 2018-08-28 13:50:43 +1200, Thomas Munro wrote: >> What bad thing would happen if we used OIDs directly as hash values in >> internal hash tables (that is, instead of uint32_hash() we'd use >> uint32_identity(), or somehow optimise it away entirely, as you can >> see in s

Re: Why hash OIDs?

2018-08-27 Thread Andres Freund
On 2018-08-28 13:50:43 +1200, Thomas Munro wrote: > I'm curious about something which may qualify as a stupid question. > > What bad thing would happen if we used OIDs directly as hash values in > internal hash tables (that is, instead of uint32_hash() we'd use > uint32_identity(), or somehow opti

Why hash OIDs?

2018-08-27 Thread Thomas Munro
Hello, I'm curious about something which may qualify as a stupid question. What bad thing would happen if we used OIDs directly as hash values in internal hash tables (that is, instead of uint32_hash() we'd use uint32_identity(), or somehow optimise it away entirely, as you can see in some C++ st

First steps to being a contributer

2018-08-27 Thread Daniel Wood
Having quit Amazon, where I was doing Postgres development, I've started looking at various things I might work on for fun. One thought is to start with something easy like the scalability of GetSnapshotData(). :-) I recently found it interesting to examine performance while running near 1 mil

Re: simplehash.h comment

2018-08-27 Thread Thomas Munro
On Tue, Aug 28, 2018 at 1:40 AM Tom Lane wrote: > Thomas Munro writes: > > ... and a patch. Thoughts? > > It's not real obvious why you're adding inclusion guards over just a > subset of the header. A short comment about that would be worthwhile. Right, thanks. I added a comment and committed

Re[2]: doc - improve description of default privileges

2018-08-27 Thread Bradley DeJong
Hi Fabien, Thanks for writing this up - in particular the psql backslash commands. comments on the patch ... 1) I think that adding the "This privilege is abbreviated ... when displayed." lines to the privilege descriptions is redundant. The abbreviations are already listed after the "The ent

Re: FailedAssertion on partprune

2018-08-27 Thread Jonathan S. Katz
> On Aug 17, 2018, at 2:49 AM, David Rowley > wrote: > > On 17 August 2018 at 06:52, Robert Haas wrote: >> I don't know whether there's actually a defect here any more. I was >> trying to dispel some perceived confusion on the part of David and Tom >> about what this code was trying to accomp

Re: Memory leak with CALL to Procedure with COMMIT.

2018-08-27 Thread Jonathan S. Katz
> On Aug 27, 2018, at 5:13 PM, Peter Eisentraut > wrote: > > On 23/08/2018 17:35, Jonathan S. Katz wrote: >> Applied the patch against head. make check passed, all the tests I ran >> earlier in this thread passed as well. >> >> Reviewed the code - looks to match above reasoning, and comments >

Re: Memory leak with CALL to Procedure with COMMIT.

2018-08-27 Thread Peter Eisentraut
On 23/08/2018 17:35, Jonathan S. Katz wrote: > Applied the patch against head. make check passed, all the tests I ran > earlier in this thread passed as well. > > Reviewed the code - looks to match above reasoning, and comments > are clear. > > From my perspective it looks good, but happy to hear

Re[3]: Adding a note to protocol.sgml regarding CopyData

2018-08-27 Thread Bradley DeJong
On 2018-08-27, Fabien COELHO wrote ... > ... references to the protocol version lacks homogeneity. > ... I'd suggest to keep "the vX.0 protocol" for a short version, > and "the version X.0 protocol" for long ... I agree. Change made. protocol.v3.patch Description: Binary data

Re: [HACKERS] Proposal to add work_mem option to postgres_fdw module

2018-08-27 Thread Fabrízio de Royes Mello
On Mon, Aug 27, 2018 at 3:35 PM Robert Haas wrote: > > On Mon, Aug 27, 2018 at 1:29 AM Michael Paquier wrote: > > It seems to me that you would pass down just a string which gets > > allocated for "options", and injection risks are something to be careful > > about. Another possibility would be

Re: More parallel pg_dump bogosities

2018-08-27 Thread Stephen Frost
Tom, * Tom Lane (t...@sss.pgh.pa.us) wrote: > Stephen Frost writes: > > * Tom Lane (t...@sss.pgh.pa.us) wrote: > >> And lastly, the ROW SECURITY items are ready because they are not marked > >> with any dependency at all, none, nada. This seems bad. In principle > >> it could mean that parallel

Re: [HACKERS] Proposal to add work_mem option to postgres_fdw module

2018-08-27 Thread Robert Haas
On Mon, Aug 27, 2018 at 1:29 AM Michael Paquier wrote: > It seems to me that you would pass down just a string which gets > allocated for "options", and injection risks are something to be careful > about. Another possibility would be an array with comma-separated > arguments, say: > options = 'o

Re: More parallel pg_dump bogosities

2018-08-27 Thread Tom Lane
Stephen Frost writes: > * Tom Lane (t...@sss.pgh.pa.us) wrote: >> And lastly, the ROW SECURITY items are ready because they are not marked >> with any dependency at all, none, nada. This seems bad. In principle >> it could mean that parallel restore would try to emit "ALTER TABLE ENABLE >> ROW L

Re: More parallel pg_dump bogosities

2018-08-27 Thread Stephen Frost
Greetings, * Tom Lane (t...@sss.pgh.pa.us) wrote: > So I started poking at the idea of sorting by size during parallel > restore instead of sorting pg_dump's TOC that way. While investigating > just where to do that, I discovered that, using the regression database > as test case, restore_toc_ent

More parallel pg_dump bogosities

2018-08-27 Thread Tom Lane
So I started poking at the idea of sorting by size during parallel restore instead of sorting pg_dump's TOC that way. While investigating just where to do that, I discovered that, using the regression database as test case, restore_toc_entries_parallel() finds these objects to be *immediately* rea

Re: pg_dump test instability

2018-08-27 Thread Stephen Frost
Greetings, * Tom Lane (t...@sss.pgh.pa.us) wrote: > Stephen Frost writes: > > * Tom Lane (t...@sss.pgh.pa.us) wrote: > >> However, at least for the directory-format case (which I think is the > >> only one supported for parallel restore), we could make it compare the > >> file sizes of the TABLE

Re: [HACKERS] WIP: long transactions on hot standby feedback replica / proof of concept

2018-08-27 Thread Alexander Korotkov
Hi! Thank you for feedback. On Sun, Aug 26, 2018 at 4:09 AM Robert Haas wrote: > On Tue, Aug 21, 2018 at 9:10 AM, Alexander Korotkov > wrote: > > After heap truncation using this algorithm, shared buffers may contain > > past-OEF buffers. But those buffers are empty (no used items) and > > cle

Re: pg_dump test instability

2018-08-27 Thread Tom Lane
Stephen Frost writes: > * Tom Lane (t...@sss.pgh.pa.us) wrote: >> However, at least for the directory-format case (which I think is the >> only one supported for parallel restore), we could make it compare the >> file sizes of the TABLE DATA items. That'd work pretty well as a proxy >> for both t

Re: pg_dump test instability

2018-08-27 Thread Stephen Frost
Greetings, * Tom Lane (t...@sss.pgh.pa.us) wrote: > Peter Eisentraut writes: > > In a non-data_only dump, the order of the tables doesn't matter, because > > the foreign keys are added at the very end. In parallel dumps, the > > tables are in addition sorted by size, so the resultant order is >

Re: some more error location support

2018-08-27 Thread Peter Eisentraut
On 27/08/2018 10:53, Fabien COELHO wrote: > There is a "make_parsestate", but no corresponding free. The usual > pattern, although there seems to be a few exception, is to "make" and > "free". > > Even if there is some under-the-hood garbage collection, I'd suggest to > add a free after the cal

Re: has_table_privilege for a table in unprivileged schema causes an error

2018-08-27 Thread David G. Johnston
On Sat, Aug 25, 2018 at 8:29 PM, Tom Lane wrote: > There's a backwards-compatibility argument for not changing this behavior, > sure, but I don't buy the other arguments you made here. And I don't > think there's all that much to the backwards-compatibility argument, > considering that the curre

simplify index tuple descriptor initialization

2018-08-27 Thread Peter Eisentraut
Whenever some pg_attribute field is added or changed, a lot of repetitive changes all over the code are necessary. Here is a small change to remove one such place. We have two code paths for initializing the tuple descriptor for a new index: For a normal index, we copy the tuple descriptor from t

Re: has_table_privilege for a table in unprivileged schema causes an error

2018-08-27 Thread Tom Lane
Yugo Nagata writes: > Tom Lane wrote: >> There's a backwards-compatibility argument for not changing this behavior, >> sure, but I don't buy the other arguments you made here. And I don't >> think there's all that much to the backwards-compatibility argument, >> considering that the current beha

Re: pg_dump test instability

2018-08-27 Thread Tom Lane
Peter Eisentraut writes: > In a non-data_only dump, the order of the tables doesn't matter, because > the foreign keys are added at the very end. In parallel dumps, the > tables are in addition sorted by size, so the resultant order is > different from a single-threaded dump. This can be seen by

Re: Built-in connection pooling

2018-08-27 Thread Konstantin Knizhnik
New versions of built-in connection pool is attached to this mail. Now client's startup package is received by one of listener workers and postmater knows database/user name of the recevied connection and so is able to marshal it to the proper connection pool. Right now SSL is not supported.

Re: simplehash.h comment

2018-08-27 Thread Tom Lane
Thomas Munro writes: > ... and a patch. Thoughts? It's not real obvious why you're adding inclusion guards over just a subset of the header. A short comment about that would be worthwhile. regards, tom lane

Re: pg_dump test instability

2018-08-27 Thread Stephen Frost
Greetings, * Peter Eisentraut (peter.eisentr...@2ndquadrant.com) wrote: > During the development of an unrelated feature, I have experienced > failures in a pg_dump test case, specifically > > t/002_pg_dump.pl ... 1825/6052 > # Failed test 'defaults_parallel: should not dump COPY > fk_refer

Re: libpq debug log

2018-08-27 Thread Yugo Nagata
On Fri, 24 Aug 2018 04:38:22 + "Iwata, Aya" wrote: > Hi, > > I'm going to propose libpq debug log for analysis of queries on the > application side. > I think that it is useful to determine whether the cause is on the > application side or the server side when a slow query occurs. Do you

Re: pg_verify_checksums -d option (was: Re: pg_verify_checksums -r option)

2018-08-27 Thread Yugo Nagata
On Mon, 27 Aug 2018 13:34:12 +0200 Michael Banck wrote: > Hi, > > On Mon, Aug 27, 2018 at 07:53:36PM +0900, Yugo Nagata wrote: > > On Fri, 24 Aug 2018 18:01:09 +0200 > > Peter Eisentraut wrote: > > > I'm curious about this option: > > > > > > -r RELFILENODE check only relation with s

Re: table_privileges view under information_schema doesn't show privileges on materialized views

2018-08-27 Thread Ashutosh Sharma
On Fri, Aug 24, 2018 at 9:06 PM, Tom Lane wrote: > Ashutosh Sharma writes: >> Currently, table_privileges view in information_schema.sql doesn't >> show privileges on materialized views for currently enabled roles. As >> per the documentation-[1], it should be showing the all privileges >> grante

pg_verify_checksums -d option (was: Re: pg_verify_checksums -r option)

2018-08-27 Thread Michael Banck
Hi, On Mon, Aug 27, 2018 at 07:53:36PM +0900, Yugo Nagata wrote: > On Fri, 24 Aug 2018 18:01:09 +0200 > Peter Eisentraut wrote: > > I'm curious about this option: > > > > -r RELFILENODE check only relation with specified relfilenode > > > > but there is no facility to specify a databa

Re: [HACKERS] Optional message to user when terminating/cancelling backend

2018-08-27 Thread Daniel Gustafsson
> On 23 Aug 2018, at 20:53, Tom Lane wrote: > I think this is just a timing problem: the signal gets sent, > but it might or might not get received before the current statement ends. > It's possible that a signal-sent-to-self can be expected to be received > synchronously on all Unix platforms, b

Re: pg_verify_checksums -r option

2018-08-27 Thread Yugo Nagata
On Fri, 24 Aug 2018 18:01:09 +0200 Peter Eisentraut wrote: > I'm curious about this option: > > -r RELFILENODE check only relation with specified relfilenode > > but there is no facility to specify a database. > > Also, referring to the relfilenode of a mapped relation seems a bit >

Re: [HACKERS] Optional message to user when terminating/cancelling backend

2018-08-27 Thread Daniel Gustafsson
> On 24 Aug 2018, at 03:37, Tom Lane wrote: > > Thomas Munro writes: >> On Fri, Aug 24, 2018 at 6:53 AM, Tom Lane wrote: >>> I think this is just a timing problem: the signal gets sent, >>> but it might or might not get received before the current statement ends. > >> How about we just wait fo

Re: Fix help option of contrib/oid2name

2018-08-27 Thread Michael Paquier
On Fri, Aug 24, 2018 at 01:32:47PM +0900, Tatsuro Yamada wrote: > I revised the patch and created new tap tests. Thanks, I have looked at the patch set. And here are some notes about issues found while reviewing. For oid2name: - Documentation had some typos, --tablename was used instead of --tab

Re: Refactor textToQualifiedNameList()

2018-08-27 Thread Yugo Nagata
On Fri, 24 Aug 2018 20:44:12 +0900 Yugo Nagata wrote: > Hi, > > When working on other patch[1], I found there are almost same > functions, texttoQualifiedNameList() and stringToQualifiedNameList(). > The only difference is the argument type, text or char*. I don't know > why these functions are

Re: has_table_privilege for a table in unprivileged schema causes an error

2018-08-27 Thread Yugo Nagata
On Sat, 25 Aug 2018 23:29:27 -0400 Tom Lane wrote: > Robert Haas writes: > > I'm not sure that it's a good idea to change this behavior. > > > In the case of an unqualified name, the permissions on the schemas in > > the search path can affect which table is chosen in the first place. > > ... S

Re: some more error location support

2018-08-27 Thread Fabien COELHO
Here are three patches to add more detailed error location support to some parts of CREATE TABLE (defaults, check constraints, partition specifications) as well as the COPY command. Examples can be seen in the regression test output. The first two are low-hanging fruit, since all the informat

Re: some more error location support

2018-08-27 Thread Fabien COELHO
Hello Peter, Here are three patches to add more detailed error location support to some parts of CREATE TABLE (defaults, check constraints, partition specifications) as well as the COPY command. Examples can be seen in the regression test output. The first two are low-hanging fruit, since al

Re: some more error location support

2018-08-27 Thread Fabien COELHO
I noticed that you provide NULL from "ALTER TABLE" which is calling the create table machinery: The structure of the ALTER TABLE code is such that it would be quite complicated to pass through the required state. It's doable, but would be a bigger and independent project. Ok, so no "easy"

Re: some more error location support

2018-08-27 Thread Peter Eisentraut
On 27/08/2018 10:41, Fabien COELHO wrote: > I noticed that you provide NULL from "ALTER TABLE" which is calling the > create table machinery: The structure of the ALTER TABLE code is such that it would be quite complicated to pass through the required state. It's doable, but would be a bigger an

Re: some more error location support

2018-08-27 Thread Fabien COELHO
Hello Peter, Here are three patches to add more detailed error location support to some parts of CREATE TABLE (defaults, check constraints, partition specifications) as well as the COPY command. Examples can be seen in the regression test output. The first two are low-hanging fruit, since al

Re: Adding a note to protocol.sgml regarding CopyData

2018-08-27 Thread Tatsuo Ishii
> Hello Bradley & Tatsuo-san, > > My 0.02€ on the text: > >> Version 2.0 of the PostgreSQL protocol >> In the v3.0 protocol, >> the 3.0 protocol >> version 3.0 of the copy-in/copy-out sub-protocol >> the V2.0 protocol. > > While reading nice English (I learned "holdover"), it occurs to me > that

some more error location support

2018-08-27 Thread Peter Eisentraut
Here are three patches to add more detailed error location support to some parts of CREATE TABLE (defaults, check constraints, partition specifications) as well as the COPY command. Examples can be seen in the regression test output. The first two are low-hanging fruit, since all the information