Re: [Patch] Make pg_checksums skip foreign tablespace directories

2020-01-31 Thread Kyotaro Horiguchi
Agreed that the tools should ignore such files. Looking excludeFile, it seems to me that basebackup makes sure to exclude only files that should harm. I'm not sure whether it's explicitly, but tablespace_map.old and backup_label.old are not excluded. The patch excludes harmless files such

Re: [Patch] Make pg_checksums skip foreign tablespace directories

2020-01-31 Thread Kyotaro Horiguchi
At Fri, 31 Jan 2020 17:30:43 +0900 (JST), Kyotaro Horiguchi wrote in > I don't think that is a problem right away, of course. It looks good > to me except for the possible excessive exclusion. So, I don't object > it if we don't mind that. That's a bit wrong. All the discussion is only on exc

Re: SimpleLruTruncate() mutual exclusion

2020-01-31 Thread Noah Misch
On Thu, Jan 30, 2020 at 04:34:33PM +0100, Dmitry Dolgov wrote: > > On Sun, Nov 17, 2019 at 10:14:26PM -0800, Noah Misch wrote: > > > I'm probably missing something, so just wanted to clarify. Do I > > > understand correctly, that thread [1] and this one are independent, and > > > it is assumed in t

Re: Hash join not finding which collation to use for string hashing

2020-01-31 Thread Amit Langote
On Fri, Jan 31, 2020 at 6:15 AM Mark Dilger wrote: > > On Jan 30, 2020, at 12:29 PM, Tom Lane wrote: > > Mark Dilger writes: > >> Would it make sense to catch a qual with unassigned collation > >> somewhere in the planner, where the qual's operator family is > >> estatblished, by checking if the

Re: Marking some contrib modules as trusted extensions

2020-01-31 Thread Dean Rasheed
On Wed, 29 Jan 2020 at 21:39, Tom Lane wrote: > > >>> pg_stat_statements > > Mmm, I'm not convinced --- the ability to see what statements are being > executed in other sessions (even other databases) is something that > paranoid installations might not be so happy about. Our previous > discussio

Re: [Patch] Make pg_checksums skip foreign tablespace directories

2020-01-31 Thread Michael Banck
Hi, Am Freitag, den 31.01.2020, 13:53 +0900 schrieb Michael Paquier: > On Thu, Jan 30, 2020 at 06:11:22PM +0100, Michael Banck wrote: > Having a past tablespace version left > around after an upgrade is a pilot error in my opinion because > pg_upgrade generates a script to cleanup past tablespaces

Re: pg_restore crash when there is a failure before all child process is created

2020-01-31 Thread Ahsan Hadi
I have applied tested both patches separately and ran regression with both. No new test cases are failing with both patches. The issues is fixed by both patches however the fix from Tom looks more elegant. I haven't done a detailed code review. On Fri, Jan 31, 2020 at 12:39 AM Tom Lane wrote: >

Re: unsupportable composite type partition keys

2020-01-31 Thread Julien Rouhaud
On Sun, Dec 22, 2019 at 10:51 PM Tom Lane wrote: > > I wrote: > > Now as far as point 1 goes, I think it's not really that awful to use > > CheckAttributeType() with a dummy attribute name. The attached > > incomplete patch uses "partition key" which causes it to emit errors > > like > > regressi

Re: [Patch] Make pg_checksums skip foreign tablespace directories

2020-01-31 Thread Bernd Helmle
Am Freitag, den 31.01.2020, 13:53 +0900 schrieb Michael Paquier: > Indeed, with a bad timing and a crash in the middle of > write_relcache_init_file(), it could be possible to have such a file > left around in the data folder. Having a past tablespace version > left > around after an upgrade is a

Re: pg_restore crash when there is a failure before all child process is created

2020-01-31 Thread vignesh C
On Fri, Jan 31, 2020 at 1:09 AM Tom Lane wrote: > > vignesh C writes: > > On Wed, Jan 29, 2020 at 6:54 PM Ahsan Hadi wrote: > >> Can you share a test case or steps that you are using to reproduce this > >> issue? Are you reproducing this using a debugger? > > > I could reproduce with the follow

Re: Physical replication slot advance is not persistent

2020-01-31 Thread Alexey Kondratov
On 30.01.2020 05:19, Michael Paquier wrote: On Wed, Jan 29, 2020 at 05:10:20PM +0900, Kyotaro Horiguchi wrote: Looks perfect. Thanks Horiguchi-san and others. Applied and back-patched down to 11. Great! Thanks for getting this done. -- Alexey Kondratov Postgres Professional https://www.pos

Re: Duplicated LSN in ReorderBuffer

2020-01-31 Thread Amit Kapila
On Fri, Jan 31, 2020 at 12:35 AM Alvaro Herrera wrote: > > On 2019-Jul-26, Andres Freund wrote: > > > Petr, Simon, see the potential issue related to fast forward at the > > bottom. > > I think we neglected this bit. I looked at the patch Simon submitted > downthread, and while I vaguely understa

Re: allow online change primary_conninfo

2020-01-31 Thread Sergei Kornilov
Hello Small rebase due to merge conflict of the tests. No functional changes since v7. PS: also it is end of current CF, I will mark patch entry as moved to the next CF.diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 2e2af9e96e..7887391bbb 100644 --- a/doc/src/sgml/config.s

Re: Data race in interfaces/libpq/fe-exec.c

2020-01-31 Thread Mark Charsley
According to folks significantly cleverer than me, this can be a problem: See section 2.4 in https://www.usenix.org/legacy/events/hotpar11/tech/final_files/Boehm.pdf tl;dr in a self-fulfilling prophecy kind of way, there are no benign data-races. So the compiler can assume no-one would write a dat

PATCH: Fix wrong size argument to pg_strncasecmp

2020-01-31 Thread Dominik Czarnota
Hello, Please find a one-liner patch in the attachment. This patch fixes a size parameter of `pg_strncasecmp` which compared a "string" literal with a variable by passing a size of 5 while the "string" literal has 6 bytes. This issue can be observed with the following query (where 'X' is any cha

Re: standby apply lag on inactive servers

2020-01-31 Thread Fujii Masao
On 2020/01/31 5:45, Alvaro Herrera wrote: On 2020-Jan-30, Kyotaro Horiguchi wrote: Agreed about backbranches. I'd like to preserve the word "transaction" as it is more familiar to users. How about something like the follows? "transactions are completed up to log time %s" That's a good poi

Re: table partitioning and access privileges

2020-01-31 Thread Fujii Masao
On 2020/01/31 13:38, Amit Langote wrote: On Fri, Jan 31, 2020 at 1:28 AM Fujii Masao wrote: On 2020/01/31 1:02, Tom Lane wrote: Fujii Masao writes: Thanks for updating the patch! Barring any objection, I will commit this fix and backport it to all supported versions. Sorry for not havin

[PATCH] Erase the distinctClause if the result is unique by definition

2020-01-31 Thread Andy Fan
Hi: I wrote a patch to erase the distinctClause if the result is unique by definition, I find this because a user switch this code from oracle to PG and find the performance is bad due to this, so I adapt pg for this as well. This patch doesn't work for a well-written SQL, but some drawback of

Re: pg_restore crash when there is a failure before all child process is created

2020-01-31 Thread ahsan hadi
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: not tested Spec compliant: tested, passed Documentation:not tested I have applied tested both patches separately and ran regression with

Re: Allow to_date() and to_timestamp() to accept localized names

2020-01-31 Thread Juan José Santamaría Flecha
On Tue, Jan 28, 2020 at 9:35 PM Peter Eisentraut < peter.eisentr...@2ndquadrant.com> wrote: > On 2020-01-28 16:47, Juan José Santamaría Flecha wrote: > > This patch targets to do something symmetrical to to_char(), which will > > just return a single value. > > I didn't fully realize while reading

Re: standby apply lag on inactive servers

2020-01-31 Thread Alvaro Herrera
On 2020-Jan-31, Fujii Masao wrote: > You're thinking to apply this change to the back branches? Sorry > if my understanding is not right. But I don't think that back-patch > is ok because it changes the documented existing behavior > of pg_last_xact_replay_timestamp(). So it looks like the behavio

Missing break in RelationFindReplTupleSeq

2020-01-31 Thread Konstantin Knizhnik
Eventually we find out that logical replication in the current version of Postgres works significantly slower on table with replica identity full than old pglogical implementation. The comment to RelationFindReplTupleSeq says:     Note that this stops on the first matching tuple. But actually

Re: Missing break in RelationFindReplTupleSeq

2020-01-31 Thread Alvaro Herrera
On 2020-Jan-31, Konstantin Knizhnik wrote: > Eventually we find out that logical replication in the current version of > Postgres works significantly slower on table with replica identity full than > old pglogical implementation. > > The comment to RelationFindReplTupleSeq says: > >     Note tha

Re: Do we need to handle orphaned prepared transactions in the server?

2020-01-31 Thread Hamid Akhtar
On Thu, Jan 30, 2020 at 8:28 AM Craig Ringer wrote: > On Thu, 30 Jan 2020 at 02:04, Hamid Akhtar wrote: > > > > So having seen the feedback on this thread, and I tend to agree with > most of what has been said here, I also agree that the server core isn't > really the ideal place to handle the o

Re: standby apply lag on inactive servers

2020-01-31 Thread Fujii Masao
On 2020/01/31 22:40, Alvaro Herrera wrote: On 2020-Jan-31, Fujii Masao wrote: You're thinking to apply this change to the back branches? Sorry if my understanding is not right. But I don't think that back-patch is ok because it changes the documented existing behavior of pg_last_xact_replay_

Re: standby apply lag on inactive servers

2020-01-31 Thread Alvaro Herrera
On 2020-Jan-31, Fujii Masao wrote: > On 2020/01/31 22:40, Alvaro Herrera wrote: > > On 2020-Jan-31, Fujii Masao wrote: > > > > > You're thinking to apply this change to the back branches? Sorry > > > if my understanding is not right. But I don't think that back-patch > > > is ok because it changes

Re: Marking some contrib modules as trusted extensions

2020-01-31 Thread Tom Lane
Dean Rasheed writes: > On Wed, 29 Jan 2020 at 21:39, Tom Lane wrote: >> The bigger picture here is that I don't want to get push-back that >> we've broken somebody's security posture by marking too many extensions >> trusted. So for anything where there's any question about security >> implicati

Re: Use compiler intrinsics for bit ops in hash

2020-01-31 Thread David Fetter
On Wed, Jan 15, 2020 at 03:45:12PM -0800, Jesse Zhang wrote: > On Tue, Jan 14, 2020 at 2:09 PM David Fetter wrote: > > > The changes in hash AM and SIMPLEHASH do look like a net positive > > > improvement. My biggest cringe might be in pg_bitutils: > > > > > > 1. Is ceil_log2_64 dead code? > > > >

Re: standby apply lag on inactive servers

2020-01-31 Thread Fujii Masao
On 2020/01/31 23:47, Alvaro Herrera wrote: On 2020-Jan-31, Fujii Masao wrote: On 2020/01/31 22:40, Alvaro Herrera wrote: On 2020-Jan-31, Fujii Masao wrote: You're thinking to apply this change to the back branches? Sorry if my understanding is not right. But I don't think that back-patch i

Re: Unix-domain socket support on Windows

2020-01-31 Thread Peter Eisentraut
On 2020-01-30 19:41, Tom Lane wrote: The code looks fine (and a big +1 for not having knowledge of DEFAULT_PGSOCKET_DIR wired into UNIXSOCK_PATH). I wonder though whether any user-facing documentation needs to be adjusted. There are no user-facing changes in this patch yet. That will come wit

Re: Extracting only the columns needed for a query

2020-01-31 Thread Melanie Plageman
I'm bumping this to the next commitfest because I haven't had a chance to address the questions posed by Dmitry. I'm sure I'll get to it in the next few weeks. > I believe it would be beneficial to add this potential API extension patch into > the thread (as an example of an interface defining how

Re: Extracting only the columns needed for a query

2020-01-31 Thread Ashwin Agrawal
On Wed, Jan 15, 2020 at 7:54 AM Dmitry Dolgov <9erthali...@gmail.com> wrote: > > For UPDATE, we need all of the columns in the table because of the > > table_lock() API's current expectation that the slot has all of the > > columns populated. If we want UPDATE to only need to insert the column > >

Re: Shared memory leak on DSM slot exhaustion

2020-01-31 Thread Robert Haas
On Thu, Jan 30, 2020 at 4:54 AM Thomas Munro wrote: > As reported over on pgsql-general[1], we leak shared memory when we > run out of DSM slots. To see this, add the random-run-out-of-slots > hack I showed in that thread, create and analyze a table t(i) with a > million integers, run with dynami

get a relations DDL server-side

2020-01-31 Thread Jordan Deitch
I would like to introduce the ability to get object DDL (server-side) by introducing a new function with roughly the following prototype: get_ddl(regclass) RETURNS text LANGUAGE C STRICT PARALLEL SAFE; A previous conversation seemed to encourage the development of this feature https://www.postg

Re: [Proposal] Global temporary tables

2020-01-31 Thread Robert Haas
On Thu, Jan 30, 2020 at 4:33 AM Konstantin Knizhnik wrote: > On 29.01.2020 21:16, Robert Haas wrote: > > On Wed, Jan 29, 2020 at 10:30 AM Konstantin Knizhnik > > wrote: > > > > I think that the idea of calling ambuild() on the fly is not going to > > work, because, again, I don't think that calli

Re: pg_restore crash when there is a failure before all child process is created

2020-01-31 Thread Tom Lane
ahsan hadi writes: > I have applied tested both patches separately and ran regression with both. > No new test cases are failing with both patches. > The issues is fixed by both patches however the fix from Tom > (0001-fix-worker-status.patch) looks more elegant. I haven't done a detailed > cod

Re: get a relations DDL server-side

2020-01-31 Thread Tom Lane
"Jordan Deitch" writes: > I would like to introduce the ability to get object DDL (server-side) by > introducing a new function with roughly the following prototype: > get_ddl(regclass) > RETURNS text > LANGUAGE C STRICT PARALLEL SAFE; Umm ... "regclass" would only be appropriate for relations.

widen vacuum buffer counters

2020-01-31 Thread Alvaro Herrera
We recently noticed that vacuum buffer counters wraparound in extreme cases, with ridiculous results. Example: 2020-01-06 16:38:38.010 EST [45625-1] app= LOG: automatic vacuum of table "somtab.sf.foobar": index scans: 17 pages: 0 removed, 207650641 remain, 0 skipped due to pins, 1341940

Re: widen vacuum buffer counters

2020-01-31 Thread Julien Rouhaud
On Fri, Jan 31, 2020 at 9:59 PM Alvaro Herrera wrote: > > We recently noticed that vacuum buffer counters wraparound in extreme > cases, with ridiculous results. Example: > > 2020-01-06 16:38:38.010 EST [45625-1] app= LOG: automatic vacuum of table > "somtab.sf.foobar": index scans: 17 >

Re: unsupportable composite type partition keys

2020-01-31 Thread Tom Lane
Julien Rouhaud writes: > On Sun, Dec 22, 2019 at 10:51 PM Tom Lane wrote: >> While poking at this, I also started to wonder why CheckAttributeType >> wasn't recursing into ranges, since those are our other kind of >> container type. And the answer is that it must, because we allow >> creation of

Re: WIP: Aggregation push-down

2020-01-31 Thread legrand legrand
Hello, Thank you for this great feature ! I hope this will be reviewed/validated soon ;o) Just a comment: set enable_agg_pushdown to true; isn't displayed in EXPLAIN (SETTINGS) syntax. The following modification seems to fix that: src/backend/utils/misc/guc.c {"enable_agg_pu

Re: widen vacuum buffer counters

2020-01-31 Thread Tom Lane
Alvaro Herrera writes: > We recently noticed that vacuum buffer counters wraparound in extreme > cases, with ridiculous results. Ugh. > I propose to backpatch this. +1 for widening these counters, but since they're global variables, -0.2 or so for back-patching. I don't know of any reason that

Re: PATCH: Fix wrong size argument to pg_strncasecmp

2020-01-31 Thread Tom Lane
Dominik Czarnota writes: > This patch fixes a size parameter of `pg_strncasecmp` which compared a > "string" literal with a variable by passing a size of 5 while the "string" > literal has 6 bytes. Pushed, thanks for the report! > By the way, the `strncasecmp` usages around the fixed line could

Re: Created feature for to_date() conversion using patterns 'YYYY-WW', 'YYYY-WW-D', 'YYYY-MM-W' and 'YYYY-MM-W-D'

2020-01-31 Thread Cleysson Lima
Em sex., 10 de jan. de 2020 às 09:22, Mark Lorenz escreveu: > Updated the chg_to_date_wwd.patch with additional tests (because it > works not only for 'D' pattern but also for all day patterns like 'Day' > or 'DY'). Added the necessary documentation change. > > (The fix_to_char_wwd.patch

Re: [Patch] pg_rewind: options to use restore_command from recovery.conf or command line

2020-01-31 Thread Alexey Kondratov
On 2020-01-24 08:50, Michael Paquier wrote: On Wed, Jan 22, 2020 at 12:55:30AM +0300, Alexander Korotkov wrote: On Sun, Jan 19, 2020 at 1:24 PM Michael Paquier wrote: +static int +RestoreArchivedWALFile(const char *path, const char *xlogfname, + off_t expectedSize, const c

Re: Created feature for to_date() conversion using patterns 'YYYY-WW', 'YYYY-WW-D', 'YYYY-MM-W' and 'YYYY-MM-W-D'

2020-01-31 Thread Tom Lane
Cleysson Lima writes: > this is a review of the patch: chg_to_date_wwd.patch > There hasn't been any problem, at least that I've been able to find. AFAICS, the point of this patch is to make to_date symmetrical with the definition of WW that the other patch wants for to_char. But the other pa

Re: Shared memory leak on DSM slot exhaustion

2020-01-31 Thread Thomas Munro
On Sat, Feb 1, 2020 at 7:37 AM Robert Haas wrote: > Whoops. The patch looks OK to me. Pushed.

What's difference among insert/write/flush lsn?

2020-01-31 Thread Jinhua Luo
Hi, pg_current_wal_flush_lsn()pg_lsnGet current write-ahead log flush location pg_current_wal_insert_lsn()pg_lsnGet current write-ahead log insert location pg_current_wal_lsn()pg_lsnGet current write-ahead log write location I guess write is about how many bytes written in shared cache, and flush

Re: dropdb --force

2020-01-31 Thread Amit Kapila
On Sat, Nov 30, 2019 at 7:46 AM Michael Paquier wrote: > > On Fri, Nov 29, 2019 at 03:31:21PM +0530, Amit Kapila wrote: > > It might be better to move it to next CF as the discussion is still active. > > OK, just did that. > I have marked this as committed in CF. This was committed some time bac

Re: Don't try fetching future segment of a TLI.

2020-01-31 Thread David Steele
On 1/28/20 8:02 PM, Kyotaro Horiguchi wrote: > At Tue, 28 Jan 2020 19:13:32 +0300, Pavel Suderevsky >> Regading influence: issue is not about the large amount of WALs to apply >> but in searching for the non-existing WALs on the remote storage, each such >> search can take 5-10 seconds while obt

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-01-31 Thread Amit Kapila
On Thu, Jan 30, 2020 at 6:10 PM Dilip Kumar wrote: > > Other comments look fine to me so I will reply to them along with the > next version of the patch. > This still needs more work, so I have moved this to the next CF. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: [PATCH] Fix Proposal - Deadlock Issue in Single User Mode When IO Failure Occurs

2020-01-31 Thread Amit Kapila
On Mon, Nov 25, 2019 at 1:17 PM Michael Paquier wrote: > > On Mon, Sep 09, 2019 at 05:34:43PM +0530, Amit Kapila wrote: > > The only difference is in the last line where for me it gives > > assertion failure when trying to do ReleaseAuxProcessResources. Below > > is the callstack: > > No need for

Re: [Proposal] Add accumulated statistics for wait event

2020-01-31 Thread Pavel Stehule
Hi st 15. 1. 2020 v 14:15 odesílatel Imai Yoshikazu napsal: > On 2020/01/13 4:11, Pavel Stehule wrote: > > The following review has been posted through the commitfest application: > > make installcheck-world: tested, passed > > Implements feature: tested, passed > > Spec compliant:

Re: Tid scan increments value of pg_stat_all_tables.seq_scan. (but not seq_tup_read)

2020-01-31 Thread Kasahara Tatsuhito
On Thu, Jan 30, 2020 at 1:49 PM Kyotaro Horiguchi wrote: > At Thu, 30 Jan 2020 13:30:56 +0900, Kasahara Tatsuhito > wrote in > > > TID scan : yes , seq_scan, , > > Here is wrong, because TID scan came to have SO_TYPE_SEQSCAN flags > > from commit 147e3722f7. > > It is refl

Re: unsupportable composite type partition keys

2020-01-31 Thread Julien Rouhaud
On Fri, Jan 31, 2020 at 04:20:36PM -0500, Tom Lane wrote: > Julien Rouhaud writes: > > On Sun, Dec 22, 2019 at 10:51 PM Tom Lane wrote: > >> While poking at this, I also started to wonder why CheckAttributeType > >> wasn't recursing into ranges, since those are our other kind of > >> container ty