Re: [HACKERS] pnstrdup considered armed and dangerous

2017-10-02 Thread Andres Freund
On 2016-10-03 14:55:24 -0700, Andres Freund wrote: > Hi, > > A colleage of me just wrote innocent looking code like > char *shardRelationName = pnstrdup(relationName, NAMEDATALEN); > which is at the moment wrong if relationName isn't preallocated to > NAMEDATALEN size. > > /* > * pnstrdu

Re: [HACKERS] proposal - Default namespaces for XPath expressions (PostgreSQL 11)

2017-10-02 Thread Kyotaro HORIGUCHI
At Mon, 2 Oct 2017 12:43:19 +0200, Pavel Stehule wrote in > > Sorry, I forgot to care about that. (And the definition of > > namespace array is of course fabricated by me). I'd like to leave > > this to committers. Anyway it is working but the syntax (or > > whether it is acceptable) is still a

Re: [HACKERS] cache lookup errors for missing replication origins

2017-10-02 Thread Masahiko Sawada
On Wed, Sep 6, 2017 at 3:51 PM, Michael Paquier wrote: > On Tue, Sep 5, 2017 at 12:59 PM, Michael Paquier > wrote: >> ERROR: 42704: replication slot "%s" does not exist > > s/slot/origin/ > >> As far as I can see, replorigin_by_oid makes no use of its missing_ok >> = false in the backend code, s

Re: [HACKERS] 64-bit queryId?

2017-10-02 Thread Andres Freund
On 2017-10-03 03:07:09 +0300, Alexander Korotkov wrote: > On Tue, Oct 3, 2017 at 12:32 AM, Tom Lane wrote: > > > Peter Geoghegan writes: > > > You need to change the SQL interface as well, although I'm not sure > > > exactly how. The problem is that you are now passing a uint64 queryId > > > to

Re: [HACKERS] [sqlsmith] crash in RestoreLibraryState during low-memory testing

2017-10-02 Thread Tom Lane
Amit Kapila writes: > Any other ideas? Given that the crash is so far down inside __dlopen(), and that there's a clear reference to the string we presumably passed to that: #11 0x7f518485e489 in _dl_open (file=0x55b692f2d2b0 "/home/smith/postgres/inst/master/lib/pgcrypto.so", mode=-21474833

Re: [HACKERS] Transactions involving multiple postgres foreign servers

2017-10-02 Thread Ashutosh Bapat
On Fri, Sep 29, 2017 at 9:12 PM, Robert Haas wrote: > > It's possible that we might find that neither of the above approaches > are practical and that the performance benefits of resolving the > transaction from the original connection are large enough that we want > to try to make it work anyhow.

Re: [HACKERS] [sqlsmith] crash in RestoreLibraryState during low-memory testing

2017-10-02 Thread Amit Kapila
On Tue, Oct 3, 2017 at 8:31 AM, Amit Kapila wrote: > On Tue, Oct 3, 2017 at 3:04 AM, Andreas Seltenreich > wrote: >> Hi, >> >> doing low-memory testing with REL_10_STABLE at 1f19550a87 also produced >> a couple of parallel worker core dumps with the backtrace below. >> Although most of the backt

Re: [HACKERS] PoC: full merge join on comparison clause

2017-10-02 Thread Ashutosh Bapat
On Thu, Sep 28, 2017 at 8:57 PM, Alexander Kuzmenkov wrote: > Hi Ashutosh, > > Thanks for the review. > > Jeff, I'm copying you because this is relevant to our discussion about what > to do with mergeopfamilies when adding new merge join types. > > You have renamed RestrictInfo member mergeopfamil

Re: [HACKERS] Conversion error

2017-10-02 Thread Tatsuo Ishii
> Tatsuo Ishii writes: >> I saw this while restoring 9.6 database to 10.0 database. >> \connect: FATAL: conversion between UTF8 and MULE_INTERNAL is not supported >> Is this expected? I saw nothing releated to this in the release notes. > > Don't think that's ever been supported. Sorry, that wa

Re: [HACKERS] Possible SSL improvements for a newcomer to tackle

2017-10-02 Thread Michael Paquier
On Tue, Oct 3, 2017 at 1:15 PM, Zeus Kronion wrote: > I previously made one minuscule contribution to the project two years ago. > I'm interested in doing some more, and I'm trying to figure out what to > focus on. Two SSL-related projects caught my attention: > 1) Allow automatic selection of SSL

Re: [HACKERS] Possible SSL improvements for a newcomer to tackle

2017-10-02 Thread Tom Lane
Zeus Kronion writes: > 2) I was surprised to learn the following from the docs: >> By default, PostgreSQL will not perform any verification of the server >> certificate. > Is there a technical reason to perform no verification by default? Wouldn't > a safer default be desirable? I'm not an SSL

Re: [HACKERS] Conversion error

2017-10-02 Thread Tom Lane
Tatsuo Ishii writes: > I saw this while restoring 9.6 database to 10.0 database. > \connect: FATAL: conversion between UTF8 and MULE_INTERNAL is not supported > Is this expected? I saw nothing releated to this in the release notes. Don't think that's ever been supported.

Re: [HACKERS] Conversion error

2017-10-02 Thread Tatsuo Ishii
> I saw this while restoring 9.6 database to 10.0 database. > > \connect: FATAL: conversion between UTF8 and MULE_INTERNAL is not supported > > Is this expected? I saw nothing releated to this in the release notes. This had been allowed in 9.6. So I think 10.0 silently drops the feature. Best

Re: [HACKERS] Logging idle checkpoints

2017-10-02 Thread Kyotaro HORIGUCHI
At Tue, 3 Oct 2017 10:23:08 +0900, Michael Paquier wrote in > On Tue, Oct 3, 2017 at 12:01 AM, Stephen Frost wrote: > > I certainly don't care for the idea of adding log messages saying we > > aren't doing anything just to match a count that's incorrectly claiming > > that checkpoints are happe

[HACKERS] Possible SSL improvements for a newcomer to tackle

2017-10-02 Thread Zeus Kronion
I previously made one minuscule contribution to the project two years ago. I'm interested in doing some more, and I'm trying to figure out what to focus on. Two SSL-related projects caught my attention: 1) Allow automatic selection of SSL client certificates from a certificate store (https://www.po

Re: [HACKERS] path toward faster partition pruning

2017-10-02 Thread Amit Langote
Hi David. Thanks a lot for your review comments and sorry it took me a while to reply. On 2017/09/28 18:16, David Rowley wrote: > On 27 September 2017 at 14:22, Amit Langote > wrote: >> - 0001 includes refactoring that Dilip proposed upthread [1] (added him as >> an author). I slightly tweake

Re: [HACKERS] Horrible CREATE DATABASE Performance in High Sierra

2017-10-02 Thread Tom Lane
Andres Freund writes: > On 2017-10-02 19:50:51 -0400, Tom Lane wrote: >> What I saw was that the backend process was consuming 100% of (one) CPU, >> while the I/O transaction rate viewed by "iostat 1" started pretty low >> --- under 10% of what the machine is capable of --- and dropped from >> the

Re: [HACKERS] [sqlsmith] crash in RestoreLibraryState during low-memory testing

2017-10-02 Thread Amit Kapila
On Tue, Oct 3, 2017 at 3:04 AM, Andreas Seltenreich wrote: > Hi, > > doing low-memory testing with REL_10_STABLE at 1f19550a87 also produced > a couple of parallel worker core dumps with the backtrace below. > Although most of the backtrace is inside the dynamic linker, it looks > like it was pass

Re: [HACKERS] [PATCH] Tests for reloptions

2017-10-02 Thread Michael Paquier
On Sun, Oct 1, 2017 at 5:18 AM, Nikolay Shaplov wrote: > src/backend/access/common/reloptions.c get only 7 lines, it was quite covered > by existing test, but all most of the access methods gets some coverage > increase: > > src/backend/access/brin 1268 -> 1280 (+18) > src/backend/access/g

Re: [HACKERS] Partition-wise join for join between (declaratively) partitioned tables

2017-10-02 Thread Robert Haas
On Thu, Sep 21, 2017 at 8:21 AM, Ashutosh Bapat wrote: > Here's set of rebased patches. The patch with extra tests is not for > committing. All other patches, except the last one, will need to be > committed together. The last patch may be committed along with other > patches or as a separate patc

Re: [HACKERS] [PATCH] Assert that the correct locks are held when calling PageGetLSN()

2017-10-02 Thread Michael Paquier
On Tue, Oct 3, 2017 at 12:54 AM, Alvaro Herrera wrote: > Michael Paquier wrote: >> On Mon, Oct 2, 2017 at 11:44 PM, Robert Haas wrote: >> > On Mon, Oct 2, 2017 at 10:19 AM, Michael Paquier >> > wrote: >> >> So those bits could be considered for integration. >> > >> > Yes, and they also tend to s

[HACKERS] Combining expr{Type,Typmod,Collation}() into one function.

2017-10-02 Thread Andres Freund
Hi, I'd recently noticed that expr* functions actually show up in profiles because we use them at some very common paths (e.g. ExecTypeFromTLInternal()) and that we commonly call all the three variants from $subject in sequence. Looking at their code I was wondering whether it's reasonable to com

Re: [JDBC] [HACKERS] Channel binding support for SCRAM-SHA-256

2017-10-02 Thread Michael Paquier
On Tue, Oct 3, 2017 at 1:30 AM, Robert Haas wrote: > On Fri, Sep 15, 2017 at 6:29 PM, Michael Paquier > wrote: >> I would like to point out that per the RFC, if the client attempts a >> SSL connection with SCRAM and that the server supports channel >> binding, then it has to publish the SASL mech

Re: [HACKERS][BUG] Cache invalidation for queries that contains const of temporary composite type

2017-10-02 Thread Kyotaro HORIGUCHI
Hi, Sorry, I saw this once but somehow my attension was blown away on the way. At Tue, 3 Oct 2017 02:41:34 +0300, Alexander Korotkov wrote in > On Tue, Oct 3, 2017 at 12:20 AM, Maksim Milyutin > wrote: > > > I have tested the following case: > > > > create type pair as (x int, y int); > > p

Re: [HACKERS] Commitfest 201709 is now closed

2017-10-02 Thread Haribabu Kommi
On Tue, Oct 3, 2017 at 3:12 AM, Robert Haas wrote: > On Mon, Oct 2, 2017 at 11:57 AM, Tom Lane wrote: > > Daniel Gustafsson writes: > >> Thanks to everyone who participated, and to everyone who have responded > to my > >> nagging via the CF app email function. This is clearly an awesome > commu

Re: [HACKERS] Logging idle checkpoints

2017-10-02 Thread Michael Paquier
On Tue, Oct 3, 2017 at 12:01 AM, Stephen Frost wrote: > I certainly don't care for the idea of adding log messages saying we > aren't doing anything just to match a count that's incorrectly claiming > that checkpoints are happening when they aren't. > > The down-thread suggestion of keeping track

Re: [HACKERS] 64-bit queryId?

2017-10-02 Thread Michael Paquier
On Tue, Oct 3, 2017 at 9:07 AM, Alexander Korotkov wrote: > +1, > I see 3 options there: > 1) Drop high-order bit, as you proposed. > 2) Allow negative queryIds. > 3) Implement unsigned 64-type. > > #1 causes minor loss of precision which looks rather insignificant in given > context. > #2 might b

Re: [HACKERS] [sqlsmith] stuck spinlock in pg_stat_get_wal_receiver after OOM

2017-10-02 Thread Michael Paquier
On Tue, Oct 3, 2017 at 6:54 AM, Tom Lane wrote: > I wrote: >> If this is the only problem then I'd agree we should stick to a spinlock >> (I assume the strings in question can't be very long). I was thinking >> more about what to do if we find other violations that are harder to fix. I don't thi

[HACKERS] Conversion error

2017-10-02 Thread Tatsuo Ishii
I saw this while restoring 9.6 database to 10.0 database. \connect: FATAL: conversion between UTF8 and MULE_INTERNAL is not supported Is this expected? I saw nothing releated to this in the release notes. Best regards, -- Tatsuo Ishii SRA OSS, Inc. Japan English: http://www.sraoss.co.jp/index_e

Re: [HACKERS] Commitfest 201709 is now closed

2017-10-02 Thread Amit Langote
On 2017/10/03 7:16, Michael Paquier wrote: > On Tue, Oct 3, 2017 at 1:23 AM, Alexander Korotkov > wrote: >> On Mon, Oct 2, 2017 at 6:57 PM, Tom Lane wrote: >>> >>> Daniel Gustafsson writes: Thanks to everyone who participated, and to everyone who have responded to my nagging via t

Re: [HACKERS] 64-bit queryId?

2017-10-02 Thread Alexander Korotkov
On Tue, Oct 3, 2017 at 12:32 AM, Tom Lane wrote: > Peter Geoghegan writes: > > You need to change the SQL interface as well, although I'm not sure > > exactly how. The problem is that you are now passing a uint64 queryId > > to Int64GetDatumFast() within pg_stat_statements_internal(). That > > w

Re: [HACKERS] Horrible CREATE DATABASE Performance in High Sierra

2017-10-02 Thread Andres Freund
On 2017-10-02 19:50:51 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2017-10-02 18:33:17 -0400, Tom Lane wrote: > >> I'm kind of surprised that machine B doesn't show obvious tanking in this > >> test given that msync() makes it suck so badly at copying a database. > >> I wonder what is di

Re: [HACKERS] Horrible CREATE DATABASE Performance in High Sierra

2017-10-02 Thread Tom Lane
Andres Freund writes: > On 2017-10-02 18:33:17 -0400, Tom Lane wrote: >> I'm kind of surprised that machine B doesn't show obvious tanking in this >> test given that msync() makes it suck so badly at copying a database. >> I wonder what is different from the kernel's standpoint ... maybe the >> sh

Re: [HACKERS][BUG] Cache invalidation for queries that contains const of temporary composite type

2017-10-02 Thread Alexander Korotkov
On Tue, Oct 3, 2017 at 12:20 AM, Maksim Milyutin wrote: > I have tested the following case: > > create type pair as (x int, y int); > prepare test as select json_populate_record(null::pair, '{"x": 1, "y": > 2}'::json); > drop type pair cascade; > > execute test; > > -- The following output is obt

Re: [HACKERS] Horrible CREATE DATABASE Performance in High Sierra

2017-10-02 Thread Andres Freund
Hi, On 2017-10-02 18:33:17 -0400, Tom Lane wrote: > Andres Freund writes: > > To demonstrate what I'm observing here, on linux with a fairly fast ssd: > > ... > > I tried to replicate this test as closely as I could on the Mac hardware > I have laying about. Thanks! > I only bothered with the

Re: [HACKERS] list of credits for release notes

2017-10-02 Thread Bruce Momjian
On Mon, Oct 2, 2017 at 02:12:50PM -0400, Stephen Frost wrote: > > How should this be handled for the Postgres 11 release notes? > > Ideally, we would let the individuals choose how to be recognized in > release notes, and anywhere else we recognize them. We have the start > of that in https://po

Re: [HACKERS] Horrible CREATE DATABASE Performance in High Sierra

2017-10-02 Thread Tom Lane
Andres Freund writes: > To demonstrate what I'm observing here, on linux with a fairly fast ssd: > ... I tried to replicate this test as closely as I could on the Mac hardware I have laying about. I only bothered with the synchronous_commit=off case, though, since you say that shows the worst ef

Re: [HACKERS] Commitfest 201709 is now closed

2017-10-02 Thread Michael Paquier
On Tue, Oct 3, 2017 at 1:23 AM, Alexander Korotkov wrote: > On Mon, Oct 2, 2017 at 6:57 PM, Tom Lane wrote: >> >> Daniel Gustafsson writes: >> > Thanks to everyone who participated, and to everyone who have responded >> > to my >> > nagging via the CF app email function. This is clearly an aweso

Re: [HACKERS] Binary search in fmgr_isbuiltin() is a bottleneck.

2017-10-02 Thread Andres Freund
On 2017-10-02 17:57:51 -0400, Tom Lane wrote: > Andres Freund writes: > > Done that way. It's a bit annoying, because we've to take care to > > initialize the "unused" part of the array with a valid signalling it's > > an unused mapping. Can't use 0 for that because fmgr_builtins[0] is a > > valid

Re: [HACKERS] Binary search in fmgr_isbuiltin() is a bottleneck.

2017-10-02 Thread Tom Lane
Andres Freund writes: > Done that way. It's a bit annoying, because we've to take care to > initialize the "unused" part of the array with a valid signalling it's > an unused mapping. Can't use 0 for that because fmgr_builtins[0] is a > valid entry. The prototype code I posted further upthread ju

Re: [HACKERS] [sqlsmith] stuck spinlock in pg_stat_get_wal_receiver after OOM

2017-10-02 Thread Tom Lane
I wrote: > If this is the only problem then I'd agree we should stick to a spinlock > (I assume the strings in question can't be very long). I was thinking > more about what to do if we find other violations that are harder to fix. I took a quick look through walreceiver.c, and there are a couple

Re: [HACKERS] Binary search in fmgr_isbuiltin() is a bottleneck.

2017-10-02 Thread Andres Freund
On 2017-09-28 19:06:27 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2017-09-28 18:52:28 -0400, Tom Lane wrote: > >> Uh, what? Access to fmgr_nbuiltins shouldn't be part of any critical path > >> anymore after this change. > > > Indeed. But the size of the the oid -> fmgr_builtins index

[HACKERS] [sqlsmith] crash in RestoreLibraryState during low-memory testing

2017-10-02 Thread Andreas Seltenreich
Hi, doing low-memory testing with REL_10_STABLE at 1f19550a87 also produced a couple of parallel worker core dumps with the backtrace below. Although most of the backtrace is inside the dynamic linker, it looks like it was passed a pointer to gone-away shared memory. regards, Andreas Core was ge

Re: [HACKERS] [sqlsmith] stuck spinlock in pg_stat_get_wal_receiver after OOM

2017-10-02 Thread Tom Lane
Andres Freund writes: > On 2017-10-02 17:30:25 -0400, Tom Lane wrote: >> Or replace the spinlock with an LWLock? > That'd probably be a good idea, but I'm loathe to do so in the back > branches. Not at this callsite, but some others, there's some potential > for contention. If this is the only p

Re: [HACKERS] 64-bit queryId?

2017-10-02 Thread Tom Lane
Peter Geoghegan writes: > You need to change the SQL interface as well, although I'm not sure > exactly how. The problem is that you are now passing a uint64 queryId > to Int64GetDatumFast() within pg_stat_statements_internal(). That > worked when queryId was a uint32, because you can easily repre

Re: [HACKERS] [sqlsmith] stuck spinlock in pg_stat_get_wal_receiver after OOM

2017-10-02 Thread Andres Freund
On 2017-10-02 17:30:25 -0400, Tom Lane wrote: > Andres Freund writes: > > Yes, that'd be a bad idea. It's not great to have memcpys in a critical > > section, but it's way better than pallocs. So we need to use some local > > buffers that this get copied to. > > Or replace the spinlock with an LW

Re: [HACKERS] [sqlsmith] stuck spinlock in pg_stat_get_wal_receiver after OOM

2017-10-02 Thread Tom Lane
Andres Freund writes: > On 2017-10-02 22:56:49 +0200, Andreas Seltenreich wrote: >> low-memory testing with REL_10_STABLE at 1f19550a87 produced the >> following PANIC: >> stuck spinlock detected at pg_stat_get_wal_receiver, walreceiver.c:1397 > Ugh. Egad. > Yes, that'd be a bad idea. It's not

Re: [HACKERS][BUG] Cache invalidation for queries that contains const of temporary composite type

2017-10-02 Thread Maksim Milyutin
Hi, Alexander! Thanks for the comments. 02.10.17 20:02, Alexander Korotkov wrote: Please, register this patch at upcoming commitfest to ensure it wouldn't be forgotten. Regression test changes (both .sql and .out) are essential parts of the patch.  I see no point in posting them separately. 

Re: [HACKERS] 64-bit queryId?

2017-10-02 Thread Peter Geoghegan
On Mon, Oct 2, 2017 at 9:10 AM, Robert Haas wrote: > On Mon, Oct 2, 2017 at 11:02 AM, Joshua D. Drake > wrote: >> +1 to both of these as well. > > OK, so here's a patch. Review appreciated. You need to change the SQL interface as well, although I'm not sure exactly how. The problem is that you

Re: [HACKERS] list of credits for release notes

2017-10-02 Thread Tatsuo Ishii
> On Wed, Sep 27, 2017 at 8:29 PM, Michael Paquier > wrote: >> Looking at this list, the first name is followed by the family name, >> so there are inconsistencies with some Japanese names: >> - Fujii Masao should be Masao Fujii. >> - KaiGai Kohei should be Kohei Kaigai. > > But his emails say Fu

Re: [HACKERS] [sqlsmith] stuck spinlock in pg_stat_get_wal_receiver after OOM

2017-10-02 Thread Andres Freund
On 2017-10-02 22:56:49 +0200, Andreas Seltenreich wrote: > Hi, > > low-memory testing with REL_10_STABLE at 1f19550a87 produced the > following PANIC: > > stuck spinlock detected at pg_stat_get_wal_receiver, walreceiver.c:1397 Ugh. > I was about to wrap the pstrdup()s with a PG_TRY block, b

Re: [HACKERS] [sqlsmith] stuck spinlock in pg_stat_get_wal_receiver after OOM

2017-10-02 Thread Thomas Munro
On Tue, Oct 3, 2017 at 9:56 AM, Andreas Seltenreich wrote: > low-memory testing with REL_10_STABLE at 1f19550a87 produced the > following PANIC: > > stuck spinlock detected at pg_stat_get_wal_receiver, walreceiver.c:1397 > > I was about to wrap the pstrdup()s with a PG_TRY block, but I can't f

[HACKERS] [sqlsmith] stuck spinlock in pg_stat_get_wal_receiver after OOM

2017-10-02 Thread Andreas Seltenreich
Hi, low-memory testing with REL_10_STABLE at 1f19550a87 produced the following PANIC: stuck spinlock detected at pg_stat_get_wal_receiver, walreceiver.c:1397 I was about to wrap the pstrdup()s with a PG_TRY block, but I can't find a spinlock being released in a PG_CATCH block anywhere, so ma

Re: [HACKERS] 64-bit queryId?

2017-10-02 Thread Gavin Flower
On 03/10/17 04:02, Joshua D. Drake wrote: On 10/01/2017 04:22 PM, Robert Haas wrote: On Sun, Oct 1, 2017 at 3:48 PM, Greg Stark wrote: Well these kinds of monitoring systems tend to be used by operations people who are a lot more practical and a lot less worried about theoretical concerns like

Re: [HACKERS] Horrible CREATE DATABASE Performance in High Sierra

2017-10-02 Thread Andres Freund
On 2017-10-02 15:59:05 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2017-10-02 15:54:43 -0400, Tom Lane wrote: > >> Should I expect there to be any difference at all? We don't enable > >> *_flush_after by default on non-Linux platforms. > > > Right, you'd have to enable that. But your p

Re: [HACKERS] Horrible CREATE DATABASE Performance in High Sierra

2017-10-02 Thread Tom Lane
Andres Freund writes: > On 2017-10-02 15:54:43 -0400, Tom Lane wrote: >> Should I expect there to be any difference at all? We don't enable >> *_flush_after by default on non-Linux platforms. > Right, you'd have to enable that. But your patch would neuter an > intentionally enabled config too, n

Re: [HACKERS] Horrible CREATE DATABASE Performance in High Sierra

2017-10-02 Thread Andres Freund
On 2017-10-02 15:54:43 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2017-10-02 15:42:25 -0400, Tom Lane wrote: > >> I experimented with this further by seeing whether the msync() code path > >> is of any value on Sierra either. The answer seems to be "no": cloning > >> a scale-1000 pgben

Re: [HACKERS] Horrible CREATE DATABASE Performance in High Sierra

2017-10-02 Thread Tom Lane
Andres Freund writes: > On 2017-10-02 15:42:25 -0400, Tom Lane wrote: >> I experimented with this further by seeing whether the msync() code path >> is of any value on Sierra either. The answer seems to be "no": cloning >> a scale-1000 pgbench database takes about 17-18 seconds on my Sierra >> la

Re: [HACKERS] Horrible CREATE DATABASE Performance in High Sierra

2017-10-02 Thread Andres Freund
On 2017-10-02 15:42:25 -0400, Tom Lane wrote: > I wrote: > > In short, therefore, APFS cannot cope with the way we're using msync(). > > I experimented with this further by seeing whether the msync() code path > is of any value on Sierra either. The answer seems to be "no": cloning > a scale-1000

Re: [HACKERS] Horrible CREATE DATABASE Performance in High Sierra

2017-10-02 Thread Brent Dearth
Thanks for this breakdown Tom! FWIW - I'm on Postgres 9.6.5 as bundled with Postgres.app (2.0.5) running on 2013 MBP (2.7GHz i7 / 16GB / SSD) setup. It looks like this might be a priority for an upcoming release, so I might try to hold out for downstream, but thanks for the patch. It will help if

Re: [HACKERS] issue: record or row variable cannot be part of multiple-item INTO list

2017-10-02 Thread Robert Haas
On Mon, Oct 2, 2017 at 12:44 PM, Tom Lane wrote: >> And I'm saying - that argument is bogus. Regardless of what people >> want or what we have historically done in the case where the >> record/row is the only INTO target, when there are multiple targets it >> seems clear that they want to match u

Re: [HACKERS] Horrible CREATE DATABASE Performance in High Sierra

2017-10-02 Thread Tom Lane
I wrote: > In short, therefore, APFS cannot cope with the way we're using msync(). I experimented with this further by seeing whether the msync() code path is of any value on Sierra either. The answer seems to be "no": cloning a scale-1000 pgbench database takes about 17-18 seconds on my Sierra l

Re: [HACKERS] generated columns

2017-10-02 Thread Nico Williams
So yes, distinguishing stored vs. not stored computed columns is useful, especially if the expression can refer to other columns of the same row, though not only then. Examples: -- useful only if stored (assuming these never get updated) inserted_at TIMESTAMP WITHOUT TIME ZONE AS (clo

Re: [HACKERS] GSoC 2017 : Patch for predicate locking in Gist index

2017-10-02 Thread Alexander Korotkov
On Mon, Oct 2, 2017 at 9:11 PM, Andrew Borodin wrote: > On Mon, Oct 2, 2017 at 8:00 PM, Alexander Korotkov > wrote: > > What happen if exactly this "continue" fires? > > > >> if (GistFollowRight(stack->page)) > >> { > >> if (!xlocked) > >> { > >> LockBuffer(stack->buffer, GIST_UNLOCK); > >> Lock

Re: [HACKERS] generated columns

2017-10-02 Thread Nico Williams
On Mon, Oct 02, 2017 at 02:30:38PM -0400, Tom Lane wrote: > Nico Williams writes: > > On Mon, Oct 02, 2017 at 12:50:14PM -0400, Adam Brusselback wrote: > >> So for me, i'd rather default to compute on read, as long storing the > >> pre-computed value is an option when necessary. > > > Sure, I agr

Re: [HACKERS] generated columns

2017-10-02 Thread David Fetter
On Mon, Oct 02, 2017 at 02:30:38PM -0400, Tom Lane wrote: > Nico Williams writes: > > On Mon, Oct 02, 2017 at 12:50:14PM -0400, Adam Brusselback wrote: > >> So for me, i'd rather default to compute on read, as long storing the > >> pre-computed value is an option when necessary. > > > Sure, I agr

Re: [HACKERS] generated columns

2017-10-02 Thread Tom Lane
Nico Williams writes: > On Mon, Oct 02, 2017 at 12:50:14PM -0400, Adam Brusselback wrote: >> So for me, i'd rather default to compute on read, as long storing the >> pre-computed value is an option when necessary. > Sure, I agree. I was just wondering whether there might be any other > differenc

Re: [HACKERS] Horrible CREATE DATABASE Performance in High Sierra

2017-10-02 Thread Tom Lane
Brent Dearth writes: > I just recently "upgraded" to High Sierra and experiencing horrendous CREATE > DATABASE performance. Creating a database from a 3G template DB used to > take ~1m but post-upgrade is taking ~22m at a sustained write of around > 4MB/s. Occasionally, attempting to create an emp

Re: [HACKERS] list of credits for release notes

2017-10-02 Thread Stephen Frost
Bruce, * Bruce Momjian (br...@momjian.us) wrote: > On Fri, Sep 29, 2017 at 12:00:05PM -0400, Peter Eisentraut wrote: > > On 9/29/17 11:35, Robert Haas wrote: > > > On Wed, Sep 27, 2017 at 8:29 PM, Michael Paquier > > > wrote: > > >> Looking at this list, the first name is followed by the family n

Re: [HACKERS] GSoC 2017 : Patch for predicate locking in Gist index

2017-10-02 Thread Andrew Borodin
On Mon, Oct 2, 2017 at 8:00 PM, Alexander Korotkov wrote: > What happen if exactly this "continue" fires? > >> if (GistFollowRight(stack->page)) >> { >> if (!xlocked) >> { >> LockBuffer(stack->buffer, GIST_UNLOCK); >> LockBuffer(stack->buffer, GIST_EXCLUSIVE); >> xlocked = true; >> /* someone migh

Re: [HACKERS] generated columns

2017-10-02 Thread Nico Williams
On Mon, Oct 02, 2017 at 12:50:14PM -0400, Adam Brusselback wrote: > I know that for my use-cases, having both options available would be very > appreciated. The vast majority of the computed columns I would use in my > database would be okay to compute on read. But there are for sure some > which

Re: [HACKERS] list of credits for release notes

2017-10-02 Thread Bruce Momjian
On Fri, Sep 29, 2017 at 12:00:05PM -0400, Peter Eisentraut wrote: > On 9/29/17 11:35, Robert Haas wrote: > > On Wed, Sep 27, 2017 at 8:29 PM, Michael Paquier > > wrote: > >> Looking at this list, the first name is followed by the family name, > >> so there are inconsistencies with some Japanese na

Re: [HACKERS] CREATE COLLATION does not sanitize ICU's BCP 47 language tags. Should it?

2017-10-02 Thread Peter Geoghegan
On Mon, Oct 2, 2017 at 9:42 AM, Peter Eisentraut wrote: > I understand where you are coming from. My experience with developing > this feature has been that it is quite fragile in some ways. We have > had this out there for testing for many months, and we have fixed many > bugs, and follow-up bu

[HACKERS] Issues with logical replication

2017-10-02 Thread Konstantin Knizhnik
I have faced two issues with logical replication. Repro scenario: 1. start two Postgres instances (I start both at local machine). 2. Initialize pgbench tables at both instances: pgbench -i -s 10 postgres 3. Create publication of pgbench_accounts table at one node: create publication pub

Re: [HACKERS] issue: record or row variable cannot be part of multiple-item INTO list

2017-10-02 Thread Pavel Stehule
2017-10-02 18:44 GMT+02:00 Tom Lane : > Robert Haas writes: > > On Mon, Oct 2, 2017 at 12:28 PM, Tom Lane wrote: > >> I'm not sure if that's true or not. I am sure, though, that since > >> we've done B for twenty years we can't just summarily change to A. > > > I agree, but so what? You said t

Re: [HACKERS][BUG] Cache invalidation for queries that contains const of temporary composite type

2017-10-02 Thread Alexander Korotkov
On Mon, Oct 2, 2017 at 3:43 PM, Maksim Milyutin wrote: > On 26.09.2017 23:25, Maksim Milyutin wrote: > > Updated patchset contains more transparent definition of composite type > for constant node and regression test for described above buggy case. > > Is there any interest on the problem in this

Re: [HACKERS] Issues with logical replication

2017-10-02 Thread Petr Jelinek
On 02/10/17 17:49, Konstantin Knizhnik wrote: > I have faced two issues with logical replication. > Reproducing scenario: > > 1. start two Postgres instances (I start both at local machine). > 2. Initialize pgbench tables at both instances: >     pgbench -i -s 10 postgres > 3. Create publication o

Re: [HACKERS] GSoC 2017: weekly progress reports (week 6)

2017-10-02 Thread Alexander Korotkov
On Sun, Oct 1, 2017 at 11:53 AM, Shubham Barai wrote: > Yes, page-level predicate locking should happen only when fast update is > off. > Actually, I forgot to put conditions in updated patch. Does everything > else look ok ? > I think that isolation tests should be improved. It doesn't seems t

Re: [HACKERS] generated columns

2017-10-02 Thread Adam Brusselback
I know that for my use-cases, having both options available would be very appreciated. The vast majority of the computed columns I would use in my database would be okay to compute on read. But there are for sure some which would be performance prohibitive to have compute on read, so i'd rather h

Re: [HACKERS] issue: record or row variable cannot be part of multiple-item INTO list

2017-10-02 Thread Tom Lane
Robert Haas writes: > On Mon, Oct 2, 2017 at 12:28 PM, Tom Lane wrote: >> I'm not sure if that's true or not. I am sure, though, that since >> we've done B for twenty years we can't just summarily change to A. > I agree, but so what? You said that we couldn't adopt Pavel's > proposal for this

Re: [HACKERS] CREATE COLLATION does not sanitize ICU's BCP 47 language tags. Should it?

2017-10-02 Thread Peter Eisentraut
On 10/1/17 14:26, Peter Geoghegan wrote: > It does seem too late. It's disappointing that we did not do better > here. This problem was entirely avoidable. I understand where you are coming from. My experience with developing this feature has been that it is quite fragile in some ways. We have h

Re: [HACKERS] issue: record or row variable cannot be part of multiple-item INTO list

2017-10-02 Thread Robert Haas
On Mon, Oct 2, 2017 at 12:28 PM, Tom Lane wrote: > Robert Haas writes: >> On Tue, Sep 19, 2017 at 3:18 PM, Tom Lane wrote: >>> That's certainly a case that we ought to support somehow. The problem is >>> staying reasonably consistent with the two-decades-old precedent of the >>> existing behavi

Re: [JDBC] [HACKERS] Channel binding support for SCRAM-SHA-256

2017-10-02 Thread Robert Haas
On Fri, Sep 15, 2017 at 6:29 PM, Michael Paquier wrote: > I would like to point out that per the RFC, if the client attempts a > SSL connection with SCRAM and that the server supports channel > binding, then it has to publish the SASL mechanism for channel > binding, aka SCRAM-PLUS. If the client

Re: [HACKERS] issue: record or row variable cannot be part of multiple-item INTO list

2017-10-02 Thread Tom Lane
Robert Haas writes: > On Tue, Sep 19, 2017 at 3:18 PM, Tom Lane wrote: >> That's certainly a case that we ought to support somehow. The problem is >> staying reasonably consistent with the two-decades-old precedent of the >> existing behavior for one target variable. > My point is that you obje

Re: [HACKERS] Commitfest 201709 is now closed

2017-10-02 Thread Alexander Korotkov
On Mon, Oct 2, 2017 at 6:57 PM, Tom Lane wrote: > Daniel Gustafsson writes: > > Thanks to everyone who participated, and to everyone who have responded > to my > > nagging via the CF app email function. This is clearly an awesome > community. > > And thanks to you for your hard work as CFM! Tha

Re: [HACKERS] issue: record or row variable cannot be part of multiple-item INTO list

2017-10-02 Thread Robert Haas
On Tue, Sep 19, 2017 at 3:18 PM, Tom Lane wrote: > Robert Haas writes: >> I think the fact that single-target INTO lists and multiple-target >> INTO lists are handled completely differently is extremely poor >> language design. It would have been far better, as you suggested >> downthread, to ha

Re: [HACKERS] [PATCH]make pg_rewind to not copy useless WAL files

2017-10-02 Thread chenhj
On 2017-10-02 23:24:30,"Alexander Korotkov" wrote: On Sun, Oct 1, 2017 at 8:27 PM, chenhj wrote: Now, this patch looks good for me. It applies cleanly, builds cleanly, passes regression tests, new functionality is covered by regression tests. Code is OK for me and docs too. I'm markin

Re: [HACKERS] [PATCH] Assert that the correct locks are held when calling PageGetLSN()

2017-10-02 Thread Robert Haas
On Mon, Oct 2, 2017 at 11:05 AM, Michael Paquier wrote: > Well, there are cases where you don't need any locking checks, and the > proposed patch ignores that. I understand that, but shouldn't we then look for a way to adjust the patch so that it doesn't have that issue any longer, rather than ju

Re: [HACKERS] Commitfest 201709 is now closed

2017-10-02 Thread Robert Haas
On Mon, Oct 2, 2017 at 11:57 AM, Tom Lane wrote: > Daniel Gustafsson writes: >> Thanks to everyone who participated, and to everyone who have responded to my >> nagging via the CF app email function. This is clearly an awesome community. > > And thanks to you for your hard work as CFM! That's te

Re: [HACKERS] 64-bit queryId?

2017-10-02 Thread Robert Haas
On Mon, Oct 2, 2017 at 11:02 AM, Joshua D. Drake wrote: > +1 to both of these as well. OK, so here's a patch. Review appreciated. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company 64-bit-queryid-v1.patch Description: Binary data -- Sent via pgsql-ha

Re: [HACKERS] Commitfest 201709 is now closed

2017-10-02 Thread Tom Lane
Daniel Gustafsson writes: > Thanks to everyone who participated, and to everyone who have responded to my > nagging via the CF app email function. This is clearly an awesome community. And thanks to you for your hard work as CFM! That's tedious and largely thankless work, but it's needed to keep

Re: [HACKERS] [PATCH] Assert that the correct locks are held when calling PageGetLSN()

2017-10-02 Thread Alvaro Herrera
Michael Paquier wrote: > On Mon, Oct 2, 2017 at 11:44 PM, Robert Haas wrote: > > On Mon, Oct 2, 2017 at 10:19 AM, Michael Paquier > > wrote: > >> So those bits could be considered for integration. > > > > Yes, and they also tend to suggest that the rest of the patch has value. > > Well, there ar

[HACKERS] Horrible CREATE DATABASE Performance in High Sierra

2017-10-02 Thread Brent Dearth
I just recently "upgraded" to High Sierra and experiencing horrendous CREATE DATABASE performance. Creating a database from a 3G template DB used to take ~1m but post-upgrade is taking ~22m at a sustained write of around 4MB/s. Occasionally, attempting to create an empty database hangs indefinitely

[HACKERS] Issues with logical replication

2017-10-02 Thread Konstantin Knizhnik
I have faced two issues with logical replication. Reproducing scenario: 1. start two Postgres instances (I start both at local machine). 2. Initialize pgbench tables at both instances: pgbench -i -s 10 postgres 3. Create publication of pgbench_accounts table at one node: create publicatio

[HACKERS] Commitfest 201709 is now closed

2017-10-02 Thread Daniel Gustafsson
We have now entered October, which marks the end of Commitfest 201709. All patches have now been dealt with and the commitfest is closed. Before starting on moving, and closing, patches the stat for this commitfest were: Needs review: 69. Waiting on Author: 22. Ready for

Re: [HACKERS] [PATCH]make pg_rewind to not copy useless WAL files

2017-10-02 Thread Alexander Korotkov
On Sun, Oct 1, 2017 at 8:27 PM, chenhj wrote: > On 2017-10-01 04:09:19,"Alexander Korotkov" > wrote: > > On Sat, Sep 30, 2017 at 8:18 PM, chenhj wrote: > >> On 2017-09-30 02:17:54,"Alexander Korotkov" > > wrote: >> >> >> Great. Now code of this patch looks good for me. >> However, we forgot a

Re: [HACKERS] generated columns

2017-10-02 Thread Nico Williams
On Thu, Aug 31, 2017 at 12:16:43AM -0400, Peter Eisentraut wrote: > In previous discussions, it has often been a source of confusion whether > these generated columns are supposed to be computed on insert/update and > stored, or computed when read. The SQL standard is not explicit, but > appears t

Re: [HACKERS] [PATCH] Assert that the correct locks are held when calling PageGetLSN()

2017-10-02 Thread Michael Paquier
On Mon, Oct 2, 2017 at 11:44 PM, Robert Haas wrote: > On Mon, Oct 2, 2017 at 10:19 AM, Michael Paquier > wrote: >> So those bits could be considered for integration. > > Yes, and they also tend to suggest that the rest of the patch has value. Well, there are cases where you don't need any lockin

Re: [HACKERS] 64-bit queryId?

2017-10-02 Thread Joshua D. Drake
On 10/01/2017 04:22 PM, Robert Haas wrote: On Sun, Oct 1, 2017 at 3:48 PM, Greg Stark wrote: Well these kinds of monitoring systems tend to be used by operations people who are a lot more practical and a lot less worried about theoretical concerns like that. +1, well said. In context the po

Re: [HACKERS] Logging idle checkpoints

2017-10-02 Thread Stephen Frost
Vik, all, * Vik Fearing (vik.fear...@2ndquadrant.com) wrote: > I recently had a sad because I noticed that checkpoint counts were > increasing in pg_stat_bgwriter, but weren't accounted for in my logs > with log_checkpoints enabled. > After some searching, I found that it was the idle checkpoints

  1   2   >