Re: Transactions involving multiple postgres foreign servers, take 2

2020-07-15 Thread Fujii Masao
On 2020/07/15 15:06, Masahiko Sawada wrote: On Tue, 14 Jul 2020 at 09:08, Masahiro Ikeda wrote: I've attached the latest version patches. I've incorporated the review comments I got so far and improved locking strategy. Thanks for updating the patch! I have three questions about the v23

Re: POC and rebased patch for CSN based snapshots

2020-07-15 Thread Andrey V. Lepikhov
On 7/13/20 11:46 AM, movead...@highgo.ca wrote: I continue to see your patch. Some code improvements see at the attachment. Questions: * csnSnapshotActive is the only member of the CSNshapshotShared struct. * The WriteAssignCSNXlogRec() routine. I din't understand why you add 20 nanosec to curr

gs_group_1 crashing on 13beta2/s390x

2020-07-15 Thread Christoph Berg
On the Debian s390x buildd, the 13beta2 build is crashing: 2020-07-15 01:19:59.149 UTC [859] LOG: server process (PID 1415) was terminated by signal 11: Segmentation fault 2020-07-15 01:19:59.149 UTC [859] DETAIL: Failed process was running: create table gs_group_1 as select g100, g10,

Re: Support for NSS as a libpq TLS backend

2020-07-15 Thread Thomas Munro
On Fri, Jul 10, 2020 at 5:10 PM Thomas Munro wrote: > -#define LARGE_OFF_T (((off_t) 1 << 62) - 1 + ((off_t) 1 << 62)) > +#define LARGE_OFF_T off_t) 1 << 31) << 31) - 1 + (((off_t) 1 << 31) << > 31)) > > I see the same when I use Debian's autoconf, but not FreeBSD's or > MacPorts', despite al

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-07-15 Thread Ajin Cherian
On Wed, Jul 15, 2020 at 2:05 PM Dilip Kumar wrote: > Please see the > latest patch set v33. > > -- > Regards, > Dilip Kumar > EnterpriseDB: http://www.enterprisedb.com > > > I have a minor comment. You've defined a new function ReorderBufferStartStreaming() but the function doesn't actually sta

file_fdw vs relative paths

2020-07-15 Thread Magnus Hagander
According to the documentation, the filename given in file_fdw must be an absolute path. Hwever, it works perfectly fine with a relative path. So either the documentation is wrong, or the code is wrong. It behaves the same at least back to 9.5, I did not try it further back than that. I can't fin

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-07-15 Thread Amit Kapila
On Wed, Jul 15, 2020 at 4:51 PM Ajin Cherian wrote: > > On Wed, Jul 15, 2020 at 2:05 PM Dilip Kumar wrote: >> >> Please see the >> latest patch set v33. >> >> >> > > I have a minor comment. You've defined a new function > ReorderBufferStartStreaming() but the function doesn't actually start >

Re: SQL/JSON: functions

2020-07-15 Thread Andrew Dunstan
On 7/14/20 9:47 PM, Nikita Glukhov wrote: > Attached 49th version of the patches with two new patches #5 and #6. > > On 15.07.2020 00:09, Andrew Dunstan wrote: >> On 7/14/20 1:00 PM, Andrew Dunstan wrote: >>> On 7/5/20 1:29 PM, Justin Pryzby wrote: On Mon, Mar 23, 2020 at 08:28:52PM +0300, N

Re: Transactions involving multiple postgres foreign servers, take 2

2020-07-15 Thread Masahiro Ikeda
On 2020-07-15 15:06, Masahiko Sawada wrote: On Tue, 14 Jul 2020 at 09:08, Masahiro Ikeda wrote: > I've attached the latest version patches. I've incorporated the review > comments I got so far and improved locking strategy. Thanks for updating the patch! I have three questions about the v23 p

Re: Partitioning and postgres_fdw optimisations for multi-tenancy

2020-07-15 Thread Etsuro Fujita
On Wed, Jul 15, 2020 at 12:12 AM Alexey Kondratov wrote: > On 2020-07-14 15:27, Ashutosh Bapat wrote: > > On Tue, Jul 14, 2020 at 12:48 AM Alexey Kondratov > > wrote: > >> Some real-life test queries show, that all single-node queries aren't > >> pushed-down to the required node. For example: > >

Re: Mark btree_gist functions as PARALLEL SAFE

2020-07-15 Thread Alexander Korotkov
On Thu, Jun 18, 2020 at 7:48 PM Winfield, Steven wrote: > Done - thanks again. This patch looks good to me. I've rechecked it marks all the functions as parallel safe by installing an extension and querying the catalog. I've also rechecked that there is nothing suspicious in these functions in

Volatile Functions in Parallel Plans

2020-07-15 Thread Zhenghua Lyu
Hi, I test some SQL in the latest Postgres master branch code (we find these issues when developing Greenplum database in the PR https://github.com/greenplum-db/gpdb/pull/10418, and my colleague come up with the following cases in Postgres): create table t3 (c1 text, c2 text); CREATE TABLE i

Have SIGHUP instead of SIGTERM for config reload in logical replication launcher

2020-07-15 Thread Bharath Rupireddy
Hi, In ApplyLauncherMain, it seems like we are having SIGTERM signal mapped for config reload. I think we should be having SIGHUP for SignalHandlerForConfigReload(). Otherwise we miss to take the updated value for wal_retrieve_retry_interval in ApplyLauncherMain. Attached is a patch having this c

Re: Have SIGHUP instead of SIGTERM for config reload in logical replication launcher

2020-07-15 Thread Dilip Kumar
On Wed, Jul 15, 2020 at 6:17 PM Bharath Rupireddy wrote: > > Hi, > > In ApplyLauncherMain, it seems like we are having SIGTERM signal > mapped for config reload. I think we should be having SIGHUP for > SignalHandlerForConfigReload(). Otherwise we miss to take the updated > value for wal_retrieve_

Which SET TYPE don't actually require a rewrite

2020-07-15 Thread Magnus Hagander
Our Fine Manual (TM) specifies: "As an exception, when changing the type of an existing column, if the USING clause does not change the column contents and the old type is either binary coercible to the new type or an unconstrained domain over the new type, a table rewrite is not needed; but any in

Re: some more pg_dump refactoring

2020-07-15 Thread Peter Eisentraut
On 2020-07-09 16:14, Peter Eisentraut wrote: On 2020-07-08 06:42, Fabien COELHO wrote: What do you think about this patch to reorganize the existing code from that old commit? I think it is a definite further improvement. Patch applies cleanly, compiles, global & pg_dump tap test ok, looks ok

renaming configure.in to configure.ac

2020-07-15 Thread Peter Eisentraut
It was mentioned elsewhere in passing that a new Autoconf release might be coming. That one will warn about the old naming "configure.in" and request "configure.ac". So we might want to rename that sometime. Before we get into the specifics, I suggest that all interested parties check whether

Re: Have SIGHUP instead of SIGTERM for config reload in logical replication launcher

2020-07-15 Thread Amit Kapila
On Wed, Jul 15, 2020 at 6:21 PM Dilip Kumar wrote: > > On Wed, Jul 15, 2020 at 6:17 PM Bharath Rupireddy > wrote: > > > > Hi, > > > > In ApplyLauncherMain, it seems like we are having SIGTERM signal > > mapped for config reload. I think we should be having SIGHUP for > > SignalHandlerForConfigRel

Re: libpq: Request Pipelining/Batching status ?

2020-07-15 Thread Matthieu Garrigues
Did my message made it to the mailing list ? or not yet ? Matthieu Garrigues On Fri, Jul 10, 2020 at 5:08 PM Matthieu Garrigues < matthieu.garrig...@gmail.com> wrote: > Hi all, > > Do you know what is the status of Request Pipelining and/or Batching in > libpq ? > > I could see that I'm not the

Re: renaming configure.in to configure.ac

2020-07-15 Thread Andrew Dunstan
On 7/15/20 9:14 AM, Peter Eisentraut wrote: > It was mentioned elsewhere in passing that a new Autoconf release > might be coming.  That one will warn about the old naming > "configure.in" and request "configure.ac".  So we might want to rename > that sometime. Before we get into the specifics, I

Re: renaming configure.in to configure.ac

2020-07-15 Thread Tom Lane
Peter Eisentraut writes: > It was mentioned elsewhere in passing that a new Autoconf release might > be coming. That one will warn about the old naming "configure.in" and > request "configure.ac". So we might want to rename that sometime. > Before we get into the specifics, I suggest that all

Re: Improve handling of parameter differences in physical replication

2020-07-15 Thread Peter Eisentraut
Here is a minimally updated new patch version to resolve a merge conflict. On 2020-06-24 10:00, Peter Eisentraut wrote: Here is another stab at this subject. This is a much simplified variant: When encountering a parameter change in the WAL that is higher than the standby's current setting, we

BUG #15285: Query used index over field with ICU collation in some cases wrongly return 0 rows

2020-07-15 Thread Jehan-Guillaume de Rorthais
Hi, I'm bumping this thread on pgsql-hacker, hopefully it will drag some more opinions/discussions. Should we try to fix this issue or not? This is clearly an upstream bug. It has been reported, including regression tests, but this doesn't move since 2 years now. If we choose not to fix it on ou

Re: Postgres is not able to handle more than 4k tables!?

2020-07-15 Thread Konstantin Knizhnik
On 15.07.2020 02:17, Alvaro Herrera wrote: On 2020-Jul-10, Konstantin Knizhnik wrote: @@ -3076,6 +3080,10 @@ relation_needs_vacanalyze(Oid relid, instuples = tabentry->inserts_since_vacuum; anltuples = tabentry->changes_since_analyze; + rel = RelationIdGet

Re: Postgres is not able to handle more than 4k tables!?

2020-07-15 Thread Alvaro Herrera
On 2020-Jul-15, Konstantin Knizhnik wrote: > > > On 15.07.2020 02:17, Alvaro Herrera wrote: > > On 2020-Jul-10, Konstantin Knizhnik wrote: > > > > > @@ -3076,6 +3080,10 @@ relation_needs_vacanalyze(Oid relid, > > > instuples = tabentry->inserts_since_vacuum; > > >

Re: Have SIGHUP instead of SIGTERM for config reload in logical replication launcher

2020-07-15 Thread Bharath Rupireddy
> > +1. I will commit this tomorrow unless someone thinks otherwise. > I think versions <= 12, have "pqsignal(SIGHUP, logicalrep_launcher_sighup)", not sure why and which commit removed logicalrep_launcher_sighup(). We might have to also backpatch this patch to version 13. With Regards, Bharath

Re: TAP tests and symlinks on Windows

2020-07-15 Thread Andrew Dunstan
On 7/14/20 1:31 AM, Michael Paquier wrote: > On Fri, Jul 10, 2020 at 07:58:02AM -0400, Andrew Dunstan wrote: >> After much frustration and gnashing of teeth here's a patch that allows >> almost all the TAP tests involving symlinks to work as expected on all >> Windows build environments, without r

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-07-15 Thread Andres Freund
Hi, On 2020-07-14 15:59:21 -0400, Robert Haas wrote: > On Tue, Jul 14, 2020 at 3:42 PM Andres Freund wrote: > > The "found xmin ... from before relfrozenxid ..." cases should all be > > fixable without needing such a function, and without it making fixing > > them significantly easier, no? As far

Re: Have SIGHUP instead of SIGTERM for config reload in logical replication launcher

2020-07-15 Thread Andres Freund
Hi, On 2020-07-15 20:33:59 +0530, Bharath Rupireddy wrote: > > > > +1. I will commit this tomorrow unless someone thinks otherwise. > > > > I think versions <= 12, have "pqsignal(SIGHUP, > logicalrep_launcher_sighup)", not sure why and which commit removed > logicalrep_launcher_sighup(). commit

Re: BUG #16419: wrong parsing BC year in to_date() function

2020-07-15 Thread David G. Johnston
On Tue, May 12, 2020 at 8:56 PM Laurenz Albe wrote: > On Tue, 2020-05-12 at 18:09 -0700, David G. Johnston wrote: > > Redirecting to -hackers for visibility. I feel there needs to be > something done here, even if just documentation (a bullet in the usage > notes section - and a code comment upd

Re: Postgres is not able to handle more than 4k tables!?

2020-07-15 Thread Konstantin Knizhnik
On 15.07.2020 18:03, Alvaro Herrera wrote: On 2020-Jul-15, Konstantin Knizhnik wrote: On 15.07.2020 02:17, Alvaro Herrera wrote: On 2020-Jul-10, Konstantin Knizhnik wrote: @@ -3076,6 +3080,10 @@ relation_needs_vacanalyze(Oid relid, instuples = tabentry->inserts_since_vacuu

Re: Support for NSS as a libpq TLS backend

2020-07-15 Thread Andrew Dunstan
On 5/15/20 4:46 PM, Daniel Gustafsson wrote: > > My plan is to keep hacking at this to have it reviewable for the 14 cycle, so > if anyone has an interest in NSS, then I would love to hear feedback on how it > works (and doesn't work). I'll be happy to help, particularly with Windows support an

Re: Dumping/restoring fails on inherited generated column

2020-07-15 Thread Tom Lane
Peter Eisentraut writes: >> Right, there were a number of combinations that were not properly >> handled. The attached patch should fix them all. It's made against >> PG12 but also works on master. See contained commit message and >> documentation for details. > committed to master and PG12 S

Dependencies for partitioned indexes are still a mess

2020-07-15 Thread Tom Lane
I've been experimenting with trying to dump-and-restore the regression database, which is a test case that for some reason we don't cover in the buildfarm (pg_upgrade is not the same thing). It seems like the dependency choices we've made for partitioned indexes are a complete failure for this purp

Re: gs_group_1 crashing on 13beta2/s390x

2020-07-15 Thread Christoph Berg
Re: To PostgreSQL Hackers > On the Debian s390x buildd, the 13beta2 build is crashing: > > 2020-07-15 01:19:59.149 UTC [859] LOG: server process (PID 1415) was > terminated by signal 11: Segmentation fault > 2020-07-15 01:19:59.149 UTC [859] DETAIL: Failed process was running: create > table g

Re: Warn when parallel restoring a custom dump without data offsets

2020-07-15 Thread Tom Lane
I wrote: > The attached 0001 rewrites your 0001 as per the above ideas (dropping > the proposed doc change for now), and includes your 0004 for simplicity. > I'm including your 0002 verbatim just so the cfbot will be able to do a > meaningful test on 0001; but as stated, I don't really want to comm

Re: gs_group_1 crashing on 13beta2/s390x

2020-07-15 Thread Tom Lane
Christoph Berg writes: >> On the Debian s390x buildd, the 13beta2 build is crashing: > I wired gdb into the build process and got this backtrace: > #0 datumCopy (typByVal=false, typLen=-1, value=0) at > ./build/../src/backend/utils/adt/datum.c:142 > vl = 0x0 > res = >

Re: Generic Index Skip Scan

2020-07-15 Thread David Rowley
On Thu, 16 Jul 2020 at 07:52, Floris Van Nee wrote: > > Besides the great efforts that Dmitry et al. are putting into the skip scan > for DISTINCT queries [1], I’m also still keen on extending the use of it > further. I’d like to address the limited cases in which skipping can occur > here. A f

Re: Support for NSS as a libpq TLS backend

2020-07-15 Thread Daniel Gustafsson
> On 15 Jul 2020, at 20:35, Andrew Dunstan > wrote: > > On 5/15/20 4:46 PM, Daniel Gustafsson wrote: >> >> My plan is to keep hacking at this to have it reviewable for the 14 cycle, so >> if anyone has an interest in NSS, then I would love to hear feedback on how >> it >> works (and doesn't wo

sys_siglist[] is causing us trouble again

2020-07-15 Thread Tom Lane
As of a couple days ago, buildfarm member caiman (Fedora rawhide) is failing like this in all the pre-v12 branches: ccache gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexces

Re: sys_siglist[] is causing us trouble again

2020-07-15 Thread Thomas Munro
On Thu, Jul 16, 2020 at 10:48 AM Tom Lane wrote: > We haven't changed anything, ergo something changed at the OS level. > > Oddly, we'd not get to this code unless configure set > HAVE_DECL_SYS_SIGLIST, so it's defined *somewhere*. I suspect the root > issue here is some rearrangement of system h

Re: sys_siglist[] is causing us trouble again

2020-07-15 Thread Tom Lane
Thomas Munro writes: > On Thu, Jul 16, 2020 at 10:48 AM Tom Lane wrote: >> Oddly, we'd not get to this code unless configure set >> HAVE_DECL_SYS_SIGLIST, so it's defined *somewhere*. > It looks like glibc very recently decided[1] to hide the declaration, > but we're using a cached configure tes

Re: Infinities in type numeric

2020-07-15 Thread Dean Rasheed
On Tue, 16 Jun 2020 at 18:24, Tom Lane wrote: > > The attached v3 patch fixes these things and also takes care of an > oversight in v2: I'd made numeric() apply typmod restrictions to Inf, > but not numeric_in() or numeric_recv(). I believe the patch itself > is in pretty good shape now, though t

Re: sys_siglist[] is causing us trouble again

2020-07-15 Thread Tom Lane
Thomas Munro writes: > On Thu, Jul 16, 2020 at 10:48 AM Tom Lane wrote: >> We haven't changed anything, ergo something changed at the OS level. > It looks like glibc very recently decided[1] to hide the declaration, > but we're using a cached configure test result. Right. So, modulo the mis-ca

"cert" + clientcert=verify-ca in pg_hba.conf?

2020-07-15 Thread Kyotaro Horiguchi
Hello. The "Certificate Authentication" section in the doc for PG12 and later describes the relation ship with clientcert as follows. > In a pg_hba.conf record specifying certificate authentication, the > authentication option clientcert is assumed to be verify-ca or > verify-full, and it cannot

Re: Improving connection scalability: GetSnapshotData()

2020-07-15 Thread Alvaro Herrera
On 2020-Jul-15, Andres Freund wrote: > It could make sense to split the conversion of > VariableCacheData->latestCompletedXid to FullTransactionId out from 0001 > into is own commit. Not sure... +1, the commit is large enough and that change can be had in advance. Note you forward-declare struct

Re: sys_siglist[] is causing us trouble again

2020-07-15 Thread Filipe Rosset
On Wed, Jul 15, 2020 at 7:48 PM Tom Lane wrote: > As of a couple days ago, buildfarm member caiman (Fedora rawhide) > is failing like this in all the pre-v12 branches: > > ccache gcc -Wall -Wmissing-prototypes -Wpointer-arith > -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribu

Re: Is it useful to record whether plans are generic or custom?

2020-07-15 Thread torikoshia
On 2020-07-15 11:44, Fujii Masao wrote: On 2020/07/14 21:24, torikoshia wrote: On 2020-07-10 10:49, torikoshia wrote: On 2020-07-08 16:41, Fujii Masao wrote: On 2020/07/08 10:14, torikoshia wrote: On 2020-07-06 22:16, Fujii Masao wrote: On 2020/06/11 14:59, torikoshia wrote: On 2020-06-10 1

Re: INSERT INTO SELECT, Why Parallelism is not selected?

2020-07-15 Thread Amit Kapila
On Wed, Jul 15, 2020 at 8:06 AM Amit Kapila wrote: > > On Wed, Jul 15, 2020 at 12:32 AM Robert Haas wrote: > > > > On Sat, Jul 11, 2020 at 8:37 AM Dilip Kumar wrote: > > > I have just notice that the parallelism is off even for the select > > > part of the query mentioned in the $subject. I see

Re: Volatile Functions in Parallel Plans

2020-07-15 Thread Amit Kapila
On Wed, Jul 15, 2020 at 6:14 PM Zhenghua Lyu wrote: > > > The first plan: > > Finalize Aggregate >-> Gather > Workers Planned: 2 > -> Partial Aggregate >-> Nested Loop > Join Filter: (t3.c1 = t4.c1) > -> Parallel

Re: Transactions involving multiple postgres foreign servers, take 2

2020-07-15 Thread Masahiko Sawada
On Tue, 14 Jul 2020 at 17:24, Masahiro Ikeda wrote: > > > I've attached the latest version patches. I've incorporated the review > > comments I got so far and improved locking strategy. > > I want to ask a question about streaming replication with 2PC. > Are you going to support 2PC with streaming

Re: Volatile Functions in Parallel Plans

2020-07-15 Thread Zhenghua Lyu
Hi, thanks for your reply. But this won't be consistent even for non-parallel plans. If we do not use the distributed law of parallel join, it seems OK. If we generate a parallel plan using the distributed law of the join, then this transformation's pre-assumption might be broken. Currently, we d

RE: Transactions involving multiple postgres foreign servers, take 2

2020-07-15 Thread tsunakawa.ta...@fujitsu.com
Hi Sawada san, I'm reviewing this patch series, and let me give some initial comments and questions. I'm looking at this with a hope that this will be useful purely as a FDW enhancement for our new use cases, regardless of whether the FDW will be used for Postgres scale-out. I don't think it

Re: Partitioning and postgres_fdw optimisations for multi-tenancy

2020-07-15 Thread Etsuro Fujita
On Wed, Jul 15, 2020 at 9:02 PM Etsuro Fujita wrote: > On Wed, Jul 15, 2020 at 12:12 AM Alexey Kondratov > wrote: > > On 2020-07-14 15:27, Ashutosh Bapat wrote: > > > On Tue, Jul 14, 2020 at 12:48 AM Alexey Kondratov > > > wrote: > > >> Some real-life test queries show, that all single-node quer

Re: renaming configure.in to configure.ac

2020-07-15 Thread Noah Misch
On Wed, Jul 15, 2020 at 09:45:54AM -0400, Tom Lane wrote: > Peter Eisentraut writes: > > It was mentioned elsewhere in passing that a new Autoconf release might > > be coming. That one will warn about the old naming "configure.in" and > > request "configure.ac". So we might want to rename that

Re: Transactions involving multiple postgres foreign servers, take 2

2020-07-15 Thread Masahiko Sawada
On Tue, 14 Jul 2020 at 11:19, Fujii Masao wrote: > > > > On 2020/07/14 9:08, Masahiro Ikeda wrote: > >> I've attached the latest version patches. I've incorporated the review > >> comments I got so far and improved locking strategy. > > > > Thanks for updating the patch! > > +1 > I'm interested in

Re: [Patch] ALTER SYSTEM READ ONLY

2020-07-15 Thread Prabhat Sahu
Hi All, I was testing the feature on top of v3 patch and found the "pg_upgrade" failure after keeping "alter system read only;" as below: -- Steps: ./initdb -D data ./pg_ctl -D data -l logs start -c ./psql postgres alter system read only; \q ./pg_ctl -D data -l logs stop -c ./initdb -D data2 ./pg

Re: Volatile Functions in Parallel Plans

2020-07-15 Thread Jesse Zhang
Hi Zhenghua, On Wed, Jul 15, 2020 at 5:44 AM Zhenghua Lyu wrote: > > Hi, > I test some SQL in the latest Postgres master branch code (we find these > issues when > developing Greenplum database in the PR > https://github.com/greenplum-db/gpdb/pull/10418, > and my colleague come up with the f

Re: gs_group_1 crashing on 13beta2/s390x

2020-07-15 Thread Andrew Gierth
> "Tom" == Tom Lane writes: Tom> It's hardly surprising that datumCopy would segfault when given a Tom> null "value" and told it is pass-by-reference. However, to get to Tom> the datumCopy call, we must have passed the MemoryContextContains Tom> check on that very same pointer value, and

Re: Resetting spilled txn statistics in pg_stat_replication

2020-07-15 Thread Amit Kapila
On Fri, Jul 10, 2020 at 2:42 PM Amit Kapila wrote: > > On Fri, Jul 10, 2020 at 7:23 AM Masahiko Sawada > wrote: > > > > On Thu, 9 Jul 2020 at 12:11, Amit Kapila wrote: > > > > > > On Wed, Jul 8, 2020 at 1:14 PM Masahiko Sawada > > > wrote: > > > > > > > > > > > > I think that using oids has ano

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-07-15 Thread Dilip Kumar
On Wed, Jul 15, 2020 at 6:59 PM Amit Kapila wrote: > > On Wed, Jul 15, 2020 at 9:29 AM Dilip Kumar wrote: > > > > > > I have reviewed your changes and those look good to me, please find > > the latest version of the patch set. > > > > I have done an additional round of review and below are the c