Re: scram-sha-256 broken with FIPS and OpenSSL 1.0.2

2020-10-14 Thread Michael Paquier
On Wed, Oct 14, 2020 at 05:18:51PM +0900, Michael Paquier wrote: > Sure, thanks. I wanted to keep things isolated in sha2_openssl.c as > that's something specific to the implementation. Thinking more about > it, your suggestion makes a lot of sense in the long-term by including > MD5 and HMAC in

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-10-14 Thread tsunakawa.ta...@fujitsu.com
From: Jamison, Kirk/ジャミソン カーク > 2. Non-recovery Performance > However, I still can't seem to find the cause of why the non-recovery > performance does not change when compared to master. (1 min 15 s for the > given test case below) ... > 5. Measure VACUUM timing > \timing > VACUUM; Oops, why are

Re: Add Information during standby recovery conflicts

2020-10-14 Thread Kyotaro Horiguchi
At Thu, 15 Oct 2020 14:28:57 +0900, Masahiko Sawada wrote in > > ereport(..(errmsg("%s", _("hogehoge" results in > > fprintf((translated("%s")), translate("hogehoge")). > > > > So your change (errmsg("%s", gettext_noop("hogehoge")) results in > > > > fprintf((translated("%s")), DONT_translat

Re: kevent latch paths don't handle postmaster death well

2020-10-14 Thread Thomas Munro
On Thu, Oct 15, 2020 at 12:55 PM Thomas Munro wrote: > On Thu, Oct 15, 2020 at 12:14 PM Tom Lane wrote: > > I did not try to test it, but there's code that purports to handle that > > in latch.c, ~ line 1150, and the behavior it's expecting mostly agrees > > with what I read in the macOS kevent m

Re: Resetting spilled txn statistics in pg_stat_replication

2020-10-14 Thread Masahiko Sawada
On Wed, 14 Oct 2020 at 12:03, Shinoda, Noriyoshi (PN Japan A&PS Delivery) wrote: > > Thanks for your comment. > > > 8.3. Character Types: > > The name type exists only for the storage of identifiers in the internal > > system catalogs > > I didn't know the policy about data types. Thank you. > Bu

Re: Online checksums verification in the backend

2020-10-14 Thread Julien Rouhaud
On Thu, Oct 1, 2020 at 1:07 PM Michael Paquier wrote: > > On Fri, Sep 25, 2020 at 06:11:47PM +0800, Julien Rouhaud wrote: > > Thanks a lot for the tests! I'm not surprised that forcing the lock > > will slow down the pg_check_relation() execution, but I'm a bit > > surprised that holding the buff

Re: speed up unicode decomposition and recomposition

2020-10-14 Thread Kyotaro Horiguchi
At Wed, 14 Oct 2020 23:06:28 -0400, Tom Lane wrote in > John Naylor writes: > > With those points in mind and thinking more broadly, I'd like to try harder > > on recomposition. Even several times faster, recomposition is still orders > > of magnitude slower than ICU, as measured by Daniel Verit

Re: Add Information during standby recovery conflicts

2020-10-14 Thread Masahiko Sawada
On Thu, 15 Oct 2020 at 12:13, Kyotaro Horiguchi wrote: > > At Wed, 14 Oct 2020 17:39:20 +0900, Masahiko Sawada > wrote in > > On Wed, 14 Oct 2020 at 07:44, Alvaro Herrera > > wrote: > > > > > > On 2020-Oct-14, Masahiko Sawada wrote: > > > > > > > I've attached the patch as an idea of fixing th

Re: Parallel INSERT (INTO ... SELECT ...)

2020-10-14 Thread Greg Nancarrow
On Fri, Oct 9, 2020 at 8:09 PM Amit Kapila wrote: > > It might be a good idea to first just get this patch committed, if > possible. So, I have reviewed the latest version of this patch: > > 0001-InsertParallelSelect I've attached an updated InsertParallelSelect patch (v2) - allowing underlying p

Re: Parallel Inserts in CREATE TABLE AS

2020-10-14 Thread Bharath Rupireddy
On Wed, Oct 14, 2020 at 6:16 PM Amit Kapila wrote: > > > If somebody expects to preserve the order of the tuples that are > > coming from GatherMerge node of the select part in CTAS or SELECT INTO > > while inserting, now if parallelism is allowed, that may not be the > > case i.e. the order of in

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2020-10-14 Thread Julien Rouhaud
On Wed, Oct 14, 2020 at 10:40 PM Bruce Momjian wrote: > > On Wed, Oct 14, 2020 at 10:34:31PM +0800, Julien Rouhaud wrote: > > On Wed, Oct 14, 2020 at 10:31 PM Tom Lane wrote: > > > > > > Bruce Momjian writes: > > > > Is there a measureable overhead when this is turned on, since it is off > > > >

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2020-10-14 Thread Andy Fan
Hi David/Alvaro: On Thu, Oct 15, 2020 at 9:09 AM David Rowley wrote: > On Thu, 15 Oct 2020 at 14:04, Andy Fan wrote: > > > > I think if it is possible to implement the detech with a NoWait option . > > > > ALTER TABLE ... DETACH PARTITION .. [NoWait]. > > > > if it can't get the lock, raise "R

Re: VACUUM PARALLEL option vs. max_parallel_maintenance_workers

2020-10-14 Thread Amit Kapila
On Mon, Oct 5, 2020 at 8:11 AM Amit Kapila wrote: > > On Sat, Oct 3, 2020 at 6:55 PM Masahiko Sawada > wrote: > > > > To make the behavior of parallel vacuum more consistent with other > > parallel maintenance commands (i.g., only parallel INDEX CREATE for > > now), as a second idea, can we make

Re: [HACKERS] logical decoding of two-phase transactions

2020-10-14 Thread Amit Kapila
On Wed, Oct 14, 2020 at 6:15 PM Ajin Cherian wrote: > I think it will be easier to review this work if we can split the patches according to the changes made in different layers. The first patch could be changes made in output plugin and the corresponding changes in test_decoding, see the similar

Re: Add Information during standby recovery conflicts

2020-10-14 Thread Kyotaro Horiguchi
At Wed, 14 Oct 2020 17:39:20 +0900, Masahiko Sawada wrote in > On Wed, 14 Oct 2020 at 07:44, Alvaro Herrera wrote: > > > > On 2020-Oct-14, Masahiko Sawada wrote: > > > > > I've attached the patch as an idea of fixing the above comments as > > > well as the comment from Alvaro. I can be applied

Re: speed up unicode decomposition and recomposition

2020-10-14 Thread Tom Lane
John Naylor writes: > With those points in mind and thinking more broadly, I'd like to try harder > on recomposition. Even several times faster, recomposition is still orders > of magnitude slower than ICU, as measured by Daniel Verite [1]. Huh. Has anyone looked into how they do it?

Re: speed up unicode decomposition and recomposition

2020-10-14 Thread John Naylor
On Wed, Oct 14, 2020 at 8:25 PM Michael Paquier wrote: > On Wed, Oct 14, 2020 at 01:06:40PM -0400, Tom Lane wrote: > > IIUC, the only place libpq uses this is to process a password-sized > string > > or two during connection establishment. It seems quite silly to add > > 26kB in order to make th

Re: Wrong statistics for size of XLOG_SWITCH during pg_waldump.

2020-10-14 Thread Kyotaro Horiguchi
At Wed, 14 Oct 2020 13:46:13 -0700, Andres Freund wrote in > Hi, > > On 2020-10-14 15:52:43 +0900, Michael Paquier wrote: > > Yeah. In its current shape, it means that only pg_waldump would be > > able to know this information. If you make this information part of > > xlogdesc.c, any consumer

Re: Minor documentation error regarding streaming replication protocol

2020-10-14 Thread Bruce Momjian
On Wed, Oct 14, 2020 at 05:10:30PM -0400, Tom Lane wrote: > Bruce Momjian writes: > > Patch attached. I would like to backpatch this to all supported > > versions so we are consistent and people don't think different PG > > versions use different return values for this. Is that safe? Looking >

Re: Add a description to the documentation that toast_tuple_target affects "Main"

2020-10-14 Thread Fujii Masao
On 2020/10/14 16:21, Shinya Okano wrote: On 2020-10-14 01:30, Fujii Masao wrote: On 2020/10/13 10:40, Kasahara Tatsuhito wrote: On Fri, Oct 9, 2020 at 5:44 PM Shinya Okano wrote: Regarding the toast_tuple_target parameter of CREATE TABLE, the documentation says that it only affects Externa

Re: jit and explain nontext

2020-10-14 Thread David Rowley
On Thu, 15 Oct 2020 at 14:43, Justin Pryzby wrote: > > On Thu, Oct 15, 2020 at 02:23:01PM +1300, David Rowley wrote: > > On Thu, 15 Oct 2020 at 14:15, Tom Lane wrote: > > > Hmm, I dunno if my opinion counts as "wisdom", but what I was arguing for > > > there was that we should print stuff if it's

Re: jit and explain nontext

2020-10-14 Thread Justin Pryzby
On Thu, Oct 15, 2020 at 02:23:01PM +1300, David Rowley wrote: > On Thu, 15 Oct 2020 at 14:15, Tom Lane wrote: > > David Rowley writes: > > > Just for some reference. Some wisdom was shared in [1], which made a > > > lot of sense to me. > > > If we apply that, then we just need to decide if displa

Re: jit and explain nontext

2020-10-14 Thread David Rowley
On Thu, 15 Oct 2020 at 14:15, Tom Lane wrote: > > David Rowley writes: > > Just for some reference. Some wisdom was shared in [1], which made a > > lot of sense to me. > > If we apply that, then we just need to decide if displaying any jit > > related fields without any jitted expressions is rele

Re: jit and explain nontext

2020-10-14 Thread Tom Lane
David Rowley writes: > Just for some reference. Some wisdom was shared in [1], which made a > lot of sense to me. > If we apply that, then we just need to decide if displaying any jit > related fields without any jitted expressions is relevant. Hmm, I dunno if my opinion counts as "wisdom", but w

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2020-10-14 Thread David Rowley
On Thu, 15 Oct 2020 at 14:04, Andy Fan wrote: > > I think if it is possible to implement the detech with a NoWait option . > > ALTER TABLE ... DETACH PARTITION .. [NoWait]. > > if it can't get the lock, raise "Resource is Busy" immediately, without > blocking others. > this should be a default b

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2020-10-14 Thread Alvaro Herrera
On 2020-Oct-15, Andy Fan wrote: > I think if it is possible to implement the detech with a NoWait option . > > ALTER TABLE ... DETACH PARTITION .. [NoWait]. > > if it can't get the lock, raise "Resource is Busy" immediately, > without blocking others. this should be a default behavior. If >

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2020-10-14 Thread Andy Fan
Hi Alvaro: On Tue, Aug 4, 2020 at 7:49 AM Alvaro Herrera wrote: > I've been working on the ability to detach a partition from a > partitioned table, without causing blockages to concurrent activity. > I think this operation is critical for some use cases. > I think if it is possible to implemen

Re: jit and explain nontext

2020-10-14 Thread David Rowley
On Thu, 15 Oct 2020 at 08:39, Justin Pryzby wrote: > > /* don't print information if no JITing happened */ > if (!ji || ji->created_functions == 0) > return; > > This applies even when (es->format != EXPLAIN_FORMAT_TEXT), which I think is > wrong. Jit use can be de

Re: gs_group_1 crashing on 13beta2/s390x

2020-10-14 Thread Andres Freund
Hi, On 2020-10-14 14:58:35 -0700, Andres Freund wrote: > I suspect that building with LDFLAGS="-Wl,-z,relro -Wl,-z,now" - which > is what I think the debian package does - creates the types of > relocations that LLVM doesn't handle for elf + s390. > > 10 release branch: > > void RuntimeDyldELF::

Re: pgsql: Restore replication protocol's duplicate command tags

2020-10-14 Thread Alvaro Herrera
On 2020-Oct-14, Alvaro Herrera wrote: > Add a test case that shows the failure. It might still succeed even > without the patch when run on a fast enough server, but it suffices to > show the bug in enough cases that it would be noticed in buildfarm. Hm, this failed on sidewinder. I think the "

Re: speed up unicode decomposition and recomposition

2020-10-14 Thread Michael Paquier
On Wed, Oct 14, 2020 at 01:06:40PM -0400, Tom Lane wrote: > John Naylor writes: >> Some other considerations: >> - As I alluded above, this adds ~26kB to libpq because of SASLPrep. Since >> the decomp array was reordered to optimize linear search, it can no longer >> be used for binary search. It

Re: kevent latch paths don't handle postmaster death well

2020-10-14 Thread Thomas Munro
On Thu, Oct 15, 2020 at 12:14 PM Tom Lane wrote: > Andres Freund writes: > > On 2020-10-15 11:10:28 +1300, Thomas Munro wrote: > >> I don't think that's a problem -- the kernel will report the event to > >> each interested kqueue object. The attached fixes the problem for me. > > > Will it do so

Re: kevent latch paths don't handle postmaster death well

2020-10-14 Thread Tom Lane
Andres Freund writes: > On 2020-10-15 11:10:28 +1300, Thomas Munro wrote: >> I don't think that's a problem -- the kernel will report the event to >> each interested kqueue object. The attached fixes the problem for me. > Will it do so even if the kqueue is created after postmaster death? I did

Re: kevent latch paths don't handle postmaster death well

2020-10-14 Thread Andres Freund
Hi, On 2020-10-15 11:10:28 +1300, Thomas Munro wrote: > I don't think that's a problem -- the kernel will report the event to > each interested kqueue object. The attached fixes the problem for me. Will it do so even if the kqueue is created after postmaster death? - Andres

Re: kevent latch paths don't handle postmaster death well

2020-10-14 Thread Tom Lane
Thomas Munro writes: > (Hmm, I wonder about that Windows process exit event.) If anyone wants to test that, I can save you a little time building infrastructure, perhaps. I used the attached program built into a .so. After creating the function, invoke it, and once it's blocked kill -9 the postm

Re: kevent latch paths don't handle postmaster death well

2020-10-14 Thread Thomas Munro
On Thu, Oct 15, 2020 at 11:18 AM Tom Lane wrote: > Thomas Munro writes: > > On Thu, Oct 15, 2020 at 8:40 AM Tom Lane wrote: > >> Seems like having that be per-WaitEventSet state is also not a great > >> idea --- if we detect PM death while waiting on one WES, and then > >> wait on another one, i

Re: kevent latch paths don't handle postmaster death well

2020-10-14 Thread Tom Lane
Thomas Munro writes: > On Thu, Oct 15, 2020 at 8:40 AM Tom Lane wrote: >> Seems like having that be per-WaitEventSet state is also not a great >> idea --- if we detect PM death while waiting on one WES, and then >> wait on another one, it won't work. A plain process-wide static >> variable would

Re: kevent latch paths don't handle postmaster death well

2020-10-14 Thread Thomas Munro
On Thu, Oct 15, 2020 at 8:40 AM Tom Lane wrote: > Thomas Munro writes: > > The process exit event is like an 'edge', not a 'level'... hmm. It > > might be enough to set report_postmaster_not_running = true the first > > time it tells us so if we try to wait again we'll treat it like a > > level.

Re: gs_group_1 crashing on 13beta2/s390x

2020-10-14 Thread Andres Freund
Hi, Christoph helped me to get access to a s390x machine - I wasn't able to reproduce exactly the error he hit. Initially all tests passed, but after recompiling with build flags more similar to Christop's I was able to hit another instance of what I assume to be the same bug. I am fairly sure th

Failures during FATAL exit

2020-10-14 Thread Tom Lane
As already noted in another thread, buildfarm member chipmunk failed today with an unexpected Assert [1]. I've now reproduced this by manually killing the postmaster during the regression tests. The stack trace looks like #0 0x91507598 in raise () from /lib64/libc.so.6 #1 0x914

Re: Minor documentation error regarding streaming replication protocol

2020-10-14 Thread Tom Lane
Bruce Momjian writes: > Patch attached. I would like to backpatch this to all supported > versions so we are consistent and people don't think different PG > versions use different return values for this. Is that safe? Looking > at the uses of this in our code, it seems so. We aren't doing BYT

Re: Minor documentation error regarding streaming replication protocol

2020-10-14 Thread Bruce Momjian
On Thu, Oct 8, 2020 at 11:17:59PM -0400, Bruce Momjian wrote: > Good point. The reporter was assuming the data would come to the client > in the bytea output format specified by the GUC, e.g. \x..., so that > doesn't happen either. As I said before, it is more raw bytes, but we > don't have an S

Re: Wrong statistics for size of XLOG_SWITCH during pg_waldump.

2020-10-14 Thread Andres Freund
Hi, On 2020-10-14 15:52:43 +0900, Michael Paquier wrote: > Yeah. In its current shape, it means that only pg_waldump would be > able to know this information. If you make this information part of > xlogdesc.c, any consumer of the WAL record descriptions would be able > to show this information,

Re: Some remaining htonl() and ntohl() calls in the code

2020-10-14 Thread Andres Freund
Hi, On 2020-10-14 14:53:03 +0900, Michael Paquier wrote: > Since 510b8cbf, we have in-core equivalents for htonl(), ntohl() & co > through pg_bswap.h that allows to compile with a built-in function if > the compiler used has one. > > All the existing calls in the code tree have been changed with

Re: More aggressive vacuuming of temporary tables

2020-10-14 Thread Andres Freund
Hi, On 2020-09-08 18:13:58 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2020-09-08 15:24:54 -0400, Tom Lane wrote: > >> Andres Freund writes: > >>> But now I do wonder why we need to know whether the command is top level > >>> or not? Why isn't the correct thing to instead look at what

Re: kevent latch paths don't handle postmaster death well

2020-10-14 Thread Heikki Linnakangas
On 14/10/2020 21:58, Tom Lane wrote: I noticed that chipmunk failed [1] with a rather interesting log: 2020-10-14 08:57:01.661 EEST [27048:6] pg_regress/prepared_xacts LOG: statement: UPDATE pxtest1 SET foobar = 'bbb' WHERE foobar = 'aaa'; 2020-10-14 08:57:01.721 EEST [27048:7] pg_regress/prep

Re: kevent latch paths don't handle postmaster death well

2020-10-14 Thread Tom Lane
Thomas Munro writes: > The process exit event is like an 'edge', not a 'level'... hmm. It > might be enough to set report_postmaster_not_running = true the first > time it tells us so if we try to wait again we'll treat it like a > level. I will look into it later today. Seems like having that

jit and explain nontext

2020-10-14 Thread Justin Pryzby
/* don't print information if no JITing happened */ if (!ji || ji->created_functions == 0) return; This applies even when (es->format != EXPLAIN_FORMAT_TEXT), which I think is wrong. Jit use can be determined by cost, so I think jit details should be shown in non-t

Re: kevent latch paths don't handle postmaster death well

2020-10-14 Thread Thomas Munro
On Thu, Oct 15, 2020 at 7:58 AM Tom Lane wrote: > We appear to have already realized that the postmaster died, since we're > inside proc_exit. WaitForBackgroundWorkerShutdown is doing this: > > rc = WaitLatch(MyLatch, >WL_LATCH_SET | WL_POSTMASTER_DEATH, 0, >

kevent latch paths don't handle postmaster death well

2020-10-14 Thread Tom Lane
I noticed that chipmunk failed [1] with a rather interesting log: 2020-10-14 08:57:01.661 EEST [27048:6] pg_regress/prepared_xacts LOG: statement: UPDATE pxtest1 SET foobar = 'bbb' WHERE foobar = 'aaa'; 2020-10-14 08:57:01.721 EEST [27048:7] pg_regress/prepared_xacts LOG: statement: SELECT * F

Re: speed up unicode decomposition and recomposition

2020-10-14 Thread Tom Lane
John Naylor writes: > Some other considerations: > - As I alluded above, this adds ~26kB to libpq because of SASLPrep. Since > the decomp array was reordered to optimize linear search, it can no longer > be used for binary search. It's possible to build two arrays, one for > frontend and one for b

Re: Global snapshots

2020-10-14 Thread Fujii Masao
On 2020/09/17 15:56, Amit Kapila wrote: On Thu, Sep 10, 2020 at 4:20 PM Fujii Masao wrote: One alternative is to add only hooks into PostgreSQL core so that we can implement the global transaction management outside. This idea was discussed before as the title "eXtensible Transaction Manag

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2020-10-14 Thread Bruce Momjian
On Wed, Oct 14, 2020 at 10:34:31PM +0800, Julien Rouhaud wrote: > On Wed, Oct 14, 2020 at 10:31 PM Tom Lane wrote: > > > > Bruce Momjian writes: > > > Is there a measureable overhead when this is turned on, since it is off > > > by default and maybe should default to on. > > > > I don't believe t

Re: Deleting older versions in unique indexes to avoid page splits

2020-10-14 Thread Peter Geoghegan
On Wed, Oct 14, 2020 at 7:07 AM Anastasia Lubennikova wrote: > The idea seems very promising, especially when extended to handle non-unique > indexes too. Thanks! > That's exactly what I wanted to discuss after the first letter. If we could > make (non)HOT-updates index specific, I think it co

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2020-10-14 Thread Julien Rouhaud
On Wed, Oct 14, 2020 at 10:31 PM Tom Lane wrote: > > Bruce Momjian writes: > > Is there a measureable overhead when this is turned on, since it is off > > by default and maybe should default to on. > > I don't believe that "default to on" can even be in the discussion. > There is no in-core featu

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2020-10-14 Thread Tom Lane
Bruce Momjian writes: > Is there a measureable overhead when this is turned on, since it is off > by default and maybe should default to on. I don't believe that "default to on" can even be in the discussion. There is no in-core feature that would use this by default. reg

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2020-10-14 Thread Bruce Momjian
On Wed, Oct 14, 2020 at 10:21:24PM +0800, Julien Rouhaud wrote: > On Wed, Oct 14, 2020 at 10:09 PM Bruce Momjian wrote: > > > > OK, I came up with the hash idea only to address one of your concerns > > > > about mismatched hashes for algorithm improvements/changes. Seems we > > > > might as well

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2020-10-14 Thread Julien Rouhaud
On Wed, Oct 14, 2020 at 10:09 PM Bruce Momjian wrote: > > On Wed, Oct 14, 2020 at 05:43:33PM +0800, Julien Rouhaud wrote: > > On Tue, Oct 13, 2020 at 4:53 AM Bruce Momjian wrote: > > > > > > On Mon, Oct 12, 2020 at 04:07:30PM -0400, Tom Lane wrote: > > > > Bruce Momjian writes: > > > > > On Mon,

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2020-10-14 Thread Bruce Momjian
On Wed, Oct 14, 2020 at 05:43:33PM +0800, Julien Rouhaud wrote: > On Tue, Oct 13, 2020 at 4:53 AM Bruce Momjian wrote: > > > > On Mon, Oct 12, 2020 at 04:07:30PM -0400, Tom Lane wrote: > > > Bruce Momjian writes: > > > > On Mon, Oct 12, 2020 at 02:26:15PM -0400, Tom Lane wrote: > > > >> Yeah, I a

Re: Deleting older versions in unique indexes to avoid page splits

2020-10-14 Thread Anastasia Lubennikova
On 08.10.2020 02:48, Peter Geoghegan wrote: On Tue, Jun 30, 2020 at 5:03 PM Peter Geoghegan wrote: Attached is a POC patch that teaches nbtree to delete old duplicate versions from unique indexes. The optimization targets non-HOT duplicate version bloat. Although the patch is rather rough, it n

RE: [PoC] Non-volatile WAL buffer

2020-10-14 Thread Takashi Menjo
Hi Gang, Thanks. I have tried to reproduce performance degrade, using your configuration, query, and steps. And today, I got some results that Original (PMEM) achieved better performance than Non-volatile WAL buffer on my Ubuntu environment. Now I work for further investigation. Best regards,

Re: [Patch] Using Windows groups for SSPI authentication

2020-10-14 Thread Russell Foster
Right after I sent that I realized that sspi-group was a bad idea, not sure if that's even a thing. Tried to cancel as it was still in moderation, but it made it through anyways! You are right, it is very windows specific. I can make it windows-group as you said, and resubmit. On Tue, Oct 13, 2020

Re: Parallel copy

2020-10-14 Thread vignesh C
On Thu, Oct 8, 2020 at 8:43 AM Greg Nancarrow wrote: > > On Thu, Oct 8, 2020 at 5:44 AM vignesh C wrote: > > > Attached v6 patch with the fixes. > > > > Hi Vignesh, > > I noticed a couple of issues when scanning the code in the following patch: > > v6-0003-Allow-copy-from-command-to-process-d

Re: Parallel copy

2020-10-14 Thread vignesh C
On Sat, Oct 3, 2020 at 6:20 AM Tomas Vondra wrote: > > Hello Vignesh, > > I've done some basic benchmarking on the v4 version of the patches (but > AFAIKC the v5 should perform about the same), and some initial review. > > For the benchmarking, I used the lineitem table from TPC-H - for 75GB > dat

Re: Parallel copy

2020-10-14 Thread vignesh C
On Fri, Oct 9, 2020 at 11:01 AM Amit Kapila wrote: > > On Thu, Oct 8, 2020 at 12:14 AM vignesh C wrote: > > > > On Mon, Sep 28, 2020 at 12:19 PM Amit Kapila > > wrote: > > > > > + */ > > > > > +typedef struct ParallelCopyLineBoundary > > > > > > > > > > Are we doing all this state management to

WIP psql \df choose functions by their arguments

2020-10-14 Thread Greg Sabino Mullane
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Improve psql \df to choose functions by their arguments == OVERVIEW Having to scroll through same-named functions with different argument types when you know exactly which one you want is annoying at best, error causing at worst. This patch e

Re: Parallel copy

2020-10-14 Thread vignesh C
On Fri, Oct 9, 2020 at 10:42 AM Amit Kapila wrote: > > On Thu, Oct 8, 2020 at 12:14 AM vignesh C wrote: > > > > On Mon, Sep 28, 2020 at 12:19 PM Amit Kapila wrote: > > > > > > > > > I am convinced by the reason given by Kyotaro-San in that another > > > thread [1] and performance data shown by P

Re: Parallel Inserts in CREATE TABLE AS

2020-10-14 Thread Amit Kapila
On Wed, Oct 14, 2020 at 2:46 PM Bharath Rupireddy wrote: > > On Tue, Oct 6, 2020 at 10:58 AM Amit Kapila wrote: > > > > > > While skimming through the patch, a small thing I noticed: > > + /* > > + * SELECT part of the CTAS is parallelizable, so we can make > > + * each parallel worker insert the

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2020-10-14 Thread Julien Rouhaud
On Wed, Oct 14, 2020 at 5:43 PM Julien Rouhaud wrote: > > On Tue, Oct 13, 2020 at 4:53 AM Bruce Momjian wrote: > > > > On Mon, Oct 12, 2020 at 04:07:30PM -0400, Tom Lane wrote: > > > Bruce Momjian writes: > > > > On Mon, Oct 12, 2020 at 02:26:15PM -0400, Tom Lane wrote: > > > >> Yeah, I agree --

RelationGetNumberOfBlocks is called every time of heap_rescan.

2020-10-14 Thread Andy Fan
As for the initscan, It looks to me that the codes and comments don't match (obviously I'm wrong, this is why I'm asking). /* * Determine the number of blocks we have to scan. * * It is sufficient to do this once at scan start, since any tuples added * while the scan is i

Re: Parallel copy

2020-10-14 Thread Bharath Rupireddy
I did performance testing on v7 patch set[1] with custom postgresql.conf[2]. The results are of the triplet form (exec time in sec, number of workers, gain) Use case 1: 10million rows, 5.2GB data, 2 indexes on integer columns, 1 index on text column, binary file (1104.898, 0, 1X), (1112.221, 1, 1X

Re: Logical replication CPU-bound with TRUNCATE/DROP/CREATE many tables

2020-10-14 Thread Dilip Kumar
On Wed, Oct 14, 2020 at 4:12 PM Amit Kapila wrote: > > On Mon, Oct 12, 2020 at 3:23 PM Dilip Kumar wrote: > > > > On Fri, Oct 9, 2020 at 2:34 PM Amit Kapila wrote: > > > > > > > > > Okay, I think this makes sense. I think we should see the performance > > > benefit for this case as well but mayb

Re: Logical replication CPU-bound with TRUNCATE/DROP/CREATE many tables

2020-10-14 Thread Amit Kapila
On Mon, Oct 12, 2020 at 3:23 PM Dilip Kumar wrote: > > On Fri, Oct 9, 2020 at 2:34 PM Amit Kapila wrote: > > > > > > Okay, I think this makes sense. I think we should see the performance > > benefit for this case as well but maybe to a bit lesser degree because > > we will exclude some of the sub

Re: Transactions involving multiple postgres foreign servers, take 2

2020-10-14 Thread Masahiko Sawada
On Wed, 14 Oct 2020 at 17:11, Kyotaro Horiguchi wrote: > > (v26 fails on the current master) Thanks, I'll update the patch. > > At Wed, 14 Oct 2020 13:52:49 +0900, Masahiko Sawada > wrote in > > On Wed, 14 Oct 2020 at 13:19, Kyotaro Horiguchi > > wrote: > > > > > > At Wed, 14 Oct 2020 12:09:

Re: partition routing layering in nodeModifyTable.c

2020-10-14 Thread Amit Langote
On Wed, Oct 14, 2020 at 6:04 PM Heikki Linnakangas wrote: > On 14/10/2020 09:44, Amit Langote wrote: > > I like the idea of storing the ResultRelInfo in ForeignScanState, but > > it would be better if we can document the fact that an FDW may not > > reliably access until IterateDirectModify(). Tha

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2020-10-14 Thread Julien Rouhaud
On Tue, Oct 13, 2020 at 4:53 AM Bruce Momjian wrote: > > On Mon, Oct 12, 2020 at 04:07:30PM -0400, Tom Lane wrote: > > Bruce Momjian writes: > > > On Mon, Oct 12, 2020 at 02:26:15PM -0400, Tom Lane wrote: > > >> Yeah, I agree --- a version number is the wrong way to think about this. > > > > > Th

Re: Add session statistics to pg_stat_database

2020-10-14 Thread Laurenz Albe
Thanks for the --- as always --- valuable review! On Tue, 2020-10-13 at 17:55 -0500, Justin Pryzby wrote: > On Tue, Oct 13, 2020 at 01:44:41PM +0200, Laurenz Albe wrote: > > Attached is v3 with improvements. > > + > + Time spent in database sessions in this database, in milliseconds.

Re: Parallel Inserts in CREATE TABLE AS

2020-10-14 Thread Bharath Rupireddy
On Tue, Oct 6, 2020 at 10:58 AM Amit Kapila wrote: > > > Yes we do a bunch of catalog changes related to the created new table. > > We will have both the txn id and command id assigned when catalogue > > changes are being made. But, right after the table is created in the > > leader, the command i

Re: partition routing layering in nodeModifyTable.c

2020-10-14 Thread Heikki Linnakangas
On 14/10/2020 09:44, Amit Langote wrote: I like the idea of storing the ResultRelInfo in ForeignScanState, but it would be better if we can document the fact that an FDW may not reliably access until IterateDirectModify(). That's because, setting it in ExecInitForeignScan() will mean *all* result

Re: Loose ends after CVE-2020-14350 (extension installation hazards)

2020-10-14 Thread Noah Misch
On Fri, Aug 14, 2020 at 02:50:32PM -0400, Tom Lane wrote: > However, in itself this can only fix references that are resolved during > execution of the extension script. I don't see a good way to use the > idea to make earthdistance's SQL functions fully secure. It won't do > to write, say, > >

Re: Add Information during standby recovery conflicts

2020-10-14 Thread Masahiko Sawada
On Wed, 14 Oct 2020 at 07:44, Alvaro Herrera wrote: > > On 2020-Oct-14, Masahiko Sawada wrote: > > > I've attached the patch as an idea of fixing the above comments as > > well as the comment from Alvaro. I can be applied on top of v4 patch. > > One note about the translation stuff. Currently you

Re: scram-sha-256 broken with FIPS and OpenSSL 1.0.2

2020-10-14 Thread Michael Paquier
On Wed, Oct 14, 2020 at 10:40:12AM +0300, Heikki Linnakangas wrote: > Since this is going to be core backend code (and also frontend), we don't > need to use the generic reource owner callback mechanism, we could add a > built-in ResourceOwnerData field and functions in resowner.c. The callback > m

Re: Two fsync related performance issues?

2020-10-14 Thread Michael Banck
Hi, Am Mittwoch, den 14.10.2020, 14:06 +0900 schrieb Michael Paquier: > On Wed, Oct 14, 2020 at 02:48:18PM +1300, Thomas Munro wrote: > > On Wed, Oct 14, 2020 at 12:53 AM Michael Banck > > wrote: > > > One question about this: Did you consider the case of a basebackup being > > > copied/restored

Re: Transactions involving multiple postgres foreign servers, take 2

2020-10-14 Thread Kyotaro Horiguchi
(v26 fails on the current master) At Wed, 14 Oct 2020 13:52:49 +0900, Masahiko Sawada wrote in > On Wed, 14 Oct 2020 at 13:19, Kyotaro Horiguchi > wrote: > > > > At Wed, 14 Oct 2020 12:09:34 +0900, Masahiko Sawada > > wrote in > > > On Wed, 14 Oct 2020 at 10:16, Kyotaro Horiguchi > > > wr

Re: Use appendStringInfoString and appendPQExpBufferStr where possible

2020-10-14 Thread David Rowley
On Mon, 28 Sep 2020 at 13:47, Hou, Zhijie wrote: > Thanks for your response, combined them as a single patchset now. I had a look over the three patches. All the changes look fine. Thanks for working on it. After a bit of grep work, I found about 6 times more calls that could be improved. Thes

Re: scram-sha-256 broken with FIPS and OpenSSL 1.0.2

2020-10-14 Thread Heikki Linnakangas
On 14/10/2020 06:29, Michael Paquier wrote: With 0001 in place, switching the SHA2 implementation of OpenSSL to use EVP is straight-forward, as the only thing that's actually needed here is to put in place a callback to clean up the EVP contexts allocated by OpenSSL. This is rather similar to wh

Re: allow partial union-all and improve parallel subquery costing

2020-10-14 Thread Luc Vlaming
Hi, It seems I ran the wrong make checks to verify everything is correct (make check instead of make installcheck-world) and this uncovered another regress test change. I also noticed the statistics are sometimes giving different row count results so I increased the row statistics target to mak

Re: Remove some unnecessary if-condition

2020-10-14 Thread Michael Paquier
On Mon, Oct 12, 2020 at 11:42:31AM +, Hou, Zhijie wrote: > Thank you for reviewing! added it to commitfest > https://commitfest.postgresql.org/30/2760/ - if (!bootstrap) - { - pgstat_bestart(); - CommitTransactionCommand(); - } + pgstat_bestart(); +

Re: Add a description to the documentation that toast_tuple_target affects "Main"

2020-10-14 Thread Shinya Okano
On 2020-10-14 01:30, Fujii Masao wrote: On 2020/10/13 10:40, Kasahara Tatsuhito wrote: On Fri, Oct 9, 2020 at 5:44 PM Shinya Okano wrote: Regarding the toast_tuple_target parameter of CREATE TABLE, the documentation says that it only affects External or Extended, but it actually affects the co

Re: Use list_delete_xxxcell O(1) instead of list_delete_ptr O(N) in some places

2020-10-14 Thread Luc Vlaming
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: not tested Spec compliant: not tested Documentation:not tested Patch applies cleanly on master & 13 and installcheck-world runs on 13 &