Re: Parallel copy

2020-09-24 Thread Bharath Rupireddy
Thanks Greg for the testing. On Thu, Sep 24, 2020 at 8:27 AM Greg Nancarrow wrote: > > > 3. Could you please run the test case 3 times at least? Just to ensure the consistency of the issue. > > Yes, have run 4 times. Seems to be a performance hit (whether normal > copy or parallel-1 copy) on the

Re: Range checks of pg_test_fsync --secs-per-test and pg_test_timing --duration

2020-09-24 Thread Michael Paquier
On Wed, Sep 23, 2020 at 08:11:59AM +0200, Peter Eisentraut wrote: > This patch mixes up unsigned int and uint32 in random ways. The variable is > uint32, but the format is %u and the max constant is UINT_MAX. > > I think just use unsigned int as the variable type. There is no need to use > the b

Re: Online checksums patch - once again

2020-09-24 Thread Daniel Gustafsson
> On 24 Sep 2020, at 06:27, Michael Paquier wrote: > > On Wed, Sep 23, 2020 at 02:34:36PM +0200, Daniel Gustafsson wrote: >> While in the patch I realized that the relationlist saved the relkind but the >> code wasn't actually using it, so I've gone ahead and removed that with a lot >> fewer pall

Re: [PATCH] Add features to pg_stat_statements

2020-09-24 Thread Katsuragi Yuta
On 2020-09-24 14:15, legrand legrand wrote: Hi, Both are probably not needed. I would prefer it accessible in a view live Event | date | victims Eviction... Reset... Part reset ... As there are other needs to track reset times. Let me confirm one thing. Is the number of records of this view f

Re: A little enhancement for hashdisk testset

2020-09-24 Thread Daniel Gustafsson
> On 24 Sep 2020, at 07:45, Hou, Zhijie wrote: > I found some tables is not dropped at the end of the sqlscript, > It does not hava any problem, but I think it's better to drop the table in > time. There is value in keeping a representative set of objects around after regression tests, as the d

Re: 回复:how to create index concurrently on partitioned table

2020-09-24 Thread Michael Paquier
On Mon, Sep 14, 2020 at 09:31:03AM -0500, Justin Pryzby wrote: > Anyway, for now this is rebased on 83158f74d. I have not thought yet about all the details of CIC and DIC and what you said upthread, but I have gone through CLUSTER for now, as a start. So, the goal of the patch, as I would define

RE: Transactions involving multiple postgres foreign servers, take 2

2020-09-24 Thread tsunakawa.ta...@fujitsu.com
From: Masahiko Sawada > So with your idea, I think we require FDW developers to not call > ereport(ERROR) as much as possible. If they need to use a function > including palloc, lappend etc that could call ereport(ERROR), they > need to use PG_TRY() and PG_CATCH() and return the control along with

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-09-24 Thread k.jami...@fujitsu.com
On Thursday, September 24, 2020 1:27 PM, Tsunakawa-san wrote: > (1) > + for (cur_blk = firstDelBlock[j]; cur_blk < > nblocks; cur_blk++) > > The right side of "cur_blk <" should not be nblocks, because nblocks is not > the number of the relation fork anymore. Right. F

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-09-24 Thread Kyotaro Horiguchi
Hello. At Wed, 23 Sep 2020 05:37:24 +, "tsunakawa.ta...@fujitsu.com" wrote in > From: Jamison, Kirk/ジャミソン カーク # Wow. I'm surprised to read it.. > > I revised the patch based from my understanding of Horiguchi-san's comment, > > but I could be wrong. > > Quoting: > > > > " > > +

RE: I'd like to discuss scaleout at PGCon

2020-09-24 Thread tsunakawa.ta...@fujitsu.com
From: MauMau > I intentionally have put little conclusion on our specification and > design. I'd like you to look at recent distributed databases, and > then think about and discuss what we want to aim for together. I feel > it's better to separate a thread per topic or group of topics. Finally

Re: Parallel copy

2020-09-24 Thread Bharath Rupireddy
> > > Have you tested your patch when encoding conversion is needed? If so, > > could you please point out the email that has the test results. > > > > We have not yet done encoding testing, we will do and post the results > separately in the coming days. > Hi Ashutosh, I ran the tests ensuring p

Re: [PATCH] Add features to pg_stat_statements

2020-09-24 Thread legrand legrand
Not limited to 3, Like an history table. Will have to think if data is saved at shutdown. -- Sent from: https://www.postgresql-archive.org/PostgreSQL-hackers-f1928748.html

Re: problem with RETURNING and update row movement

2020-09-24 Thread Etsuro Fujita
On Thu, Sep 24, 2020 at 2:47 PM Amit Langote wrote: > On Thu, Sep 24, 2020 at 4:25 AM Etsuro Fujita wrote: > BTW, the discussion so far on the other thread is oblivious to the > issue being discussed here, where we need to find a way to transfer > system attributes between a pair of partitions th

Re: Logical replication from PG v13 and below to PG v14 (devel version) is not working.

2020-09-24 Thread Dilip Kumar
On Thu, Sep 24, 2020 at 11:55 AM Amit Kapila wrote: > > On Tue, Sep 22, 2020 at 5:15 PM Dilip Kumar wrote: > > > > On Tue, Sep 22, 2020 at 12:02 PM Amit Kapila > > wrote: > > > > > > > > > I am not sure if this suggestion makes it better than what is purposed > > > by Dilip but I think we can d

Re: Logical replication from PG v13 and below to PG v14 (devel version) is not working.

2020-09-24 Thread Amit Kapila
On Thu, Sep 24, 2020 at 4:31 PM Dilip Kumar wrote: > > On Thu, Sep 24, 2020 at 11:55 AM Amit Kapila wrote: > > > > On Tue, Sep 22, 2020 at 5:15 PM Dilip Kumar wrote: > > > > > > On Tue, Sep 22, 2020 at 12:02 PM Amit Kapila > > > wrote: > > > > > > > > > > > > I am not sure if this suggestion m

Re: Logical replication from PG v13 and below to PG v14 (devel version) is not working.

2020-09-24 Thread Dilip Kumar
On Thu, Sep 24, 2020 at 4:45 PM Amit Kapila wrote: > > On Thu, Sep 24, 2020 at 4:31 PM Dilip Kumar wrote: > > > > On Thu, Sep 24, 2020 at 11:55 AM Amit Kapila > > wrote: > > > > > > On Tue, Sep 22, 2020 at 5:15 PM Dilip Kumar wrote: > > > > > > > > On Tue, Sep 22, 2020 at 12:02 PM Amit Kapila

Re: Report error position in partition bound check

2020-09-24 Thread Amit Langote
On Thu, Sep 24, 2020 at 7:19 AM Tom Lane wrote: > I looked this over and pushed it with some minor adjustments. Thank you. > However, while I was looking at it I couldn't help noticing that > transformPartitionBoundValue's handling of collation concerns seems > less than sane. There are two thi

Re: Logical replication from PG v13 and below to PG v14 (devel version) is not working.

2020-09-24 Thread Amit Kapila
On Thu, Sep 24, 2020 at 5:11 PM Dilip Kumar wrote: > > On Thu, Sep 24, 2020 at 4:45 PM Amit Kapila wrote: > > > > > > Have you checked what will function walrcv_server_version() will > > return? I was thinking that if we know that subscriber is connected to > > Publisher version < 14 then we can

Re: Resetting spilled txn statistics in pg_stat_replication

2020-09-24 Thread Amit Kapila
On Sat, Sep 19, 2020 at 1:48 PM Amit Kapila wrote: > > On Tue, Sep 8, 2020 at 7:02 PM Amit Kapila wrote: > > > > On Tue, Sep 8, 2020 at 7:53 AM Masahiko Sawada > > wrote: > > I have fixed these review comments in the attached patch. > > > Apart from the above, > (a) fixed one bug in ReorderBuffe

Re: Logical replication from PG v13 and below to PG v14 (devel version) is not working.

2020-09-24 Thread Ashutosh Sharma
Hi Amit, On Thu, Sep 24, 2020 at 11:55 AM Amit Kapila wrote: > > On Tue, Sep 22, 2020 at 5:15 PM Dilip Kumar wrote: > > > > On Tue, Sep 22, 2020 at 12:02 PM Amit Kapila > > wrote: > > > > > > > > > I am not sure if this suggestion makes it better than what is purposed > > > by Dilip but I thin

Re: fixing old_snapshot_threshold's time->xid mapping

2020-09-24 Thread Hamid Akhtar
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: not tested Spec compliant: not tested Documentation:not tested Patch looks good to me.

Re: PostgreSQL 13 Release Timeline

2020-09-24 Thread Jonathan S. Katz
On 9/10/20 1:08 PM, Jonathan S. Katz wrote: > Hi, > > On 9/2/20 2:13 PM, Jonathan S. Katz wrote: > >> * PostgreSQL 13 Release Candidate 1 (RC1) will be released on September >> 17, 2020. >> >> * In absence of any critical issues, PostgreSQL 13 will become generally >> available on September 24, 2

Re: Parallel copy

2020-09-24 Thread Ashutosh Sharma
On Thu, Sep 24, 2020 at 3:00 PM Bharath Rupireddy wrote: > > > > > > Have you tested your patch when encoding conversion is needed? If so, > > > could you please point out the email that has the test results. > > > > > > > We have not yet done encoding testing, we will do and post the results > >

Re: PostgreSQL 13 Release Timeline

2020-09-24 Thread Simon Riggs
> I am pleased to report that PostgreSQL 13 is now GA: > > https://www.postgresql.org/about/news/2077/ > > Thank you everyone for your hard work and congratulations on another > successful release! Well done team! -- Simon Riggshttp://www.2ndQuadrant.com/ Mission Critical Databas

Re: problem with RETURNING and update row movement

2020-09-24 Thread Amit Langote
On Thu, Sep 24, 2020 at 7:30 PM Etsuro Fujita wrote: > On Thu, Sep 24, 2020 at 2:47 PM Amit Langote wrote: > > On Thu, Sep 24, 2020 at 4:25 AM Etsuro Fujita > > wrote: > > > Yeah, but for the other issue, I started thinking that we should just > > > forbid referencing xmin/xmax/cmin/cmax in 12,

Re: PostgreSQL 13 Release Timeline

2020-09-24 Thread James Coleman
On Thu, Sep 24, 2020 at 9:30 AM Jonathan S. Katz wrote: > > I am pleased to report that PostgreSQL 13 is now GA: > > https://www.postgresql.org/about/news/2077/ I'm not sure if there was a "release announcement draft" thread (searching my folder didn't find one for GA, but it's also easily possib

Re: PostgreSQL 13 Release Timeline

2020-09-24 Thread Jonathan S. Katz
On 9/24/20 10:11 AM, James Coleman wrote: > On Thu, Sep 24, 2020 at 9:30 AM Jonathan S. Katz wrote: >> >> I am pleased to report that PostgreSQL 13 is now GA: >> >> https://www.postgresql.org/about/news/2077/ > > I'm not sure if there was a "release announcement draft" thread > (searching my fold

Re: scram-sha-256 broken with FIPS and OpenSSL 1.0.2

2020-09-24 Thread Daniel Gustafsson
> On 24 Sep 2020, at 04:53, Michael Paquier wrote: > Enabling FIPS with OpenSSL 1.0.2 causes direct calls to the SHAXXX > routines to fail: > "Low level API call to digest SHA256 forbidden in fips mode" Confirmed by running 1.0.2s-fips with fips_mode=yes in the alg_section in openssl.cnf. > Thi

Re: [patch] Fix checksum verification in base backups for zero page headers

2020-09-24 Thread Anastasia Lubennikova
On 22.09.2020 17:30, Michael Banck wrote: Hi, Am Dienstag, den 22.09.2020, 16:26 +0200 schrieb Michael Banck: Am Mittwoch, den 02.09.2020, 16:50 +0300 schrieb Anastasia Lubennikova: I've looked through the previous discussion. As far as I got it, most of the controversy was about online chec

Re: Refactor pg_rewind code and make it work against a standby

2020-09-24 Thread Heikki Linnakangas
Thanks for the review! I'll post a new version shortly, with your comments incorporated. More detailed response to a few of them below: On 18/09/2020 10:41, Kyotaro Horiguchi wrote: I don't think filemap_finalize needs to iterate over filemap twice. True, but I thought it's more clear that wa

Re: Report error position in partition bound check

2020-09-24 Thread Tom Lane
[ cc'ing Peter, since his opinion seems to have got us here in the first place ] Amit Langote writes: > On Thu, Sep 24, 2020 at 7:19 AM Tom Lane wrote: >> However, while I was looking at it I couldn't help noticing that >> transformPartitionBoundValue's handling of collation concerns seems >> le

Re: "cert" + clientcert=verify-ca in pg_hba.conf?

2020-09-24 Thread Bruce Momjian
On Thu, Sep 24, 2020 at 12:44:01PM +0900, Michael Paquier wrote: > On Tue, Sep 01, 2020 at 10:27:03PM -0400, Bruce Momjian wrote: > > OK, good. Let's wait a few days and I will then apply it for PG 14. > > It has been a few days, and nothing has happened here. I have not > looked at the patch in

Custom options for building extensions with --with--llvm

2020-09-24 Thread Konstantin Knizhnik
Hi, In my extension I want to define some custom options for compiler. I do it in the following way: ifdef USE_DISK CUSTOM_COPT += -DIMCS_DISK_SUPPORT endif So if I want to enable disk support, I am building my extension as make USE_DISK=1 clean all install and it normally works... unless Po

Re: scram-sha-256 broken with FIPS and OpenSSL 1.0.2

2020-09-24 Thread Heikki Linnakangas
On 24/09/2020 17:21, Daniel Gustafsson wrote: If we really want to support it (which would require more evidence of it being a problem IMO), using the non-OpenSSL sha256 code would be one option I guess? That would technically work, but wouldn't it make the product as whole not FIPS compliant?

Re: scram-sha-256 broken with FIPS and OpenSSL 1.0.2

2020-09-24 Thread Daniel Gustafsson
> On 24 Sep 2020, at 18:21, Heikki Linnakangas wrote: > > On 24/09/2020 17:21, Daniel Gustafsson wrote: >> If we really want to support it (which would require more evidence of it >> being >> a problem IMO), using the non-OpenSSL sha256 code would be one option I >> guess? > > That would techn

Incremental sort docs and release announcement

2020-09-24 Thread James Coleman
I'm breaking out a few questions I'd posed on another thread about the release timeline [1] into this new thread. I noticed on the PG13 release announcement that the link for incremental sort goes to the GUC docs [2] because (as Jonathan Katz confirmed in the linked thread) that page actually has

Re: Refactor pg_rewind code and make it work against a standby

2020-09-24 Thread Heikki Linnakangas
On 20/09/2020 23:44, Soumyadeep Chakraborty wrote: Before getting into the code review for the patch, I wanted to know why we don't use a Bitmapset for target_modified_pages? Bitmapset is not available in client code. Perhaps it could be moved to src/common with some changes, but doesn't seem

Re: proposal: possibility to read dumped table's name from file

2020-09-24 Thread Pavel Stehule
Hi rebase + minor change - using pg_get_line_buf instead pg_get_line_append Regards Pavel diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml index e09ed0a4c3..9d7ebaf922 100644 --- a/doc/src/sgml/ref/pg_dump.sgml +++ b/doc/src/sgml/ref/pg_dump.sgml @@ -757,6 +757,56 @@ Po

Re: scram-sha-256 broken with FIPS and OpenSSL 1.0.2

2020-09-24 Thread Peter Eisentraut
On 2020-09-24 18:21, Heikki Linnakangas wrote: That would technically work, but wouldn't it make the product as whole not FIPS compliant? I'm not a FIPS lawyer, but as I understand it the point of FIPS is that all the crypto code is encapsulated in a certified module. Having your own SHA-256 impl

Re: Custom options for building extensions with --with--llvm

2020-09-24 Thread Andres Freund
Hi, On 2020-09-24 19:15:22 +0300, Konstantin Knizhnik wrote: > In my extension I want to define some custom options for compiler. > I do it in the following way: > > ifdef USE_DISK > CUSTOM_COPT += -DIMCS_DISK_SUPPORT > endif Why aren't you adding it to PG_CPPFLAGS? That should work, and I think

Re: proposal: schema variables

2020-09-24 Thread Pavel Stehule
t does not apply >> anymore. This has not attracted much the attention of committers as >> well. >> > > I'll fix it today > fixed patch attached Regards Pavel > -- >> Michael >> > schema-variables-20200924.patch.gz Description: application/gzip

[patch] Concurrent table reindex per-index progress reporting

2020-09-24 Thread Matthias van de Meent
Hi, While working on a PG12-instance I noticed that the progress reporting of concurrent index creation for non-index relations fails to update the index/relation OIDs that it's currently working on in the pg_stat_progress_create_index view after creating the indexes. Please find attached a patch

Re: scram-sha-256 broken with FIPS and OpenSSL 1.0.2

2020-09-24 Thread Robert Haas
On Thu, Sep 24, 2020 at 1:57 PM Peter Eisentraut wrote: > Depends on what one considers to be covered by FIPS. The entire rest of > SCRAM is custom code, so running it on top of the world's greatest > SHA-256 implementation isn't going to make the end product any more > trustworthy. I mean, the

Re: Compatible defaults for LEAD/LAG

2020-09-24 Thread Pavel Stehule
út 22. 9. 2020 v 2:33 odesílatel Tom Lane napsal: > Pavel Stehule writes: > > I see few possibilities how to finish and close this issue: > > 1. use anycompatible type and add note to documentation so expression of > > optional argument can change a result type, and so this is Postgres > > speci

Re: scram-sha-256 broken with FIPS and OpenSSL 1.0.2

2020-09-24 Thread Daniel Gustafsson
> On 24 Sep 2020, at 21:22, Robert Haas wrote: > > On Thu, Sep 24, 2020 at 1:57 PM Peter Eisentraut > wrote: >> Depends on what one considers to be covered by FIPS. The entire rest of >> SCRAM is custom code, so running it on top of the world's greatest >> SHA-256 implementation isn't going to

Re: fixing old_snapshot_threshold's time->xid mapping

2020-09-24 Thread Robert Haas
On Wed, Sep 23, 2020 at 9:16 PM Thomas Munro wrote: > LGTM. Committed. Thomas, with respect to your part of this patch set, I wonder if we can make the functions that you're using to write tests safe enough that we could add them to contrib/old_snapshot and let users run them if they want. As yo

Re: Custom options for building extensions with --with--llvm

2020-09-24 Thread Konstantin Knizhnik
On 24.09.2020 21:37, Andres Freund wrote: Hi, On 2020-09-24 19:15:22 +0300, Konstantin Knizhnik wrote: In my extension I want to define some custom options for compiler. I do it in the following way: ifdef USE_DISK CUSTOM_COPT += -DIMCS_DISK_SUPPORT endif Why aren't you adding it to PG_CPP

Re: Inconsistent Japanese name order in v13 contributors list

2020-09-24 Thread Bruce Momjian
On Mon, Sep 21, 2020 at 10:31:31AM -0300, Alvaro Herrera wrote: > On 2020-Sep-18, Bruce Momjian wrote: > > > This thread from 2015 is the most comprehensive discussion I remember of > > Japanese name ordering, including a suggestion to use small caps: > > > > > > https://www.postgresql.org/m

Re: [PATCH] Automatic HASH and LIST partition creation

2020-09-24 Thread Anastasia Lubennikova
On 24.09.2020 06:27, Michael Paquier wrote: On Mon, Sep 14, 2020 at 02:38:56PM +0300, Anastasia Lubennikova wrote: Fixed. This was also caught by cfbot. This version should pass it clean. Please note that regression tests are failing, because of 6b2c4e59. -- Michael Thank you. Updated patch i

Re: WIP: BRIN multi-range indexes

2020-09-24 Thread John Naylor
On Mon, Sep 21, 2020 at 3:56 PM Tomas Vondra wrote: > > On Mon, Sep 21, 2020 at 01:42:34PM -0400, John Naylor wrote: > >While playing around with the numbers I had an epiphany: At the > >defaults, the filter already takes up ~4.3kB, over half the page. > >There is no room for another tuple, so if

Re: Add session statistics to pg_stat_database

2020-09-24 Thread Soumyadeep Chakraborty
Hello Laurenz, Thanks for submitting this! Please find my feedback below. * Are we trying to capture ONLY client initiated disconnects in m_aborted (we are not handling other disconnects by not accounting for EOF..like if psql was killed)? If yes, why? * pgstat_send_connstats(): How about renami

RE: Transactions involving multiple postgres foreign servers, take 2

2020-09-24 Thread tsunakawa.ta...@fujitsu.com
From: Ashutosh Bapat > The way I am looking at is to put the parallelism in the resolution > worker and not in the FDW. If we use multiple resolution workers, they > can fire commit/abort on multiple foreign servers at a time. From a single session's view, yes. However, the requests from multipl

Re: Fix inconsistency in jsonpath .datetime()

2020-09-24 Thread Alexander Korotkov
On Sun, Sep 20, 2020 at 2:23 AM Nikita Glukhov wrote: > The beta-tester of PG13 reported a inconsistency in our current jsonpath > datetime() method implementation. By the standard format strings in > datetime() > allows only characters "-./,':; " to be used as separators in format strings. > Bu

Re: history file on replica and double switchover

2020-09-24 Thread David Zhang
Hi, My understanding is that the "archiver" won't even start if "archive_mode = on" has been set on a "replica". Therefore, either (XLogArchiveMode == ARCHIVE_MODE_ALWAYS) or (XLogArchiveMode != ARCHIVE_MODE_OFF) will produce the same result. Please see how the "archiver" is started in src/

Re: Handing off SLRU fsyncs to the checkpointer

2020-09-24 Thread Thomas Munro
On Wed, Sep 23, 2020 at 1:56 PM Thomas Munro wrote: > As for the ShutdownXXX() functions, I haven't yet come up with any > reason for this code to exist. Emboldened by a colleague's inability > to explain to me what that code is doing for us, here is a new version > that just rips it all out. Re

Re: WIP: BRIN multi-range indexes

2020-09-24 Thread Tomas Vondra
On Thu, Sep 24, 2020 at 05:18:03PM -0400, John Naylor wrote: On Mon, Sep 21, 2020 at 3:56 PM Tomas Vondra wrote: On Mon, Sep 21, 2020 at 01:42:34PM -0400, John Naylor wrote: >While playing around with the numbers I had an epiphany: At the >defaults, the filter already takes up ~4.3kB, over

Re: Refactor pg_rewind code and make it work against a standby

2020-09-24 Thread Soumyadeep Chakraborty
On Thu, Sep 24, 2020 at 10:27 AM Heikki Linnakangas wrote: > >> /* > >> * If this is a relation file, copy the modified blocks. > >> * > >> * This is in addition to any other changes. > >> */ > >> iter = datapagemap_iterate(&entry->target_modified_pages); > >> while (datapagemap_next(iter,

Re: history file on replica and double switchover

2020-09-24 Thread Fujii Masao
On 2020/09/25 8:15, David Zhang wrote: Hi, My understanding is that the "archiver" won't even start if "archive_mode = on" has been set on a "replica". Therefore, either (XLogArchiveMode == ARCHIVE_MODE_ALWAYS) or (XLogArchiveMode != ARCHIVE_MODE_OFF) will produce the same result. Yes, th

Re: Handing off SLRU fsyncs to the checkpointer

2020-09-24 Thread Tom Lane
Thomas Munro writes: > Tom, do you have any thoughts on ShutdownCLOG() etc? Hm, if we cannot reach that without first completing a shutdown checkpoint, it does seem a little pointless. It'd likely be a good idea to add a comment to CheckPointCLOG et al explaining that we expect $what-exactly to

Re: "cert" + clientcert=verify-ca in pg_hba.conf?

2020-09-24 Thread Kyotaro Horiguchi
At Thu, 24 Sep 2020 11:43:40 -0400, Bruce Momjian wrote in > On Thu, Sep 24, 2020 at 12:44:01PM +0900, Michael Paquier wrote: > > On Tue, Sep 01, 2020 at 10:27:03PM -0400, Bruce Momjian wrote: > > > OK, good. Let's wait a few days and I will then apply it for PG 14. > > > > It has been a few da

Re: Handing off SLRU fsyncs to the checkpointer

2020-09-24 Thread Thomas Munro
On Fri, Sep 25, 2020 at 12:05 PM Tom Lane wrote: > Thomas Munro writes: > > Tom, do you have any thoughts on ShutdownCLOG() etc? > > Hm, if we cannot reach that without first completing a shutdown checkpoint, > it does seem a little pointless. Thanks for the sanity check. > It'd likely be a goo

pg_upgrade: fail early if a tablespace dir already exists for new cluster version

2020-09-24 Thread Justin Pryzby
This should be caught during --check, or earlier in the upgrade, rather than only after dumping the schema. Typically, the tablespace dir would be left behind by a previous failed upgrade attempt, causing a cascade of failured upgrades. I guess I may not be the only one with a 3 year old wrapper

Re: "cert" + clientcert=verify-ca in pg_hba.conf?

2020-09-24 Thread Michael Paquier
On Thu, Sep 24, 2020 at 11:43:40AM -0400, Bruce Momjian wrote: > I will handle it. Thanks. I have switched the patch as waiting on author due to the complaint of the CF bot for now, but if you feel that this does not require an extra round of review after the new rebase, of course please feel fre

Re: Memory allocation abstraction in pgcrypto

2020-09-24 Thread Michael Paquier
On Wed, Sep 23, 2020 at 04:57:44PM +0900, Michael Paquier wrote: > I doubt that anybody has been compiling with PX_OWN_ALLOC in the last > years, so let's remove this abstraction. And +1 for your patch. So, I have looked at that this morning again, and applied the thing. -- Michael signature.as

Re: Logical replication from PG v13 and below to PG v14 (devel version) is not working.

2020-09-24 Thread Dilip Kumar
On Thu, Sep 24, 2020 at 5:31 PM Amit Kapila wrote: > > On Thu, Sep 24, 2020 at 5:11 PM Dilip Kumar wrote: > > > > On Thu, Sep 24, 2020 at 4:45 PM Amit Kapila wrote: > > > > > > > > > Have you checked what will function walrcv_server_version() will > > > return? I was thinking that if we know tha

Re: fixing old_snapshot_threshold's time->xid mapping

2020-09-24 Thread Michael Paquier
On Thu, Sep 24, 2020 at 03:46:14PM -0400, Robert Haas wrote: > Committed. Cool, thanks. > Thomas, with respect to your part of this patch set, I wonder if we > can make the functions that you're using to write tests safe enough > that we could add them to contrib/old_snapshot and let users run th

Re: "cert" + clientcert=verify-ca in pg_hba.conf?

2020-09-24 Thread Tom Lane
Kyotaro Horiguchi writes: > Thank you Bruce, Michael. This is a rebased version. I really strongly object to all the encoded data in this patch. One cannot read it, one cannot even easily figure out how long it is until the tests break by virtue of the certificates expiring. One can, however, be

Re: Logical replication from PG v13 and below to PG v14 (devel version) is not working.

2020-09-24 Thread Amit Kapila
On Fri, Sep 25, 2020 at 7:21 AM Dilip Kumar wrote: > > On Thu, Sep 24, 2020 at 5:31 PM Amit Kapila wrote: > > > > On Thu, Sep 24, 2020 at 5:11 PM Dilip Kumar wrote: > > > > > > On Thu, Sep 24, 2020 at 4:45 PM Amit Kapila > > > wrote: > > > > > > > > > > > > Have you checked what will function

Re: Logical replication from PG v13 and below to PG v14 (devel version) is not working.

2020-09-24 Thread Amit Kapila
On Thu, Sep 24, 2020 at 6:33 PM Ashutosh Sharma wrote: > > Hi Amit, > > > Here, I think instead of using MySubscription->stream, we should use > > server/walrecv version number as we used at one place in tablesync.c. > > Should subscribers be setting the LR protocol value based on what is > the pu

Re: New statistics for tuning WAL buffer size

2020-09-24 Thread Masahiro Ikeda
On 2020-09-18 11:11, Kyotaro Horiguchi wrote: At Fri, 18 Sep 2020 09:40:11 +0900, Masahiro Ikeda wrote in Thanks. I confirmed that it causes HOT pruning or killing of dead index tuple if DecodeCommit() is called. As you said, DecodeCommit() may access the system table. ... The wals are gener

Re: Parallel INSERT (INTO ... SELECT ...)

2020-09-24 Thread Amit Kapila
On Thu, Sep 24, 2020 at 7:51 AM Andres Freund wrote: > > On 2020-09-22 14:55:21 +1000, Greg Nancarrow wrote: > > > > diff --git a/src/backend/access/heap/heapam.c > > b/src/backend/access/heap/heapam.c > > index 1585861..94c8507 100644 > > --- a/src/backend/access/heap/heapam.c > > +++ b/src/back

Re: scram-sha-256 broken with FIPS and OpenSSL 1.0.2

2020-09-24 Thread Michael Paquier
On Thu, Sep 24, 2020 at 06:28:25PM +0200, Daniel Gustafsson wrote: > Doh, of course, I blame a lack of caffeine this afternoon. Having a private > local sha256 implementation using the EVP_* API inside scram-common would > maintain FIPS compliance and ABI compatibility, but would also be rather ug

Re: scram-sha-256 broken with FIPS and OpenSSL 1.0.2

2020-09-24 Thread Michael Paquier
On Thu, Sep 24, 2020 at 09:44:57PM +0200, Daniel Gustafsson wrote: > On 24 Sep 2020, at 21:22, Robert Haas wrote: >> I mean, the issue here, as is so often the case, is not what is >> actually more secure, but what meets the terms of some security >> standard. > > Correct, IIUC in order to be FIP

Re: Parallel INSERT (INTO ... SELECT ...)

2020-09-24 Thread Greg Nancarrow
> > What if this > > ends up being invoked from inside C code? > > > > I think it shouldn't be a problem unless one is trying to do something > like insert into foreign key table. So, probably we can have an Assert > to catch it if possible. Do you have any other idea? > Note that the planner code

Re: history file on replica and double switchover

2020-09-24 Thread David Zhang
On 2020-09-24 5:00 p.m., Fujii Masao wrote: On 2020/09/25 8:15, David Zhang wrote: Hi, My understanding is that the "archiver" won't even start if "archive_mode = on" has been set on a "replica". Therefore, either (XLogArchiveMode == ARCHIVE_MODE_ALWAYS) or (XLogArchiveMode != ARCHIVE_MODE

Re: scram-sha-256 broken with FIPS and OpenSSL 1.0.2

2020-09-24 Thread Michael Paquier
On Fri, Sep 25, 2020 at 12:19:44PM +0900, Michael Paquier wrote: > Even if we'd try to force our internal implementation of SHA256 on > already-released branches instead of the one of OpenSSL, this would be > an ABI break for compiled modules expected to work on this released > branch as OpenSSL's

Re: scram-sha-256 broken with FIPS and OpenSSL 1.0.2

2020-09-24 Thread Tom Lane
Michael Paquier writes: > On Fri, Sep 25, 2020 at 12:19:44PM +0900, Michael Paquier wrote: >> Even if we'd try to force our internal implementation of SHA256 on >> already-released branches instead of the one of OpenSSL, this would be >> an ABI break for compiled modules expected to work on this r

Re: Parallel INSERT (INTO ... SELECT ...)

2020-09-24 Thread Greg Nancarrow
Hi Andres, On Thu, Sep 24, 2020 at 12:21 PM Andres Freund wrote: > >I think it'd be good if you outlined what your approach is to make this >safe. Some prior work has already been done to establish the necessary infrastructure to allow parallel INSERTs, in general, to be safe, except for cases w

Feature improvement of tab completion for DEALLOCATE

2020-09-24 Thread btnakamichin
Hello! I’d like to improve the deallocate tab completion. The deallocate tab completion can’t complement “ALL”. Therefore, this patch fixes the problem. Regards, Naoki Nakamichidiff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c index 9c6f5ecb6a..d877cc86f0 100644 --- a/sr

Re: Retry Cached Remote Connections for postgres_fdw in case remote backend gets killed/goes away

2020-09-24 Thread Bharath Rupireddy
On Wed, Sep 23, 2020 at 8:19 PM Fujii Masao wrote: > > > Please let me know if okay with the above agreed points, I will work on the > > new patch. > > Yes, please work on the patch! Thanks! I may revisit the above points later, > though ;) > Thanks, attaching v4 patch. Please consider this for

Feature improvement for FETCH tab completion

2020-09-24 Thread btnakamichin
Hello! I’d like to improve the FETCH tab completion. The FETCH tab completion . Therefore, this patch fixes the problem. Previous function completed one of FORWARD, BACKWARD, RELATIVE, ABSOLUTE, but now it completes one of FORWARD, BACKWARD, RELATIVE, ABSOLUTE, ALL, NEXT, PRIOR, FIRST, LAST a

Re: scram-sha-256 broken with FIPS and OpenSSL 1.0.2

2020-09-24 Thread Peter Eisentraut
On 2020-09-24 21:44, Daniel Gustafsson wrote: On 24 Sep 2020, at 21:22, Robert Haas wrote: On Thu, Sep 24, 2020 at 1:57 PM Peter Eisentraut wrote: Depends on what one considers to be covered by FIPS. The entire rest of SCRAM is custom code, so running it on top of the world's greatest SHA-25

Re: Feature improvement of tab completion for DEALLOCATE

2020-09-24 Thread Fujii Masao
On 2020/09/25 13:45, btnakamichin wrote: Hello! I’d like to improve the deallocate tab completion. The deallocate tab completion can’t complement “ALL”. Therefore, this patch fixes the problem. Thanks for the patch! It looks basically good, but I think it's better to add newline just befo

Re: scram-sha-256 broken with FIPS and OpenSSL 1.0.2

2020-09-24 Thread Tom Lane
Peter Eisentraut writes: > On 2020-09-24 21:44, Daniel Gustafsson wrote: >> Correct, IIUC in order to be FIPS compliant all cryptographic modules used >> must >> be FIPS certified. > As I read FIPS 140-2, it just specifies what must be true of > cryptographic modules that claim to follow that s

Re: vs formatting in the docs

2020-09-24 Thread Peter Eisentraut
On 2020-07-10 17:05, Peter Eisentraut wrote: On 2020-06-21 18:57, Dagfinn Ilmari Mannsåker wrote: Attached are two patches: the first adds the missing tags, the second adds to all the SQL commands (specifically anything with 7). I have committed the first one. I have some concerns about the

Re: [PATCH] Add features to pg_stat_statements

2020-09-24 Thread Katsuragi Yuta
On 2020-09-24 18:55, legrand legrand wrote: Not limited to 3, Like an history table. Will have to think if data is saved at shutdown. Thanks. This design might introduce some additional complexity and things to be considered. For example, the maximum size of "history table", how to evict entr

Re: Range checks of pg_test_fsync --secs-per-test and pg_test_timing --duration

2020-09-24 Thread Peter Eisentraut
On 2020-09-24 09:12, Michael Paquier wrote: On Wed, Sep 23, 2020 at 08:11:59AM +0200, Peter Eisentraut wrote: This patch mixes up unsigned int and uint32 in random ways. The variable is uint32, but the format is %u and the max constant is UINT_MAX. I think just use unsigned int as the variable

Re: Logical replication from PG v13 and below to PG v14 (devel version) is not working.

2020-09-24 Thread Ashutosh Sharma
On Fri, Sep 25, 2020 at 8:12 AM Amit Kapila wrote: > > On Thu, Sep 24, 2020 at 6:33 PM Ashutosh Sharma wrote: > > > > Hi Amit, > > > > > Here, I think instead of using MySubscription->stream, we should use > > > server/walrecv version number as we used at one place in tablesync.c. > > > > Should

Re: Feature improvement of tab completion for DEALLOCATE

2020-09-24 Thread btnakamichin
2020-09-25 14:30 に Fujii Masao さんは書きました: On 2020/09/25 13:45, btnakamichin wrote: Hello! I’d like to improve the deallocate tab completion. The deallocate tab completion can’t complement “ALL”. Therefore, this patch fixes the problem. Thanks for the patch! It looks basically good, but I thi

Problem of ko.po on branch REL9_5_STABLE

2020-09-24 Thread Yang, Rong
Hi, When I checked the encoding of the Po files, I noticed that src/bin/pg_config/po/ko.po on REL9_5_STABLE branch seemed a little different. The ‘Content-Type’ of this file and file’s encoding are different from those under other modules, as follows:   src/bin/pg_config/

Re: Load TIME fields - proposed performance improvement

2020-09-24 Thread Peter Smith
The patch has been re-implemented based on previous advice. Please see attached. ~ Test: A test table was created and 20 million rows inserted as follows: test=# create table t1 (id int, a timestamp, b time without time zone default '01:02:03', c date default CURRENT_DATE, d time with time zon

Re: Feature improvement for FETCH tab completion

2020-09-24 Thread Fujii Masao
On 2020/09/25 14:24, btnakamichin wrote: Hello! I’d like to improve the FETCH tab completion. The FETCH tab completion . Therefore, this patch fixes the problem. Previous function completed one of FORWARD, BACKWARD, RELATIVE, ABSOLUTE, but now it completes one of FORWARD, BACKWARD, RELATIV

Re: [patch] Concurrent table reindex per-index progress reporting

2020-09-24 Thread Michael Paquier
On Thu, Sep 24, 2020 at 09:19:18PM +0200, Matthias van de Meent wrote: > While working on a PG12-instance I noticed that the progress reporting of > concurrent index creation for non-index relations fails to update the > index/relation OIDs that it's currently working on in the > pg_stat_progress_c

Re: [patch] Fix checksum verification in base backups for zero page headers

2020-09-24 Thread Michael Banck
Hi, Am Donnerstag, den 24.09.2020, 17:24 +0300 schrieb Anastasia Lubennikova: > So I mark this one as ReadyForCommitter. Thanks! > The only minor problem is a typo (?) in the proposed commit message. > "If a page is all zero, consider that a checksum failure." It should be > "If a page is NOT a

Re: scram-sha-256 broken with FIPS and OpenSSL 1.0.2

2020-09-24 Thread Michael Paquier
On Fri, Sep 25, 2020 at 01:36:44AM -0400, Tom Lane wrote: > Peter Eisentraut writes: >> However, again, the SCRAM >> implementation would already appear to fail that requirement because it >> uses a custom HMAC implementation, and HMAC is listed in FIPS 140-2 as a >> covered algorithm. > > Ugh