Re: logical streaming of xacts via test_decoding is broken

2020-11-09 Thread Dilip Kumar
On Mon, Nov 9, 2020 at 11:31 AM Amit Kapila wrote: > > On Mon, Nov 9, 2020 at 11:21 AM Dilip Kumar wrote: > > > > On Mon, Nov 9, 2020 at 11:04 AM Dilip Kumar wrote: > > > > > > On Mon, Nov 9, 2020 at 11:00 AM Amit Kapila > > > wrote: > > > > > > > > Michael reported a BF failure [1] related to

Re: pg_upgrade analyze script

2020-11-09 Thread Michael Paquier
On Mon, Nov 02, 2020 at 02:18:32PM +0100, Magnus Hagander wrote: > On Fri, Oct 30, 2020 at 5:10 PM Georgios Kokolatos > wrote: >> I did notice that the cfbot [1] is failing for this patch. >> Please try to address the issues for the upcoming Commitfest. > > Thanks for the notice -- PFA a rebased

Re: abstract Unix-domain sockets

2020-11-09 Thread Peter Eisentraut
On 2020-11-09 07:08, Michael Paquier wrote: As abstract namespaces don't have permissions, anyone knowing the name of the path, which should be unique, can have an access to the server. Do you think that the documentation should warn the user about that? This feature is about easing the managemen

Re: [HACKERS] logical decoding of two-phase transactions

2020-11-09 Thread Masahiko Sawada
On Mon, Nov 9, 2020 at 3:23 PM Peter Smith wrote: > > > > 4. > > > +static void > > > +apply_handle_prepare_txn(LogicalRepPrepareData * prepare_data) > > > +{ > > > + Assert(prepare_data->prepare_lsn == remote_final_lsn); > > > + > > > + /* The synchronization worker runs in single transaction. */

Performance regressions

2020-11-09 Thread Vik Fearing
Hello, I've been doing some benchmarking on recent version of PostgreSQL and I'm seeing some regressions. The benchmark setup is as described in [1] except it looks like I got lucky in the runs used for that article. After many more runs, I get these NOPM averages (hopefully formatting will surv

Useless string ouput in error message

2020-11-09 Thread Tang, Haiying
Hi I think I found a typo for the output of an error message which may cause building warning. Please refer to the attachment for the detail. Previous discussion: https://www.postgresql.org/message-id/alpine.DEB.2.21.1910311939430.27369@lancre Best regards Tang 0001-Remove-useless-s.patch D

Re: [PATCH] Add features to pg_stat_statements

2020-11-09 Thread Seino Yuki
2020-11-09 15:39 に Seino Yuki さんは書きました: However, let me confirm the following. Is this information really useful? If there is no valid use case for this, I'd like to drop it. Thought? I thought it would be easy for users to see at a glance that if there is a case I assumed, if the last modif

Re: logical streaming of xacts via test_decoding is broken

2020-11-09 Thread Dilip Kumar
On Mon, Nov 9, 2020 at 1:34 PM Dilip Kumar wrote: > > On Mon, Nov 9, 2020 at 11:31 AM Amit Kapila wrote: > > > > On Mon, Nov 9, 2020 at 11:21 AM Dilip Kumar wrote: > > > > > > On Mon, Nov 9, 2020 at 11:04 AM Dilip Kumar wrote: > > > > > > > > On Mon, Nov 9, 2020 at 11:00 AM Amit Kapila > > >

Re: Protect syscache from bloating with negative cache entries

2020-11-09 Thread Kyotaro Horiguchi
At Fri, 6 Nov 2020 10:42:15 +0200, Heikki Linnakangas wrote in > Do you need the "ntaccess == 2" test? You could always increment the > counter, and in the code that uses ntaccess to decide what to evict, > treat all values >= 2 the same. > > Need to handle integer overflow somehow. Or maybe no

Re: Useless string ouput in error message

2020-11-09 Thread Magnus Hagander
On Mon, Nov 9, 2020 at 9:39 AM Tang, Haiying wrote: > > Hi > > I think I found a typo for the output of an error message which may cause > building warning. > Please refer to the attachment for the detail. > > Previous discussion: > https://www.postgresql.org/message-id/alpine.DEB.2.21.1910311939

Re: default result formats setting

2020-11-09 Thread Peter Eisentraut
On 2020-11-05 22:03, Peter Eisentraut wrote: Independently of that, how would you implement "says otherwise" here, ie do a single-query override of the session's prevailing setting? Maybe the right thing for that is to define -1 all the way down to the protocol level as meaning "use the session's

Re: pg_upgrade analyze script

2020-11-09 Thread Magnus Hagander
On Mon, Nov 9, 2020 at 8:53 AM Michael Paquier wrote: > > On Mon, Nov 02, 2020 at 02:18:32PM +0100, Magnus Hagander wrote: > > On Fri, Oct 30, 2020 at 5:10 PM Georgios Kokolatos > > wrote: > >> I did notice that the cfbot [1] is failing for this patch. > >> Please try to address the issues for th

Re: Multi Inserts in CREATE TABLE AS - revived patch

2020-11-09 Thread Bharath Rupireddy
On Tue, Nov 3, 2020 at 4:54 PM Bharath Rupireddy wrote: > > If the approach followed in the patch looks okay, I can work on a separate > patch for multi inserts in refresh materialized view cases. > Hi, I'm attaching a v2 patch that has multi inserts for CTAS as well as REFRESH MATERIALiZED VIEW

Re: -O switch

2020-11-09 Thread Magnus Hagander
On Wed, Nov 4, 2020 at 2:10 AM Tom Lane wrote: > > Magnus Hagander writes: > > [ remove_option_o_2.patch ] > > This seems committable to me now, although ... > > > On Mon, Nov 2, 2020 at 6:58 PM Tom Lane wrote: > >> Magnus Hagander writes: > >>> Initially I kept the dynamic argv/argc in even th

Re: Dumping/restoring fails on inherited generated column

2020-11-09 Thread Peter Eisentraut
On 2020-11-06 04:55, Masahiko Sawada wrote: Both of these result in the same change to the dump output. Both of them have essentially the same idea. The first one adds the conditionals during the information gathering phase of pg_dump, the second one adds the conditionals during the output phas

Re: logical streaming of xacts via test_decoding is broken

2020-11-09 Thread Amit Kapila
On Mon, Nov 9, 2020 at 3:01 PM Dilip Kumar wrote: > > On Mon, Nov 9, 2020 at 1:34 PM Dilip Kumar wrote: > > > > On Mon, Nov 9, 2020 at 11:31 AM Amit Kapila wrote: > > > > > > On Mon, Nov 9, 2020 at 11:21 AM Dilip Kumar wrote: > > > > > > > > On Mon, Nov 9, 2020 at 11:04 AM Dilip Kumar > > > >

Re: Asymmetric partition-wise JOIN

2020-11-09 Thread Anastasia Lubennikova
On 21.08.2020 09:02, Andrey V. Lepikhov wrote: On 7/1/20 2:10 PM, Daniel Gustafsson wrote: On 27 Dec 2019, at 08:34, Kohei KaiGai wrote: The attached v2 fixed the problem, and regression test finished correctly. This patch no longer applies to HEAD, please submit an rebased version. Markin

Re: Feature improvement for pg_stat_statements

2020-11-09 Thread Seino Yuki
Thank you for pointing that out. I'll post a fixed patch. + SpinLockAcquire(&pgss->mutex); You might noticed, but there a purpose of using the following idiom. Without that, compiler might optimize out the comparison assuming *pgss won't change. volatile pgssSharedState

Re: pg_upgrade analyze script

2020-11-09 Thread Magnus Hagander
On Mon, Nov 9, 2020 at 11:22 AM Magnus Hagander wrote: > > On Mon, Nov 9, 2020 at 8:53 AM Michael Paquier wrote: > > > > On Mon, Nov 02, 2020 at 02:18:32PM +0100, Magnus Hagander wrote: > > > On Fri, Oct 30, 2020 at 5:10 PM Georgios Kokolatos > > > wrote: > > >> I did notice that the cfbot [1] i

Re: [HACKERS] logical decoding of two-phase transactions

2020-11-09 Thread Amit Kapila
On Mon, Nov 9, 2020 at 1:38 PM Masahiko Sawada wrote: > > On Mon, Nov 9, 2020 at 3:23 PM Peter Smith wrote: > > > > I've looked at the patches and done some tests. Here is my comment and > question I realized during testing and reviewing. > > +static void > +DecodePrepare(LogicalDecodingContext *

Re: Prevent printing "next step instructions" in initdb and pg_upgrade

2020-11-09 Thread Magnus Hagander
On Mon, Nov 2, 2020 at 2:23 PM Magnus Hagander wrote: > > On Tue, Oct 27, 2020 at 12:40 PM Bruce Momjian wrote: > > > > On Tue, Oct 27, 2020 at 12:35:56PM +0100, Peter Eisentraut wrote: > > > On 2020-10-27 11:53, Bruce Momjian wrote: > > > > On Tue, Oct 27, 2020 at 11:35:25AM +0100, Peter Eisentr

Re: logical streaming of xacts via test_decoding is broken

2020-11-09 Thread Dilip Kumar
On Mon, Nov 9, 2020 at 4:21 PM Amit Kapila wrote: > > On Mon, Nov 9, 2020 at 3:01 PM Dilip Kumar wrote: > > > > On Mon, Nov 9, 2020 at 1:34 PM Dilip Kumar wrote: > > > > > > On Mon, Nov 9, 2020 at 11:31 AM Amit Kapila > > > wrote: > > > > > > > > On Mon, Nov 9, 2020 at 11:21 AM Dilip Kumar >

Re: logical streaming of xacts via test_decoding is broken

2020-11-09 Thread Dilip Kumar
On Mon, Nov 9, 2020 at 5:37 PM Dilip Kumar wrote: > > On Mon, Nov 9, 2020 at 4:21 PM Amit Kapila wrote: > > > > On Mon, Nov 9, 2020 at 3:01 PM Dilip Kumar wrote: > > > > > > On Mon, Nov 9, 2020 at 1:34 PM Dilip Kumar wrote: > > > > > > > > On Mon, Nov 9, 2020 at 11:31 AM Amit Kapila > > > > w

Re: Prevent printing "next step instructions" in initdb and pg_upgrade

2020-11-09 Thread Anastasia Lubennikova
On 02.11.2020 16:23, Magnus Hagander wrote: On Tue, Oct 27, 2020 at 11:35:25AM +0100, Peter Eisentraut wrote: On 2020-10-06 12:26, Magnus Hagander wrote: I went with the name --no-instructions to have the same name for both initdb and pg_upgrade. The downside is that "no-instructions" also caus

Re: Prevent printing "next step instructions" in initdb and pg_upgrade

2020-11-09 Thread Magnus Hagander
On Mon, Nov 9, 2020 at 2:18 PM Anastasia Lubennikova wrote: > > On 02.11.2020 16:23, Magnus Hagander wrote: > > On Tue, Oct 27, 2020 at 11:35:25AM +0100, Peter Eisentraut wrote: > > On 2020-10-06 12:26, Magnus Hagander wrote: > > I went with the name --no-instructions to have the same name for bot

Re: document pg_settings view doesn't display custom options

2020-11-09 Thread John Naylor
On Mon, Nov 9, 2020 at 2:12 AM Fujii Masao wrote: > Pushed. Thanks! > Thank you! -- John Naylor EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: Making cancellations safe

2020-11-09 Thread Dave Cramer
On Wed, 4 Nov 2020 at 10:50, Shay Rojansky wrote: > Hi all. > > Back in 2016 I started a thread about making cancellations safer[1], I'd > like to try to pick this up again. Here a summary of the previous > conversation: > > The main ask here is to allow clients to specify which command to cancel

Re: Reduce the number of special cases to build contrib modules on windows

2020-11-09 Thread Alvaro Herrera
On 2020-Nov-06, David Rowley wrote: > +# Handle makefile rules for when file to be added to the project > +# does not exist. Returns 1 when the original file add should be > +# skipped. > +sub FindAndAddAdditionalFiles > +{ > + my $self = shift; > + my $fname = shift; > + my ($ext) =

Re: Prevent printing "next step instructions" in initdb and pg_upgrade

2020-11-09 Thread Alvaro Herrera
On 2020-Nov-09, Magnus Hagander wrote: > But for usability that makes less sense. For the delete script, the > wrapper (that the switch is intended for) knows more than pg_upgrade > about how to delete it, so it can do a better job, and thus it makes > sense to silence it. But for something like t

Re: WIP: BRIN multi-range indexes

2020-11-09 Thread John Naylor
On Sat, Nov 7, 2020 at 4:38 PM Tomas Vondra wrote: > Overall, I think there's very little difference, particularly in the > "match" cases when we're searching for a value that we know is in the > table. The one-hash variant seems to perform a bit better, but the > difference is fairly small. > >

Re: Prevent printing "next step instructions" in initdb and pg_upgrade

2020-11-09 Thread Magnus Hagander
On Mon, Nov 9, 2020 at 3:29 PM Alvaro Herrera wrote: > > On 2020-Nov-09, Magnus Hagander wrote: > > > But for usability that makes less sense. For the delete script, the > > wrapper (that the switch is intended for) knows more than pg_upgrade > > about how to delete it, so it can do a better job,

Re: Rethinking LOCK TABLE's behavior on views

2020-11-09 Thread Alvaro Herrera
On 2020-Nov-07, Noah Misch wrote: > On Sat, Nov 07, 2020 at 11:57:20AM -0500, Tom Lane wrote: > > A completely different approach we could consider is to weaken the > > permissions requirements for LOCK on a view, say "allow it if either > > the calling user or the view owner has the needed permi

Re: pg_upgrade analyze script

2020-11-09 Thread Peter Eisentraut
On 2020-11-09 11:22, Magnus Hagander wrote: I have spotted one small-ish thing. This patch is missing to update the following code in vcregress.pl: print "\nSetting up stats on new cluster\n\n"; system(".\\analyze_new_cluster.bat") == 0 or exit 1; Ah, nice catch -- thanks! I guess thi

Re: vacuum -vs reltuples on insert only index

2020-11-09 Thread Jehan-Guillaume de Rorthais
On Wed, 4 Nov 2020 18:44:03 -0800 Peter Geoghegan wrote: > On Mon, Nov 2, 2020 at 10:03 AM Peter Geoghegan wrote: > > Actually, it seems better to always count num_index_tuples the old way > > during cleanup-only index VACUUMs, despite the inaccuracy that that > > creates with posting list tuple

Re: Prevent printing "next step instructions" in initdb and pg_upgrade

2020-11-09 Thread Alvaro Herrera
On 2020-Nov-09, Magnus Hagander wrote: > On Mon, Nov 9, 2020 at 3:29 PM Alvaro Herrera wrote: > > > > How about a switch like "--with-scripts=" where the list can be > > "all" to include everything (default), "none" to include nothing, or a > > comma-separated list of things to include? (Also "-

Re: Disable WAL logging to speed up data loading

2020-11-09 Thread Stephen Frost
Greetings, * osumi.takami...@fujitsu.com (osumi.takami...@fujitsu.com) wrote: > On Tuesday, Nov 3, 2020 3:02 AM Stephen Frost wrote: > > I'm not sure that wal_level=none is really the right way to address this > > use-case. We already have unlogged tables and that's pretty clean and > > meets th

Re: -Wformat-signedness

2020-11-09 Thread Tom Lane
Peter Eisentraut writes: > 1. enums are unsigned by default in gcc, so all those internal error > messages "unrecognized blah kind: %d" need to be changed to %u. Do we have reason to think that that is true in every C compiler? My own preference for this would be to leave the messages as-is and

Re: Reduce the time required for a database recovery from archive.

2020-11-09 Thread Dmitry Shulga
Hello Stephen, > On 19 Oct 2020, at 23:25, Stephen Frost wrote: > > Greetings, > Implementation of this approach assumes running several background processes (bgworkers) each of which runs a shell command specified by the parameter restore_command to deliver an archiv

Re: remove spurious CREATE INDEX CONCURRENTLY wait

2020-11-09 Thread Dmitry Dolgov
> On Tue, Nov 03, 2020 at 07:14:47PM +0100, Dmitry Dolgov wrote: > > On Thu, Aug 20, 2020 at 03:11:19PM +0900, Michael Paquier wrote: > > On Wed, Aug 19, 2020 at 02:16:46PM -0400, Alvaro Herrera wrote: > > > I did not set the flag in REINDEX CONCURRENTLY, but as I understand it > > > can be done to

Re: -O switch

2020-11-09 Thread Tom Lane
Magnus Hagander writes: > On Wed, Nov 4, 2020 at 2:10 AM Tom Lane wrote: >> ... looking at this again, BackendRun certainly looks ridiculously >> over-engineered for what it still does. > Yeah, looking at it again, I agree. PFA an updated patch, which I'll > go ahead and push shortly. LGTM. >

Re: abstract Unix-domain sockets

2020-11-09 Thread Andreas Karlsson
On 11/9/20 9:04 AM, Peter Eisentraut wrote: On 2020-11-09 07:08, Michael Paquier wrote: As abstract namespaces don't have permissions, anyone knowing the name of the path, which should be unique, can have an access to the server. Do you think that the documentation should warn the user about tha

Re: PATCH: Report libpq version and configuration

2020-11-09 Thread Alvaro Herrera
On 2020-Oct-27, Craig Ringer wrote: > On Tue, Oct 27, 2020 at 12:56 AM Tom Lane wrote: > > > +1. Are we concerned about translatability of these strings? I think > > I'd vote against, as it would complicate applications, but it's worth > > thinking about it now not later. > > It's necessary no

Re: Reduce the time required for a database recovery from archive.

2020-11-09 Thread Stephen Frost
Greetings, * Dmitry Shulga (d.shu...@postgrespro.ru) wrote: > > On 19 Oct 2020, at 23:25, Stephen Frost wrote: > Implementation of this approach assumes running several background > processes (bgworkers) > each of which runs a shell command specified by the parameter > resto

Re: PATCH: Report libpq version and configuration

2020-11-09 Thread Tom Lane
Alvaro Herrera writes: > Well, if we can make it run in more systems than just Linux, then it > seems worth having. The submitted patch seems a little bit on the > naughty side. I agree that the facility seems possibly useful, as long as we can minimize its platform dependency. Just embedding s

Misuse of TimestampDifference() in the autoprewarm feature of pg_prewarm

2020-11-09 Thread Alexey Kondratov
Hi Hackers, Today I have accidentally noticed that autoprewarm feature of pg_prewarm used TimestampDifference()'s results in a wrong way. First, it used *seconds* result from it as a *milliseconds*. It was causing it to make dump file autoprewarm.blocks ~every second with default setting of

Re: Disable WAL logging to speed up data loading

2020-11-09 Thread David G. Johnston
On Mon, Nov 9, 2020 at 8:18 AM Stephen Frost wrote: > Presently, my feeling is that we could address this use-case without > having to introduce a new cluster-wide WAL level, and that's the > direction I'd want to see this going. Perhaps I'm missing something > about why the approach I've set fo

Re: Disable WAL logging to speed up data loading

2020-11-09 Thread Stephen Frost
Greetings, * David G. Johnston (david.g.johns...@gmail.com) wrote: > On Mon, Nov 9, 2020 at 8:18 AM Stephen Frost wrote: > > Presently, my feeling is that we could address this use-case without > > having to introduce a new cluster-wide WAL level, and that's the > > direction I'd want to see this

Re: WIP: BRIN multi-range indexes

2020-11-09 Thread Tomas Vondra
On 11/9/20 3:29 PM, John Naylor wrote: > On Sat, Nov 7, 2020 at 4:38 PM Tomas Vondra > mailto:tomas.von...@enterprisedb.com>> > wrote: > >> Overall, I think there's very little difference, particularly in the >> "match" cases when we're searching for a value that we know is in the >> table. The

Re: automatic analyze: readahead - add "IO read time" log message

2020-11-09 Thread Stephen Frost
Greetings, * Tomas Vondra (tomas.von...@2ndquadrant.com) wrote: > On 11/4/20 5:02 PM, Stephen Frost wrote: > >* Tomas Vondra (tomas.von...@2ndquadrant.com) wrote: > >>>If you highlight "738754560" in the output it appears to duplicate the > >>>syscalls issued until it preads() - in case of "738754

Re: Misuse of TimestampDifference() in the autoprewarm feature of pg_prewarm

2020-11-09 Thread Jeevan Ladhe
Hi, On Mon, Nov 9, 2020 at 10:15 PM Alexey Kondratov wrote: > Hi Hackers, > > Today I have accidentally noticed that autoprewarm feature of pg_prewarm > used TimestampDifference()'s results in a wrong way. > > First, it used *seconds* result from it as a *milliseconds*. It was > causing it to ma

Re: automatic analyze: readahead - add "IO read time" log message

2020-11-09 Thread Tomas Vondra
On 11/9/20 7:06 PM, Stephen Frost wrote: > Greetings, > > * Tomas Vondra (tomas.von...@2ndquadrant.com) wrote: >> On 11/4/20 5:02 PM, Stephen Frost wrote: >>> * Tomas Vondra (tomas.von...@2ndquadrant.com) wrote: > If you highlight "738754560" in the output it appears to duplicate the > sys

Re: Disable WAL logging to speed up data loading

2020-11-09 Thread David G. Johnston
On Mon, Nov 9, 2020 at 10:36 AM Stephen Frost wrote: > * David G. Johnston (david.g.johns...@gmail.com) wrote: > > > If the commit doesn't complete all of the newly created pages are junk. > > Otherwise, you have a crash-recoverable state for those tables as regards > > those specific pages. > >

Re: RE: Delaying/avoiding BTreeTupleGetNAtts() call within _bt_compare()

2020-11-09 Thread Peter Geoghegan
On Mon, Nov 2, 2020 at 1:04 PM Peter Geoghegan wrote: > if Andres cannot spend any time on this in the foreseeable future then > I'll withdraw the patch. I intend to formally withdraw the patch on > November 9th, provided no new information comes to light. I have now formally withdrawn the patch

Re: PATCH: Attempt to make dbsize a bit more consistent

2020-11-09 Thread Soumyadeep Chakraborty
Hey Georgios, Thanks for looking for more avenues to invoke tableAM APIS! Please find my review below: On Tue, Oct 13, 2020 at 6:28 AM wrote: 1. > /* > - * heap size, including FSM and VM > + * table size, including FSM and VM > */ We should not mention FSM and VM in dbsize.c at all as thes

Re: Misuse of TimestampDifference() in the autoprewarm feature of pg_prewarm

2020-11-09 Thread Tom Lane
Alexey Kondratov writes: > After fixing this issue I have noticed that it still dumps blocks twice > at each timeout (here I set autoprewarm_interval to 15s): > ... > This happens because at timeout time we were using continue, but > actually we still have to wait the entire autoprewarm_interval

Re: WIP: BRIN multi-range indexes

2020-11-09 Thread John Naylor
On Mon, Nov 9, 2020 at 1:39 PM Tomas Vondra wrote: > > > While investigating the failures, I've tried increasing the values a > lot, without observing any measurable increase in runtime. IIRC I've > even used (10 * target_partlen) or something like that. That tells me > it's not very sensitive par

Re: Disable WAL logging to speed up data loading

2020-11-09 Thread Stephen Frost
Greetings, * David G. Johnston (david.g.johns...@gmail.com) wrote: > On Mon, Nov 9, 2020 at 10:36 AM Stephen Frost wrote: > > * David G. Johnston (david.g.johns...@gmail.com) wrote: > > > If the commit doesn't complete all of the newly created pages are junk. > > > Otherwise, you have a crash-rec

Re: Skip ExecCheckRTPerms in CTAS with no data

2020-11-09 Thread Anastasia Lubennikova
On 29.09.2020 14:39, Bharath Rupireddy wrote: On Mon, Sep 28, 2020 at 7:48 PM Tom Lane wrote: Bharath Rupireddy writes: In case of CTAS with no data, we actually do not insert the tuples into the created table, so we can skip checking for the insert permissions. Anyways, the insert permission

Re: public schema default ACL

2020-11-09 Thread Bruce Momjian
On Mon, Nov 2, 2020 at 11:05:15PM -0800, Noah Misch wrote: > On Mon, Nov 02, 2020 at 12:42:26PM -0500, Tom Lane wrote: > > Robert Haas writes: > > > On Mon, Nov 2, 2020 at 5:51 AM Peter Eisentraut > > > wrote: > > >> I'm not convinced, however, that this would would really move the needle > > >>

Re: public schema default ACL

2020-11-09 Thread Bruce Momjian
On Mon, Nov 2, 2020 at 01:41:09PM -0500, Stephen Frost wrote: > At least from seeing the users that start out with PG and then come to > the Slack or IRC channel asking questions, the on-boarding experience > today typically consists of 'apt install postgresql' and then complaints > that they aren

Re: Misuse of TimestampDifference() in the autoprewarm feature of pg_prewarm

2020-11-09 Thread Alexey Kondratov
On 2020-11-09 21:53, Tom Lane wrote: Alexey Kondratov writes: After fixing this issue I have noticed that it still dumps blocks twice at each timeout (here I set autoprewarm_interval to 15s): ... This happens because at timeout time we were using continue, but actually we still have to wait th

Re: Misuse of TimestampDifference() in the autoprewarm feature of pg_prewarm

2020-11-09 Thread Tom Lane
Alexey Kondratov writes: > On 2020-11-09 21:53, Tom Lane wrote: >> 0002 seems like a pretty clear bug fix, though I wonder if this is exactly >> what we want to do going forward. It seems like a very large fraction of >> the callers of TimestampDifference would like to have the value in msec, >>

Re: Useless string ouput in error message

2020-11-09 Thread Fabien COELHO
I think I found a typo for the output of an error message which may cause building warning. Please refer to the attachment for the detail. Indeed. Thanks for the fix! -- Fabien.

Re: Error on failed COMMIT

2020-11-09 Thread Dave Cramer
On Wed, 30 Sep 2020 at 18:14, Andrew Dunstan wrote: > > On 8/4/20 12:19 PM, Dave Cramer wrote: > > Attached is the rebased patch for consideration. > > > > > > > It's a bit sad this has been hanging around so long without attention. > > > The previous discussion seems to give the patch a clean bi

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-11-09 Thread David Rowley
On Mon, 9 Nov 2020 at 16:29, Andy Fan wrote: > I think either version is OK for me and I like this patch overall. That's good to know. Thanks. > However I believe v9 > should be no worse than v8 all the time, Is there any theory to explain > your result? Nothing jumps out at me from looking a

Re: -Wformat-signedness

2020-11-09 Thread Thomas Munro
On Tue, Nov 10, 2020 at 4:25 AM Tom Lane wrote: > Peter Eisentraut writes: > > 1. enums are unsigned by default in gcc, so all those internal error > > messages "unrecognized blah kind: %d" need to be changed to %u. > > Do we have reason to think that that is true in every C compiler? > My own pr

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-11-09 Thread Alvaro Herrera
On 2020-Nov-10, David Rowley wrote: > On Mon, 9 Nov 2020 at 16:29, Andy Fan wrote: > > However I believe v9 > > should be no worse than v8 all the time, Is there any theory to explain > > your result? > > Nothing jumps out at me from looking at profiles. The only thing I > noticed was the t

Re: Additional Chapter for Tutorial

2020-11-09 Thread David G. Johnston
On Sun, Nov 8, 2020 at 8:56 AM Jürgen Purtz wrote: > > Good catches. Everything applied. > Reviewed the first three sections. template0 - I would remove the schema portions of this and simply note this as being a pristine recovery database in the diagram. I would drop the word "more" and just

Re: Add important info about ANALYZE after create Functional Index

2020-11-09 Thread Bruce Momjian
On Tue, Oct 27, 2020 at 12:12:00AM -0700, Nikolay Samokhvalov wrote: > On Mon, Oct 26, 2020 at 3:08 PM Fabrízio de Royes Mello < > fabriziome...@gmail.com> wrote: > > Would be nice if add some information about it into our docs but not sure > where. I'm thinking about: > - doc/src/sgml

Re: Add important info about ANALYZE after create Functional Index

2020-11-09 Thread Fabrízio de Royes Mello
On Mon, 9 Nov 2020 at 20:27 Bruce Momjian wrote: > > I see REINDEX CONCURRENTLY was fixed in head, but the docs didn't get > updated to mention the need to run ANALYZE or wait for autovacuum before > expression indexes can be fully used by the optimizer. Instead of > putting this mention in the

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-11-09 Thread Tom Lane
Alvaro Herrera writes: > Are you taking into account the possibility that generated machine code > is a small percent slower out of mere bad luck? I remember someone > suggesting that they can make code 2% faster or so by inserting random > no-op instructions in the binary, or something like that

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-11-09 Thread David Rowley
On Tue, 10 Nov 2020 at 12:49, Tom Lane wrote: > > Alvaro Herrera writes: > > Are you taking into account the possibility that generated machine code > > is a small percent slower out of mere bad luck? I remember someone > > suggesting that they can make code 2% faster or so by inserting random >

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-11-09 Thread Peter Geoghegan
On Mon, Nov 9, 2020 at 3:49 PM Tom Lane wrote: > Alvaro Herrera writes: > > Are you taking into account the possibility that generated machine code > > is a small percent slower out of mere bad luck? I remember someone > > suggesting that they can make code 2% faster or so by inserting random >

Re: MultiXact\SLRU buffers configuration

2020-11-09 Thread Tomas Vondra
Hi, After the issue reported in [1] got fixed, I've restarted the multi-xact stress test, hoping to reproduce the issue. But so far no luck :-( I've started slightly different tests on two machines - on one machine I've done this: a) init.sql create table t (a int); insert into t select i

Make Append Cost aware of some run time partition prune case

2020-11-09 Thread Andy Fan
Currently the cost model of append path sums the cost/rows for all the subpaths, it usually works well until we run into the run-time partition prune case. The first result is that generic plans will rarely be used for some cases. For instance, SELECT * FROM p WHERE pkey = $1; The custom plan wil

RE: POC: postgres_fdw insert batching

2020-11-09 Thread tsunakawa.ta...@fujitsu.com
Hello, The attached patch implements the new bulk insert routine for postgres_fdw and the executor utilizing it. It passes make check-world. I measured performance in a basic non-partitioned case by modifying Tomas-san's scripts. They perform an INSERT SELECT statement that copies one millio

Re: Disable WAL logging to speed up data loading

2020-11-09 Thread Kyotaro Horiguchi
At Mon, 9 Nov 2020 10:18:08 -0500, Stephen Frost wrote in > Greetings, > > * osumi.takami...@fujitsu.com (osumi.takami...@fujitsu.com) wrote: > > When I consider the use case is the system of data warehouse > > as described upthread, the size of each table can be large. > > Thus, changing the st

Adding an aminsert() hint that triggers bottom-up index deletion for UPDATEs that can't use HOT

2020-11-09 Thread Peter Geoghegan
I am working on a project called "bottom-up index deletion" (I was calling it "delete deduplication" until recently). This is another nbtree project. The patch series adds a mechanism that controls old duplicate versions caused by non-HOT UPDATEs. This is very effective. In fact, it prevents almost

Re: pg_upgrade analyze script

2020-11-09 Thread Michael Paquier
On Mon, Nov 09, 2020 at 03:47:22PM +0100, Peter Eisentraut wrote: > You should just remove those calls. There is no need to replace them with > vacuumdb calls. The reason those calls were there is that they were testing > the generated script itself. If the script is gone, there is no more need.

Re: remove spurious CREATE INDEX CONCURRENTLY wait

2020-11-09 Thread Michael Paquier
On Mon, Nov 09, 2020 at 04:47:43PM +0100, Dmitry Dolgov wrote: > > On Tue, Nov 03, 2020 at 07:14:47PM +0100, Dmitry Dolgov wrote: > > > On Thu, Aug 20, 2020 at 03:11:19PM +0900, Michael Paquier wrote: > > > On Wed, Aug 19, 2020 at 02:16:46PM -0400, Alvaro Herrera wrote: > > > > I did not set the fl

Re: remove spurious CREATE INDEX CONCURRENTLY wait

2020-11-09 Thread Tom Lane
Michael Paquier writes: >> +LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE); >> +MyProc->vacuumFlags |= PROC_IN_SAFE_IC; >> +ProcGlobal->vacuumFlags[MyProc->pgxactoff] = >> MyProc->vacuumFlags; >> +LWLockRelease(ProcArrayLock); > I can't help noticing t

Re: remove spurious CREATE INDEX CONCURRENTLY wait

2020-11-09 Thread Michael Paquier
On Mon, Nov 09, 2020 at 08:32:13PM -0500, Tom Lane wrote: > Do we really need exclusive lock on the ProcArray to make this flag > change? That seems pretty bad from a concurrency standpoint. Any place where we update vacuumFlags acquires an exclusive LWLock on ProcArrayLock. That's held for a ve

Re: remove spurious CREATE INDEX CONCURRENTLY wait

2020-11-09 Thread Tom Lane
Michael Paquier writes: > On Mon, Nov 09, 2020 at 08:32:13PM -0500, Tom Lane wrote: >> Do we really need exclusive lock on the ProcArray to make this flag >> change? That seems pretty bad from a concurrency standpoint. > Any place where we update vacuumFlags acquires an exclusive LWLock on > Pro

Avoiding useless SHA256 initialization with backup manifests, breaking base backups with FIPS

2020-11-09 Thread Michael Paquier
Hi all, Trying to use OpenSSL with FIPS breaks if one attempts to call the low-level SHA2 routines we currently use in sha2_openssl.c (upstream calls that OpenSSLDie()), forcing a crash of PG. The actual way to fix that is to use EVP as I solved here: https://commitfest.postgresql.org/30/2762/ U

Re: remove spurious CREATE INDEX CONCURRENTLY wait

2020-11-09 Thread Alvaro Herrera
On 2020-Nov-09, Tom Lane wrote: > Michael Paquier writes: > > On Mon, Nov 09, 2020 at 08:32:13PM -0500, Tom Lane wrote: > >> Do we really need exclusive lock on the ProcArray to make this flag > >> change? That seems pretty bad from a concurrency standpoint. > > > Any place where we update vacu

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-11-09 Thread Andy Fan
On Tue, Nov 10, 2020 at 7:55 AM David Rowley wrote: > On Tue, 10 Nov 2020 at 12:49, Tom Lane wrote: > > > > Alvaro Herrera writes: > > > Are you taking into account the possibility that generated machine code > > > is a small percent slower out of mere bad luck? I remember someone > > > sugges

Re: remove spurious CREATE INDEX CONCURRENTLY wait

2020-11-09 Thread Michael Paquier
On Mon, Nov 09, 2020 at 11:31:15PM -0300, Alvaro Herrera wrote: > Yeah ... it would be much better if we can make it use atomics instead. > Currently it's an uint8, and in PGPROC itself it's probably not a big > deal to enlarge that, but I fear that quadrupling the size of the > mirroring array in

Re: logical streaming of xacts via test_decoding is broken

2020-11-09 Thread Amit Kapila
On Mon, Nov 9, 2020 at 6:00 PM Dilip Kumar wrote: > > On Mon, Nov 9, 2020 at 5:37 PM Dilip Kumar wrote: > > > > On Mon, Nov 9, 2020 at 4:21 PM Amit Kapila wrote: > > > > The bigger question is do we want to give users an option > > > for skip_empty_streams similar to skip_empty_xacts? I would a

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-11-09 Thread k.jami...@fujitsu.com
> From: k.jami...@fujitsu.com > On Thursday, October 22, 2020 3:15 PM, Kyotaro Horiguchi > wrote: > > I'm not sure about the exact steps of the test, but it can be expected > > if we have many small relations to truncate. > > > > Currently BUF_DROP_FULL_SCAN_THRESHOLD is set to Nbuffers / 512, >

Re: remove spurious CREATE INDEX CONCURRENTLY wait

2020-11-09 Thread Tom Lane
Alvaro Herrera writes: > Yeah ... it would be much better if we can make it use atomics instead. I was thinking more like "do we need any locking at all". Assuming that a proc's vacuumFlags can be set by only the process itself, there's no write conflicts to worry about. On the read side, there

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-11-09 Thread Amit Kapila
On Tue, Nov 10, 2020 at 8:19 AM k.jami...@fujitsu.com wrote: > > > From: k.jami...@fujitsu.com > > On Thursday, October 22, 2020 3:15 PM, Kyotaro Horiguchi > > wrote: > > > I'm not sure about the exact steps of the test, but it can be expected > > > if we have many small relations to truncate. >

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-11-09 Thread Kyotaro Horiguchi
At Tue, 10 Nov 2020 08:33:26 +0530, Amit Kapila wrote in > On Tue, Nov 10, 2020 at 8:19 AM k.jami...@fujitsu.com > wrote: > > > > I repeated the recovery performance test for vacuum. (I made a mistake > > previously in NBuffers/128) > > The 3 kinds of thresholds are almost equally performant.

Re: warn_unused_results

2020-11-09 Thread Michael Paquier
On Mon, Nov 09, 2020 at 08:23:31AM +0100, Peter Eisentraut wrote: > On 2020-11-09 07:56, Michael Paquier wrote: >> This is accepted by clang, and MSVC has visibly an equivalent for >> that, as of VS 2012: >> #elif defined(_MSC_VER) && (_MSC_VER >= 1700) >> #define pg_nodiscard _Check_return_ >> We

Re: list of extended statistics on psql

2020-11-09 Thread Tatsuro Yamada
Hi Tomas, I took a look at this today, and I think the code is ready, but the regression test needs a bit more work: Thanks for taking your time. :-D 1) It's probably better to use somewhat more specific names for the objects, especially when created in public schema. It decreases the chanc

Re: upcoming API changes for LLVM 12

2020-11-09 Thread Andres Freund
Hi, On 2020-11-08 18:22:50 -0500, Tom Lane wrote: > Andres Freund writes: > > Yea, I'll try to do that in the next few days I pushed the change to master. If that doesn't show any problems, I'll backpatch in a week or so. Seawasp runs only on master, so it should satisfy the buildfarm at least.

Re: upcoming API changes for LLVM 12

2020-11-09 Thread Tom Lane
Andres Freund writes: > I pushed the change to master. Thanks! > If that doesn't show any problems, I'll > backpatch in a week or so. Seawasp runs only on master, so it should > satisfy the buildfarm at least. Yeah, sounds like a good plan. FWIW, master builds clean for me.

Re: Refactor MD5 implementations and switch to EVP for OpenSSL

2020-11-09 Thread Michael Paquier
On Fri, Nov 06, 2020 at 04:34:34PM +0900, Michael Paquier wrote: > The attached patch set does a bit of rework to make the Postgres code > more consistent with OpenSSL, similarly to the work I did for all the > SHA2 implementations with EVP in [1]: > - 0001 is something stolen from the SHA2 set, ad

Re: [HACKERS] logical decoding of two-phase transactions

2020-11-09 Thread Masahiko Sawada
On Mon, Nov 9, 2020 at 8:21 PM Amit Kapila wrote: > > On Mon, Nov 9, 2020 at 1:38 PM Masahiko Sawada wrote: > > > > On Mon, Nov 9, 2020 at 3:23 PM Peter Smith wrote: > > > > > > > I've looked at the patches and done some tests. Here is my comment and > > question I realized during testing and re

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-11-09 Thread Thomas Munro
On Sat, Nov 7, 2020 at 12:40 AM Amit Kapila wrote: > I think one of the problems is returning fewer rows and that too > without any warning or error, so maybe that is a bigger problem but we > seem to be okay with it as that is already a known thing though I > think that is not documented anywhere

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-11-09 Thread David Rowley
On Tue, 10 Nov 2020 at 15:38, Andy Fan wrote: > While I have interest about what caused the tiny difference, I admit that > what direction > this patch should go is more important. Not sure if anyone is convinced that > v8 and v9 have a similar performance. The current data show it is similar.

  1   2   >