Refactor MD5 implementations and switch to EVP for OpenSSL

Hi all, Please find attached a patch set to sanitize the use of MD5 we have in the core tree. As of now, there are two duplicated implementations of MD5, one in contrib/pgcrypto/ used as a fallback when not compiling with OpenSSL, and one in src/common/ used by the backend when compiling with *or

Re: libpq compression

> 6 нояб. 2020 г., в 00:22, Peter Eisentraut > написал(а): > > On 2020-11-02 20:50, Andres Freund wrote: >> On 2020-10-31 22:25:36 +0500, Andrey Borodin wrote: >>> But the price of compression is 1 cpu for 500MB/s (zstd). With a >>> 20Gbps network adapters cost of recompressing all traffic is

Re: libpq compression

On 05.11.2020 22:22, Peter Eisentraut wrote: On 2020-11-02 20:50, Andres Freund wrote: On 2020-10-31 22:25:36 +0500, Andrey Borodin wrote: But the price of compression is 1 cpu for 500MB/s (zstd). With a 20Gbps network adapters cost of recompressing all traffic is at most ~4 cores. It's no

Re: libpq compression

On 05.11.2020 21:07, Matthias van de Meent wrote: On Thu, 5 Nov 2020 at 17:01, Konstantin Knizhnik wrote: Sorry, I do not understand your point. This view reports network traffic from server's side. But client's traffic information is "mirror" of this statistic: server_tx=client_rx and visa

Re: [Patch] Optimize dropping of relation buffers using dlist

On Fri, Nov 6, 2020 at 5:09 PM Amit Kapila wrote: > On Fri, Nov 6, 2020 at 5:02 AM Thomas Munro wrote: > > Here's a devil's advocate position I thought about: It's OK to leave > > stray buffers (clean or dirty) in the buffer pool if files are > > truncated underneath us by gremlins, as long as y

Re: default result formats setting

čt 5. 11. 2020 v 21:48 odesílatel Peter Eisentraut < peter.eisentr...@2ndquadrant.com> napsal: > On 2020-10-26 09:45, Pavel Stehule wrote: > > The attached patch implements this. For example, to get int2, int4, > > int8 in binary by default, you could set > > > > SET default_result_fo

Re: Why does to_json take "anyelement" rather than "any"?

pá 6. 11. 2020 v 1:39 odesílatel David G. Johnston < david.g.johns...@gmail.com> napsal: > On Thu, Nov 5, 2020 at 3:43 PM Mohamed Wael Khobalatte < > mkhobala...@grubhub.com> wrote: > >> You can always cast to text yourself, of course, but I am not familiar >> with the type hierarchy enough to te

Re: REFRESH MATERIALIZED VIEW and completion tag output

> On Nov 5, 2020, at 4:45 PM, Fujii Masao wrote: > > > > On 2020/11/06 1:56, Mark Dilger wrote: >>> On Nov 5, 2020, at 8:20 AM, Fujii Masao wrote: >>> >>> The patch that makes pg_stat_statements track the number of rows that >>> REFRESH MATERIALIZED VIEW command processes was proposed at [

Re: Implementing Incremental View Maintenance

On Mon, Oct 05, 2020 at 06:16:18PM +0900, Yugo NAGATA wrote: > Hi, > > Attached is the rebased patch (v18) to add support for Incremental This needs to be rebased again - the last version doesn't apply anymore. http://cfbot.cputube.org/yugo-nagata.html I looked though it a bit and attach some fi

Re: [Patch] Optimize dropping of relation buffers using dlist

On Fri, Nov 6, 2020 at 5:02 AM Thomas Munro wrote: > > On Thu, Nov 5, 2020 at 10:47 PM Amit Kapila wrote: > > I still feel 'cached' is a better name. > > Amusingly, this thread is hitting all the hardest problems in computer > science according to the well known aphorism... > > Here's a devil's a

Re: Dumping/restoring fails on inherited generated column

On Wed, Nov 4, 2020 at 4:23 AM Peter Eisentraut wrote: > > On 2020-08-27 13:30, Masahiko Sawada wrote: > > This issue is not fixed yet. I've attached the updated version patch > > and registered it to commit fest so as not to forget. Please review > > it. > > A few fixes have been committed in thi

Re: Why does to_json take "anyelement" rather than "any"?

On 11/5/20 8:58 PM, Tom Lane wrote: "any" is a dinosaur IMO. It's definitely lower-level than anyelement; for example the function has to be prepared to deal with raw "unknown" literals. So I feel like the proposed solution here is a bit of a hack. I see what you are saying, but since the cod

RE: [Patch] Optimize dropping of relation buffers using dlist

On Thursday, October 22, 2020 3:15 PM, Kyotaro Horiguchi wrote: > I'm not sure about the exact steps of the test, but it can be expected if we > have many small relations to truncate. > > Currently BUF_DROP_FULL_SCAN_THRESHOLD is set to Nbuffers / 512, > which is quite arbitrary that comes from

Re: [HACKERS] logical decoding of two-phase transactions

On Wed, Nov 4, 2020 at 9:31 PM Amit Kapila wrote: > Okay, so it is mentioned in the comment why we need to execute > invalidations even when the database is not the same. So, we are > probably good here if we are executing the invalidations whenever we > skip to decode the prepared xact. Updated

subscribing

subscribing

Re: Add Information during standby recovery conflicts

On Fri, Oct 30, 2020 at 6:02 PM Drouvot, Bertrand wrote: > > Hi, > > On 10/30/20 4:25 AM, Fujii Masao wrote: > > CAUTION: This email originated from outside of the organization. Do > > not click links or open attachments unless you can confirm the sender > > and know the content is safe. > > > > >

RE: pgbench: option delaying queries till connections establishment?

Dear Fabien, > Indeed. I scanned the file but did not find other places that needed > updating. > Yes. > Not sure either. I'm not for having too many braces anyway, so I removed > them. I checked your fixes and I think it's OK. Finally, please move the doc fixes to patch B in order to separ

Re: Why does to_json take "anyelement" rather than "any"?

Nikhil Benesch writes: > On 11/5/20 7:38 PM, David G. Johnston wrote: >> My understanding is that "any" is defined to accept that behavior - allowing >> any pseudo-type and unknown.  The "anyelement" polymorphic pseudo-type is >> defined such that only concrete known types are allowed to match -

Re: Why does to_json take "anyelement" rather than "any"?

On 11/5/20 7:38 PM, David G. Johnston wrote: My understanding is that "any" is defined to accept that behavior - allowing any pseudo-type and unknown.  The "anyelement" polymorphic pseudo-type is defined such that only concrete known types are allowed to match - and then the rules of polymorphi

Re: Add statistics to pg_stat_wal view for wal related parameter tuning

On 2020-10-30 11:50, Fujii Masao wrote: On 2020/10/29 17:03, Masahiro Ikeda wrote: Hi, Thanks for your comments and advice. I updated the patch. On 2020-10-21 18:03, Kyotaro Horiguchi wrote: At Tue, 20 Oct 2020 16:11:29 +0900, Masahiro Ikeda wrote in On 2020-10-20 12:46, Amit Kapila wrote:

Re: REFRESH MATERIALIZED VIEW and completion tag output

On 2020/11/06 1:56, Mark Dilger wrote: On Nov 5, 2020, at 8:20 AM, Fujii Masao wrote: The patch that makes pg_stat_statements track the number of rows that REFRESH MATERIALIZED VIEW command processes was proposed at [1]. When reviewing the patch, I just wondered why the completion tag out

Re: Why does to_json take "anyelement" rather than "any"?

On Thu, Nov 5, 2020 at 3:43 PM Mohamed Wael Khobalatte < mkhobala...@grubhub.com> wrote: > You can always cast to text yourself, of course, but I am not familiar > with the type hierarchy enough to tell why `to_json` can't deduce that as > text whereas the other function can. > My understanding

Re: list_free() in index_get_partition()

On Thu, Nov 05, 2020 at 02:36:06PM -0600, Justin Pryzby wrote: > Seems to be missing. Any code paths calling index_get_partition() is in tablecmds.c, involving commands that are not that hot with such lookups, but that could be an issue if this gets called by some out-of-core code if memory contex

RE: extension patch of CREATE OR REPLACE TRIGGER

Hi, On Thursday, November 5, 2020 1:22 PM Peter Smith wrote: > On Wed, Nov 4, 2020 at 2:53 PM osumi.takami...@fujitsu.com > wrote: > > > (1) COMMENT > > > File: NA > > > Maybe next time consider using format-patch to make the patch. Then > > > you can include a comment to give the background/mo

Re: Move OpenSSL random under USE_OPENSSL_RANDOM

On Thu, Nov 05, 2020 at 01:59:11PM +0100, Daniel Gustafsson wrote: > Not yet, and potentially never will. Given the consequences of a PRNG which > hasn't been properly initialized I think it's ok to be defensive in this > codepath however. + /* +* In case the backend is using the PRNG from

Re: [Patch] Optimize dropping of relation buffers using dlist

On Thu, Nov 5, 2020 at 10:47 PM Amit Kapila wrote: > I still feel 'cached' is a better name. Amusingly, this thread is hitting all the hardest problems in computer science according to the well known aphorism... Here's a devil's advocate position I thought about: It's OK to leave stray buffers

Re: Why does to_json take "anyelement" rather than "any"?

On Tue, Nov 3, 2020 at 11:54 AM Nikhil Benesch wrote: > to_json is declared as taking "anyelement" as input, which means > you can't pass it something of unknown type: > > postgres=# SELECT to_json('foo'); > ERROR: could not determine polymorphic type because input has type > unknown >

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

On Mon, 2 Nov 2020 at 20:43, David Rowley wrote: > > On Tue, 20 Oct 2020 at 22:30, David Rowley wrote: > I did some further tests this time with some tuple deforming. Again, > it does seem that v9 is slower than v8. > > Graphs attached > > Looking at profiles, I don't really see any obvious reas

Re: Collation versioning

On Wed, Nov 4, 2020 at 9:11 PM Michael Paquier wrote: > On Wed, Nov 04, 2020 at 08:44:15AM +0100, Juan José Santamaría Flecha wrote: > > We could create a static table with the conversion based on what was > > discussed for commit a169155, please find attached a spreadsheet with the > > compariso

Re: Reduce the number of special cases to build contrib modules on windows

Thank you for looking at this. On Tue, 3 Nov 2020 at 09:49, Andres Freund wrote: > > diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm > > index 90594bd41b..491a465e2f 100644 > > --- a/src/tools/msvc/Mkvcbuild.pm > > +++ b/src/tools/msvc/Mkvcbuild.pm > > @@ -32,16 +32,13 @@ m

Re: Fix brin_form_tuple to properly detoast data

On 2020-Nov-05, Tomas Vondra wrote: > On 11/5/20 6:17 PM, Alvaro Herrera wrote: > > There are recent changes in vacuum for temp tables (commit 94bc27b57680?) > > that would maybe make this stable enough, without having to have the CIC > > there. At least, I tried it locally a few times and it ap

Re: default result formats setting

On 2020-10-26 15:35, Tom Lane wrote: In the discussion in [0], I pondered a new protocol message for that, but after further thought, a GUC setting would do just as well. I think a GUC is conceptually the wrong level ... It does feel that way, but it gets the job done well and you can use all

Re: default result formats setting

On 2020-10-26 09:45, Pavel Stehule wrote: The attached patch implements this.  For example, to get int2, int4, int8 in binary by default, you could set SET default_result_formats = '21=1,23=1,20=1'; Using SET statement for this case looks very obscure :/ This is a protocol related

list_free() in index_get_partition()

Seems to be missing. The 2nd patch does some more cleanup - Before, a failed syscache lookup would ERROR, but I don't think that's supposed to happen. get_rel_relispartition() would instead return false, and we won't call get_partition_parent(). commit a8ad949f22b8dd7b23049b0b0704e5be9233e319 Au

Re: libpq compression

On 2020-11-02 20:50, Andres Freund wrote: On 2020-10-31 22:25:36 +0500, Andrey Borodin wrote: But the price of compression is 1 cpu for 500MB/s (zstd). With a 20Gbps network adapters cost of recompressing all traffic is at most ~4 cores. It's not quite that simple, because presumably each conn

Re: Move catalog toast table and index declarations

On 2020-11-05 12:59, John Naylor wrote: I think we're talking past eachother. Here's a concrete example: #define BKI_ROWTYPE_OID(oid,oidmacro) #define DECLARE_TOAST(name,toastoid,indexoid) extern int no_such_variable I understand these to be functionally equivalent as far as what the C compile

Re: libpq compression

On Thu, 5 Nov 2020 at 17:01, Konstantin Knizhnik wrote: > > Sorry, I do not understand your point. > This view reports network traffic from server's side. > But client's traffic information is "mirror" of this statistic: > server_tx=client_rx and visa versa. > > Yes, first few bytes exchanged by

Re: Fix brin_form_tuple to properly detoast data

On 11/5/20 6:17 PM, Alvaro Herrera wrote: On 2020-Nov-04, Tomas Vondra wrote: The first test is fairly trivial - it simply builds index on toasted data and then shows how an insert and select fail. There's a caveat, that this requires a DELETE + VACUUM, and the VACUUM actually has to cleanup

Re: Yet another fast GiST build

On Thu, Nov 05, 2020 at 10:11:52PM +0500, Andrey Borodin wrote: > To test that functions are actually called for sorting build we should > support directive sorting build like "CREATE INDEX ON A USING GIST(B) > WITH(sorting=surely,and fail if not)". Maybe you could add a DEBUG1 message for that,

Re: Fix brin_form_tuple to properly detoast data

On 2020-Nov-04, Tomas Vondra wrote: > The first test is fairly trivial - it simply builds index on toasted data > and then shows how an insert and select fail. There's a caveat, that this > requires a DELETE + VACUUM, and the VACUUM actually has to cleanup the rows. > So there must be no concurren

Re: Yet another fast GiST build

> 2 нояб. 2020 г., в 19:45, Heikki Linnakangas написал(а): > >> How do you think, should this patch and patch with pageinspect GiST >> functions be registered on commitfest? > > Yeah, that'd be good. I've registered both patches on January CF. pageinspect patch's code looks goot to me (besid

Re: REFRESH MATERIALIZED VIEW and completion tag output

> On Nov 5, 2020, at 8:20 AM, Fujii Masao wrote: > > The patch that makes pg_stat_statements track the number of rows that > REFRESH MATERIALIZED VIEW command processes was proposed at [1]. > When reviewing the patch, I just wondered why the completion tag output > for REFRESH MATERIALIZED VIE

REFRESH MATERIALIZED VIEW and completion tag output

Hi, The patch that makes pg_stat_statements track the number of rows that REFRESH MATERIALIZED VIEW command processes was proposed at [1]. When reviewing the patch, I just wondered why the completion tag output for REFRESH MATERIALIZED VIEW doesn't include the row count. Is this an intentional be

Re: libpq compression

On 05.11.2020 15:40, Matthias van de Meent wrote: How about Xx_logical_bytes for raw the pg command stream data, and keeping Xx_compressed_bytes for the compressed data in/out? Frankly speaking I do not like work "logical" in this context. It is in any case physical bytes, received from the

Re: redundant error messages

On 2020-Nov-05, Isaac Morland wrote: > In principle, the client knows the database name. In practice, if it's > coming from PGDATABASE or via a service configuration, one may be confused > about the database; having the error message be explicit will avoid many > problems. I can easily imagine tha

Re: How to retain lesser paths at add_path()?

> On Tue, Jan 14, 2020 at 12:46:02AM +0900, Kohei KaiGai wrote: > The v2 patch is attached. > > This adds two dedicated lists on the RelOptInfo to preserve lesser paths > if extension required to retain the path-node to be removed in usual manner. > These lesser paths are kept in the separated list

Re: enable pg_stat_statements to track rows processed by REFRESH MATERIALIZED VIEW

2020-11-02 20:01 に Fujii Masao さんは書きました: On 2020/11/02 14:02, Yuki Seino wrote: The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: tested, passed Documentation:

Re: redundant error messages

On Thu, 5 Nov 2020 at 08:34, Euler Taveira wrote: Is the database name important for this message? You should inform which > database you want to connect for all client tools except pg_dumpall. > Hence, you > already know which database has the connection problem. IMO the pg_dumpall > message sho

Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)

On Wed, Oct 28, 2020 at 02:34:02PM -0500, Justin Pryzby wrote: > On Tue, Sep 08, 2020 at 02:51:26PM -0500, Justin Pryzby wrote: > > On Sat, Jul 18, 2020 at 03:15:32PM -0500, Justin Pryzby wrote: > > > Still waiting for feedback from a committer. > > > > This patch has been waiting for input from a

Re: redundant error messages

On Thu, 5 Nov 2020 at 09:27, Peter Eisentraut < peter.eisentr...@enterprisedb.com> wrote: > A few client tools duplicate error messages already provided by libpq, > such as > > pg_rewind: fatal: could not connect to server: could not connect to > server: No such file or directory > > Good catch!

RE: Parallel copy

Hi > > my $bytes = $ARGV[0]; > for(my $i = 0; $i < $bytes; $i+=8){ > print "longdata"; > } > print "\n"; > > > postgres=# copy longdata from program 'perl /tmp/longdata.pl 1' > with (parallel 2); > > This gets stuck forever (or at least I didn't have the patience to wait >

Re: Move OpenSSL random under USE_OPENSSL_RANDOM

> On 5 Nov 2020, at 13:28, Michael Paquier wrote: > It seems to me that this one would become incorrect if compiling with > OpenSSL but select a random source that requires an initialization, as > it would enforce only OpenSSL initialization all the time. Right, how about something like the atta

Re: Move OpenSSL random under USE_OPENSSL_RANDOM

On Thu, Nov 5, 2020 at 1:28 PM Michael Paquier wrote: > > On Thu, Nov 05, 2020 at 01:18:15PM +0100, Daniel Gustafsson wrote: > > What about the (hypothetical) situation where USE_OPENSSL_RANDOM is used > > without USE_OPENSSL? Wouldn't the below make sure we cover all bases? > > You can actually t

Re: ModifyTable overheads in generic plans

On Wed, Nov 4, 2020 at 11:32 AM Amit Langote wrote: > On Tue, Nov 3, 2020 at 9:05 PM Heikki Linnakangas wrote: > > A) We could change FDW API so that BeginDirectModify takes the same > > arguments as BeginForeignModify(). That avoids the assumption that it's > > a ForeignScan node, because BeginF

Re: libpq compression

On Mon, 2 Nov 2020 at 20:20, Konstantin Knizhnik wrote: > > On 02.11.2020 19:53, Matthias van de Meent wrote: > > This is the result of network traffic of two backends one with enabled > >> compression and another with disable compression > >> after execution of "select * from pg_class" command: >

Re: Move OpenSSL random under USE_OPENSSL_RANDOM

On Thu, Nov 05, 2020 at 01:18:15PM +0100, Daniel Gustafsson wrote: > What about the (hypothetical) situation where USE_OPENSSL_RANDOM is used > without USE_OPENSSL? Wouldn't the below make sure we cover all bases? You can actually try that combination, because it is possible today to compile witho

redundant error messages

A few client tools duplicate error messages already provided by libpq, such as pg_rewind: fatal: could not connect to server: could not connect to server: No such file or directory pg_basebackup: error: could not connect to server: could not connect to server: No such file or directory psq

Re: Move OpenSSL random under USE_OPENSSL_RANDOM

> On 5 Nov 2020, at 13:12, Michael Paquier wrote: > > On Thu, Nov 05, 2020 at 10:49:45AM +0100, Daniel Gustafsson wrote: >> This must check for USE_OPENSSL as well as per my original patch, since we'd >> otherwise fail to perform post-fork initialization in case one use OpenSSL >> with >> anothe

Re: Move OpenSSL random under USE_OPENSSL_RANDOM

On Thu, Nov 05, 2020 at 10:49:45AM +0100, Daniel Gustafsson wrote: > This must check for USE_OPENSSL as well as per my original patch, since we'd > otherwise fail to perform post-fork initialization in case one use OpenSSL > with > anothe PRNG for pg_strong_random. That might be theoretical at th

Re: Move catalog toast table and index declarations

On Thu, Nov 5, 2020 at 4:24 AM Peter Eisentraut < peter.eisentr...@2ndquadrant.com> wrote: > On 2020-10-27 13:12, John Naylor wrote: > > There's nothing wrong; it's just a minor point of consistency. For the > > first part, I mean defined symbols in this file that are invisible to > > the C compil

Re: Multi Inserts in CREATE TABLE AS - revived patch

On Tue, Nov 3, 2020 at 4:54 PM Bharath Rupireddy < bharath.rupireddyforpostg...@gmail.com> wrote: > > Use case 1- 100mn tuples, 2 integer columns, exec time in sec: > HEAD: 131.507 when the select part is not parallel, 128.832 when the select part is parallel > Patch: 98.925 when the select part is

Re: Move OpenSSL random under USE_OPENSSL_RANDOM

> On 5 Nov 2020, at 04:44, Michael Paquier wrote: > > On Wed, Nov 04, 2020 at 10:05:48AM +0100, Magnus Hagander wrote: >> Yes, we should absolutely do that. We already do this for >> pg_strong_random() itself, and we should definitely repeat the pattern >> in the init function. > > This poked at

Re: [Patch] Optimize dropping of relation buffers using dlist

On Thu, Nov 5, 2020 at 1:59 PM Kyotaro Horiguchi wrote: > > At Thu, 5 Nov 2020 11:07:21 +0530, Amit Kapila > wrote in > > On Thu, Nov 5, 2020 at 8:26 AM k.jami...@fujitsu.com > > wrote: > > > > > Few comments on patches: > > > > > == > > > > > v29-0002-Add-bool-param-in-smgr

Re: Some doubious code in pgstat.c

On Thu, Nov 5, 2020 at 2:13 PM Kyotaro Horiguchi wrote: > > At Thu, 5 Nov 2020 11:48:24 +0530, Amit Kapila > wrote in > > On Thu, Nov 5, 2020 at 9:44 AM Masahiko Sawada > > wrote: > > > > > > On Thu, Nov 5, 2020 at 11:18 AM Kyotaro Horiguchi > > > wrote: > > > > As another issue, just replace

Re: Protect syscache from bloating with negative cache entries

On 19/11/2019 12:48, Kyotaro Horiguchi wrote: 1. Inserting a branch in SearchCatCacheInternal. (CatCache_Pattern_1.patch) This is the most straightforward way to add an alternative feature. pattern 1 | 8459.73 | 28.15 # 9% (>> 1%) slower than 7757.58 pattern 1 | 8504.83 | 55.61 pattern 1 |

Re: overriding current_timestamp

so I used this method here: https://gist.github.com/thehesiod/d0314d599c721216f075375c667e2d9a and indeed does not work for current_timestamp and the like functions, and there's another major problems, it doesn't seem to work f

Re: Some doubious code in pgstat.c

At Thu, 5 Nov 2020 11:48:24 +0530, Amit Kapila wrote in > On Thu, Nov 5, 2020 at 9:44 AM Masahiko Sawada wrote: > > > > On Thu, Nov 5, 2020 at 11:18 AM Kyotaro Horiguchi > > wrote: > > > As another issue, just replace memcpy with strlcpy makes compiler > > > complain of type mismatch, as the f

Re: [PoC] Non-volatile WAL buffer

Hi Gang, I appreciate your patience. I reproduced the results you reported to me, on my environment. First of all, the condition you gave to me was a little unstable on my environment, so I made the values of {max_,min_,nv}wal_size larger and the pre-warm duration longer to get stable performance

Re: [Patch] Optimize dropping of relation buffers using dlist

At Thu, 5 Nov 2020 11:07:21 +0530, Amit Kapila wrote in > On Thu, Nov 5, 2020 at 8:26 AM k.jami...@fujitsu.com > wrote: > > > > Few comments on patches: > > > > == > > > > v29-0002-Add-bool-param-in-smgrnblocks-for-cached-blocks > > > > --

Re: Move catalog toast table and index declarations

On 2020-10-27 13:12, John Naylor wrote: There's nothing wrong; it's just a minor point of consistency. For the first part, I mean defined symbols in this file that are invisible to the C compiler are written #define SOMETHING() If some are written #define SOMETHING() extern int no_such_varia