Re: explicit_bzero for sslpassword

2020-05-19 Thread Michael Paquier
On Tue, May 19, 2020 at 02:33:40PM +0200, Daniel Gustafsson wrote: > Since commit 74a308cf5221f we use explicit_bzero on pgpass and connhost > password in libpq, but not sslpassword which seems an oversight. The attached > performs an explicit_bzero before freeing like the pattern for other passwo

Re: SyncRepLock acquired exclusively in default configuration

2020-05-19 Thread Michael Paquier
On Tue, May 19, 2020 at 08:56:13AM -0700, Ashwin Agrawal wrote: > Sure, add it to commit fest. > Seems though it should be backpatched to relevant branches as well. It does not seem to be listed yet. Are you planning to add it under the section for bug fixes? -- Michael signature.asc Descriptio

Re: Problem with pg_atomic_compare_exchange_u64 at 32-bit platformwd

2020-05-19 Thread Andres Freund
Hi, On May 19, 2020 8:05:00 PM PDT, Noah Misch wrote: >On Tue, May 19, 2020 at 04:07:29PM +0300, Konstantin Knizhnik wrote: >> Definition of pg_atomic_compare_exchange_u64 requires alignment of >expected >> pointer on 8-byte boundary. >> >> pg_atomic_compare_exchange_u64(volatile pg_atomic_uint

Re: MultiXact\SLRU buffers configuration

2020-05-19 Thread Kyotaro Horiguchi
At Fri, 15 May 2020 14:01:46 +0500, "Andrey M. Borodin" wrote in > > > > 15 мая 2020 г., в 05:03, Kyotaro Horiguchi > > написал(а): > > > > At Thu, 14 May 2020 11:44:01 +0500, "Andrey M. Borodin" > > wrote in > >>> GetMultiXactIdMembers believes that 4 is successfully done if 2 > >>> ret

Re: Is it useful to record whether plans are generic or custom?

2020-05-19 Thread Kyotaro Horiguchi
At Tue, 19 May 2020 22:56:17 +0900, Atsushi Torikoshi wrote in > On Sat, May 16, 2020 at 6:01 PM legrand legrand > wrote: > > > > To track executed plan types, I think execution layer hooks > > > are appropriate. > > > These hooks, however, take QueryDesc as a param and it does > > > not inclu

Re: Trouble with hashagg spill I/O pattern and costing

2020-05-19 Thread Jeff Davis
On Tue, 2020-05-19 at 19:53 +0200, Tomas Vondra wrote: > > And if there a way to pre-allocate larger chunks? Presumably we could > assign the blocks to tape in larger chunks (e.g. 128kB, i.e. 16 x > 8kB) > instead of just single block. I haven't seen anything like that in > tape.c, though ... It

Re: Warn when parallel restoring a custom dump without data offsets

2020-05-19 Thread Justin Pryzby
I started fooling with this at home while our ISP is broke (pardon my brevity). Maybe you also saw commit b779ea8a9a2dc3a089b3ac152b1ec4568bfeb26f "Fix pg_restore so parallel restore doesn't fail when the input file doesn't contain data offsets (which it won't, if pg_dump thought its output wasn't

Re: Parallel Seq Scan vs kernel read ahead

2020-05-19 Thread Thomas Munro
On Wed, May 20, 2020 at 2:23 PM Amit Kapila wrote: > Good experiment. IIRC, we have discussed a similar idea during the > development of this feature but we haven't seen any better results by > allocating in ranges on the systems we have tried. So, we want with > the current approach which is mo

Re: Problem with pg_atomic_compare_exchange_u64 at 32-bit platformwd

2020-05-19 Thread Noah Misch
On Tue, May 19, 2020 at 04:07:29PM +0300, Konstantin Knizhnik wrote: > Definition of pg_atomic_compare_exchange_u64 requires alignment of expected > pointer on 8-byte boundary. > > pg_atomic_compare_exchange_u64(volatile pg_atomic_uint64 *ptr, >                                uint64 *expected, uin

Re: Parallel Seq Scan vs kernel read ahead

2020-05-19 Thread Amit Kapila
On Wed, May 20, 2020 at 7:24 AM Thomas Munro wrote: > > Hello hackers, > > Parallel sequential scan relies on the kernel detecting sequential > access, but we don't make the job easy. The resulting striding > pattern works terribly on strict next-block systems like FreeBSD UFS, > and degrades rap

Parallel Seq Scan vs kernel read ahead

2020-05-19 Thread Thomas Munro
Hello hackers, Parallel sequential scan relies on the kernel detecting sequential access, but we don't make the job easy. The resulting striding pattern works terribly on strict next-block systems like FreeBSD UFS, and degrades rapidly when you add too many workers on sliding window systems like

Re: pg_stat_wal_receiver and flushedUpto/writtenUpto

2020-05-19 Thread Fujii Masao
On 2020/05/20 8:31, Michael Paquier wrote: On Tue, May 19, 2020 at 11:38:52PM +0900, Fujii Masao wrote: I found that "received_lsn" is still used in high-availability.sgml. We should apply the following change in high-availability? - view's received_lsn indicates that WAL is being +

Re: factorial function/phase out postfix operators?

2020-05-19 Thread Tom Lane
I wrote: > However, we do have to have a benefit to show those people whose > queries we break. Hence my insistence on having a working AS fix > (or some other benefit) before not after. I experimented with this a bit more, and came up with the attached. It's not a working patch, just a set of gr

Re: pg_stat_wal_receiver and flushedUpto/writtenUpto

2020-05-19 Thread Michael Paquier
On Tue, May 19, 2020 at 11:38:52PM +0900, Fujii Masao wrote: > I found that "received_lsn" is still used in high-availability.sgml. > We should apply the following change in high-availability? > > - view's received_lsn indicates that WAL is being > + view's flushed_lsn indicates that WAL i

Re: Extension ownership and misuse of SET ROLE/SET SESSION AUTHORIZATION

2020-05-19 Thread Daniel Gustafsson
> On 19 May 2020, at 17:34, Tom Lane wrote: > > Daniel Gustafsson writes: >>> On 13 Feb 2020, at 23:55, Tom Lane wrote: >>> Given the current behavior of SET ROLE and SET SESSION AUTHORIZATION, >>> I don't actually see any way that we could get these features to >>> play together. > >> Is this

Re: Two fsync related performance issues?

2020-05-19 Thread Thomas Munro
On Wed, May 20, 2020 at 12:51 AM Robert Haas wrote: > On Mon, May 11, 2020 at 8:43 PM Paul Guo wrote: > > I have this concern since I saw an issue in a real product environment that > > the startup process needs 10+ seconds to start wal replay after relaunch > > due to elog(PANIC) (it was seen

Re: factorial function/phase out postfix operators?

2020-05-19 Thread Tom Lane
Robert Haas writes: > On Tue, May 19, 2020 at 2:30 PM Tom Lane wrote: >> Anyway, the bottom-line conclusion remains the same: let's make sure >> we know what we'd do after getting rid of postfix ops, before we do >> that. > Well, I don't think we really need to get too conservative here. > ... I

Re: factorial function/phase out postfix operators?

2020-05-19 Thread Robert Haas
On Tue, May 19, 2020 at 2:30 PM Tom Lane wrote: > Might work. My main concern would be if we have to forbid those keywords > as column names --- for words like "year", in particular, that'd be a > disaster. If the net effect is only that they can't be AS-less col labels, > it won't break any cas

Re: factorial function/phase out postfix operators?

2020-05-19 Thread Tom Lane
Robert Haas writes: > On Tue, May 19, 2020 at 11:32 AM Tom Lane wrote: >> Before we go much further on this, we should have some proof >> that there's actually material benefit to be gained. I spent some >> time just now trying to relax the AS restriction by ripping out >> postfix ops, and the r

Re: factorial function/phase out postfix operators?

2020-05-19 Thread Robert Haas
On Tue, May 19, 2020 at 11:32 AM Tom Lane wrote: > Before we go much further on this, we should have some proof > that there's actually material benefit to be gained. I spent some > time just now trying to relax the AS restriction by ripping out > postfix ops, and the results were not too promisi

Re: Trouble with hashagg spill I/O pattern and costing

2020-05-19 Thread Tomas Vondra
On Tue, May 19, 2020 at 09:27:34AM -0700, Jeff Davis wrote: On Tue, 2020-05-19 at 17:12 +0200, Tomas Vondra wrote: I think there are two related problem - with costing and with excessive I/O due to using logical tapes. Thank you for the detailed analysis. I am still digesting this information.

Re: PostgreSQL 13 Beta 1 Release Announcement Draft

2020-05-19 Thread Lukas Fittl
On Mon, May 18, 2020 at 7:29 PM Jonathan S. Katz wrote: > Attached is a draft of the release announcement for the PostgreSQL 13 > Beta 1 release this week. > We could call out the additional commits that Tom has done for wait event renaming, re: compatibility - next to "Rename some recovery-rela

Re: Trouble with hashagg spill I/O pattern and costing

2020-05-19 Thread Jeff Davis
On Tue, 2020-05-19 at 17:12 +0200, Tomas Vondra wrote: > I think there are two related problem - with costing and with > excessive > I/O due to using logical tapes. Thank you for the detailed analysis. I am still digesting this information. > This kinda makes me question whether logical tapes are

Re: SyncRepLock acquired exclusively in default configuration

2020-05-19 Thread Ashwin Agrawal
On Mon, May 18, 2020 at 7:41 PM Masahiko Sawada < masahiko.saw...@2ndquadrant.com> wrote: > This item is for PG14, right? If so I'd like to add this item to the > next commit fest. > Sure, add it to commit fest. Seems though it should be backpatched to relevant branches as well.

Re: Missing grammar production for WITH TIES

2020-05-19 Thread Alvaro Herrera
On 2020-May-19, Tom Lane wrote: > Yeah, that would have been better per project protocol: if a tarball > re-wrap becomes necessary then it would be messy not to include this > change along with fixing whatever urgent bug there might be. > > However, I thought the case for delaying this fix till p

Re: Extension ownership and misuse of SET ROLE/SET SESSION AUTHORIZATION

2020-05-19 Thread Tom Lane
Daniel Gustafsson writes: >> On 13 Feb 2020, at 23:55, Tom Lane wrote: >> Given the current behavior of SET ROLE and SET SESSION AUTHORIZATION, >> I don't actually see any way that we could get these features to >> play together. > Is this being worked on for the 13 cycle such that it should be

Re: factorial function/phase out postfix operators?

2020-05-19 Thread Tom Lane
Vik Fearing writes: > I'm -1 on keeping ! around as a hard-coded postfix operator. Before we go much further on this, we should have some proof that there's actually material benefit to be gained. I spent some time just now trying to relax the AS restriction by ripping out postfix ops, and the r

Re: Extension ownership and misuse of SET ROLE/SET SESSION AUTHORIZATION

2020-05-19 Thread Daniel Gustafsson
> On 13 Feb 2020, at 23:55, Tom Lane wrote: Is this being worked on for the 13 cycle such that it should be an open item? > Given the current behavior of SET ROLE and SET SESSION AUTHORIZATION, > I don't actually see any way that we could get these features to > play together. SET SESSION AUTHO

Re: factorial function/phase out postfix operators?

2020-05-19 Thread Stephen Frost
Greetings, * Vik Fearing (v...@postgresfriends.org) wrote: > On 5/19/20 4:03 AM, Tom Lane wrote: > > Peter Eisentraut writes: > >> What are the thoughts about then marking the postfix operator deprecated > >> and eventually removing it? > > > > If we do this it'd require a plan. We'd have to a

Re: factorial function/phase out postfix operators?

2020-05-19 Thread Tom Lane
Robert Haas writes: > The ambiguity doesn't come from the mere existence of postfix > operators. It comes from the fact that, when we lex the input, we > can't tell whether a particular operator that we happen to encounter > is prefix, infix, or postfix. So hard-coding, for example, a rule that >

Re: Performance penalty when requesting text values in binary format

2020-05-19 Thread Jack Christensen
On Mon, May 18, 2020 at 7:07 AM Laurenz Albe wrote: > Did you profile your benchmark? > It would be interesting to know where the time is spent. > Unfortunately, I have not. Fortunately, it appears that Tom Lane recognized this as a part of another issue and has prepared a patch. https://www.po

Re: factorial function/phase out postfix operators?

2020-05-19 Thread Robert Haas
On Tue, May 19, 2020 at 10:36 AM Vik Fearing wrote: > So if I make a complex UDT where a NOT operator makes a lot of sense[*], > why wouldn't I be allowed to make a prefix operator ! for it? All for > what? That one person in the corner over there who doesn't want to > rewrite their query to use

Re: pg_stat_wal_receiver and flushedUpto/writtenUpto

2020-05-19 Thread Fujii Masao
On 2020/05/17 10:08, Michael Paquier wrote: On Sat, May 16, 2020 at 10:15:47AM +0900, Michael Paquier wrote: Thanks. If there are no objections, I'll revisit that tomorrow and apply it with those changes, just in time for beta1. Okay, done this part then. I found that "received_lsn" is s

Re: factorial function/phase out postfix operators?

2020-05-19 Thread Vik Fearing
On 5/19/20 4:22 PM, Robert Haas wrote: > On Tue, May 19, 2020 at 9:51 AM Tom Lane wrote: >> Uh ... what exactly would be the point of that? The real reason to do >> this at all is not that we have it in for '!', but that we want to >> drop the possibility of postfix operators from the grammar alt

Re: ldap tls test fails in some environments

2020-05-19 Thread Christoph Berg
Re: Thomas Munro > In your transcript for test 20, it looks like the client (PostgreSQL) > is hanging up without even sending a TLS ClientHello: Maybe tests 19 and 20 are failing because 18 was already bad. (But probably not.) > I wonder how to figure out why... does this tell you anything? > >

Re: factorial function/phase out postfix operators?

2020-05-19 Thread Robert Haas
On Tue, May 19, 2020 at 9:51 AM Tom Lane wrote: > Uh ... what exactly would be the point of that? The real reason to do > this at all is not that we have it in for '!', but that we want to > drop the possibility of postfix operators from the grammar altogether, > which will remove a boatload of a

Re: Is it useful to record whether plans are generic or custom?

2020-05-19 Thread Atsushi Torikoshi
On Sat, May 16, 2020 at 6:01 PM legrand legrand wrote: > > To track executed plan types, I think execution layer hooks > > are appropriate. > > These hooks, however, take QueryDesc as a param and it does > > not include cached plan information. > > It seems that the same QueryDesc entry is reused

Re: factorial function/phase out postfix operators?

2020-05-19 Thread Tom Lane
Robert Haas writes: > I think it's generally a good idea, though perhaps we should consider > continuing to allow '!' as a postfix operator and just removing > support for any other. Uh ... what exactly would be the point of that? The real reason to do this at all is not that we have it in for '

Re: Expand the use of check_canonical_path() for more GUCs

2020-05-19 Thread Tom Lane
Michael Paquier writes: > While digging into my backlog, I have found this message from Peter E > mentioning about $subject: > https://www.postgresql.org/message-id/e6aac026-174c-9952-689f-6bee76f9a...@2ndquadrant.com > It seems to me that it would be a good idea to make those checks more > consi

Re: factorial function/phase out postfix operators?

2020-05-19 Thread Kenneth Marshall
> > I won't lose a lot of sleep if we decide to rip out '!' as well, but I > don't think that continuing to support it would cost us much. > +1 for keeping ! and nuking the rest, if possible. Regards, Ken

Re: Missing grammar production for WITH TIES

2020-05-19 Thread Tom Lane
Michael Paquier writes: > On Tue, May 19, 2020 at 12:41:39AM -0400, Tom Lane wrote: >> Michael Paquier writes: >>> This has been committed just after beta1 has been stamped. So it >>> means that it won't be included in it, right? >> Right. > Still, wouldn't it be better to wait until the versi

Problem with pg_atomic_compare_exchange_u64 at 32-bit platformwd

2020-05-19 Thread Konstantin Knizhnik
Definition of pg_atomic_compare_exchange_u64 requires alignment of expected pointer on 8-byte boundary. pg_atomic_compare_exchange_u64(volatile pg_atomic_uint64 *ptr,                                uint64 *expected, uint64 newval) { #ifndef PG_HAVE_ATOMIC_U64_SIMULATION     AssertPointerAlignmen

Re: Warn when parallel restoring a custom dump without data offsets

2020-05-19 Thread Justin Pryzby
On Sat, May 16, 2020 at 04:57:46PM -0400, David Gilman wrote: > If pg_dump can't seek on its output stream when writing a dump in the > custom archive format (possibly because you piped its stdout to a file) > it can't update that file with data offsets. These files will often > break parallel rest

Re: some grammar refactoring

2020-05-19 Thread Robert Haas
On Tue, May 19, 2020 at 2:43 AM Peter Eisentraut wrote: > Here is a series of patches to do some refactoring in the grammar around > the commands COMMENT, DROP, SECURITY LABEL, and ALTER EXTENSION ... > ADD/DROP. In the grammar, these commands (with some exceptions) > basically just take a refere

Re: Two fsync related performance issues?

2020-05-19 Thread Robert Haas
On Mon, May 11, 2020 at 8:43 PM Paul Guo wrote: > I have this concern since I saw an issue in a real product environment that > the startup process needs 10+ seconds to start wal replay after relaunch due > to elog(PANIC) (it was seen on postgres based product Greenplum but it is a > common iss

Re: factorial function/phase out postfix operators?

2020-05-19 Thread Pavel Stehule
út 19. 5. 2020 v 14:27 odesílatel Robert Haas napsal: > On Mon, May 18, 2020 at 10:42 AM Peter Eisentraut > wrote: > > What are the thoughts about then marking the postfix operator deprecated > > and eventually removing it? > > I wrote a little bit about this last year: > > > http://postgr.es/m/

explicit_bzero for sslpassword

2020-05-19 Thread Daniel Gustafsson
Since commit 74a308cf5221f we use explicit_bzero on pgpass and connhost password in libpq, but not sslpassword which seems an oversight. The attached performs an explicit_bzero before freeing like the pattern for other password variables. cheers ./daniel sslpassword_bzero.patch Description: Bi

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-05-19 Thread Amit Kapila
On Fri, May 15, 2020 at 2:48 PM Dilip Kumar wrote: > > On Wed, May 13, 2020 at 4:50 PM Amit Kapila wrote: > > > > > 3. > > And, during catalog scan we can check the status of the xid and > > + * if it is aborted we will report a specific error that we can ignore. We > > + * might have already st

Re: factorial function/phase out postfix operators?

2020-05-19 Thread Robert Haas
On Mon, May 18, 2020 at 10:42 AM Peter Eisentraut wrote: > What are the thoughts about then marking the postfix operator deprecated > and eventually removing it? I wrote a little bit about this last year: http://postgr.es/m/CA+TgmoarLfSQcLCh7jx0737SZ28qwbuy+rUWT6rSHAO=b-6...@mail.gmail.com I th

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-05-19 Thread Amit Kapila
On Fri, May 15, 2020 at 2:47 PM Dilip Kumar wrote: > > On Tue, May 12, 2020 at 4:39 PM Amit Kapila wrote: > > > > > 4. > > +static void > > +stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn) > > +{ > > + LogicalDecodingContext *ctx = cache->private_data; > > + LogicalErrorCallb

Re: Add A Glossary

2020-05-19 Thread Peter Eisentraut
On 2020-05-19 08:17, Laurenz Albe wrote: The term "cluster" is unfortunate, because to most people it suggests a group of machines, so the term "instance" is better, but that ship has sailed long ago. I don't see what would stop us from renaming some things, with some care. -- Peter Eisentraut

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-05-19 Thread Amit Kapila
On Tue, May 19, 2020 at 3:31 PM Dilip Kumar wrote: > > On Tue, May 19, 2020 at 2:34 PM Amit Kapila wrote: > > > > On Mon, May 18, 2020 at 5:57 PM Amit Kapila wrote: > > > > > > > > > 3. > > > + /* > > > + * If streaming is enable and we have serialized this transaction because > > > + * it had i

Re: Expand the use of check_canonical_path() for more GUCs

2020-05-19 Thread Peter Eisentraut
On 2020-05-19 09:09, Michael Paquier wrote: While digging into my backlog, I have found this message from Peter E mentioning about $subject: https://www.postgresql.org/message-id/e6aac026-174c-9952-689f-6bee76f9a...@2ndquadrant.com It seems to me that it would be a good idea to make those checks

Re: Optimizer docs typos

2020-05-19 Thread Etsuro Fujita
On Mon, May 18, 2020 at 7:45 PM Richard Guo wrote: > In this same README doc, another suspicious typo to me, which happens in > section "Optimizer Functions", is in the prefix to query_planner(), > we should have three dashes, rather than two, since query_planner() is > called within grouping_plan

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-05-19 Thread Dilip Kumar
On Tue, May 19, 2020 at 2:34 PM Amit Kapila wrote: > > On Mon, May 18, 2020 at 5:57 PM Amit Kapila wrote: > > > > > > 3. > > + /* > > + * If streaming is enable and we have serialized this transaction because > > + * it had incomplete tuple. So if now we have got the complete tuple we > > + * ca

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-05-19 Thread Amit Kapila
On Mon, May 18, 2020 at 5:57 PM Amit Kapila wrote: > > > 3. > + /* > + * If streaming is enable and we have serialized this transaction because > + * it had incomplete tuple. So if now we have got the complete tuple we > + * can stream it. > + */ > + if (ReorderBufferCanStream(rb) && can_stream &

RE: PostgresSQL project

2020-05-19 Thread Luke Porter
Hi Peter Thanks for the prompt response. Yes, this is an open source project which will be shared with the community. We will also consider hiring appropriate consultants. At a summary level, we have a proven approach for how a relational database can provide comprehensive logical insert, upda

Expand the use of check_canonical_path() for more GUCs

2020-05-19 Thread Michael Paquier
Hi all, While digging into my backlog, I have found this message from Peter E mentioning about $subject: https://www.postgresql.org/message-id/e6aac026-174c-9952-689f-6bee76f9a...@2ndquadrant.com It seems to me that it would be a good idea to make those checks more consistent, and attached is a p