Re: proposal \gcsv

2020-04-01 Thread Vik Fearing
On 4/1/20 6:29 PM, Tom Lane wrote: > I'd modify my first proposal so far as to make it > > \g ( pset-option pset-value ... ) filename-or-pipe > > That is, require spaces around the parens I think requiring spaces inside the parentheses is a severe POLA violation and I vote strongly against

Re: SLRU statistics

2020-04-01 Thread Tomas Vondra
Hi, I've pushed this after some minor cleanup and improvements. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: snapshot too old issues, first around wraparound and then more.

2020-04-01 Thread Andres Freund
Hi, On 2020-04-01 16:59:51 -0700, Andres Freund wrote: > The primary issue here is that there is no TestForOldSnapshot() in > heap_hot_search_buffer(). Therefore index fetches will never even try to > detect that tuples it needs actually have already been pruned away. bitmap heap scan doesn't hav

Re: snapshot too old issues, first around wraparound and then more.

2020-04-01 Thread Peter Geoghegan
On Wed, Apr 1, 2020 at 5:54 PM Andres Freund wrote: > As far as I can tell there's not sufficient in-tree explanation of when > code needs to test for an old snapshot. There's just the following > comment above TestForOldSnapshot(): > * Check whether the given snapshot is too old to have safely r

Re: Add A Glossary

2020-04-01 Thread Justin Pryzby
On Tue, Mar 31, 2020 at 03:26:02PM -0400, Corey Huinker wrote: > Just so I can prioritize my work, which of these things, along with your > suggestions in previous emails, would you say is a barrier to considering > this ready for a committer? To answer your off-list inquiry, I'm not likely to mar

Re: control max length of parameter values logged

2020-04-01 Thread Justin Pryzby
Thanks for updating the patch. On Thu, Apr 02, 2020 at 01:29:04AM +0100, Alexey Bashtanov wrote: > +If greater than zero, bind parameter values reported in non-error > +statement-logging messages are trimmed to no more than this many > bytes. > +If this value is specified

Re: recovery_target_action=pause with confusing hint

2020-04-01 Thread movead...@highgo.ca
>So I'd like to propose the attached patch. The patch changes the message >logged when a promotion is requested, based on whether the recovery is >in paused state or not. It is a compromise, we should notice it in document I think. Regards, Highgo Software (Canada/China/Pakistan) URL : www.hi

Re: Add A Glossary

2020-04-01 Thread Alvaro Herrera
On 2020-Apr-01, Justin Pryzby wrote: > planner/optimizer: ... I propose we define "planner" and make "optimizer" a entry. I further propose not to define the term "normalized", at least not for now. That seems a very deep rabbit hole. -- Álvaro Herrerahttps://www.2ndQuadrant.

Re: snapshot too old issues, first around wraparound and then more.

2020-04-01 Thread Andres Freund
Hi, On 2020-04-01 17:54:06 -0700, Andres Freund wrote: > * Check whether the given snapshot is too old to have safely read the given > * page from the given table. If so, throw a "snapshot too old" error. > * > * This test generally needs to be performed after every BufferGetPage() call > *

RE: SLRU statistics

2020-04-01 Thread Shinoda, Noriyoshi (PN Japan A&PS Delivery)
Hi, Thank you for developing great features. The attached patch is a small fix to the committed documentation for the data type name of blks_hit column. Best regards, Noriyoshi Shinoda -Original Message- From: Tomas Vondra [mailto:tomas.von...@2ndquadrant.com] Sent: Thursday, April 2,

Re: Planning counters in pg_stat_statements (using pgss_store)

2020-04-01 Thread Fujii Masao
On 2020/04/01 18:19, Fujii Masao wrote: On 2020/04/01 3:42, Julien Rouhaud wrote: On Wed, Apr 01, 2020 at 02:43:10AM +0900, Fujii Masao wrote: On 2020/03/31 16:33, Julien Rouhaud wrote: v12 attached! Thanks for updating the patch! The patch looks good to me. I applied minor and cosm

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-01 Thread Tomas Vondra
On Wed, Apr 01, 2020 at 10:09:20PM -0400, James Coleman wrote: On Wed, Apr 1, 2020 at 5:42 PM Tomas Vondra wrote: ... I've realized the way get_useful_pathkeys_for_relation() is coded kinda works against the fastpath we added when comparing pathkeys. That depends on comparing pointers to the li

Re: pg_stat_statements issue with parallel maintenance (Was Re: WAL usage calculation patch)

2020-04-01 Thread Amit Kapila
Adding Peter G. On Wed, Apr 1, 2020 at 12:41 PM Dilip Kumar wrote: > > I have done some testing for the parallel "create index". > > postgres[99536]=# show maintenance_work_mem ; > maintenance_work_mem > -- > 1MB > (1 row) > > CREATE TABLE test (a int, b int); > INSERT INTO

Re: pg_stat_statements issue with parallel maintenance (Was Re: WAL usage calculation patch)

2020-04-01 Thread Peter Geoghegan
On Wed, Apr 1, 2020 at 7:52 PM Amit Kapila wrote: > Peter, Is this behavior expected? > > Let me summarize the situation so that it would be easier for Peter to > comment. Julien has noticed that parallel vacuum and parallel create > index doesn't seem to report correct values for buffer usage st

Re: [HACKERS] advanced partition matching algorithm for partition-wise join

2020-04-01 Thread Etsuro Fujita
Hi Tomas and Ashutosh, On Thu, Apr 2, 2020 at 1:51 AM Ashutosh Bapat wrote: > On Thu, 26 Mar 2020 at 05:47, Tomas Vondra > wrote: >> three more comments after eye-balling the code for a bit longer. >> >> 1) The patch probably needs to tweak config.sgml which says this about >> the enable_parti

Re: Berserk Autovacuum (let's save next Mandrill)

2020-04-01 Thread Tom Lane
Dean Rasheed writes: > Yeah, that makes sense. I still can't see what might be causing those > failures. The tests that were doing an ALTER COLUMN and then expecting > to see the results of a non-analysed table ought to be fixed by > 0936d1b6f, but that doesn't match the buildfarm failures. Possib

Re: Should we add xid_current() or a int8->xid cast?

2020-04-01 Thread Thomas Munro
On Sat, Mar 21, 2020 at 11:14 AM Thomas Munro wrote: > * updated OIDs to avoid collisions > * added btequalimage to btree/xid8_ops Here's the version I'm planning to commit tomorrow, if no one objects. Changes: * txid.c renamed to xid8funcs.c * remaining traces of "txid" replaced various intern

Re: Add A Glossary

2020-04-01 Thread Corey Huinker
> > 2. I found out that "see xyz" and "see also" have bespoke markup in > Docbook -- and . I changed some glossentries > to use those, removing some glossdefs and changing a couple of paras to > glossseealsos. I also removed all "id" properties from glossentries > that are just , because I think

Re: pg_stat_statements issue with parallel maintenance (Was Re: WAL usage calculation patch)

2020-04-01 Thread Dilip Kumar
On Thu, Apr 2, 2020 at 8:34 AM Peter Geoghegan wrote: > > On Wed, Apr 1, 2020 at 7:52 PM Amit Kapila wrote: > > Peter, Is this behavior expected? > > > > Let me summarize the situation so that it would be easier for Peter to > > comment. Julien has noticed that parallel vacuum and parallel creat

Re: Add A Glossary

2020-04-01 Thread Corey Huinker
> > I propose we define "planner" and make "optimizer" a entry. > I have no objection to more entries, or edits to entries, but am concerned that the process leads to someone having to manually merge several start-from-scratch patches, with no clear sense of when we'll be done. I may make sense t

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-04-01 Thread Noah Misch
On Mon, Mar 30, 2020 at 11:37:57PM -0700, Andres Freund wrote: > On 2020-03-30 23:28:54 -0700, Noah Misch wrote: > > On Mon, Mar 30, 2020 at 04:43:00PM +0900, Michael Paquier wrote: > > > On Sun, Mar 29, 2020 at 09:41:01PM -0700, Noah Misch wrote: > > > > I think attached v41nm is ready for commit.

Re: BUG #16109: Postgres planning time is high across version (Expose buffer usage during planning in EXPLAIN)

2020-04-01 Thread Fujii Masao
On 2020/04/02 3:47, Julien Rouhaud wrote: On Wed, Apr 1, 2020 at 7:51 PM Fujii Masao wrote: On 2020/03/31 10:31, Justin Pryzby wrote: On Wed, Jan 29, 2020 at 12:15:59PM +0100, Julien Rouhaud wrote: Rebase due to conflict with 3ec20c7091e97. This is failing to apply probably since 4a539

Re: Proposal: Expose oldest xmin as SQL function for monitoring

2020-04-01 Thread Craig Ringer
On Thu, 2 Apr 2020 at 07:57, Tom Lane wrote: > Alvaro Herrera writes: > > On 2020-Apr-01, Tom Lane wrote: > >> The fact that I had to use max(age(...)) in that sample query > >> hints at one reason: it's really hard to do arithmetic correctly > >> on raw XIDs. Dealing with wraparound is a probl

Re: Allow continuations in "pg_hba.conf" files

2020-04-01 Thread Kyotaro Horiguchi
At Thu, 02 Apr 2020 00:20:12 +, David Zhang wrote in > Hi Fabien, > Should we consider the case "\ ", i.e. one or more spaces after the backslash? > For example, if I replace a user map > "mymap /^(.*)@mydomain\.com$ \1" with > "mymap /^(.*)@mydomain\.com$ \ " > "\1" > by add

Re: pg_stat_statements issue with parallel maintenance (Was Re: WAL usage calculation patch)

2020-04-01 Thread Dilip Kumar
On Thu, Apr 2, 2020 at 9:13 AM Dilip Kumar wrote: > > On Thu, Apr 2, 2020 at 8:34 AM Peter Geoghegan wrote: > > > > On Wed, Apr 1, 2020 at 7:52 PM Amit Kapila wrote: > > > Peter, Is this behavior expected? > > > > > > Let me summarize the situation so that it would be easier for Peter to > > > c

User Interface for WAL usage data

2020-04-01 Thread Amit Kapila
Hi, In thread [1], we are discussing to expose WAL usage data for each statement in a way quite similar to how we expose BufferUsage data. The way it exposes seems reasonable to me and no one else raises any objection. It could be that it appears fine to others who have reviewed the patch but I t

Re: NOT IN subquery optimization

2020-04-01 Thread Andrey Lepikhov
You should do small rebase (conflict with 911e7020770) and pgindent of the patch to repair problems with long lines and backspaces. I am reviewing your patch in small steps. Questions: 1. In the find_innerjoined_rels() routine you stop descending on JOIN_FULL node type. I think it is wrong beca

Re: Autovacuum vs vac_update_datfrozenxid() vs ?

2020-04-01 Thread Noah Misch
On Mon, Mar 23, 2020 at 04:50:36PM -0700, Andres Freund wrote: > I think there's also another (even larger?) race in > vac_update_datfrozenxid(): Unless I miss something, two backends can > concurrently run through the scan in vac_update_datfrozenxid() for two > different tables in the same databas

Re: Some problems of recovery conflict wait events

2020-04-01 Thread Masahiko Sawada
On Wed, 1 Apr 2020 at 22:32, Fujii Masao wrote: > > > > On 2020/03/30 20:10, Masahiko Sawada wrote: > > On Fri, 27 Mar 2020 at 17:54, Fujii Masao > > wrote: > >> > >> > >> > >> On 2020/03/04 14:31, Masahiko Sawada wrote: > >>> On Wed, 4 Mar 2020 at 13:48, Fujii Masao > >>> wrote: > >

Re: Allow continuations in "pg_hba.conf" files

2020-04-01 Thread Fabien COELHO
Hello, FWIW, I don't think so. Generally a trailing backspace is an escape character for the following newline. And '\ ' is a escaped space, which is usualy menas a space itself. In this case escape character doesn't work generally and I think it is natural that a backslash in the middle of

Re: WAL usage calculation patch

2020-04-01 Thread Amit Kapila
On Wed, Apr 1, 2020 at 8:00 PM Julien Rouhaud wrote: > > On Wed, Apr 01, 2020 at 04:29:16PM +0530, Amit Kapila wrote: > > 3. Doing some testing with and without parallelism to ensure WAL usage > > data is correct would be great and if possible, share the results? > > > I just saw that Dilip did so

Re: User Interface for WAL usage data

2020-04-01 Thread Justin Pryzby
On Thu, Apr 02, 2020 at 10:13:18AM +0530, Amit Kapila wrote: > In thread [1], we are discussing to expose WAL usage data for each > statement in a way quite similar to how we expose BufferUsage data. > The way it exposes seems reasonable to me and no one else raises any > objection. It could be th

Re: WAL usage calculation patch

2020-04-01 Thread Amit Kapila
On Thu, Apr 2, 2020 at 11:07 AM Amit Kapila wrote: > > On Wed, Apr 1, 2020 at 8:00 PM Julien Rouhaud wrote: > Also, I forgot to mention that let's not base this on buffer usage patch for create index (v10-0002-Allow-parallel-index-creation-to-accumulate-buff) because as per recent discussion I a

Re: Allow continuations in "pg_hba.conf" files

2020-04-01 Thread Justin Pryzby
On Thu, Apr 02, 2020 at 07:25:36AM +0200, Fabien COELHO wrote: > > Hello, > > > FWIW, I don't think so. Generally a trailing backspace is an escape > > character for the following newline. And '\ ' is a escaped space, > > which is usualy menas a space itself. > > > > In this case escape charact

Re: User Interface for WAL usage data

2020-04-01 Thread Kyotaro Horiguchi
At Thu, 2 Apr 2020 00:41:20 -0500, Justin Pryzby wrote in > On Thu, Apr 02, 2020 at 10:13:18AM +0530, Amit Kapila wrote: > > In thread [1], we are discussing to expose WAL usage data for each > > statement in a way quite similar to how we expose BufferUsage data. > > The way it exposes seems rea

Re: BUG #16109: Postgres planning time is high across version (Expose buffer usage during planning in EXPLAIN)

2020-04-01 Thread Julien Rouhaud
On Thu, Apr 02, 2020 at 01:05:56PM +0900, Fujii Masao wrote: > > > On 2020/04/02 3:47, Julien Rouhaud wrote: > > On Wed, Apr 1, 2020 at 7:51 PM Fujii Masao > > wrote: > > > > > > > > > On 2020/03/31 10:31, Justin Pryzby wrote: > > > > On Wed, Jan 29, 2020 at 12:15:59PM +0100, Julien Rouhaud w

Re: User Interface for WAL usage data

2020-04-01 Thread Amit Kapila
On Thu, Apr 2, 2020 at 11:28 AM Kyotaro Horiguchi wrote: > > At Thu, 2 Apr 2020 00:41:20 -0500, Justin Pryzby wrote > in > > On Thu, Apr 02, 2020 at 10:13:18AM +0530, Amit Kapila wrote: > > > In thread [1], we are discussing to expose WAL usage data for each > > > statement in a way quite simila

Re: pgbench - add \aset to store results of a combined query

2020-04-01 Thread Fabien COELHO
Bonjour Michaël, Except for the addition of a test case to skip empty results when \aset is used, I think that we are pretty good here. While hacking on the patch more by myself, I found that mixing tests for \gset and \aset was rather messy. A test for an empty result leads also to a failur

Re: Berserk Autovacuum (let's save next Mandrill)

2020-04-01 Thread David Rowley
On Thu, 2 Apr 2020 at 16:13, Tom Lane wrote: > > Dean Rasheed writes: > > Yeah, that makes sense. I still can't see what might be causing those > > failures. The tests that were doing an ALTER COLUMN and then expecting > > to see the results of a non-analysed table ought to be fixed by > > 0936d1

Re: ALTER tbl rewrite loses CLUSTER ON index

2020-04-01 Thread Michael Paquier
On Wed, Mar 18, 2020 at 11:48:37AM +0900, Michael Paquier wrote: > Anyway, Tom, Alvaro, are you planning to look at what is proposed on > this thread? I don't want to step on your toes if that's the case and > it seems to me that the approach taken by the patch is sound, using as > basic fix the a

Re: potential stuck lock in SaveSlotToPath()

2020-04-01 Thread Michael Paquier
On Wed, Apr 01, 2020 at 04:26:25PM +0200, Peter Eisentraut wrote: > Good catch. How about the attached patch? WFM. Another trick would be to call LWLockRelease() after generating the log, but I find your patch more consistent with the surroundings. -- Michael signature.asc Description: PGP sig

Re: A bug when use get_bit() function for a long bytea string

2020-04-01 Thread movead...@highgo.ca
>I don't think this has really solved the overflow hazards. For example, >in binary_encode we've got >resultlen = enc->encode_len(VARDATA_ANY(data), datalen); >result = palloc(VARHDRSZ + resultlen); >and all you've done about that is changed resultlen from int to int64. >On a 64-bit machine,

Re: User Interface for WAL usage data

2020-04-01 Thread Julien Rouhaud
Thanks all for the feedback. On Thu, Apr 2, 2020 at 8:02 AM Amit Kapila wrote: > > On Thu, Apr 2, 2020 at 11:28 AM Kyotaro Horiguchi > wrote: > > > > At Thu, 2 Apr 2020 00:41:20 -0500, Justin Pryzby > > wrote in > > > On Thu, Apr 02, 2020 at 10:13:18AM +0530, Amit Kapila wrote: > > > > In thre

Re: Some problems of recovery conflict wait events

2020-04-01 Thread Fujii Masao
On 2020/04/02 14:25, Masahiko Sawada wrote: On Wed, 1 Apr 2020 at 22:32, Fujii Masao wrote: On 2020/03/30 20:10, Masahiko Sawada wrote: On Fri, 27 Mar 2020 at 17:54, Fujii Masao wrote: On 2020/03/04 14:31, Masahiko Sawada wrote: On Wed, 4 Mar 2020 at 13:48, Fujii Masao wrote:

Re: WAL usage calculation patch

2020-04-01 Thread Dilip Kumar
On Wed, Apr 1, 2020 at 8:00 PM Julien Rouhaud wrote: > > Hi, > > I'm replying here to all reviews that have been sent, thanks a lot! > > On Wed, Apr 01, 2020 at 04:29:16PM +0530, Amit Kapila wrote: > > On Wed, Apr 1, 2020 at 1:32 PM Julien Rouhaud wrote: > > > > > > So here's a v9, rebased on top

Re: User Interface for WAL usage data

2020-04-01 Thread Justin Pryzby
On Thu, Apr 02, 2020 at 11:32:16AM +0530, Amit Kapila wrote: > On Thu, Apr 2, 2020 at 11:28 AM Kyotaro Horiguchi > wrote: > > > > At Thu, 2 Apr 2020 00:41:20 -0500, Justin Pryzby > > wrote in > > > Regarding v10-0004-Add-option-to-report-WAL-usage-in-EXPLAIN-and-au.patch: > > > I think there sh

Re: pgbench - add pseudo-random permutation function

2020-04-01 Thread Fabien COELHO
Hello David, Attached is an attempt at improving things. I have added a explicit note and hijacked an existing example to better illustrate the purpose of the function. This patch does not build on Linux due to some unused functions and variables: http://cfbot.cputube.org/patch_27_1712.log

Re: Allow continuations in "pg_hba.conf" files

2020-04-01 Thread Fabien COELHO
Hi Justin, There are no assumption about backslash escaping, quotes and such, which seems reasonable given the lexing structure of the files, i.e. records of space-separated words, and # line comments. Quoting does allow words containing spaces: Ok. I meant that the continuation handling

Re: ALTER tbl rewrite loses CLUSTER ON index

2020-04-01 Thread Justin Pryzby
On Thu, Apr 02, 2020 at 03:14:21PM +0900, Michael Paquier wrote: > Now, regarding patch 0002, note that you have a problem for this part: > -tuple = SearchSysCache1(INDEXRELID, ObjectIdGetDatum(indexOid)); > -if (!HeapTupleIsValid(tuple))/* probably can't happen */ > -

Re: BUG #16109: Postgres planning time is high across version (Expose buffer usage during planning in EXPLAIN)

2020-04-01 Thread Fujii Masao
On 2020/04/02 15:01, Julien Rouhaud wrote: On Thu, Apr 02, 2020 at 01:05:56PM +0900, Fujii Masao wrote: On 2020/04/02 3:47, Julien Rouhaud wrote: On Wed, Apr 1, 2020 at 7:51 PM Fujii Masao wrote: On 2020/03/31 10:31, Justin Pryzby wrote: On Wed, Jan 29, 2020 at 12:15:59PM +0100, Julie

Re: Some problems of recovery conflict wait events

2020-04-01 Thread Masahiko Sawada
On Thu, 2 Apr 2020 at 15:34, Fujii Masao wrote: > > > > On 2020/04/02 14:25, Masahiko Sawada wrote: > > On Wed, 1 Apr 2020 at 22:32, Fujii Masao > > wrote: > >> > >> > >> > >> On 2020/03/30 20:10, Masahiko Sawada wrote: > >>> On Fri, 27 Mar 2020 at 17:54, Fujii Masao > >>> wrote: > >

<    1   2