Re: [PATCH] /src/backend/access/transam/xlog.c, tiny improvements

2020-01-26 Thread Michael Paquier
On Sun, Jan 26, 2020 at 06:47:57PM -0800, Mark Dilger wrote: > There is something unusual about comparing a XLogSegNo variable in > this way, but it seems to go back to 2014 when the replication slots > were introduced in commit 858ec11858a914d4c380971985709b6d6b7dd6fc, > and XLogSegNo was unsigned

Re: can we use different function in place of atoi in vacuumdb.c file

2020-01-26 Thread Surafel Temesgen
Hi, On Thu, Jan 23, 2020 at 3:56 PM Mahendra Singh Thalor wrote: > Hi all, > While reviewing one patch, I found that if we give any non-integer string > to atoi (say aa), then it is returning zero(0) as output so we are not > giving any error(assuming 0 as valid argument) and continuing our > ope

RE: [PoC] Non-volatile WAL buffer

2020-01-26 Thread Takashi Menjo
Hello Heikki, > I have the same comments on this that I had on the previous patch, see: > > https://www.postgresql.org/message-id/2aec6e2a-6a32-0c39-e4e2-aad854543aa8%40iki.fi Thanks. I re-read your messages [1][2]. What you meant, AFAIU, is how about using memory-mapped WAL segment files as W

Re: error context for vacuum to include block number

2020-01-26 Thread Masahiko Sawada
On Mon, 27 Jan 2020 at 14:38, Justin Pryzby wrote: > > On Sun, Jan 26, 2020 at 12:29:38PM -0800, Andres Freund wrote: > > > postgres=# SET client_min_messages=debug;SET statement_timeout=99; VACUUM > > > (VERBOSE, PARALLEL 0) t; > > > INFO: vacuuming "public.t" > > > DEBUG: "t_a_idx": vacuuming

Re: error context for vacuum to include block number

2020-01-26 Thread Justin Pryzby
It occured to me that there's an issue with sharing vacrelstats between scan/vacuum, since blkno and stage are set by the heap/index vacuum routines, but not reset on their return to heap scan. Not sure if we should reset them, or go back to using a separate struct, like it was here: https://www.p

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-01-26 Thread Kyotaro Horiguchi
By the way, the previous version looks somewhat different from what I thought I posted.. At Sun, 26 Jan 2020 20:57:00 -0800, Noah Misch wrote in > On Mon, Jan 27, 2020 at 01:44:13PM +0900, Kyotaro Horiguchi wrote: > > > The purpose of this loop is to create relcache entries for rels locked in >

Re: Setting min/max TLS protocol in clientside libpq

2020-01-26 Thread Michael Paquier
On Fri, Jan 24, 2020 at 12:19:31PM +0100, Daniel Gustafsson wrote: > Attached is a v5 of the patch which hopefully address all the comments raised, > sorry for the delay. Thanks for the new version. psql: error: could not connect to server: invalid or unsupported maximum protocol version specifie

Re: error context for vacuum to include block number

2020-01-26 Thread Justin Pryzby
On Sun, Jan 26, 2020 at 12:29:38PM -0800, Andres Freund wrote: > > postgres=# SET client_min_messages=debug;SET statement_timeout=99; VACUUM > > (VERBOSE, PARALLEL 0) t; > > INFO: vacuuming "public.t" > > DEBUG: "t_a_idx": vacuuming index > > 2020-01-20 15:47:36.338 CST [20139] ERROR: canceling

Tid scan increments value of pg_stat_all_tables.seq_scan. (but not seq_tup_read)

2020-01-26 Thread Kasahara Tatsuhito
Hi, I noticed that since PostgreSQL 12, Tid scan increments value of pg_stat_all_tables.seq_scan. (but not seq_tup_read) The following is an example. CREATE TABLE t (c int); INSERT INTO t SELECT 1; SET enable_seqscan to off; (v12 -) =# EXPLAIN ANALYZE SELECT * FROM t WHERE ctid = '(0,1)';

Re: table partitioning and access privileges

2020-01-26 Thread Amit Langote
Fujii-san, On Mon, Jan 27, 2020 at 11:19 AM Fujii Masao wrote: > On 2020/01/23 22:14, Fujii Masao wrote: > > Thanks for updating the patch! Barring any objection, > > I will commit this fix and backport it to all supported versions. > > Attached are the back-port versions of the patches. > > > Th

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-01-26 Thread Noah Misch
On Mon, Jan 27, 2020 at 01:44:13PM +0900, Kyotaro Horiguchi wrote: > At Sun, 26 Jan 2020 20:22:01 -0800, Noah Misch wrote in > > Diffing the two latest versions of one patch: > > > --- v32-0002-Fix-the-defect-1.patch 2020-01-18 14:32:47.499129940 > > > -0800 > > > +++ v33-0002-Fix-the-defe

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-01-26 Thread Kyotaro Horiguchi
Thanks! At Sun, 26 Jan 2020 20:22:01 -0800, Noah Misch wrote in > Diffing the two latest versions of one patch: > > --- v32-0002-Fix-the-defect-1.patch 2020-01-18 14:32:47.499129940 -0800 > > +++ v33-0002-Fix-the-defect-1.patch 2020-01-26 16:23:52.846391035 -0800 > > +@@ -2978,8 +3054,8 @@ Asser

Re: [HACKERS] WAL logging problem in 9.4.3?

2020-01-26 Thread Noah Misch
Diffing the two latest versions of one patch: > --- v32-0002-Fix-the-defect-1.patch 2020-01-18 14:32:47.499129940 -0800 > +++ v33-0002-Fix-the-defect-1.patch 2020-01-26 16:23:52.846391035 -0800 > +@@ -2978,8 +3054,8 @@ AssertPendingSyncs_RelationCache(void) > + LOCKTAG_RELAT

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2020-01-26 Thread Kyotaro Horiguchi
Hello. At Sun, 26 Jan 2020 12:22:03 -0800, Andres Freund wrote in > Hi, I feel the same on the specific issues brought in upthread. > On 2020-01-26 16:20:03 +0100, Magnus Hagander wrote: > > On Sun, Jan 26, 2020 at 1:44 AM Andres Freund wrote: > > > On 2020-01-25 15:43:41 +0100, Magnus Hagand

Re: [PATCH] /src/backend/access/transam/xlog.c, tiny improvements

2020-01-26 Thread Mark Dilger
> On Jan 24, 2020, at 12:48 PM, Ranier Vilela wrote: > > 3. At function KeepLogSeg (line 9357) the test if (slotSegNo <= 0), the var > slotSegNo is uint64 and not can be < 0. There is something unusual about comparing a XLogSegNo variable in this way, but it seems to go back to 2014 when t

Re: Add MAIN_RELATION_CLEANUP and SECONDARY_RELATION_CLEANUP options to VACUUM

2020-01-26 Thread Michael Paquier
On Fri, Jan 24, 2020 at 09:31:26PM +, Bossart, Nathan wrote: > On 1/21/20, 9:02 PM, "Michael Paquier" wrote: >> On Tue, Jan 21, 2020 at 09:21:46PM +, Bossart, Nathan wrote: >>> I've attached a patch for a couple of new options for VACUUM: >>> MAIN_RELATION_CLEANUP and SECONDARY_RELATION_CL

RE: [PoC] Non-volatile WAL buffer

2020-01-26 Thread Takashi Menjo
Hello Fabien, Thank you for your +1 :) > Is it possible to emulate somthing without the actual hardware, at least > for testing purposes? Yes, you can emulate PMEM using DRAM on Linux, via "memmap=nnG!ssG" kernel parameter. Please see [1] and [2] for emulation details. If your emulation does n

Re: table partitioning and access privileges

2020-01-26 Thread Fujii Masao
On 2020/01/23 22:14, Fujii Masao wrote: On 2020/01/22 16:54, Amit Langote wrote: Fujii-san, Thanks for taking a look. On Fri, Jan 10, 2020 at 10:29 AM Fujii Masao wrote: On Tue, Jan 7, 2020 at 5:15 PM Amit Langote wrote: I tend to agree that TRUNCATE's permission model for inheritance

Re: [PATCH] Resolve Parallel Hash Join Performance Issue

2020-01-26 Thread Thomas Munro
On Tue, Jan 21, 2020 at 6:20 PM Thomas Munro wrote: > On Fri, Jan 10, 2020 at 1:52 PM Deng, Gang wrote: > > Thank you for the comment. Yes, I agree the alternative of using > > '(!parallel)', so that no need to test the bit. Will someone submit patch > > to for it accordingly? > > Here's a patc

Re: making the backend's json parser work in frontend code

2020-01-26 Thread Mark Dilger
> On Jan 26, 2020, at 5:51 PM, Andrew Dunstan > wrote: > >>> 0007 adds testing. >>> >>> I would appreciate somebody looking at the portability issues for 0007 >>> on Windows. >>> >> >> We'll need at a minimum something added to src/tools/msvc to build the >> test program, maybe some other s

Re: [PATCH] Windows port, fix some resources leaks

2020-01-26 Thread Michael Paquier
On Fri, Jan 24, 2020 at 09:37:25AM -0300, Ranier Vilela wrote: > Em sex., 24 de jan. de 2020 às 04:13, Michael Paquier > escreveu: >> There is some progress. You should be careful about your patches, >> as they generate compiler warnings. Here is one quote from gcc-9: >> logging.c:87:13: warning

Re: making the backend's json parser work in frontend code

2020-01-26 Thread Andrew Dunstan
> > 0007 adds testing. > > > > I would appreciate somebody looking at the portability issues for 0007 > > on Windows. > > > > We'll need at a minimum something added to src/tools/msvc to build the > test program, maybe some other stuff too. I'll take a look. Patch complains that the 0007 patch is

Re: EXPLAIN's handling of output-a-field-or-not decisions

2020-01-26 Thread Andres Freund
Hi, On 2020-01-26 17:53:09 -0500, Tom Lane wrote: > Andres Freund writes: > > I've previously wondered about adding a REGRESS option to EXPLAIN would > > not actually be a good one, so we can move the magic into that, rather > > than options that are also otherwise relevant. > > I'd be inclined

Re: making the backend's json parser work in frontend code

2020-01-26 Thread Mark Dilger
> On Jan 26, 2020, at 5:09 PM, Andrew Dunstan > wrote: > > We'll need at a minimum something added to src/tools/msvc to build the > test program, maybe some other stuff too. I'll take a look. Thanks, much appreciated. — Mark Dilger EnterpriseDB: http://www.enterprisedb.com The Enterprise Po

Re: making the backend's json parser work in frontend code

2020-01-26 Thread Andrew Dunstan
On Mon, Jan 27, 2020 at 5:54 AM Mark Dilger wrote: > > > > > On Jan 22, 2020, at 10:53 AM, Robert Haas wrote: > > > > 0004 is a substantially cleaned up version of the patch to make the > > JSON parser return a result code rather than throwing errors. Names > > have been fixed, interfaces have be

Re: making the backend's json parser work in frontend code

2020-01-26 Thread Andrew Dunstan
On Sat, Jan 25, 2020 at 6:20 AM Mark Dilger wrote: > > > > > On Jan 24, 2020, at 10:43 AM, Robert Haas wrote: > > > > Since 0001-0003 have been reviewed by multiple people and nobody's > > objected, I have committed those. > > I think 0004-0005 have been reviewed and accepted by both me and Andre

Re: Implementing Incremental View Maintenance

2020-01-26 Thread Takuma Hoshiai
On Mon, 20 Jan 2020 16:57:58 +0900 Yugo NAGATA wrote: > On Fri, 17 Jan 2020 14:10:32 -0700 (MST) > legrand legrand wrote: > > > Hello, > > > > It seems that patch v11 doesn't apply any more. > > Problem with "scanRTEForColumn" maybe because of change: > > Thank you for your reporting! We will

Re: Parallel leader process info in EXPLAIN

2020-01-26 Thread Thomas Munro
On Mon, Jan 27, 2020 at 11:49 AM Tom Lane wrote: > I've occasionally wondered whether we'd be better off presenting > this info as if the leader were "worker 0" and then the N workers > are workers 1 to N. I've not worked out the implications of that > in any detail though. It's fairly easy to s

Re: Duplicate Workers entries in some EXPLAIN plans

2020-01-26 Thread Tom Lane
I wrote: > Andres Freund writes: >> I wonder if we could introduce a debug GUC that makes parallel worker >> acquisition just retry in a loop, for a time determined by the GUC. That >> obviously would be a bad idea to do in a production setup, but it could >> be good enough for regression tests?

Re: EXPLAIN's handling of output-a-field-or-not decisions

2020-01-26 Thread Tom Lane
Andres Freund writes: > On 2020-01-26 16:54:58 -0500, Tom Lane wrote: >> ... how are you going to square that desire with not breaking the >> regression tests? > Well, that's how we arrived at turning off JIT information when COSTS > OFF, because that's already something all the EXPLAINs in the r

Delaying/avoiding BTreeTupleGetNAtts() call within _bt_compare()

2020-01-26 Thread Peter Geoghegan
Andres and I discussed bottlenecks in the nbtree code during the recent PgDay SF community event. Andres observed that the call to BTreeTupleGetNAtts() in _bt_compare() can become somewhat of a bottleneck. I came up with the very simple attached POC-quality patches, which Andres tested and profiled

Re: Parallel leader process info in EXPLAIN

2020-01-26 Thread Tom Lane
Thomas Munro writes: > I think I'm going to abandon 0002 for now, because that stuff is being > refactored independently over here, so rebasing would be futile: > https://www.postgresql.org/message-id/flat/CAOtHd0AvAA8CLB9Xz0wnxu1U%3DzJCKrr1r4QwwXi_kcQsHDVU%3DQ%40mail.gmail.com Yeah, your 0002 ne

Re: EXPLAIN's handling of output-a-field-or-not decisions

2020-01-26 Thread Andres Freund
Hi, On 2020-01-26 16:54:58 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2020-01-26 15:13:49 -0500, Tom Lane wrote: > >> The other offender is the JIT stuff: it prints if COSTS is on and > >> there's some JIT activity to report, and otherwise you get nothing. > >> This is OK for text mode

Re: EXPLAIN's handling of output-a-field-or-not decisions

2020-01-26 Thread Tom Lane
Andres Freund writes: > On 2020-01-26 15:13:49 -0500, Tom Lane wrote: >> The other offender is the JIT stuff: it prints if COSTS is on and >> there's some JIT activity to report, and otherwise you get nothing. >> This is OK for text mode but it's bogus for the other formats. >> Since we just rearr

Re: EXPLAIN's handling of output-a-field-or-not decisions

2020-01-26 Thread Andres Freund
Hi, On 2020-01-26 15:13:49 -0500, Tom Lane wrote: > The other offender is the JIT stuff: it prints if COSTS is on and > there's some JIT activity to report, and otherwise you get nothing. > This is OK for text mode but it's bogus for the other formats. > Since we just rearranged EXPLAIN's JIT outp

Re: Online checksums patch - once again

2020-01-26 Thread Andres Freund
Hi, On 2020-01-23 12:23:09 -0500, Robert Haas wrote: > On Thu, Jan 23, 2020 at 6:19 AM Daniel Gustafsson wrote: > > A bigger question is how to handle the offline capabilities. pg_checksums > > can > > enable or disable checksums in an offline cluster, which will put the > > cluster > > in a s

Re: Ought to use heap_multi_insert() for pg_attribute/depend insertions?

2020-01-26 Thread Andres Freund
Hi, On 2020-01-22 23:18:12 +0100, Daniel Gustafsson wrote: > > On 26 Nov 2019, at 06:44, Michael Paquier wrote: > > Re this patch being in WoA state for some time [0]: > > > The regression tests of contrib/test_decoding are still failing here: > > +ERROR: could not resolve cmin/cmax of catalog

Re: error context for vacuum to include block number

2020-01-26 Thread Andres Freund
Hi, On 2020-01-20 15:49:29 -0600, Justin Pryzby wrote: > On Mon, Jan 20, 2020 at 11:11:20AM -0800, Andres Freund wrote: > On Mon, Jan 20, 2020 at 11:11:20AM -0800, Andres Freund wrote: > > Alternatively we could push another context for each index inside > > lazy_vacuum_all_indexes(). There's been

Re: error context for vacuum to include block number

2020-01-26 Thread Andres Freund
Hi, On 2020-01-22 17:17:26 -0600, Justin Pryzby wrote: > On Mon, Jan 20, 2020 at 11:11:20AM -0800, Andres Freund wrote: > > > @@ -966,8 +986,11 @@ lazy_scan_heap(Relation onerel, VacuumParams > > > *params, LVRelStats *vacrelstats, > > > /* Work on all the indexes, then the heap

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2020-01-26 Thread Andres Freund
Hi, On 2020-01-26 16:20:03 +0100, Magnus Hagander wrote: > On Sun, Jan 26, 2020 at 1:44 AM Andres Freund wrote: > > On 2020-01-25 15:43:41 +0100, Magnus Hagander wrote: > > > On Fri, Jan 24, 2020 at 8:52 PM Andres Freund wrote: > > > > Lastly, I don't understand what the point of sending fixed s

Re: Strange coding in _mdfd_openseg()

2020-01-26 Thread Thomas Munro
On Sun, Jan 26, 2020 at 8:23 AM Noah Misch wrote: > Agreed. The rest of md.c won't cope with a hole in this array, so allowing > less-than-or-equal here is futile. The patch in the original post looks fine. Thanks. Pushed.

EXPLAIN's handling of output-a-field-or-not decisions

2020-01-26 Thread Tom Lane
I believe that the design intention for EXPLAIN's non-text output formats is that a given field should appear, or not, depending solely on the plan shape, EXPLAIN options, and possibly GUC settings. It's not okay to suppress a field just because it's empty or zero or otherwise uninteresting, becaus

Re: making the backend's json parser work in frontend code

2020-01-26 Thread Mark Dilger
> On Jan 22, 2020, at 10:53 AM, Robert Haas wrote: > > 0004 is a substantially cleaned up version of the patch to make the > JSON parser return a result code rather than throwing errors. Names > have been fixed, interfaces have been tidied up, and the thing is > better integrated with the surro

Re: Add %x to PROMPT1 and PROMPT2

2020-01-26 Thread Vik Fearing
On 26/01/2020 19:48, Fabien COELHO wrote: > > Hello Vik, > >> I cannot ever think of a time when I don't want to know if I'm in a >> transaction or not (and what its state is).  Every new setup I do, I add >> %x to the psql prompt. >> >> I think it should be part of the default prompt.  Path atta

Re: Add %x to PROMPT1 and PROMPT2

2020-01-26 Thread Fabien COELHO
Hello Vik, I cannot ever think of a time when I don't want to know if I'm in a transaction or not (and what its state is). Every new setup I do, I add %x to the psql prompt. I think it should be part of the default prompt. Path attached. Isn't there examples in the documentation which use

Re: proposal: schema variables

2020-01-26 Thread Pavel Stehule
DEFAULT; 2. LET var = (query); In first case I have not any query, that I can assign, and in this case the LET statement is really only UTILITY. I did comment there Regards Pavel > > > regards > > -- > Tomas Vondra http://www.2ndQuadrant.com > Postgr

Re: explain HashAggregate to report bucket and memory stats

2020-01-26 Thread Justin Pryzby
On Sun, Jan 26, 2020 at 08:14:25AM -0600, Justin Pryzby wrote: > On Fri, Jan 03, 2020 at 10:19:25AM -0600, Justin Pryzby wrote: > > On Sun, Feb 17, 2019 at 11:29:56AM -0500, Jeff Janes wrote: > > https://www.postgresql.org/message-id/CAMkU%3D1zBJNVo2DGYBgLJqpu8fyjCE_ys%2Bmsr6pOEoiwA7y5jrA%40mail.gm

Re: proposal: type info support functions for functions that use "any" type

2020-01-26 Thread Pavel Stehule
Hi út 14. 1. 2020 v 22:09 odesílatel Tom Lane napsal: > Pavel Stehule writes: > > [ parser-support-function-with-demo-20191128.patch ] > > TBH, I'm still not convinced that this is a good idea. Restricting > the support function to only change the function's return type is > safer than the or

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2020-01-26 Thread Magnus Hagander
On Sun, Jan 26, 2020 at 1:44 AM Andres Freund wrote: > > Hi, > > On 2020-01-25 15:43:41 +0100, Magnus Hagander wrote: > > On Fri, Jan 24, 2020 at 8:52 PM Andres Freund wrote: > > > Additionally pg_stat_bgwriter.buffers_backend also counts writes done by > > > autovacuum et al. > > > > I think it'

Re: explain HashAggregate to report bucket and memory stats

2020-01-26 Thread Justin Pryzby
On Fri, Jan 03, 2020 at 10:19:25AM -0600, Justin Pryzby wrote: > On Sun, Feb 17, 2019 at 11:29:56AM -0500, Jeff Janes wrote: > https://www.postgresql.org/message-id/CAMkU%3D1zBJNVo2DGYBgLJqpu8fyjCE_ys%2Bmsr6pOEoiwA7y5jrA%40mail.gmail.com > > What would I find very useful is [...] if the HashAggrega

vacuum verbose: show pages marked allvisible/frozen/hintbits

2020-01-26 Thread Justin Pryzby
I'm forking this thread since it's separate topic, and since keeping in a single branch hasn't made maintaining the patches any easier. https://www.postgresql.org/message-id/CAMkU%3D1xAyWnwnLGORBOD%3Dpyv%3DccEkDi%3DwKeyhwF%3DgtB7QxLBwQ%40mail.gmail.com On Sun, Dec 29, 2019 at 01:15:24PM -0500, Jeff

Re: Add %x to PROMPT1 and PROMPT2

2020-01-26 Thread Christoph Berg
Re: Vik Fearing 2020-01-26 <09502c40-cfe1-bb29-10f9-4b3fa7b2b...@2ndquadrant.com> > I cannot ever think of a time when I don't want to know if I'm in a > transaction or not (and what its state is). Every new setup I do, I add > %x to the psql prompt. > > I think it should be part of the default

Add %x to PROMPT1 and PROMPT2

2020-01-26 Thread Vik Fearing
I cannot ever think of a time when I don't want to know if I'm in a transaction or not (and what its state is). Every new setup I do, I add %x to the psql prompt. I think it should be part of the default prompt. Path attached. -- Vik Fearing >From 6118b8b2ab4cfc70525666b8d57eaa351d6c2a3d Mon Se

Re: Remove page-read callback from XLogReaderState.

2020-01-26 Thread Heikki Linnakangas
On 21/01/2020 13:33, Craig Ringer wrote: On Tue, 21 Jan 2020 at 18:46, Kyotaro Horiguchi wrote: It seems to me that it works perfectly, and everything looks good I seem to remember some considerable pain in this area when it came to timeline switches. Especially with logical decoding and xlog

Re: Remove page-read callback from XLogReaderState.

2020-01-26 Thread Heikki Linnakangas
On 21/01/2020 12:45, Kyotaro Horiguchi wrote: At Mon, 20 Jan 2020 17:24:07 +0900 (JST), Kyotaro Horiguchi wrote in Separating XLogBeginRead seems reasonable. The annoying recptr trick is no longer needed. In particular some logical-decoding stuff become far cleaner by the patch. fetching_ckp