Range checks of pg_test_fsync --secs-per-test and pg_test_timing --duration

2020-08-05 Thread Michael Paquier
Hi all, As $subject says, pg_test_fsync and pg_test_timing don't really check the range of option values specified. It is possible for example to make pg_test_fsync run an infinite amount of time, and pg_test_timing does not handle overflows with --duration at all. These are far from being criti

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-08-05 Thread Ashutosh Sharma
On Thu, Aug 6, 2020 at 1:29 AM Robert Haas wrote: > > On Wed, Aug 5, 2020 at 9:42 AM Ashutosh Sharma wrote: > > Yeah, it's being tested on the main table, not on a toast table. I've > > removed this test-case and also restricted direct access to the toast > > table using heap_force_kill/freeze fu

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-08-05 Thread Ashutosh Sharma
On Thu, Aug 6, 2020 at 1:04 AM Robert Haas wrote: > > On Mon, Aug 3, 2020 at 12:13 PM Ashutosh Sharma wrote: > > If the above path is taken that means none of the items in the page > > got changed. > > Oops. I didn't realize that, sorry. Maybe it would be a little more > clear if instead of "int

Re: public schema default ACL

2020-08-05 Thread Noah Misch
On Mon, Aug 03, 2020 at 07:46:02PM +0200, Peter Eisentraut wrote: > The important things in my mind are that you keep an easy onboarding > experience (you can do SQL things without having to create and unlock a > bunch of things first) and that advanced users can do the things they want > to do *so

Re: public schema default ACL

2020-08-05 Thread Noah Misch
On Mon, Aug 03, 2020 at 09:46:23AM -0400, Robert Haas wrote: > On Mon, Aug 3, 2020 at 2:30 AM Noah Misch wrote: > > Between (b)(2)(X) and (b)(3)(X), what are folks' preferences? Does anyone > > strongly favor some other option (including the option of changing nothing) > > over both of those two?

Re: pg_rewind is not crash safe

2020-08-05 Thread Andrey M. Borodin
> 5 авг. 2020 г., в 23:13, Heikki Linnakangas написал(а): > > A colleague of mine brought to my attention that pg_rewind is not crash safe. > If it is interrupted for any reason, it leaves behind a data directory with a > mix of data from the source and target images. If you're "lucky", the

Re: Amcheck: do rightlink verification with lock coupling

2020-08-05 Thread Andrey M. Borodin
> 6 авг. 2020 г., в 04:25, Peter Geoghegan написал(а): > > * Added _bt_checkpage() calls for buffers, as is standard practice in nbtree. > > * Added protection against locking the same page a second time in the > event of a faulty sibling link -- we should avoid a self-deadlock in > the event

Re: Parallel copy

2020-08-05 Thread vignesh C
On Tue, Aug 4, 2020 at 9:51 PM Tomas Vondra wrote: > > On Mon, Aug 03, 2020 at 12:33:48PM +0530, Bharath Rupireddy wrote: > >On Sat, Aug 1, 2020 at 9:55 AM vignesh C wrote: > >> > >> The patches were not applying because of the recent commits. > >> I have rebased the patch over head & attached. >

Re: Which SET TYPE don't actually require a rewrite

2020-08-05 Thread Noah Misch
On Wed, Aug 05, 2020 at 02:52:42PM +0200, Magnus Hagander wrote: > On Sat, Jul 18, 2020 at 4:57 AM Noah Misch wrote: > > Such a doc addition is fine with me.  I agree with Tom that it will be prone > > to staleness, but I don't conclude that the potential for staleness reduces > > its net value be

Re: FailedAssertion("pd_idx == pinfo->nparts", File: "execPartition.c", Line: 1689)

2020-08-05 Thread Tom Lane
Andy Fan writes: > On Thu, Aug 6, 2020 at 2:22 AM Tom Lane wrote: >> In the longer term, it's annoying that we have no test methodology >> for this other than "manually set a breakpoint here". > One of the methods I see is we can just add some GUC variable for some > action injection. basicall

Re: FailedAssertion("pd_idx == pinfo->nparts", File: "execPartition.c", Line: 1689)

2020-08-05 Thread Andy Fan
On Thu, Aug 6, 2020 at 2:22 AM Tom Lane wrote: > Robert Haas writes: > > On Wed, Aug 5, 2020 at 1:30 PM Tom Lane wrote: > >> I'm strongly tempted to convert the trailing Assert to an actual > >> test-and-elog, too, but didn't do so here. > > > I was thinking about that, too. +1 for taking that

Re: FailedAssertion("pd_idx == pinfo->nparts", File: "execPartition.c", Line: 1689)

2020-08-05 Thread Amit Langote
On Thu, Aug 6, 2020 at 2:30 AM Tom Lane wrote: > Amit Langote writes: > > The attached patch should fix that. > > I don't like this patch at all though; I do not think it is being nearly > careful enough to ensure that it's matched the surviving relation OIDs > correctly. In particular it blithe

Re: WIP: WAL prefetch (another approach)

2020-08-05 Thread Thomas Munro
On Tue, Aug 4, 2020 at 3:47 AM Tomas Vondra wrote: > On Thu, Jul 02, 2020 at 03:09:29PM +1200, Thomas Munro wrote: > >FYI I am still trying to reproduce and understand the problem Tomas > >reported; more soon. > > Any luck trying to reproduce thigs? Should I try again and collect some > additional

Re: Add MAIN_RELATION_CLEANUP and SECONDARY_RELATION_CLEANUP options to VACUUM

2020-08-05 Thread Michael Paquier
On Wed, Aug 05, 2020 at 12:56:48AM +, Bossart, Nathan wrote: > My main motive for adding the MAIN_RELATION_CLEANUP option is to allow > table owners to easily vacuum only a relation's TOAST table. Roles do > not have access to the pg_toast schema by default, so they might be > restricted from

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-08-05 Thread k.jami...@fujitsu.com
On Saturday, August 1, 2020 5:24 AM, Andres Freund wrote: Hi, Thank you for your constructive review and comments. Sorry for the late reply. > Hi, > > On 2020-07-31 15:50:04 -0400, Tom Lane wrote: > > Andres Freund writes: > > > Indeed. The buffer mapping hashtable already is visible as a major

Re: PROC_IN_ANALYZE stillborn 13 years ago

2020-08-05 Thread Andres Freund
Hi, On 2020-08-05 19:55:49 -0400, Alvaro Herrera wrote: > Back in the 8.3 cycle (2007) when the autovacuum launcher/worker split > was done, we annoyed people because it blocked DDL. That led to an > effort to cancel autovac automatically when that was detected, by Simon > Riggs. > https://postgr

Re: Reg. Postgres 13

2020-08-05 Thread Bruce Momjian
On Wed, Aug 5, 2020 at 06:08:15AM +, Joel Mariadasan (jomariad) wrote: > Hi, > > I see that release 13 is currently in beta. > > When will be the official production release of 13 be out? > > We need to see if we can include this as part of our product release cycle. Look here: ht

PROC_IN_ANALYZE stillborn 13 years ago

2020-08-05 Thread Alvaro Herrera
Back in the 8.3 cycle (2007) when the autovacuum launcher/worker split was done, we annoyed people because it blocked DDL. That led to an effort to cancel autovac automatically when that was detected, by Simon Riggs. https://postgr.es/m/1191526327.4223.204.ca...@ebony.site https://postgr.es/m/1192

Re: Amcheck: do rightlink verification with lock coupling

2020-08-05 Thread Peter Geoghegan
On Tue, Aug 4, 2020 at 9:33 AM Andrey M. Borodin wrote: > BTW, reviewing this patch again I cannot understand why we verify link > coherence only on leaf level but not for internal pages? > I do not see any actual problems here. Well, I thought that it might be a good idea to limit it to the lea

Re: Replace remaining StrNCpy() by strlcpy()

2020-08-05 Thread David Rowley
On Tue, 4 Aug 2020 at 00:12, Tom Lane wrote: > > I wrote: > > David Rowley writes: > >> Will mean that we'll now no longer zero the full length of the m_xlog > >> field after the end of the string. Won't that mean we'll start writing > >> junk bytes to the stats collector? > > > StrNCpy doesn't z

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-08-05 Thread David Rowley
On Thu, 6 Aug 2020 at 08:13, Robert Haas wrote: > > This is cool work; I am going to bikeshed on the name for a minute. I > don't think Result Cache is terrible, but I have two observations: Thanks > 1. It might invite confusion with a feature of some other database > systems where they cache th

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-08-05 Thread Peter Geoghegan
On Wed, May 20, 2020 at 4:44 AM David Rowley wrote: > Does it seem like something we might want for PG14? Minor terminology issue: "Hybrid Hash Join" is a specific hash join algorithm which is unrelated to what you propose to do here. I hope that confusion can be avoided, possibly by not using th

Re: Support for NSS as a libpq TLS backend

2020-08-05 Thread Andrew Dunstan
On 8/4/20 5:42 PM, Daniel Gustafsson wrote: >> On 3 Aug 2020, at 21:18, Andrew Dunstan >> wrote: >> On 8/3/20 12:46 PM, Andrew Dunstan wrote: >>> On 7/31/20 4:44 PM, Andrew Dunstan wrote: OK, here is an update of your patch that compiles and runs against NSS under Windows (VS2019). >

Re: new heapcheck contrib module

2020-08-05 Thread Peter Geoghegan
On Wed, Aug 5, 2020 at 7:09 AM Robert Haas wrote: > Not sure I agree with this. As a homeowner, the distinction between 0 > and 1 is less significant to me than the distinction between a few > (preferably in places where I'll never see them) and whole lot. I > agree with you to an extent though: a

Re: FailedAssertion("pd_idx == pinfo->nparts", File: "execPartition.c", Line: 1689)

2020-08-05 Thread Robert Haas
On Wed, Aug 5, 2020 at 4:19 PM Tom Lane wrote: > After thinking about it for a little bit, I'm envisioning a test module > that can be loaded into a session, and then it gets into the planner_hook, > and what it does is after each planner execution, take and release an > advisory lock with some se

Re: FailedAssertion("pd_idx == pinfo->nparts", File: "execPartition.c", Line: 1689)

2020-08-05 Thread Tom Lane
Robert Haas writes: > On Wed, Aug 5, 2020 at 2:22 PM Tom Lane wrote: >> In the longer term, it's annoying that we have no test methodology >> for this other than "manually set a breakpoint here". If we're going >> to allow plan-relevant DDL changes to happen with less than full table >> lock, I

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-08-05 Thread Robert Haas
On Wed, May 20, 2020 at 7:44 AM David Rowley wrote: > I've attached a patch which implements this. The new node type is > called "Result Cache". I'm not particularly wedded to keeping that > name, but if I change it, I only want to do it once. I've got a few > other names I mind, but I don't fee

Re: FailedAssertion("pd_idx == pinfo->nparts", File: "execPartition.c", Line: 1689)

2020-08-05 Thread Robert Haas
On Wed, Aug 5, 2020 at 2:22 PM Tom Lane wrote: > In the longer term, it's annoying that we have no test methodology > for this other than "manually set a breakpoint here". If we're going > to allow plan-relevant DDL changes to happen with less than full table > lock, I think we need to improve th

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-08-05 Thread Robert Haas
On Wed, Aug 5, 2020 at 9:42 AM Ashutosh Sharma wrote: > Yeah, it's being tested on the main table, not on a toast table. I've > removed this test-case and also restricted direct access to the toast > table using heap_force_kill/freeze functions. I think we shouldn't be > using these functions to d

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-08-05 Thread Robert Haas
On Mon, Aug 3, 2020 at 12:13 PM Ashutosh Sharma wrote: > If the above path is taken that means none of the items in the page > got changed. Oops. I didn't realize that, sorry. Maybe it would be a little more clear if instead of "int nSkippedItems" you had "bool did_modify_page"? Then you could in

Re: FailedAssertion("pd_idx == pinfo->nparts", File: "execPartition.c", Line: 1689)

2020-08-05 Thread Tom Lane
Robert Haas writes: > On Wed, Aug 5, 2020 at 1:30 PM Tom Lane wrote: >> I'm strongly tempted to convert the trailing Assert to an actual >> test-and-elog, too, but didn't do so here. > I was thinking about that, too. +1 for taking that step. Will do. In the longer term, it's annoying that we h

pg_rewind is not crash safe

2020-08-05 Thread Heikki Linnakangas
A colleague of mine brought to my attention that pg_rewind is not crash safe. If it is interrupted for any reason, it leaves behind a data directory with a mix of data from the source and target images. If you're "lucky", the server will start up, but it can be in an inconsistent state. That's

Re: FailedAssertion("pd_idx == pinfo->nparts", File: "execPartition.c", Line: 1689)

2020-08-05 Thread Robert Haas
On Wed, Aug 5, 2020 at 1:30 PM Tom Lane wrote: > I don't like this patch at all though; I do not think it is being nearly > careful enough to ensure that it's matched the surviving relation OIDs > correctly. In particular it blithely assumes that a zero in relid_map > *must* match the immediately

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-08-05 Thread Andres Freund
Hi, On 2020-08-04 10:05:25 +1200, David Rowley wrote: > I'd like to push the 0002 patch quite soon as I think it's an > improvement to simplehash.h regardless of if we get Result Cache. It > reuses the SH_LOOKUP function for deletes. Also, if we ever get around > to giving up performing a lookup

Re: FailedAssertion("pd_idx == pinfo->nparts", File: "execPartition.c", Line: 1689)

2020-08-05 Thread Tom Lane
Amit Langote writes: > The crash reported here is in the other case where the concurrently > added partitions cause the execution-time PartitionDesc to have more > partitions than the one that PartitionedRelPruneInfo is based on. > I was able to reproduce such a crash as follows: Yeah, I can repe

Re: Replace remaining StrNCpy() by strlcpy()

2020-08-05 Thread Tom Lane
Peter Eisentraut writes: > Okay, here is a new patch with improved implementations of namecpy() and > namestrcpy(). I didn't see any other places that relied on the > zero-filling behavior of strncpy(). I've looked through this patch, and I concur with your conclusion that noplace else is depe

Re: Allow some recovery parameters to be changed with reload

2020-08-05 Thread Robert Haas
On Sat, Mar 28, 2020 at 7:21 AM Sergei Kornilov wrote: > So... > We call restore_command only when walreceiver is stopped. > We use restore_command only in startup process - so we have no race condition > between processes. > We have some issues here? Or we can just make restore_command reloadabl

Re: Yet another issue with step generation in partition pruning

2020-08-05 Thread Etsuro Fujita
Amit-san, On Wed, Aug 5, 2020 at 5:13 PM Amit Langote wrote: > Thanks a lot for your time on fixing these multi-column range > partition pruning issues. I'm sorry that I failed to notice the > previous two reports on -bugs for which you committed a fix last week. No problem. > On Tue, Aug 4, 2

Re: new heapcheck contrib module

2020-08-05 Thread Robert Haas
On Tue, Aug 4, 2020 at 9:06 PM Peter Geoghegan wrote: > of messed-up with indexes in my experience. The first error really > does tell you most of what you need to know about any given corrupt > index. Kind of like how you can bucket the number of cockroaches in > your home into perhaps three mean

Re: [HACKERS] [PATCH] Generic type subscripting

2020-08-05 Thread Dmitry Dolgov
> On Sun, Aug 02, 2020 at 12:50:12PM +0200, Pavel Stehule wrote: > > > > > Maybe this could be salvaged by flushing 0005 in its current form and > > > having the jsonb subscript executor do something like "if the current > > > value-to-be-subscripted is a JSON array, then try to convert the textual

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-08-05 Thread Ashutosh Sharma
Hi Robert, Thanks for the review. Please find my comments inline: On Sat, Aug 1, 2020 at 12:18 AM Robert Haas wrote: > > On Fri, Jul 31, 2020 at 8:52 AM Ashutosh Sharma wrote: > > Attached is the new version of patch that addresses the comments from > > Andrey and Beena. > > +PGFILEDESC = "pg_

Re: [PATCH] - Provide robust alternatives for replace_string

2020-08-05 Thread Alvaro Herrera
On 2020-Aug-05, Asim Praveen wrote: > Please find attached a StringInfo based solution to this problem. It > uses fgetln instead of fgets such that a line is read in full, without > ever splitting it. never heard of fgetln, my system doesn't have a manpage for it, and we don't use it anywhere AF

Re: Can a background worker exist without shared memory access for EXEC_BACKEND cases?

2020-08-05 Thread Robert Haas
On Wed, Aug 5, 2020 at 9:02 AM Bharath Rupireddy wrote: > Will the proposed patch(v2) having some info in bgworker.sgml and > bgworker.h be ever useful to the users in some way? Well, it says things that aren't true, so, no, it's not useful. Your patch claims that "the worker fails to receive the

Re: Can a background worker exist without shared memory access for EXEC_BACKEND cases?

2020-08-05 Thread Bharath Rupireddy
On Wed, Aug 5, 2020 at 5:16 PM Robert Haas wrote: > > On Wed, Aug 5, 2020 at 7:24 AM Bharath Rupireddy > wrote: > > In EXEC_BACKEND cases, right after PGSharedMemoryDetach(), the bg > > worker will no longer be able to access the backend parameters, see > > below(I tried this on my Ubuntu machine

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-08-05 Thread Amit Kapila
On Tue, Aug 4, 2020 at 12:42 PM Dilip Kumar wrote: > > On Tue, Aug 4, 2020 at 10:12 AM Amit Kapila wrote: > > > > > 4. I think we can explain the problems (like we can see the wrong > > tuple or see two versions of the same tuple or whatever else wrong can > > happen, if possible with some examp

Re: Which SET TYPE don't actually require a rewrite

2020-08-05 Thread Magnus Hagander
On Sat, Jul 18, 2020 at 4:57 AM Noah Misch wrote: > On Fri, Jul 17, 2020 at 04:08:36PM +0200, Magnus Hagander wrote: > > On Fri, Jul 17, 2020 at 5:40 AM Noah Misch wrote: > > > On Wed, Jul 15, 2020 at 02:54:37PM +0200, Magnus Hagander wrote: > > > > Our Fine Manual (TM) specifies: > > > > "As an

Re: Can a background worker exist without shared memory access for EXEC_BACKEND cases?

2020-08-05 Thread Robert Haas
On Wed, Aug 5, 2020 at 7:24 AM Bharath Rupireddy wrote: > In EXEC_BACKEND cases, right after PGSharedMemoryDetach(), the bg > worker will no longer be able to access the backend parameters, see > below(I tried this on my Ubuntu machine with a bgworker with no > BGWORKER_SHMEM_ACCESS flag and defin

Re: Can a background worker exist without shared memory access for EXEC_BACKEND cases?

2020-08-05 Thread Bharath Rupireddy
On Tue, Aug 4, 2020 at 10:20 PM Robert Haas wrote: > > On Tue, Aug 4, 2020 at 7:27 AM Bharath Rupireddy > wrote: > > I could get these points earlier in my initial analysis. In fact, I > > could figure out the flow on Windows, how these parameters are shared > > using a shared file(CreateFileMapp

Re: display offset along with block number in vacuum errors

2020-08-05 Thread Robert Haas
On Sun, Aug 2, 2020 at 10:43 PM Masahiko Sawada wrote: > I think that since the user who uses this errcontext information is > likely to know more or less the internal of PostgreSQL I think 0-based > block number and 1-based offset will not be a problem. However, I > expected these are documented

Re: LSM tree for Postgres

2020-08-05 Thread Dmitry Dolgov
> On Tue, Aug 04, 2020 at 11:22:13AM +0300, Konstantin Knizhnik wrote: > > Then I think about implementing ideas of LSM using standard Postgres > nbtree. > > We need two indexes: one small for fast inserts and another - big > (main) index. This top index is small enough to fit in memory so > insert

Re: Add MAIN_RELATION_CLEANUP and SECONDARY_RELATION_CLEANUP options to VACUUM

2020-08-05 Thread Masahiko Sawada
On Mon, 3 Aug 2020 at 15:47, Michael Paquier wrote: > > On Tue, Jul 14, 2020 at 05:34:01AM +, k.jami...@fujitsu.com wrote: > > I've also confirmed those through regression + tap test in my own env > > and they've passed. I'll look into deeply again if I find problems. > > + VACOPT_TOAST_CLEA

Re: [PATCH v1] elog.c: Remove special case which avoided %*s format strings..

2020-08-05 Thread Michael Paquier
On Tue, Aug 04, 2020 at 09:06:16PM +1200, David Rowley wrote: > FWIW, the tests I did to check this when initially working on it are > in [1]. Justin, it would be good if you could verify you're making as > bad as what's mentioned on that thread again. Ouch. Thanks for the reference. Indeed it

Re: Yet another issue with step generation in partition pruning

2020-08-05 Thread Amit Langote
Fujita-san, Thanks a lot for your time on fixing these multi-column range partition pruning issues. I'm sorry that I failed to notice the previous two reports on -bugs for which you committed a fix last week. On Tue, Aug 4, 2020 at 9:46 PM Etsuro Fujita wrote: > Commit 13838740f fixed some issu

Re: Reg. Postgres 13

2020-08-05 Thread Magnus Hagander
On Wed, Aug 5, 2020 at 8:08 AM Joel Mariadasan (jomariad) < jomar...@cisco.com> wrote: > Hi, > > > > I see that release 13 is currently in beta. > > When will be the official production release of 13 be out? > > > > We need to see if we can include this as part of our product release cycle. > > >

Re: LSM tree for Postgres

2020-08-05 Thread Konstantin Knizhnik
On 04.08.2020 20:44, Tomas Vondra wrote: IMO the 6x difference is rather misleading, as it very much depends on the duration of the benchmark and how much data it ends up with. I think it's better to test 'stable states' i.e. with small data set that does not exceed RAM during the whole test,

Re: [PATCH] - Provide robust alternatives for replace_string

2020-08-05 Thread Asim Praveen
> On 03-Aug-2020, at 8:36 PM, Alvaro Herrera wrote: > > On 2020-Aug-03, Asim Praveen wrote: > >> Thank you Alvaro for reviewing the patch! >> >>> On 01-Aug-2020, at 7:22 AM, Alvaro Herrera wrote: >>> >>> What happens if a replacement string happens to be split in the middle >>> by the fgets