Re: Remove unnecessary use of PROCEDURAL

2019-02-25 Thread Peter Eisentraut
On 2019-02-20 15:39, Peter Eisentraut wrote: > I propose the attached patch to remove some unnecessary, legacy-looking > use of the PROCEDURAL keyword before LANGUAGE. We mostly don't use this > anymore, so some of these look a bit old. committed -- Peter Eisentraut http://www.2ndQ

RE: Protect syscache from bloating with negative cache entries

2019-02-25 Thread Tsunakawa, Takayuki
From: Robert Haas [mailto:robertmh...@gmail.com] > I don't understand the idea that we would add something to PostgreSQL > without proving that it has value. Sure, other systems have somewhat > similar systems, and they have knobs to tune them. But, first, we > don't know that those other systems

Re: unconstify equivalent for volatile

2019-02-25 Thread Peter Eisentraut
On 2019-02-22 21:31, Andres Freund wrote: > On 2019-02-22 12:38:35 +0100, Peter Eisentraut wrote: >> On 2019-02-19 18:02, Andres Freund wrote: >>> But even if we were to decide we'd want to keep a volatile in SetLatch() >>> - which I think really would only serve to hide bugs - that'd not mean >>>

Re: some ri_triggers.c cleanup

2019-02-25 Thread Peter Eisentraut
On 2019-02-24 00:34, Corey Huinker wrote: > As I suspected, the code for SET NULL and SET DEFAULT are highly similar > (see .diff), the major difference being two constants, the order of some > variable declarations, and the recheck in the set-default case. > > The changes were so simple that I fe

RE: Protect syscache from bloating with negative cache entries

2019-02-25 Thread Tsunakawa, Takayuki
From: Kyotaro HORIGUCHI [mailto:horiguchi.kyot...@lab.ntt.co.jp] > - If you find the process too much "bloat"s and you (intuirively) > suspect the cause is system cache, set it to certain shorter > value, say 1 minutes, and set the catalog_cache_memory_target > to allowable amount of memory f

Re: [patch] Add schema total size to psql \dn+

2019-02-25 Thread Gilles Darold
Le 22/02/2019 à 19:21, Tom Lane a écrit : > Robert Haas writes: >> On Wed, Feb 20, 2019 at 5:26 PM Gilles Darold >> wrote: >>> The attached simple patch adds this feature. Is there any cons adding >>> this information? >> Well, it'll take time to compute, maybe a lot of time if the database >> i

Re: [patch] Add schema total size to psql \dn+

2019-02-25 Thread Gilles Darold
Le 22/02/2019 à 17:06, Robert Haas a écrit : > On Wed, Feb 20, 2019 at 5:26 PM Gilles Darold > wrote: >> The attached simple patch adds this feature. Is there any cons adding >> this information? > Well, it'll take time to compute, maybe a lot of time if the database > is big and the server is bu

Re: Referential Integrity Checks with Statement-level Triggers

2019-02-25 Thread Antonin Houska
Corey Huinker wrote: > Have you considered bulk processing of individual rows by row-level trigger? > For IMMEDIATE constraints we'd have to ensure that the trigger is notified > that the current row is the last one from the current query, but that might > not be difficult. > > I'm not sure

Re: Should we increase the default vacuum_cost_limit?

2019-02-25 Thread Komяpa
I support rising the default. >From standpoint of no-clue database admin, it's easier to give more resources to Postgres and google what process called "autovacuum" does than to learn why is it being slow on read. It's also tricky that index only scans depend on working autovacuum, and autovacuum

Re: reloption to prevent VACUUM from truncating empty pages at the end of relation

2019-02-25 Thread Michael Paquier
On Mon, Feb 25, 2019 at 03:59:21PM +0900, Masahiko Sawada wrote: > Also, I think that this test may fail in case where concurrent > transactions are running. So maybe should not run it in parallel to > other tests. That's why autovacuum is disabled in this specific test, no? A comment may be a go

Re: reloption to prevent VACUUM from truncating empty pages at the end of relation

2019-02-25 Thread Michael Paquier
On Mon, Feb 25, 2019 at 08:47:28AM +0100, Julien Rouhaud wrote: > Ah good point. We could also use something like > pg_relation_size('reloptions_test') / > current_setting('block_size')::bigint but >0 should be enough for this > test. Also, shouldn't the relopt check happen in should_attempt_trun

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Adrien NAYRAT
On 2/22/19 7:10 PM, Robert Haas wrote: On Fri, Feb 22, 2019 at 12:35 PM Fujii Masao wrote: -1 for the removal. I think that there are still many users of an exclusive backup API, and it's not so easy to migrate their backup scripts so that only non-exclusive one is used because of the reason I

Re: Problems with plan estimates in postgres_fdw

2019-02-25 Thread Etsuro Fujita
(2019/02/23 0:21), Antonin Houska wrote: Etsuro Fujita wrote: (2019/02/08 2:04), Antonin Houska wrote: * regression tests: I think test(s) should be added for queries that have ORDER BY clause but do not have GROUP BY (and also no LIMIT / OFFSET) clause. I haven't noticed such tests. I no

Re: Segfault when restoring -Fd dump on current HEAD

2019-02-25 Thread Dmitry Dolgov
> On Mon, Feb 25, 2019 at 8:45 AM hubert depesz lubaczewski > wrote: > > I did upgrade of my test pg. Part of this is pg_dump -Fd of each > database, then upgrade binaries, then initdb, and pg_restore. > > But - I can't restore any database that has any data - I get segfaults. Thank for reportin

RE: reloption to prevent VACUUM from truncating empty pages at the end of relation

2019-02-25 Thread Tsunakawa, Takayuki
From: Michael Paquier [mailto:mich...@paquier.xyz] On Mon, Feb 25, 2019 at 03:59:21PM +0900, Masahiko Sawada wrote: > > Also, I think that this test may fail in case where concurrent > > transactions are running. So maybe should not run it in parallel to > > other tests. > > That's why autovacuum

Re: reloption to prevent VACUUM from truncating empty pages at the end of relation

2019-02-25 Thread Masahiko Sawada
On Mon, Feb 25, 2019 at 7:01 PM Tsunakawa, Takayuki wrote: > > From: Michael Paquier [mailto:mich...@paquier.xyz] > On Mon, Feb 25, 2019 at 03:59:21PM +0900, Masahiko Sawada wrote: > > > Also, I think that this test may fail in case where concurrent > > > transactions are running. So maybe should

Re: Problems with plan estimates in postgres_fdw

2019-02-25 Thread Etsuro Fujita
(2019/02/22 22:54), Antonin Houska wrote: Etsuro Fujita wrote: Maybe, my explanation in that thread was not enough, but the changes proposed by the patch posted there wouldn't obsolete the above. Let me explain using a foreign-table variant of the example shown in the comments for make_group_i

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Dagfinn Ilmari Mannsåker
David Steele writes: > On 2/25/19 12:35 AM, Christophe Pettus wrote: >> >> >>> On Feb 24, 2019, at 14:19, Stephen Frost wrote: >>> You say above that the new interface is unquestionably an improvement >>> and here say that we shouldn't deprecate the old one in favor of it >>> (even though we act

Re: postgres_fdw: another oddity in costing aggregate pushdown paths

2019-02-25 Thread Etsuro Fujita
(2019/02/22 23:10), Antonin Houska wrote: Etsuro Fujita wrote: As mentioned in the near thread, I think there is another oversight in the cost estimation for aggregate pushdown paths in postgres_fdw, IIUC. When costing an aggregate pushdown path using local statistics, we re-use the estimated

Re: psql show URL with help

2019-02-25 Thread Peter Eisentraut
On 2019-02-21 18:28, Peter Eisentraut wrote: > => \h analyze > URL: https://www.postgresql.org/docs/12/sql-analyze.html > Here is a patch. It doesn't deal with the "devel" paths yet. Discussion there is still ongoing a bit. -- Peter Eisentraut http://www.2ndQuadrant.com/ Pos

RE: Protect syscache from bloating with negative cache entries

2019-02-25 Thread Ideriha, Takeshi
>>From: Tsunakawa, Takayuki >>Ideriha-san, >>Could you try simplifying the v15 patch set to see how simple the code >>would look or not? That is: >> >>* 0001: add dlist_push_tail() ... as is >>* 0002: memory accounting, with correction based on feedback >>* 0003: merge the original 0003 and 0005,

Re: psql show URL with help

2019-02-25 Thread Peter Eisentraut
On 2019-02-22 12:07, Magnus Hagander wrote: > (Won't actually work because the web site isn't serving "12" URLs yet, > but that's something that could probably be sorted out.) > > > Why not just link to /devel/ when it's a devel version? The 12 docs will > be up alongside the first beta v

Re: shared-memory based stats collector

2019-02-25 Thread Arthur Zakirov
On 25.02.2019 07:52, Kyotaro HORIGUCHI wrote: It is fixed by moving StartTransactionCommand to before the first pgstat_f_s_dbentry(), which looks better not having this problem. Thank you. Still there are couple TAP-test which don't pass: 002_archiving.pl and 010_pg_basebackup.pl. I think the

Re: psql show URL with help

2019-02-25 Thread Pavel Stehule
po 25. 2. 2019 v 12:06 odesílatel Peter Eisentraut < peter.eisentr...@2ndquadrant.com> napsal: > On 2019-02-22 12:07, Magnus Hagander wrote: > > (Won't actually work because the web site isn't serving "12" URLs > yet, > > but that's something that could probably be sorted out.) > > > > > >

Re: [PROPOSAL] Shared Ispell dictionaries

2019-02-25 Thread Arthur Zakirov
On 21.02.2019 19:13, Robert Haas wrote: So I think it's better to have each backend locally make a decision about when that particular backend no longer needs the dictionary, and then let the system automatically clean up the ones that are needed by nobody. Yep, it wouldn't be hard to implement

Re: Temporal Table Proposal

2019-02-25 Thread Ibrar Ahmed
Hi Paul, On Sat, Feb 23, 2019 at 2:16 AM Paul Jungwirth wrote: > On 2/22/19 11:31 AM, Euler Taveira wrote: > > Em sex, 22 de fev de 2019 às 15:41, Ibrar Ahmed > > escreveu: > >> > >> While working on another PostgreSQL feature, I was thinking that we > could use a temporal table in PostgreSQL.

Re: NOT IN subquery optimization

2019-02-25 Thread David Rowley
On Fri, 22 Feb 2019 at 09:44, David Rowley wrote: > I've attached the rebased and still broken version. I set about trying to make a less broken version of this. A quick reminder of the semantics of NOT IN: 1. WHERE NOT IN(SELECT FROM table); If table is non-empty: will filter out rows where

Re: INSERT ... OVERRIDING USER VALUE vs GENERATED ALWAYS identity columns

2019-02-25 Thread Peter Eisentraut
On 2019-02-22 15:12, Dean Rasheed wrote: > In particular, Syntax Rule 11b of section 14.11 says that an INSERT > statement on a GENERATED ALWAYS identity column must specify an > overriding clause, but it doesn't place any restriction on the type of > overriding clause allowed. In other words it sh

Re: Should we increase the default vacuum_cost_limit?

2019-02-25 Thread Joe Conway
On 2/25/19 1:17 AM, Peter Geoghegan wrote: > On Sun, Feb 24, 2019 at 9:42 PM David Rowley > wrote: >> The current default vacuum_cost_limit of 200 seems to be 15 years old >> and was added in f425b605f4e. >> >> Any supporters for raising the default? > > I also think that the current default limi

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Stephen Frost
Greetings, * Adrien NAYRAT (adrien.nay...@anayrat.info) wrote: > On 2/22/19 7:10 PM, Robert Haas wrote: > >On Fri, Feb 22, 2019 at 12:35 PM Fujii Masao wrote: > >>-1 for the removal. I think that there are still many users of an exclusive > >>backup API, and it's not so easy to migrate their back

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Stephen Frost
Greetings, * Dagfinn Ilmari Mannsåker (ilm...@ilmari.org) wrote: > David Steele writes: > > On 2/25/19 12:35 AM, Christophe Pettus wrote: > >>> On Feb 24, 2019, at 14:19, Stephen Frost wrote: > >>> You say above that the new interface is unquestionably an improvement > >>> and here say that we s

Re: Should we increase the default vacuum_cost_limit?

2019-02-25 Thread David Rowley
On Tue, 26 Feb 2019 at 02:06, Joe Conway wrote: > > On 2/25/19 1:17 AM, Peter Geoghegan wrote: > > On Sun, Feb 24, 2019 at 9:42 PM David Rowley > > wrote: > >> The current default vacuum_cost_limit of 200 seems to be 15 years old > >> and was added in f425b605f4e. > >> > >> Any supporters for rai

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Stephen Frost
Greetings, * Andres Freund (and...@anarazel.de) wrote: > On 2019-02-24 17:19:22 -0500, Stephen Frost wrote: > > You say above that the new interface is unquestionably an improvement > > FWIW, I think we didn't design it even remotely as well as we ought to > have. It was both a mistake to not off

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Laurenz Albe
Stephen Frost wrote: > > It will be annoying if after this removal, companies must change their > > backup strategy by using specific postgres tools (pgbackrest, barman). > > You don't write your own database system using CSV files and shell > magic, do you? I have to say that it continues to bog

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Stephen Frost
Greetings, * Laurenz Albe (laurenz.a...@cybertec.at) wrote: > Stephen Frost wrote: > > > It will be annoying if after this removal, companies must change their > > > backup strategy by using specific postgres tools (pgbackrest, barman). > > > > You don't write your own database system using CSV f

Re: INSERT ... OVERRIDING USER VALUE vs GENERATED ALWAYS identity columns

2019-02-25 Thread Dean Rasheed
On Mon, 25 Feb 2019 at 12:47, Peter Eisentraut wrote: > > - and the column in new rows will automatically have values from the > > - sequence assigned to it. > > + and new rows in the column will automatically have values from the > > + sequence assigned to them. > > The "it" r

Re: Segfault when restoring -Fd dump on current HEAD

2019-02-25 Thread Justin Pryzby
On Mon, Feb 25, 2019 at 11:01:05AM +0100, Dmitry Dolgov wrote: > > On Mon, Feb 25, 2019 at 8:45 AM hubert depesz lubaczewski > > wrote: > > > > I did upgrade of my test pg. Part of this is pg_dump -Fd of each > > database, then upgrade binaries, then initdb, and pg_restore. > > > > But - I can't

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Christophe Pettus
> On Feb 25, 2019, at 05:14, Stephen Frost wrote: > > You don't write your own database system using CSV files and shell > magic, do you? I have to say that it continues to boggle my mind that > people insist that *this* part of the system has to be able to be > implementable using shell scri

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Christophe Pettus
> On Feb 25, 2019, at 05:18, Stephen Frost wrote: > So.. We've more-or-less come full circle back to where the thread had > left off last time- the plan is to update the documentation to make it > clearer that the exclusive mode is deprecated and that it's going to be > removed, and then we'll

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Christophe Pettus
> On Feb 24, 2019, at 22:49, David Steele wrote: > This is a good idea -- but I'm still a bit mystified why the ability to run a > backup in shell script is considered to be a hard requirement. Just about any > language you can name performs non-exclusive backups with ease -- except > shell.

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Stephen Frost
Greetings, * Christophe Pettus (x...@thebuild.com) wrote: > > On Feb 25, 2019, at 05:18, Stephen Frost wrote: > > So.. We've more-or-less come full circle back to where the thread had > > left off last time- the plan is to update the documentation to make it > > clearer that the exclusive mode i

readdir is incorrectly implemented at Windows

2019-02-25 Thread Konstantin Knizhnik
Hi hackers, Small issue with readir implementation for Windows. Right now it returns ENOENT in case of any error returned by FindFirstFile. So all places in Postgres where opendir/listdir are used will assume that directory is empty and do nothing without reporting any error. It is not so good

Re: Should we increase the default vacuum_cost_limit?

2019-02-25 Thread Laurenz Albe
David Rowley wrote: > On Tue, 26 Feb 2019 at 02:06, Joe Conway wrote: > > > > On 2/25/19 1:17 AM, Peter Geoghegan wrote: > > > On Sun, Feb 24, 2019 at 9:42 PM David Rowley > > > wrote: > > >> The current default vacuum_cost_limit of 200 seems to be 15 years old > > >> and was added in f425b605f4e

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Robert Haas
On Sun, Feb 24, 2019 at 3:00 PM Stephen Frost wrote: > Ok, then please do so, and please be prepared to continue to maintain > the documentation of both methods moving forward, because others have > tried and have (rightfully, in my opinion) decided that it's frankly not > worth the effort and ult

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Robert Haas
On Mon, Feb 25, 2019 at 1:49 AM David Steele wrote: > Exclusive backup will not be removed for PG12. There wasn't support for > it so I push it out to PG13. > > There does appear to be support for removing it in PG13, though, and I'd > like to see that done sooner than later. Given the additiona

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Sun, Feb 24, 2019 at 3:00 PM Stephen Frost wrote: > > Ok, then please do so, and please be prepared to continue to maintain > > the documentation of both methods moving forward, because others have > > tried and have (rightfully, in my

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Robert Haas
On Mon, Feb 25, 2019 at 8:42 AM Stephen Frost wrote: > Alright, then how about we provide a bit of help for everyone who's got > a system built around recovery.conf today, instead of just completely > ripping that out? > > If we're happy to do that then I really can't agree with these arguments >

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Robert Haas
On Fri, Feb 22, 2019 at 6:18 PM Stephen Frost wrote: > What we need is a backup tool included in core that users feel > comfortable using instead of trying to write their own. I agree. That's a great idea. Let's talk about how to make that happen. Providing a tool that gives people MORE AND BE

Re: some ri_triggers.c cleanup

2019-02-25 Thread Corey Huinker
> > Right, this makes a lot of sense, similar to how ri_restrict() combines > RESTRICT and NO ACTION. > I'm pretty sure that's where I got the idea, yes.

Re: Segfault when restoring -Fd dump on current HEAD

2019-02-25 Thread Tom Lane
Justin Pryzby writes: > What's crashing for me is restoring the (12dev) dumpfile using v11 psql: Yeah, I can reproduce that here, using either -Fc or -Fd format dumps. The immediate problem in your example is that the "defn" field of a TABLE DATA entry is now null where it used to be an empty str

Re: Segfault when restoring -Fd dump on current HEAD

2019-02-25 Thread hubert depesz lubaczewski
On Mon, Feb 25, 2019 at 08:45:39AM +0100, hubert depesz lubaczewski wrote: > Hi, > I did upgrade of my test pg. Part of this is pg_dump -Fd of each > database, then upgrade binaries, then initdb, and pg_restore. Sorry, please disregard this problem. Error was sitting on a chair. Best regards, d

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Mon, Feb 25, 2019 at 8:42 AM Stephen Frost wrote: > > Alright, then how about we provide a bit of help for everyone who's got > > a system built around recovery.conf today, instead of just completely > > ripping that out? > > > > If we'

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Robert Haas
On Mon, Feb 25, 2019 at 11:09 AM Stephen Frost wrote: > I do think we can ask that people who wish to have a capability included > in PG (or continue to be included when there are serious known issues > with it...) be prepared to either build and maintain it themselves or to > convince someone els

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Fri, Feb 22, 2019 at 6:18 PM Stephen Frost wrote: > > What we need is a backup tool included in core that users feel > > comfortable using instead of trying to write their own. > > I agree. That's a great idea. Let's talk about how t

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Robert Haas
On Mon, Feb 25, 2019 at 11:23 AM Stephen Frost wrote: > If that argument did matter, we could go back and find the prior > discussions about the issues around the exclusive backup mode and about > removing it, or next year we could point to this thread about it, or the > year after, and say "well,

Re: Should we increase the default vacuum_cost_limit?

2019-02-25 Thread Julien Rouhaud
On Mon, Feb 25, 2019 at 4:44 PM Laurenz Albe wrote: > > David Rowley wrote: > > On Tue, 26 Feb 2019 at 02:06, Joe Conway wrote: > > > > > > On 2/25/19 1:17 AM, Peter Geoghegan wrote: > > > > On Sun, Feb 24, 2019 at 9:42 PM David Rowley > > > > wrote: > > > >> The current default vacuum_cost_limi

Re: Referential Integrity Checks with Statement-level Triggers

2019-02-25 Thread Corey Huinker
> > > In order to avoid per-row calls of the constraint trigger functions, we > could > try to "aggregate" the constraint-specific events somehow, but I think a > separate queue would be needed for the constraint-specific events. > > In general, the (after) triggers and constraints have too much in

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Robert Haas
On Mon, Feb 25, 2019 at 11:33 AM Stephen Frost wrote: > > Removing an option that people are currently using, and which they > > find better than other available options for reasons with which I > > understand that you disagree, will make users more sad. Happy is > > better. > > I don't want to s

Re: Should we increase the default vacuum_cost_limit?

2019-02-25 Thread Robert Haas
On Mon, Feb 25, 2019 at 8:39 AM David Rowley wrote: > I decided to do the times by 10 option that I had mentioned Ensue > debate about that... +1 for raising the default substantially. In my experience, and it seems others are in a similar place, nobody ever gets into trouble because the def

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Mon, Feb 25, 2019 at 11:23 AM Stephen Frost wrote: > > If that argument did matter, we could go back and find the prior > > discussions about the issues around the exclusive backup mode and about > > removing it, or next year we could p

Re: pgsql: Avoid creation of the free space map for small heap relations, t

2019-02-25 Thread Tom Lane
Amit Kapila writes: > Avoid creation of the free space map for small heap relations, take 2. I think this patch still has some issues. Note the following two recent buildfarm failures: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=petalura&dt=2019-02-20%2004%3A20%3A01 https://buildfar

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Mon, Feb 25, 2019 at 11:33 AM Stephen Frost wrote: > > > Removing an option that people are currently using, and which they > > > find better than other available options for reasons with which I > > > understand that you disagree, will

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Mon, Feb 25, 2019 at 11:09 AM Stephen Frost wrote: > > I do think we can ask that people who wish to have a capability included > > in PG (or continue to be included when there are serious known issues > > with it...) be prepared to eit

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Christophe Pettus
> On Feb 25, 2019, at 08:55, Stephen Frost wrote: > > I honestly do doubt that they have had the same experiences that I have > had Well, I guarantee you that no two people on this list have had identical experiences. :) I certainly have been bitten by the problems with the current system.

Re: pgsql: Avoid creation of the free space map for small heap relations, t

2019-02-25 Thread Tom Lane
I wrote: > Amit Kapila writes: >> Avoid creation of the free space map for small heap relations, take 2. > I think this patch still has some issues. Just out of curiosity ... how can it possibly be even a little bit sane that fsm_local_map is a single static data structure, without even any indi

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Fujii Masao
On Mon, Feb 25, 2019 at 10:49 PM Laurenz Albe wrote: > > Stephen Frost wrote: > > > It will be annoying if after this removal, companies must change their > > > backup strategy by using specific postgres tools (pgbackrest, barman). > > > > You don't write your own database system using CSV files a

Auxiliary Processes and MyAuxProc

2019-02-25 Thread Mike Palmiotto
Greetings, I've been looking through the startup code for postmaster forks and saw a couple of mechanisms used. Most forks seem to be using StartChildProcess with a MyAuxProc emumerator, but then some (autovacuum launcher/worker, syslogger, bgworker, archiver) are forked through their own start fu

Re: POC: converting Lists into arrays

2019-02-25 Thread Robert Haas
On Sat, Feb 23, 2019 at 9:24 PM Tom Lane wrote: > Every time this has come up, I've opined that the right fix is to jack > up the List API and drive a new implementation underneath, as we did > once before (cf commit d0b4399d81). I thought maybe it was about time > to provide some evidence for th

Re: POC: converting Lists into arrays

2019-02-25 Thread Tom Lane
Robert Haas writes: > On Sat, Feb 23, 2019 at 9:24 PM Tom Lane wrote: >> Every time this has come up, I've opined that the right fix is to jack >> up the List API and drive a new implementation underneath, as we did >> once before (cf commit d0b4399d81). > I'm not really convinced that this is t

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread David Steele
On 2/25/19 7:50 PM, Fujii Masao wrote: On Mon, Feb 25, 2019 at 10:49 PM Laurenz Albe wrote: I'm not playing devil's advocate here to annoy you. I see the problems with the exclusive backup, and I see how it can hurt people. I just think that removing exclusive backup without some kind of help

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Bruce Momjian
On Mon, Feb 25, 2019 at 11:33:33AM -0500, Stephen Frost wrote: > I don't want to see more users stumbling over the issues with the > exclusive backup interface. A better interface exists, and has existed > since 9.6. Do you really think we would be having this discussion if the non-exclusive back

Re: Auxiliary Processes and MyAuxProc

2019-02-25 Thread Tom Lane
Mike Palmiotto writes: > I've been looking through the startup code for postmaster forks and > saw a couple of mechanisms used. Most forks seem to be using > StartChildProcess with a MyAuxProc emumerator, but then some > (autovacuum launcher/worker, syslogger, bgworker, archiver) are forked > thro

Re: POC: converting Lists into arrays

2019-02-25 Thread Robert Haas
On Mon, Feb 25, 2019 at 1:17 PM Tom Lane wrote: > I'm not following your point here. If we change key data structures > (i.e. parsetrees, plan trees, execution trees) to use some other list-ish > API, that *in itself* breaks everything that accesses those data > structures. The approach I propos

Re: Auxiliary Processes and MyAuxProc

2019-02-25 Thread Mike Palmiotto
On Mon, Feb 25, 2019 at 1:29 PM Tom Lane wrote: > > Mike Palmiotto writes: > > > > For some context, I'm trying to come up with a patch to set the > > process identifier (MyAuxProc/am_autovacuumworker/launcher, > > am_archiver, etc.) immediately after forking, > > Don't we do that already? Kind

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread David Steele
Hi Christophe, On 2/25/19 7:24 PM, Christophe Pettus wrote: On Feb 25, 2019, at 08:55, Stephen Frost wrote: I honestly do doubt that they have had the same experiences that I have had Well, I guarantee you that no two people on this list have had identical experiences. :) I certainly ha

Re: POC: converting Lists into arrays

2019-02-25 Thread Tom Lane
Robert Haas writes: > On Mon, Feb 25, 2019 at 1:17 PM Tom Lane wrote: >> I'm not following your point here. If we change key data structures >> (i.e. parsetrees, plan trees, execution trees) to use some other list-ish >> API, that *in itself* breaks everything that accesses those data >> structu

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Stephen Frost
Greetings, * Christophe Pettus (x...@thebuild.com) wrote: > > On Feb 25, 2019, at 08:55, Stephen Frost wrote: > > > > I honestly do doubt that they have had the same experiences that I have > > had > > Well, I guarantee you that no two people on this list have had identical > experiences. :)

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Stephen Frost
Greetings, * David Steele (da...@pgmasters.net) wrote: > On 2/25/19 7:50 PM, Fujii Masao wrote: > >Thought? > > Here's the really obvious bad thing: if users do not update their procedures > and we ignore backup_label.pending on startup then they will end up with a > corrupt database because it w

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Bruce Momjian
On Mon, Feb 25, 2019 at 02:24:18PM -0500, Stephen Frost wrote: > * Bruce Momjian (br...@momjian.us) wrote: > > On Mon, Feb 25, 2019 at 11:33:33AM -0500, Stephen Frost wrote: > > > I don't want to see more users stumbling over the issues with the > > > exclusive backup interface. A better interface

Re: \describe*

2019-02-25 Thread Robert Haas
On Sat, Feb 23, 2019 at 7:19 PM Andres Freund wrote: > Sure, but it was late, and we have far more patches than we can deal > with. Many of them much much older than this. More importantly, at least in my opinion, is that this is one of those questions that people tend to have very strong feeling

Re: POC: converting Lists into arrays

2019-02-25 Thread Peter Geoghegan
On Mon, Feb 25, 2019 at 10:59 AM Tom Lane wrote: > Because it involves touching ten times more code (and that's a very > conservative estimate). Excluding changes in pg_list.h + list.c, > what I posted touches approximately 600 lines of code (520 insertions, > 644 deletions to be exact). For com

Re: POC: converting Lists into arrays

2019-02-25 Thread Andres Freund
Hi, On 2019-02-25 13:02:03 -0500, Robert Haas wrote: > On Sat, Feb 23, 2019 at 9:24 PM Tom Lane wrote: > > Every time this has come up, I've opined that the right fix is to jack > > up the List API and drive a new implementation underneath, as we did > > once before (cf commit d0b4399d81). I tho

Re: [HACKERS] generated columns

2019-02-25 Thread Peter Eisentraut
On 2019-01-15 08:13, Michael Paquier wrote: >>> Ah, the volatility checking needs some improvements. I'll address that >>> in the next patch version. >> >> ok > > The same problem happens for stored and virtual columns. This is fixed in v8. > It would be nice to add a test with composite types,

Re: POC: converting Lists into arrays

2019-02-25 Thread Tom Lane
Peter Geoghegan writes: > If we knew that the list code was the bottleneck in a handful of > cases, then I'd come down on Robert's side here. It would then be > possible to update the relevant bottlenecked code in an isolated > fashion, while getting the lion's share of the benefit. However, I > d

Re: POC: converting Lists into arrays

2019-02-25 Thread Andres Freund
Hi, On 2019-02-25 13:59:36 -0500, Tom Lane wrote: > Robert Haas writes: > > On Mon, Feb 25, 2019 at 1:17 PM Tom Lane wrote: > >> I'm not following your point here. If we change key data structures > >> (i.e. parsetrees, plan trees, execution trees) to use some other list-ish > >> API, that *in

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Andres Freund
Hi, On 2019-02-25 08:14:16 -0500, Stephen Frost wrote: > > It will be annoying if after this removal, companies must change their > > backup strategy by using specific postgres tools (pgbackrest, barman). > > You don't write your own database system using CSV files and shell > magic, do you? I h

Re: POC: converting Lists into arrays

2019-02-25 Thread Tom Lane
Andres Freund writes: > Yea, it'd be more convincing. I'm not convinced it'd be a no-brainer > though. Unless you've been hacking PG for a fair bit, the pg_list.h APIs > are very hard to understand / remember. Given this change essentially > requires auditing all code that uses List, ISTM we'd be

Re: POC: converting Lists into arrays

2019-02-25 Thread Andres Freund
Hi, On 2019-02-25 11:59:06 -0800, Peter Geoghegan wrote: > On Mon, Feb 25, 2019 at 10:59 AM Tom Lane wrote: > > Because it involves touching ten times more code (and that's a very > > conservative estimate). Excluding changes in pg_list.h + list.c, > > what I posted touches approximately 600 lin

Re: POC: converting Lists into arrays

2019-02-25 Thread Tom Lane
Andres Freund writes: > FWIW, rewrites of this kind can be quite nicely automated using > coccinelle [1]. One sometimes needs to do a bit of mop-up with variable > names, but otherwise it should be mostly complete. I'm getting slightly annoyed by arguments that reject a live, workable patch in fa

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Andres Freund
On 2019-02-25 08:42:02 -0500, Stephen Frost wrote: > Greetings, > > * Andres Freund (and...@anarazel.de) wrote: > > On 2019-02-24 17:19:22 -0500, Stephen Frost wrote: > > > You say above that the new interface is unquestionably an improvement > > > > FWIW, I think we didn't design it even remotel

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Andres Freund
Hi, On 2019-02-25 20:43:01 +0200, David Steele wrote: > fsync() is the major corruption issue we are facing right now but that > doesn't mean there aren't other sources of corruption we should be thinking > about. I've thought about this one a lot and it scares me. FWIW, I think this kind of iss

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Christophe Pettus
> On Feb 25, 2019, at 11:24, Stephen Frost wrote: > Aren't they going to need to make a change for v12 now anyway? > > Hopefully they're regularly testing their backups by doing a restore of > them, and dropping a recovery.conf into the directory of a v12 system > after restore will do exactly

Re: POC: converting Lists into arrays

2019-02-25 Thread Andres Freund
Hi, On 2019-02-25 16:03:43 -0500, Tom Lane wrote: > Andres Freund writes: > > FWIW, rewrites of this kind can be quite nicely automated using > > coccinelle [1]. One sometimes needs to do a bit of mop-up with variable > > names, but otherwise it should be mostly complete. > > I'm getting slightl

Re: POC: converting Lists into arrays

2019-02-25 Thread Peter Geoghegan
On Mon, Feb 25, 2019 at 1:04 PM Tom Lane wrote: > I'm getting slightly annoyed by arguments that reject a live, workable > patch in favor of pie-in-the-sky proposals. Both you and Robert seem > to be advocating solutions that don't exist and would take a very large > amount of work to create. If

[RFC] [PATCH] Flexible "partition pruning" hook

2019-02-25 Thread Mike Palmiotto
Greetings, Attached is a patch which attempts to solve a few problems: 1) Filtering out partitions flexibly based on the results of an external function call (supplied by an extension). 2) Filtering out partitions from pg_inherits based on the same function call. 3) Filtering out partitions from

Re: POC: converting Lists into arrays

2019-02-25 Thread Andres Freund
Hi, On 2019-02-25 13:21:30 -0800, Peter Geoghegan wrote: > ISTM that we should separate the question of whether or not the List > API needs to continue to work without needing to change code in third > party extensions from the question of whether or not the List API > needs to be replaced whole c

Re: POC: converting Lists into arrays

2019-02-25 Thread Tomas Vondra
On 2/25/19 10:03 PM, Andres Freund wrote: > Hi, > > On 2019-02-25 11:59:06 -0800, Peter Geoghegan wrote: >> On Mon, Feb 25, 2019 at 10:59 AM Tom Lane wrote: >>> Because it involves touching ten times more code (and that's a very >>> conservative estimate). Excluding changes in pg_list.h + lis

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread David Steele
On 2/25/19 11:20 PM, Christophe Pettus wrote: On Feb 25, 2019, at 11:24, Stephen Frost wrote: Aren't they going to need to make a change for v12 now anyway? Hopefully they're regularly testing their backups by doing a restore of them, and dropping a recovery.conf into the directory of a v12

Re: POC: converting Lists into arrays

2019-02-25 Thread Andres Freund
Hi, On 2019-02-25 22:35:37 +0100, Tomas Vondra wrote: > So let's say we want to measure the improvement this patch gives us. > What would be some reasonable (and corner) cases to benchmark? I do have > some ideas, but as you've been looking at this in the past, perhaps you > have something better.

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread David Steele
Hi Andres, On 2/25/19 10:57 PM, Andres Freund wrote: Hi, On 2019-02-25 08:14:16 -0500, Stephen Frost wrote: It will be annoying if after this removal, companies must change their backup strategy by using specific postgres tools (pgbackrest, barman). You don't write your own database system u

  1   2   >