Re: [RFC] [PATCH] Flexible "partition pruning" hook

2019-02-25 Thread Michael Paquier
On Tue, Feb 26, 2019 at 06:55:30AM +, Tsunakawa, Takayuki wrote: > What concrete problems would you expect this patch to solve? What > kind of extensions do you imagine? I'd like to hear about the > examples. For example, "PostgreSQL 12 will not be able to filter > out enough partitions when

Re: Unused parameters & co in code

2019-02-25 Thread Michael Paquier
On Thu, Jan 31, 2019 at 03:47:59PM +0900, Michael Paquier wrote: > - 0001 cleans up port in SendAuthRequest. This one is disappointing, > so I am fine to discard it. > - 0002 works on _bt_relbuf, whose callers don't actually benefit from > the cleanup as the relation worked on is always used for d

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Laurenz Albe
Fujii Masao wrote: > Another idea is to improve an exclusive backup method so that it will never > cause such issue. What about changing an exclusive backup mode of > pg_start_backup() so that it creates something like backup_label.pending file > instead of backup_label? Then if the database cluste

inted RE: Timeout parameters

2019-02-25 Thread Nagaura, Ryohei
Hi, kirk-san. > From: Nagaura, Ryohei > This is my bad. I'll remake it. > Very sorry for the same mistake. I remade the patches and attached in this mail. In socket_timeout patch, I replaced "atoi" to "parse_int_param" and inserted spaces just after some comma. There are a few changes about docu

RE: [RFC] [PATCH] Flexible "partition pruning" hook

2019-02-25 Thread Tsunakawa, Takayuki
From: Mike Palmiotto [mailto:mike.palmio...@crunchydata.com] > Attached is a patch which attempts to solve a few problems: > > 1) Filtering out partitions flexibly based on the results of an external > function call (supplied by an extension). > 2) Filtering out partitions from pg_inherits based o

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread David Steele
On 2/26/19 6:51 AM, Michael Paquier wrote: On Mon, Feb 25, 2019 at 08:17:27PM +0200, David Steele wrote: Here's the really obvious bad thing: if users do not update their procedures and we ignore backup_label.pending on startup then they will end up with a corrupt database because it will not re

Re: Reaping Temp tables to avoid XID wraparound

2019-02-25 Thread Michael Paquier
On Fri, Feb 22, 2019 at 04:01:02PM +0100, Magnus Hagander wrote: > I did the "insert column in the middle of pg_stat_get_activity", I'm not > sure that is right -- how do we treate that one? Do we just append at the > end because people are expected to use the pg_stat_activity view? It's a > nontri

RE: reloption to prevent VACUUM from truncating empty pages at the end of relation

2019-02-25 Thread Tsunakawa, Takayuki
From: Masahiko Sawada [mailto:sawada.m...@gmail.com] > This test expects that the inserted tuple is always reclaimed by > subsequent vacuum, but it's not always true if there are concurrent > transactions. So size of the reloptions_test table will not be 0 if > the tuple is not vacuumed. In my envi

Re: No-rewrite timestamp<->timestamptz conversions

2019-02-25 Thread Noah Misch
On Thu, Feb 05, 2015 at 08:36:18PM -0500, Noah Misch wrote: > On Tue, Nov 05, 2013 at 05:02:58PM -0800, Josh Berkus wrote: > > I'd also love some way of doing a no-rewrite conversion between > > timestamp and timestamptz, based on the assumption that the original > > values are UTC time. That's on

Re: Segfault when restoring -Fd dump on current HEAD

2019-02-25 Thread Michael Paquier
On Tue, Feb 26, 2019 at 12:16:35AM -0500, Tom Lane wrote: > Well, if we didn't want to fix this, a reasonable way to go about > it would be to bump the archive version number in pg_dump output, > so that old versions would issue a useful complaint instead of crashing. > However, I repeat that this

Re: POC: converting Lists into arrays

2019-02-25 Thread Tom Lane
David Rowley writes: > Using the attached patch (as text file so as not to upset the CFbot), > which basically just measures and logs the time taken to run > pg_plan_query. ... > Surprisingly it took 1.13% longer. I did these tests on an AWS > md5.large instance. Interesting. Seems to suggest t

Re: [HACKERS] generated columns

2019-02-25 Thread Michael Paquier
On Mon, Feb 25, 2019 at 09:46:35PM +0100, Peter Eisentraut wrote: > The virtual generated column part is still a bit iffy. I'm still > finding places here and there where virtual columns are not being > expanded correctly. Maybe it needs more refactoring. One big unsolved > issue is how the stor

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Mark Kirkwood
On 26/02/19 5:41 PM, Stephen Frost wrote: Greetings Mark, * Mark Kirkwood (mark.kirkw...@catalyst.net.nz) wrote: ISTM that the onus should be on the patch submitter to provide additions to pg_basebackup that make it as painless as possible for those people *not* using pgBackRest to continue m

Re: Segfault when restoring -Fd dump on current HEAD

2019-02-25 Thread Tom Lane
Michael Paquier writes: > On Mon, Feb 25, 2019 at 11:20:14AM -0500, Tom Lane wrote: >> It appears to me that f831d4acc required a good deal more adult >> supervision than it actually got. That was alleged to be a small >> notational refactoring, not a redefinition of what gets put into >> dump fi

Re: [HACKERS] generated columns

2019-02-25 Thread Michael Paquier
On Mon, Feb 25, 2019 at 09:51:52PM +0100, Peter Eisentraut wrote: > On 2019-01-15 08:13, Michael Paquier wrote: >> + boolhas_generated_stored; >> + boolhas_generated_virtual; >> } TupleConstr; >> Could have been more simple to use a char as representation here. > > Seems confu

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Michael Paquier
On Mon, Feb 25, 2019 at 08:17:27PM +0200, David Steele wrote: > Here's the really obvious bad thing: if users do not update their procedures > and we ignore backup_label.pending on startup then they will end up with a > corrupt database because it will not replay from the correct checkpoint. If >

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Stephen Frost
Greetings Mark, * Mark Kirkwood (mark.kirkw...@catalyst.net.nz) wrote: > ISTM that the onus should be on the patch submitter to provide additions to > pg_basebackup that make it as painless as possible for those people *not* > using pgBackRest to continue making backups. Breaking this is just not

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Mark Kirkwood
On 26/02/19 4:53 PM, Robert Haas wrote: On Mon, Feb 25, 2019 at 4:38 PM David Steele wrote: FWIW, if you weren't selling backrest quite so hard everywhere backups are mentioned, I'd find this thread a lot more convicing. pgBackRest has not used exclusive backups since the new API was introdu

Re: Segfault when restoring -Fd dump on current HEAD

2019-02-25 Thread Michael Paquier
On Mon, Feb 25, 2019 at 11:20:14AM -0500, Tom Lane wrote: > It appears to me that f831d4acc required a good deal more adult > supervision than it actually got. That was alleged to be a small > notational refactoring, not a redefinition of what gets put into > dump files. How much consistent do we

Re: [HACKERS] Block level parallel vacuum

2019-02-25 Thread Haribabu Kommi
On Thu, Feb 14, 2019 at 9:17 PM Masahiko Sawada wrote: > Thank you. Attached the rebased patch. > I ran some performance tests to compare the parallelism benefits, but I got some strange results of performance overhead, may be it is because, I tested it on my laptop. FYI, Table schema: create

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Stephen Frost
Robert, * Robert Haas (robertmh...@gmail.com) wrote: > Hmm, so what you're saying is that you'd like to disable an API that > some non-backrest users are relying upon but which no backrest users > are relying upon. And you don't understand why some non-backrest > users are opposed to that plan.

Re: Pluggable Storage - Andres's take

2019-02-25 Thread Amit Khandekar
On Sat, 23 Feb 2019 at 01:22, Robert Haas wrote: > > On Fri, Feb 22, 2019 at 11:19 AM Amit Khandekar > wrote: > > Thanks for the review. Attached v2. > > Thanks. I took this, combined it with Andres's > v12-0040-WIP-Move-xid-horizon-computation-for-page-level-.patch, did > some polishing of the

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Robert Haas
On Mon, Feb 25, 2019 at 4:38 PM David Steele wrote: > > FWIW, if you weren't selling backrest quite so hard everywhere backups > > are mentioned, I'd find this thread a lot more convicing. > > pgBackRest has not used exclusive backups since the new API was > introduced in 9.6 so this is not an iss

Re: pgsql: Avoid creation of the free space map for small heap relations, t

2019-02-25 Thread Amit Kapila
On Tue, Feb 26, 2019 at 8:38 AM John Naylor wrote: > > On Tue, Feb 26, 2019 at 8:59 AM Amit Kapila wrote: > > I will look into it today and respond back with my findings. John, > > see if you also get time to investigate this. > > Will do, but it will likely be tomorrow > No problem, thanks! -

Re: pgsql: Avoid creation of the free space map for small heap relations, t

2019-02-25 Thread Amit Kapila
On Mon, Feb 25, 2019 at 11:14 PM Tom Lane wrote: > > I wrote: > > Amit Kapila writes: > >> Avoid creation of the free space map for small heap relations, take 2. > > > I think this patch still has some issues. > > Just out of curiosity ... how can it possibly be even a little bit sane > that fsm_

Re: pgsql: Avoid creation of the free space map for small heap relations, t

2019-02-25 Thread John Naylor
On Tue, Feb 26, 2019 at 8:59 AM Amit Kapila wrote: > I will look into it today and respond back with my findings. John, > see if you also get time to investigate this. Will do, but it will likely be tomorrow -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x

Re: POC: converting Lists into arrays

2019-02-25 Thread David Rowley
On Sun, 24 Feb 2019 at 15:24, Tom Lane wrote: > I haven't attempted any performance measurements on this, but at > least in principle it should speed things up a bit, especially > for complex test cases involving longer Lists. I don't have any > very suitable test cases at hand, anyway. I've not

RE: SQL statement PREPARE does not work in ECPG

2019-02-25 Thread Takahashi, Ryohei
Hi Matsumura-san, Thank you for your explaining. > An important point of the route is that it calls PQprepare() and PQprepare() > needs type-Oid list. (Idea-1) If we fix for Prepare statement with AS clause > and > with parameter list to walk through the route, preprocessor must parse the > pa

Re: JIT on FreeBSD ARMv7

2019-02-25 Thread Andrew Gierth
> "Andrew" == Andrew Gierth writes: Andrew> This seems to be the cleanest way to get a JIT build working on Andrew> FreeBSD on the armv7 platform. Without this, it fails because Andrew> the ABI functions __aeabi_ldivmod and so on are not found by Andrew> the symbol lookup for JITted code.

Re: POC: converting Lists into arrays

2019-02-25 Thread Andres Freund
Hi, On 2019-02-25 17:55:46 -0800, Andres Freund wrote: > Hm, I wonder if that's necessary / whether we can just work around user > visible breakage at a small cost. I think I'm mostly concerned with two > allocations for the very common case of small (1-3 entries) lists. We > could just allocate t

Re: pgsql: Avoid creation of the free space map for small heap relations, t

2019-02-25 Thread Amit Kapila
On Mon, Feb 25, 2019 at 10:32 PM Tom Lane wrote: > > Amit Kapila writes: > > Avoid creation of the free space map for small heap relations, take 2. > > I think this patch still has some issues. > I will look into it today and respond back with my findings. John, see if you also get time to inve

Re: POC: converting Lists into arrays

2019-02-25 Thread Andres Freund
Hi, On 2019-02-25 18:41:17 -0500, Tom Lane wrote: > I wrote: > > Andres Freund writes: > >>> 1. This still involves at least two palloc's for every nonempty List, > >>> because I kept the header and the data array separate. Perhaps it's > >>> worth allocating those in one palloc. > > >> Hm, I t

RE: Timeout parameters

2019-02-25 Thread Nagaura, Ryohei
Hi, kirk-san. > From: Jamison, Kirk > $ git apply ../socket_timeout_v5.patch > fatal: corrupt patch at line 24 > --> l24: diff --git a/src/interfaces/libpq/fe-connect.c > --> b/src/interfaces/libpq/fe-connect.c How about applying by "patch -p1"? I have confirmed that my patch could be applied in

crosstab/repivot...any interest?

2019-02-25 Thread Merlin Moncure
On Fri, Jan 25, 2019 at 9:14 PM Morris de Oryx wrote: > > Hello, I'm not a C coder and can't helpbut I love cross-tab/pivot tables. They're the best, and just fantastic for preparing data to feed into various analysis tools. The tablefunc module is helpful, but a bit awkward to use (as I remem

Re: current_logfiles not following group access and instead follows log_file_mode permissions

2019-02-25 Thread Haribabu Kommi
On Mon, Feb 4, 2019 at 12:16 PM Haribabu Kommi wrote: > > And regarding current_logfiles permissions, I feel this file should have > permissions of data directory files as it is present in the data directory > whether it stores the information of log file, until this file is > completely > remove

RE: Timeout parameters

2019-02-25 Thread Jamison, Kirk
On Monday, February 25, 2019 1:49 PM (GMT+9), Nagaura, Ryohei wrote: > Thank you for discussion. > I made documentation about socket_timeout and reflected Tsunakawa-san's > comment in the new patch. > # Mainly only fixing documentation... > The current documentation of socket_timeout is as follows

Re: Allowing extensions to supply operator-/function-specific info

2019-02-25 Thread Tom Lane
Paul Ramsey writes: > On Mon, Feb 25, 2019 at 3:01 PM Tom Lane wrote: >> It's whichever one the index column's opclass belongs to. Basically what >> you're trying to do here is verify whether the index will support the >> optimization you want to perform. > * If I have tbl1.geom > * and I have

Re: POC: converting Lists into arrays

2019-02-25 Thread Tom Lane
I wrote: > Andres Freund writes: >>> 1. This still involves at least two palloc's for every nonempty List, >>> because I kept the header and the data array separate. Perhaps it's >>> worth allocating those in one palloc. >> Hm, I think if we force external code to audit their code, we better >>

Re: Allowing extensions to supply operator-/function-specific info

2019-02-25 Thread Paul Ramsey
On Mon, Feb 25, 2019 at 3:01 PM Tom Lane wrote: > > Looking at the examples, they are making use of the opfamily that > > comes in SupportRequestIndexCondition.opfamily. > > That opfamily Oid is the first one in the IndexOptInfo.opfamily array. > > Here's where my thread of understanding fails to

Re: NOT IN subquery optimization

2019-02-25 Thread David Rowley
On Tue, 26 Feb 2019 at 11:51, Li, Zheng wrote: > Resend the patch with a whitespace removed so that "git apply patch" works > directly. I had a quick look at this and it seems to be broken for the empty table case I mentioned up thread. Quick example: Setup: create table t1 (a int); create ta

Re: POC: converting Lists into arrays

2019-02-25 Thread Tom Lane
Andres Freund writes: > Btw, should we remove the ENABLE_LIST_COMPAT stuff independent of this > discussion? Seems like we could even just do that for 12. +1. I took it out in the POC patch, but I see no very good reason not to do it sooner than that. >> The most critical thing that we lose by

Re: Allowing extensions to supply operator-/function-specific info

2019-02-25 Thread Tom Lane
Paul Ramsey writes: > So... trying to figure out how to use SupportRequestIndexCondition to > convert a call to Intersects() in to a call that also has the operator > && as well. OK. > Looking at the examples, they are making use of the opfamily that > comes in SupportRequestIndexCondition.opfam

Re: POC: converting Lists into arrays

2019-02-25 Thread Andres Freund
Hi, On 2019-02-23 21:24:40 -0500, Tom Lane wrote: > For quite some years now there's been dissatisfaction with our List > data structure implementation. Because it separately palloc's each > list cell, it chews up lots of memory, and it's none too cache-friendly > because the cells aren't necessa

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Stephen Frost
Greetings, * Andres Freund (and...@anarazel.de) wrote: > On 2019-02-25 08:42:02 -0500, Stephen Frost wrote: > > * Andres Freund (and...@anarazel.de) wrote: > > > On 2019-02-24 17:19:22 -0500, Stephen Frost wrote: > > > > You say above that the new interface is unquestionably an improvement > > >

Re: Allowing extensions to supply operator-/function-specific info

2019-02-25 Thread Paul Ramsey
On Mon, Jan 28, 2019 at 9:51 AM Tom Lane wrote: > is people like PostGIS, who already cleared that bar. I hope that > we'll soon have a bunch of examples, like those in the 0004 patch, > that people can look at to see how to do things in this area. I see > no reason to believe it'll be all that

Re: Auxiliary Processes and MyAuxProc

2019-02-25 Thread Mike Palmiotto
On Mon, Feb 25, 2019 at 1:41 PM Mike Palmiotto wrote: > > > > > > If memory serves, StartChildProcess already was an attempt to unify > > the treatment of postmaster children. It's possible that another > > round of unification would be productive, but I think you'll find > > that there are rand

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread David Steele
On 2/25/19 11:59 PM, Christophe Pettus wrote: On Feb 25, 2019, at 13:38, Andres Freund wrote: I think you might be right about this specific issue. But to me it sounds like you also don't appreciate that development resources are really constrained too, and providing endless backward compati

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Stephen Frost
Greetings, * Andres Freund (and...@anarazel.de) wrote: > On 2019-02-25 08:14:16 -0500, Stephen Frost wrote: > > > It will be annoying if after this removal, companies must change their > > > backup strategy by using specific postgres tools (pgbackrest, barman). > > > > You don't write your own da

Re: POC: converting Lists into arrays

2019-02-25 Thread Peter Geoghegan
On Mon, Feb 25, 2019 at 1:51 PM Andres Freund wrote: > Those could trivially support distinguisiong at least between lists > containing pointer, int, oid, or node. But even optionally doing more > than that would be fairly easy. It's not those modules don't currently > know the types of elements t

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Christophe Pettus
> On Feb 25, 2019, at 13:38, Andres Freund wrote: > > I think you might be right about this specific issue. But to me it > sounds like you also don't appreciate that development resources are > really constrained too, and providing endless backward compatibility for > everything is going to us

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread David Steele
On 2/25/19 11:38 PM, Andres Freund wrote: Hi, On 2019-02-25 09:24:32 -0800, Christophe Pettus wrote: But the resistance to major version upgrades is *huge*, and I'm strongly biased against anything that will make that harder. I'm not sure I'm communicating how big a problem telling many large

Re: POC: converting Lists into arrays

2019-02-25 Thread Peter Geoghegan
On Mon, Feb 25, 2019 at 1:31 PM Andres Freund wrote: > > Andres said that he doesn't like the pg_list.h API. It's not pretty, > > but is it really that bad? > > Yes. The function names alone confound anybody new to postgres, we tend > to forget that after a few years. A lot of the function return

Re: POC: converting Lists into arrays

2019-02-25 Thread Andres Freund
Hi, On 2019-02-25 13:41:48 -0800, Peter Geoghegan wrote: > On Mon, Feb 25, 2019 at 1:31 PM Andres Freund wrote: > > > Andres said that he doesn't like the pg_list.h API. It's not pretty, > > > but is it really that bad? > > > > Yes. The function names alone confound anybody new to postgres, we te

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread David Steele
Hi Andres, On 2/25/19 10:57 PM, Andres Freund wrote: Hi, On 2019-02-25 08:14:16 -0500, Stephen Frost wrote: It will be annoying if after this removal, companies must change their backup strategy by using specific postgres tools (pgbackrest, barman). You don't write your own database system u

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Magnus Hagander
On Mon, Feb 25, 2019 at 10:14 PM Andres Freund wrote: > On 2019-02-25 08:42:02 -0500, Stephen Frost wrote: > > Greetings, > > > > * Andres Freund (and...@anarazel.de) wrote: > > > On 2019-02-24 17:19:22 -0500, Stephen Frost wrote: > > > > You say above that the new interface is unquestionably an

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Andres Freund
Hi, On 2019-02-25 09:24:32 -0800, Christophe Pettus wrote: > But the resistance to major version upgrades is *huge*, and I'm > strongly biased against anything that will make that harder. I'm not > sure I'm communicating how big a problem telling many large > installations, "If you move to v12/13

Re: POC: converting Lists into arrays

2019-02-25 Thread Andres Freund
Hi, On 2019-02-25 22:35:37 +0100, Tomas Vondra wrote: > So let's say we want to measure the improvement this patch gives us. > What would be some reasonable (and corner) cases to benchmark? I do have > some ideas, but as you've been looking at this in the past, perhaps you > have something better.

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread David Steele
On 2/25/19 11:20 PM, Christophe Pettus wrote: On Feb 25, 2019, at 11:24, Stephen Frost wrote: Aren't they going to need to make a change for v12 now anyway? Hopefully they're regularly testing their backups by doing a restore of them, and dropping a recovery.conf into the directory of a v12

Re: POC: converting Lists into arrays

2019-02-25 Thread Tomas Vondra
On 2/25/19 10:03 PM, Andres Freund wrote: > Hi, > > On 2019-02-25 11:59:06 -0800, Peter Geoghegan wrote: >> On Mon, Feb 25, 2019 at 10:59 AM Tom Lane wrote: >>> Because it involves touching ten times more code (and that's a very >>> conservative estimate). Excluding changes in pg_list.h + lis

Re: POC: converting Lists into arrays

2019-02-25 Thread Andres Freund
Hi, On 2019-02-25 13:21:30 -0800, Peter Geoghegan wrote: > ISTM that we should separate the question of whether or not the List > API needs to continue to work without needing to change code in third > party extensions from the question of whether or not the List API > needs to be replaced whole c

[RFC] [PATCH] Flexible "partition pruning" hook

2019-02-25 Thread Mike Palmiotto
Greetings, Attached is a patch which attempts to solve a few problems: 1) Filtering out partitions flexibly based on the results of an external function call (supplied by an extension). 2) Filtering out partitions from pg_inherits based on the same function call. 3) Filtering out partitions from

Re: POC: converting Lists into arrays

2019-02-25 Thread Peter Geoghegan
On Mon, Feb 25, 2019 at 1:04 PM Tom Lane wrote: > I'm getting slightly annoyed by arguments that reject a live, workable > patch in favor of pie-in-the-sky proposals. Both you and Robert seem > to be advocating solutions that don't exist and would take a very large > amount of work to create. If

Re: POC: converting Lists into arrays

2019-02-25 Thread Andres Freund
Hi, On 2019-02-25 16:03:43 -0500, Tom Lane wrote: > Andres Freund writes: > > FWIW, rewrites of this kind can be quite nicely automated using > > coccinelle [1]. One sometimes needs to do a bit of mop-up with variable > > names, but otherwise it should be mostly complete. > > I'm getting slightl

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Christophe Pettus
> On Feb 25, 2019, at 11:24, Stephen Frost wrote: > Aren't they going to need to make a change for v12 now anyway? > > Hopefully they're regularly testing their backups by doing a restore of > them, and dropping a recovery.conf into the directory of a v12 system > after restore will do exactly

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Andres Freund
Hi, On 2019-02-25 20:43:01 +0200, David Steele wrote: > fsync() is the major corruption issue we are facing right now but that > doesn't mean there aren't other sources of corruption we should be thinking > about. I've thought about this one a lot and it scares me. FWIW, I think this kind of iss

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Andres Freund
On 2019-02-25 08:42:02 -0500, Stephen Frost wrote: > Greetings, > > * Andres Freund (and...@anarazel.de) wrote: > > On 2019-02-24 17:19:22 -0500, Stephen Frost wrote: > > > You say above that the new interface is unquestionably an improvement > > > > FWIW, I think we didn't design it even remotel

Re: POC: converting Lists into arrays

2019-02-25 Thread Tom Lane
Andres Freund writes: > FWIW, rewrites of this kind can be quite nicely automated using > coccinelle [1]. One sometimes needs to do a bit of mop-up with variable > names, but otherwise it should be mostly complete. I'm getting slightly annoyed by arguments that reject a live, workable patch in fa

Re: POC: converting Lists into arrays

2019-02-25 Thread Andres Freund
Hi, On 2019-02-25 11:59:06 -0800, Peter Geoghegan wrote: > On Mon, Feb 25, 2019 at 10:59 AM Tom Lane wrote: > > Because it involves touching ten times more code (and that's a very > > conservative estimate). Excluding changes in pg_list.h + list.c, > > what I posted touches approximately 600 lin

Re: POC: converting Lists into arrays

2019-02-25 Thread Tom Lane
Andres Freund writes: > Yea, it'd be more convincing. I'm not convinced it'd be a no-brainer > though. Unless you've been hacking PG for a fair bit, the pg_list.h APIs > are very hard to understand / remember. Given this change essentially > requires auditing all code that uses List, ISTM we'd be

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Andres Freund
Hi, On 2019-02-25 08:14:16 -0500, Stephen Frost wrote: > > It will be annoying if after this removal, companies must change their > > backup strategy by using specific postgres tools (pgbackrest, barman). > > You don't write your own database system using CSV files and shell > magic, do you? I h

Re: POC: converting Lists into arrays

2019-02-25 Thread Andres Freund
Hi, On 2019-02-25 13:59:36 -0500, Tom Lane wrote: > Robert Haas writes: > > On Mon, Feb 25, 2019 at 1:17 PM Tom Lane wrote: > >> I'm not following your point here. If we change key data structures > >> (i.e. parsetrees, plan trees, execution trees) to use some other list-ish > >> API, that *in

Re: POC: converting Lists into arrays

2019-02-25 Thread Tom Lane
Peter Geoghegan writes: > If we knew that the list code was the bottleneck in a handful of > cases, then I'd come down on Robert's side here. It would then be > possible to update the relevant bottlenecked code in an isolated > fashion, while getting the lion's share of the benefit. However, I > d

Re: [HACKERS] generated columns

2019-02-25 Thread Peter Eisentraut
On 2019-01-15 08:13, Michael Paquier wrote: >>> Ah, the volatility checking needs some improvements. I'll address that >>> in the next patch version. >> >> ok > > The same problem happens for stored and virtual columns. This is fixed in v8. > It would be nice to add a test with composite types,

Re: POC: converting Lists into arrays

2019-02-25 Thread Andres Freund
Hi, On 2019-02-25 13:02:03 -0500, Robert Haas wrote: > On Sat, Feb 23, 2019 at 9:24 PM Tom Lane wrote: > > Every time this has come up, I've opined that the right fix is to jack > > up the List API and drive a new implementation underneath, as we did > > once before (cf commit d0b4399d81). I tho

Re: POC: converting Lists into arrays

2019-02-25 Thread Peter Geoghegan
On Mon, Feb 25, 2019 at 10:59 AM Tom Lane wrote: > Because it involves touching ten times more code (and that's a very > conservative estimate). Excluding changes in pg_list.h + list.c, > what I posted touches approximately 600 lines of code (520 insertions, > 644 deletions to be exact). For com

Re: \describe*

2019-02-25 Thread Robert Haas
On Sat, Feb 23, 2019 at 7:19 PM Andres Freund wrote: > Sure, but it was late, and we have far more patches than we can deal > with. Many of them much much older than this. More importantly, at least in my opinion, is that this is one of those questions that people tend to have very strong feeling

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Bruce Momjian
On Mon, Feb 25, 2019 at 02:24:18PM -0500, Stephen Frost wrote: > * Bruce Momjian (br...@momjian.us) wrote: > > On Mon, Feb 25, 2019 at 11:33:33AM -0500, Stephen Frost wrote: > > > I don't want to see more users stumbling over the issues with the > > > exclusive backup interface. A better interface

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Stephen Frost
Greetings, * David Steele (da...@pgmasters.net) wrote: > On 2/25/19 7:50 PM, Fujii Masao wrote: > >Thought? > > Here's the really obvious bad thing: if users do not update their procedures > and we ignore backup_label.pending on startup then they will end up with a > corrupt database because it w

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Stephen Frost
Greetings, * Christophe Pettus (x...@thebuild.com) wrote: > > On Feb 25, 2019, at 08:55, Stephen Frost wrote: > > > > I honestly do doubt that they have had the same experiences that I have > > had > > Well, I guarantee you that no two people on this list have had identical > experiences. :)

Re: POC: converting Lists into arrays

2019-02-25 Thread Tom Lane
Robert Haas writes: > On Mon, Feb 25, 2019 at 1:17 PM Tom Lane wrote: >> I'm not following your point here. If we change key data structures >> (i.e. parsetrees, plan trees, execution trees) to use some other list-ish >> API, that *in itself* breaks everything that accesses those data >> structu

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread David Steele
Hi Christophe, On 2/25/19 7:24 PM, Christophe Pettus wrote: On Feb 25, 2019, at 08:55, Stephen Frost wrote: I honestly do doubt that they have had the same experiences that I have had Well, I guarantee you that no two people on this list have had identical experiences. :) I certainly ha

Re: Auxiliary Processes and MyAuxProc

2019-02-25 Thread Mike Palmiotto
On Mon, Feb 25, 2019 at 1:29 PM Tom Lane wrote: > > Mike Palmiotto writes: > > > > For some context, I'm trying to come up with a patch to set the > > process identifier (MyAuxProc/am_autovacuumworker/launcher, > > am_archiver, etc.) immediately after forking, > > Don't we do that already? Kind

Re: POC: converting Lists into arrays

2019-02-25 Thread Robert Haas
On Mon, Feb 25, 2019 at 1:17 PM Tom Lane wrote: > I'm not following your point here. If we change key data structures > (i.e. parsetrees, plan trees, execution trees) to use some other list-ish > API, that *in itself* breaks everything that accesses those data > structures. The approach I propos

Re: Auxiliary Processes and MyAuxProc

2019-02-25 Thread Tom Lane
Mike Palmiotto writes: > I've been looking through the startup code for postmaster forks and > saw a couple of mechanisms used. Most forks seem to be using > StartChildProcess with a MyAuxProc emumerator, but then some > (autovacuum launcher/worker, syslogger, bgworker, archiver) are forked > thro

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Bruce Momjian
On Mon, Feb 25, 2019 at 11:33:33AM -0500, Stephen Frost wrote: > I don't want to see more users stumbling over the issues with the > exclusive backup interface. A better interface exists, and has existed > since 9.6. Do you really think we would be having this discussion if the non-exclusive back

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread David Steele
On 2/25/19 7:50 PM, Fujii Masao wrote: On Mon, Feb 25, 2019 at 10:49 PM Laurenz Albe wrote: I'm not playing devil's advocate here to annoy you. I see the problems with the exclusive backup, and I see how it can hurt people. I just think that removing exclusive backup without some kind of help

Re: POC: converting Lists into arrays

2019-02-25 Thread Tom Lane
Robert Haas writes: > On Sat, Feb 23, 2019 at 9:24 PM Tom Lane wrote: >> Every time this has come up, I've opined that the right fix is to jack >> up the List API and drive a new implementation underneath, as we did >> once before (cf commit d0b4399d81). > I'm not really convinced that this is t

Re: POC: converting Lists into arrays

2019-02-25 Thread Robert Haas
On Sat, Feb 23, 2019 at 9:24 PM Tom Lane wrote: > Every time this has come up, I've opined that the right fix is to jack > up the List API and drive a new implementation underneath, as we did > once before (cf commit d0b4399d81). I thought maybe it was about time > to provide some evidence for th

Auxiliary Processes and MyAuxProc

2019-02-25 Thread Mike Palmiotto
Greetings, I've been looking through the startup code for postmaster forks and saw a couple of mechanisms used. Most forks seem to be using StartChildProcess with a MyAuxProc emumerator, but then some (autovacuum launcher/worker, syslogger, bgworker, archiver) are forked through their own start fu

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Fujii Masao
On Mon, Feb 25, 2019 at 10:49 PM Laurenz Albe wrote: > > Stephen Frost wrote: > > > It will be annoying if after this removal, companies must change their > > > backup strategy by using specific postgres tools (pgbackrest, barman). > > > > You don't write your own database system using CSV files a

Re: pgsql: Avoid creation of the free space map for small heap relations, t

2019-02-25 Thread Tom Lane
I wrote: > Amit Kapila writes: >> Avoid creation of the free space map for small heap relations, take 2. > I think this patch still has some issues. Just out of curiosity ... how can it possibly be even a little bit sane that fsm_local_map is a single static data structure, without even any indi

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Christophe Pettus
> On Feb 25, 2019, at 08:55, Stephen Frost wrote: > > I honestly do doubt that they have had the same experiences that I have > had Well, I guarantee you that no two people on this list have had identical experiences. :) I certainly have been bitten by the problems with the current system.

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Mon, Feb 25, 2019 at 11:09 AM Stephen Frost wrote: > > I do think we can ask that people who wish to have a capability included > > in PG (or continue to be included when there are serious known issues > > with it...) be prepared to eit

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Mon, Feb 25, 2019 at 11:33 AM Stephen Frost wrote: > > > Removing an option that people are currently using, and which they > > > find better than other available options for reasons with which I > > > understand that you disagree, will

Re: pgsql: Avoid creation of the free space map for small heap relations, t

2019-02-25 Thread Tom Lane
Amit Kapila writes: > Avoid creation of the free space map for small heap relations, take 2. I think this patch still has some issues. Note the following two recent buildfarm failures: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=petalura&dt=2019-02-20%2004%3A20%3A01 https://buildfar

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Mon, Feb 25, 2019 at 11:23 AM Stephen Frost wrote: > > If that argument did matter, we could go back and find the prior > > discussions about the issues around the exclusive backup mode and about > > removing it, or next year we could p

Re: Should we increase the default vacuum_cost_limit?

2019-02-25 Thread Robert Haas
On Mon, Feb 25, 2019 at 8:39 AM David Rowley wrote: > I decided to do the times by 10 option that I had mentioned Ensue > debate about that... +1 for raising the default substantially. In my experience, and it seems others are in a similar place, nobody ever gets into trouble because the def

Re: Remove Deprecated Exclusive Backup Mode

2019-02-25 Thread Robert Haas
On Mon, Feb 25, 2019 at 11:33 AM Stephen Frost wrote: > > Removing an option that people are currently using, and which they > > find better than other available options for reasons with which I > > understand that you disagree, will make users more sad. Happy is > > better. > > I don't want to s

Re: Referential Integrity Checks with Statement-level Triggers

2019-02-25 Thread Corey Huinker
> > > In order to avoid per-row calls of the constraint trigger functions, we > could > try to "aggregate" the constraint-specific events somehow, but I think a > separate queue would be needed for the constraint-specific events. > > In general, the (after) triggers and constraints have too much in

Re: Should we increase the default vacuum_cost_limit?

2019-02-25 Thread Julien Rouhaud
On Mon, Feb 25, 2019 at 4:44 PM Laurenz Albe wrote: > > David Rowley wrote: > > On Tue, 26 Feb 2019 at 02:06, Joe Conway wrote: > > > > > > On 2/25/19 1:17 AM, Peter Geoghegan wrote: > > > > On Sun, Feb 24, 2019 at 9:42 PM David Rowley > > > > wrote: > > > >> The current default vacuum_cost_limi

  1   2   >