Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-17 Thread David Rowley
On 17 December 2017 at 16:22, Robert Haas wrote: > On Fri, Dec 15, 2017 at 5:18 PM, Alvaro Herrera > wrote: >> We have two options for marking valid: >> >> 1. after each ALTER INDEX ATTACH, verify whether the set of partitions >> that contain the index is complete; if so, mark it valid, otherwis

Re: GSoC 2018

2017-12-17 Thread Alvaro Hernandez
On 15/12/17 13:57, Aleksander Alekseev wrote: Hi Stephen, HA/fail-over is a very broad topic, with a lot of pieces that need to be done such that I'm not sure it's really viable, but perhaps a precursor project (synchronous logical replication seems like a prereq, no?) would make more sense.

Small typo in comment in json_agg_transfn

2017-12-17 Thread David Rowley
The attached fixed a small typo in json_agg_transfn. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services json_agg_transfn_comment_fix.patch Description: Binary data

Parallel Aggregates for string_agg and array_agg

2017-12-17 Thread David Rowley
Hi, While working on partial aggregation a few years ago, I didn't really think it was worthwhile allowing partial aggregation of string_agg and array_agg. I soon realised that I was wrong about that and allowing parallelisation of these aggregates still could be very useful when many rows are fil

Re: pg_(total_)relation_size and partitioned tables

2017-12-17 Thread David Rowley
On 17 December 2017 at 16:24, Robert Haas wrote: > On Thu, Dec 14, 2017 at 12:23 AM, Amit Langote > wrote: >> You may have guessed from $subject that the two don't work together. > > It works exactly as documented: > > pg_total_relation_size(regclass) - Total disk space used by the > specified ta

Re: Why does array_position_common bitwise NOT an Oid type?

2017-12-17 Thread David Rowley
On 17 December 2017 at 14:53, Tom Lane wrote: > David Rowley writes: >> I was puzzled to see the following code: > >> my_extra->element_type = ~element_type; > > If memory serves, the idea was to force the subsequent datatype-lookup > path to be taken, even if for some reason element_type is Inva

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-17 Thread Alvaro Herrera
Robert Haas wrote: > On Fri, Dec 15, 2017 at 5:18 PM, Alvaro Herrera > wrote: > > We have two options for marking valid: > > > > 1. after each ALTER INDEX ATTACH, verify whether the set of partitions > > that contain the index is complete; if so, mark it valid, otherwise do > > nothing. This suc

Re: Top-N sorts verses parallelism

2017-12-17 Thread Robert Haas
On Fri, Dec 15, 2017 at 4:13 PM, Thomas Munro wrote: > Looks right to me. Commit 3452dc52 just forgot to tell the planner. > I'm pleased about that because it makes this a slam-dunk bug-fix and > not some confusing hard to justify costing problem. Jeff Janes inquired off-list about other places

Re: Backfill bgworker Extension?

2017-12-17 Thread Jeremy Finzel
On Sat, Dec 16, 2017 at 8:31 AM Peter Eisentraut < peter.eisentr...@2ndquadrant.com> wrote: > On 12/15/17 23:50, Jeremy Finzel wrote: > > The common ground is some column in some table needs to be bulk updated. > > I may not be explaining well, but in our environment we have done > > hundreds of t

Re: [HACKERS] Something for the TODO list: deprecating abstime and friends

2017-12-17 Thread David Fetter
On Wed, Dec 13, 2017 at 12:05:06AM -0800, Andres Freund wrote: > On 2017-07-15 18:00:34 -0400, Tom Lane wrote: > > * contrib/spi/timetravel depends on abstime columns to represent what > > would nowadays be better done as a tstzrange. I'd have thought we > > could maybe toss that example module ov

Re: [sqlsmith] Parallel worker executor crash on master

2017-12-17 Thread Thomas Munro
On Sun, Dec 17, 2017 at 12:26 PM, Andreas Seltenreich wrote: > Thomas Munro writes: > >> On Sat, Dec 16, 2017 at 10:13 PM, Andreas Seltenreich >> wrote: >>> Core was generated by `postgres: smith regression [local] SELECT >>> '. >>> Program terminated with signal SIGSEGV,

Re: Protect syscache from bloating with negative cache entries

2017-12-17 Thread Robert Haas
On Sat, Dec 16, 2017 at 11:42 PM, Andres Freund wrote: >> I'm not sure we should regard very quick bloating as a problem in need >> of solving. Doesn't that just mean we need the cache to be bigger, at >> least temporarily? > > Leaving that aside, is that actually not at least to a good degree, >

Re: pg_(total_)relation_size and partitioned tables

2017-12-17 Thread Robert Haas
On Sun, Dec 17, 2017 at 9:54 AM, David Rowley wrote: > I'd also vote to leave the relation_size functions alone. > > Perhaps it's worth thinking of changing pg_table_size() instead. We > have taken measures to try and hide the fact that a table is made up > of a bunch of partitions from the user i

Re: pg_(total_)relation_size and partitioned tables

2017-12-17 Thread Michael Paquier
On Sun, Dec 17, 2017 at 11:54 PM, David Rowley wrote: > I'd also vote to leave the relation_size functions alone. Count me in that bucket as well. > Perhaps it's worth thinking of changing pg_table_size() instead. We > have taken measures to try and hide the fact that a table is made up > of a b

Re: worker_spi example BGW code GUC tweak

2017-12-17 Thread Robert Haas
On Thu, Dec 14, 2017 at 5:41 PM, Chapman Flack wrote: > Would this sample code make an even better teaching example if it > used the existing GUC way to declare that worker_spi.naptime is > in units of seconds? > > Or does it not do that for some reason I've overlooked? Making it use GUC_UNIT_S s

Re: es_query_dsa is broken

2017-12-17 Thread Thomas Munro
On Thu, Dec 7, 2017 at 12:51 PM, Thomas Munro wrote: > On Wed, Dec 6, 2017 at 10:16 PM, Amit Kapila wrote: >> On Wed, Dec 6, 2017 at 1:14 AM, Robert Haas wrote: >>> On Tue, Dec 5, 2017 at 7:24 AM, Amit Kapila wrote: + EState *estate = gatherstate->ps.state; + + /* Install our DSA

Re: worker_spi example BGW code GUC tweak

2017-12-17 Thread Chapman Flack
On 12/17/2017 07:32 PM, Robert Haas wrote: > Making it use GUC_UNIT_S seems like a good idea to me, but removing > the mention of seconds from the description doesn't seem like a good > idea to me. I took for my model a quick survey of existing GUCs that use GUC_UNIT_(MS|S|MIN) - most of which do

Re: genomic locus

2017-12-17 Thread Robert Haas
On Fri, Dec 15, 2017 at 2:49 PM, Gene Selkov wrote: > I need a data type to represent genomic positions, which will consist of a > string and a pair of integers with interval logic and access methods. Sort > of like my seg type, but more straightforward. > > I noticed somebody took a good care of

Re: worker_spi example BGW code GUC tweak

2017-12-17 Thread Robert Haas
On Sun, Dec 17, 2017 at 8:30 PM, Chapman Flack wrote: > On 12/17/2017 07:32 PM, Robert Haas wrote: >> Making it use GUC_UNIT_S seems like a good idea to me, but removing >> the mention of seconds from the description doesn't seem like a good >> idea to me. > > I took for my model a quick survey of

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-17 Thread Robert Haas
On Sun, Dec 17, 2017 at 10:50 AM, Alvaro Herrera wrote: >> If ALTER INDEX .. ATTACH is already taking AEL on the parent, then I >> think it might as well try to validate while it's at it. But if not >> then we might want to go with #2. > > The problem I have with it is that restoring a dump conta

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-17 Thread Robert Haas
On Sun, Dec 17, 2017 at 5:29 AM, David Rowley wrote: >> If ALTER INDEX .. ATTACH is already taking AEL on the parent, then I >> think it might as well try to validate while it's at it. But if not >> then we might want to go with #2. > > I'm now not that clear on what the behaviour is if the ONLY

Re: Using ProcSignal to get memory context stats from a running backend

2017-12-17 Thread Robert Haas
On Thu, Dec 14, 2017 at 9:34 PM, Craig Ringer wrote: > On 15 December 2017 at 09:24, Greg Stark wrote: >> Another simpler option would be to open up a new file in the log >> directory > > ... if we have one. > > We might be logging to syslog, or whatever else. > > I'd rather keep it simple(ish).

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-17 Thread David Rowley
On 18 December 2017 at 15:04, Robert Haas wrote: > On Sun, Dec 17, 2017 at 5:29 AM, David Rowley > wrote: >> I'm now not that clear on what the behaviour is if the ONLY keyword is >> not specified on the CREATE INDEX for the partitioned index. Does that >> go and create each leaf partition index

Re: pg_(total_)relation_size and partitioned tables

2017-12-17 Thread Michael Paquier
On Mon, Dec 18, 2017 at 9:29 AM, Michael Paquier wrote: > The barrier here is thin. What's proposed here is already doable with > a WITH RECURSIVE query. So why not just documenting this query and be > done with it instead of complicating the code? It seems to me that the > performance in calling

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-17 Thread Robert Haas
On Sun, Dec 17, 2017 at 9:38 PM, David Rowley wrote: > On 18 December 2017 at 15:04, Robert Haas wrote: >> On Sun, Dec 17, 2017 at 5:29 AM, David Rowley >> wrote: >>> I'm now not that clear on what the behaviour is if the ONLY keyword is >>> not specified on the CREATE INDEX for the partitioned

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-17 Thread David Rowley
On 18 December 2017 at 15:59, Robert Haas wrote: > On Sun, Dec 17, 2017 at 9:38 PM, David Rowley > wrote: >> On 18 December 2017 at 15:04, Robert Haas wrote: >>> On Sun, Dec 17, 2017 at 5:29 AM, David Rowley >>> wrote: I'm now not that clear on what the behaviour is if the ONLY keyword is

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-12-17 Thread Robert Haas
On Sun, Dec 17, 2017 at 10:05 PM, David Rowley wrote: > I think you feel quite strongly about not having the code select a > random matching index, so if we want to stick to that rule, then we'll > need to create a set of new leaf indexes rather than select a random > one. I feel quite strongly a

Re: [HACKERS] Assertion failure when the non-exclusive pg_stop_backup aborted.

2017-12-17 Thread Robert Haas
On Thu, Dec 14, 2017 at 7:51 PM, Masahiko Sawada wrote: > +1 from me. Works for me, too, although I still don't really follow how it's happening in the present coding. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: [HACKERS] Assertion failure when the non-exclusive pg_stop_backup aborted.

2017-12-17 Thread Michael Paquier
On Mon, Dec 18, 2017 at 12:14 PM, Robert Haas wrote: > On Thu, Dec 14, 2017 at 7:51 PM, Masahiko Sawada > wrote: >> +1 from me. > > Works for me, too, although I still don't really follow how it's > happening in the present coding. Craig has mentioned at least one way upthread: https://www.post

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2017-12-17 Thread Masahiko Sawada
On Sun, Dec 17, 2017 at 12:27 PM, Robert Haas wrote: > On Thu, Dec 14, 2017 at 5:45 AM, Masahiko Sawada > wrote: >> Here is the result. >> I've measured the through-put with some cases on my virtual machine. >> Each client loads 48k file to each different relations located on >> either xfs files

Re: pg_(total_)relation_size and partitioned tables

2017-12-17 Thread Amit Langote
Thanks all for your thoughts. I agree with the Robert's point which both David and Michael seem to agree with that we shouldn't really be changing what pg_relation_size() is doing under the covers. And I guess the same for pg_table_size(), too. Both of those functions and their siblings work wit

Re: pg_(total_)relation_size and partitioned tables

2017-12-17 Thread Michael Paquier
On Mon, Dec 18, 2017 at 2:17 PM, Amit Langote wrote: > Do you (and/or others) think that's something that we can wrap inside a > built-in function(s), that is, one defined in system_views.sql? Or if we > decide to have new functions, say, pg_get_partitions() and/or > pg_get_partition_sizes(), we

Re: [HACKERS] replace GrantObjectType with ObjectType

2017-12-17 Thread Rushabh Lathia
On Sat, Dec 16, 2017 at 12:40 AM, Robert Haas wrote: > On Fri, Dec 15, 2017 at 12:40 PM, Peter Eisentraut > wrote: > > On 12/14/17 22:59, Rushabh Lathia wrote: > >> I noted that no_priv_msg and not_owner_msg array been removed > >> and code fitted the code into aclcheck_error(). Actually that >