Re: [HACKERS] WIP: Upper planner pathification

2016-03-02 Thread David Rowley
e upper planner changes in first, but soon! not at the end of March 'fest, as doing so would most likely kill parallel aggregate for 9.6, and I kinda think that would be silly as (I think) it's pretty much the biggest missing piece of the parallel query set. -- David Rowley

Re: [HACKERS] WIP: Upper planner pathification

2016-03-03 Thread David Rowley
On 3 March 2016 at 18:04, Tom Lane wrote: > David Rowley writes: >> I agree that it would be good to get this in as soon as possible. I'm >> currently very close to being done with writing Parallel Aggregate on >> top of the upper planner changes. So far this ver

Re: [HACKERS] WIP: Upper planner pathification

2016-03-03 Thread David Rowley
On 3 March 2016 at 22:57, David Rowley wrote: > On 3 March 2016 at 18:04, Tom Lane wrote: >> If you come across any points where it seems like it could be >> done better or more easily, that would be tremendously valuable feedback >> at this stage. > > Well since y

Re: [HACKERS] WIP: Upper planner pathification

2016-03-03 Thread David Rowley
t;leaf node" in the path tree. Propagating these would mean I could remove that parameter again. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Parallel Aggregate

2016-03-03 Thread David Rowley
istence of the outstanding issues above, I feel like I might be posting the patch a little earlier, but wanted to do so since this is quite a hot area in the code at the moment and I wanted to post for transparency. To apply the patch please apply [1] first. [1] http://www.postgresql.org/message-id

Re: [HACKERS] Performance improvement for joins where outer side is unique

2016-03-05 Thread David Rowley
On 5 March 2016 at 10:43, Alvaro Herrera wrote: > I wonder why do we have two identical copies of clause_sides_match_join ... Yeah, I noticed the same a while back, and posted about it. Here was the response: http://www.postgresql.org/message-id/26820.1405522...@sss.pgh.pa.us -- David Row

Re: [HACKERS] Parallel Aggregate

2016-03-06 Thread David Rowley
't see why we need to give the impression that there is in EXPLAIN. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make chan

Re: [HACKERS] WIP: Upper planner pathification

2016-03-07 Thread David Rowley
On 8 March 2016 at 10:01, Tom Lane wrote: > I've pushed it now; we'll soon see if the buildfarm finds any problems. Fantastic! I'm looking forward to all the future optimisation opportunities that this opens up. Thanks for making this happen. -- David Rowley

Re: [HACKERS] Parallel Aggregate

2016-03-07 Thread David Rowley
On 5 March 2016 at 07:25, Robert Haas wrote: > On Thu, Mar 3, 2016 at 11:00 PM, David Rowley >> 3. The code never attempts to mix and match Grouping Agg and Hash Agg >> plans. e.g it could be an idea to perform Partial Hash Aggregate -> >> Gather -> Sort -> Finalize

[HACKERS] Fix misspelling of "parallel"

2016-03-07 Thread David Rowley
The attached fixes a small spelling error in a comment. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services parallel_spelling_fix.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hac

[HACKERS] Typo in logicaldecoding docs

2016-03-07 Thread David Rowley
The attached fixes a small error in the logicaldecoding docs. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services logicaldecoding_docs_typo_fix.patch Description: Binary data -- Sent via pgsql-hackers mailing list (p

Re: [HACKERS] Parallel Aggregate

2016-03-08 Thread David Rowley
On 9 March 2016 at 04:06, Robert Haas wrote: > On Mon, Mar 7, 2016 at 5:15 PM, David Rowley > wrote: >> My concerns are: >> 1. Since there's no cheapest_partial_path in RelOptInfo the code is >> currently considering every partial_path for parallel hash aggregate.

Re: [HACKERS] Performance improvement for joins where outer side is unique

2016-03-08 Thread David Rowley
On 23 January 2016 at 05:36, Tomas Vondra wrote: > Hi, > > On 12/17/2015 02:17 PM, David Rowley wrote: >> >> On 17 December 2015 at 19:11, Simon Riggs > <mailto:si...@2ndquadrant.com>> wrote: >> >> On 17 December 2015 at 00:17, Tomas Vond

Re: [HACKERS] Performance improvement for joins where outer side is unique

2016-03-08 Thread David Rowley
t... YAML seems most compact. [1] http://www.postgresql.org/message-id/8907.1440383...@sss.pgh.pa.us -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)

Re: [HACKERS] Parallel Aggregate

2016-03-10 Thread David Rowley
On 9 March 2016 at 04:06, Robert Haas wrote: > On Mon, Mar 7, 2016 at 5:15 PM, David Rowley > wrote: >> 3. Nothing in create_grouping_paths() looks at the force_parallel_mode >> GUC. I had a quick look at this GUC and was a bit surprised to see 3 >> possible states, bu

Re: [HACKERS] Parallel Aggregate

2016-03-10 Thread David Rowley
On 8 March 2016 at 11:15, David Rowley wrote: > The setrefs.c parts of the patch remain completely broken. I've not > had time to look at this again yet, sorry. Ok, so again, apologies for previously sending such a broken patch. I've since managed to free up a bit of time to wor

Re: [HACKERS] Parallel Aggregate

2016-03-11 Thread David Rowley
On 11 March 2016 at 03:39, David Rowley wrote: > A couple of things which I'm not 100% happy with. > > 1. make_partialgroup_input_target() doing lookups to the syscache. > Perhaps this job can be offloaded to a new function in a more suitable > location. Ideally the Aggref wou

Re: [HACKERS] Performance improvement for joins where outer side is unique

2016-03-12 Thread David Rowley
in RelOptInfo, this would also allow rel_supports_distinctness() to do something a bit smarter than just return false if there's no indexes. That might not buy us much though, but at least relations tend to have very little unique indexes, even when they have lots of indexes. -- David

Re: [HACKERS] Performance improvement for joins where outer side is unique

2016-03-13 Thread David Rowley
be possible to analyse the join rel too and search for a base rel which supports the distinctness, and then ensure none of the other rels which make up the join rel can cause tuple duplication of that rel. But this just causes missed optimisation opportunities. -- David Rowley

Re: [HACKERS] Parallel Aggregate

2016-03-13 Thread David Rowley
On 12 March 2016 at 16:31, David Rowley wrote: > I've attached an updated patch which is based on commit 7087166, > things are really changing fast in the grouping path area at the > moment, but hopefully the dust is starting to settle now. The attached patch fixes a harmless co

Re: [HACKERS] Parallel Aggregate

2016-03-13 Thread David Rowley
run EXPLAIN ANALYZE on this with the 6 workers, does the actual number of Gather rows come out at 105? I'd just like to get an idea of my cost estimate for the Gather are going to be accurate for real world data sets. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Parallel Aggregate

2016-03-13 Thread David Rowley
hat function which states: /* * Limit the degree of parallelism logarithmically based on the size of the * relation. This probably needs to be a good deal more sophisticated, but we * need something here for now. */ So this will likely see some revision at some point, after 9.6. -- David Row

Re: [HACKERS] Parallel Aggregate

2016-03-13 Thread David Rowley
dding a column which explicitly stores that value (1), or collecting stats on the expression (2), or teaching the planner about the internals of that function, which is likely just never going to happen. (3) is just going to make the outlook of a hash plan look a little brighter, although you'

Re: [HACKERS] Parallel Aggregate

2016-03-13 Thread David Rowley
ime currently? There is nothing in the planner yet, or any patch that I know of to push the Partial Aggregate node to below an Append node. That will most likely come in 9.7. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Serv

Re: [HACKERS] Parallel Aggregate

2016-03-14 Thread David Rowley
for queries which estimate a larger number of groups than partial path rows. > + aggstate->finalizeAggs == true) > > We usually just say if (a) not if (a == true) when it's a boolean. > Similarly !a rather than a == false. hmm, thanks. It appear

Re: [HACKERS] Parallel Aggregate

2016-03-14 Thread David Rowley
parallel aggregate. > 4) Is clauses.h the right place for PartialAggType? I'm not sure that it is to be honest. I just put it there because the patch never persisted the value of a PartialAggType in any struct field anywhere and checks it later in some other file. In all place

Re: [HACKERS] Parallel Aggregate

2016-03-14 Thread David Rowley
t want this one getting bloated with debates on that. It's not code I'm planning on going anywhere near for this patch. I'll start a thread... -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] Choosing parallel_degree

2016-03-14 Thread David Rowley
4151882 MB, 4054 GB) For 1594323001 pages there will be 14 workers (rel size 12455648 MB, 12163 GB) [1] http://www.postgresql.org/message-id/CANkGpBtUvzpdvF2=_iq64ujmvrpycs6d4i9-wepbusq1sq+...@mail.gmail.com -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development

Re: [HACKERS] Choosing parallel_degree

2016-03-14 Thread David Rowley
only 4 pages. I've not tested in Postgres, but if you do this and: SET parallel_setup_cost = 0; then I'd imagine it should generate a parallel plan. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent v

Re: [HACKERS] Parallel Aggregate

2016-03-15 Thread David Rowley
On 15 March 2016 at 13:48, David Rowley wrote: > An updated patch will follow soon. I've attached an updated patch which addresses some of Robert's and Tomas' concerns. I've not done anything about the exprCollation() yet, as I'm still unsure of what it should do. I

Re: [HACKERS] Combining Aggregates

2016-03-15 Thread David Rowley
l, and it's tricky for regular users to deduce that as it's related to > the type of the internal state and not to the input types. An example of > that is the SUM(bigint) example mentioned above. I agree. I will look for a suitable place. -- David Rowley htt

Re: [HACKERS] Parallel Aggregate

2016-03-15 Thread David Rowley
On 16 March 2016 at 09:23, Robert Haas wrote: > On Mon, Mar 14, 2016 at 7:56 PM, David Rowley > wrote: >> A comment does explain this, but perhaps it's not good enough, so I've >> rewritten it to become. >> >> /* >> * PartialAggref >> * >&

Re: [HACKERS] Parallel Aggregate

2016-03-15 Thread David Rowley
f, aggref)) break; } if equals() compared the aggpartial then this code would fail to find the Aggref in the subnode due to the aggpartial field being true on one and false on the other Aggref. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training

Re: [HACKERS] Parallel Aggregate

2016-03-15 Thread David Rowley
On 16 March 2016 at 12:58, Robert Haas wrote: > On Tue, Mar 15, 2016 at 6:55 PM, David Rowley > wrote: >> On 16 March 2016 at 11:00, Robert Haas wrote: >>> I don't see why we would need to leave aggpartial out of the equals() >>> check. I m

Re: [HACKERS] Choosing parallel_degree

2016-03-15 Thread David Rowley
low the parallel_setup_cost to make parallel plans look less favourable for smaller relations. I assume that this is so that we don't burden the planner with the overhead of generating parallel paths for smaller relations? -- David Rowley http://www.2ndQuadrant.com/ Postg

Re: [HACKERS] Parallel Aggregate

2016-03-15 Thread David Rowley
On 16 March 2016 at 13:42, Robert Haas wrote: > On Tue, Mar 15, 2016 at 8:04 PM, David Rowley > wrote: >> On 16 March 2016 at 12:58, Robert Haas wrote: >>> ...and why would one be true and the other false? >> >> One would be the combine aggregate (having aggpa

Re: [HACKERS] Parallel Aggregate

2016-03-15 Thread David Rowley
On 16 March 2016 at 14:08, Robert Haas wrote: > On Tue, Mar 15, 2016 at 8:55 PM, David Rowley > wrote: >> On 16 March 2016 at 13:42, Robert Haas wrote: >>> On Tue, Mar 15, 2016 at 8:04 PM, David Rowley >>> wrote: >>>> On 16 March 2016 at 12:58, Robert

Re: [HACKERS] Parallel Aggregate

2016-03-16 Thread David Rowley
same way, by returning the newly created Var instead of doing that in fix_combine_agg_expr_mutator(), as the last version did. Thanks for the suggestion. New patch attached. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Combining Aggregates

2016-03-16 Thread David Rowley
On 16 March 2016 at 23:54, Haribabu Kommi wrote: > On Wed, Mar 16, 2016 at 8:34 AM, David Rowley > wrote: >> Yes me too, so I spent several hours yesterday writing all of the >> combine functions and serialisation/deserialisation that are required >> for all of SUM(),

Re: [HACKERS] Parallel Aggregate

2016-03-18 Thread David Rowley
aggregate functions in to a rather inefficient random number generator. This is because the combine Aggref->args still point to "num", instead of sum(num). > Back at make_partialgroup_input_target, the comment says "so that they > can be found later in Combine Aggregate n

Re: [HACKERS] Parallel Aggregate

2016-03-18 Thread David Rowley
On 19 March 2016 at 05:46, Robert Haas wrote: > On Wed, Mar 16, 2016 at 5:05 PM, David Rowley > wrote: >>> Cool! Why not initialize aggpartialtype always? >> >> Because the follow-on patch sets that to either the serialtype or the >> aggtranstype, dependin

Re: [HACKERS] Parallel Aggregate

2016-03-18 Thread David Rowley
On 18 March 2016 at 01:22, Amit Kapila wrote: > On Thu, Mar 17, 2016 at 10:35 AM, David Rowley > wrote: >> >> On 17 March 2016 at 01:19, Amit Kapila wrote: >> > Few assorted comments: >> > >> > 2. >> > AggPath * >> &g

Re: [HACKERS] Choosing parallel_degree

2016-03-19 Thread David Rowley
stuff takes time. It would be really nice if it were possible to drop the setting really low, so that combined with a low parallel_setup_cost we could enable parallel query on small tables in the regression test suite. -- David Rowley http://www.2ndQuadrant.com/ Postgr

Re: [HACKERS] Parallel Aggregate

2016-03-19 Thread David Rowley
any possible use cases for this, so it makes little sense to give it a name according to a single use case. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hacke

Re: [HACKERS] Parallel Aggregate

2016-03-19 Thread David Rowley
* We also convert any Aggrefs which we do find and put them into partial mode, * this adjusts the Aggref's return type so that the partially calculated * aggregate value can make its way up the execution tree up to the Finalize * Aggregate node. */ I will post an updated patch once I've addressed Amit's points. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Parallel Aggregate

2016-03-19 Thread David Rowley
On 17 March 2016 at 00:57, Robert Haas wrote: > On Wed, Mar 16, 2016 at 6:49 AM, David Rowley > wrote: >> On 16 March 2016 at 15:04, Robert Haas wrote: >>> I don't think I'd be objecting if you made PartialAggref a real >>> alternative to Aggref.

Re: [HACKERS] Parallel Aggregate

2016-03-19 Thread David Rowley
On 17 March 2016 at 18:05, David Rowley wrote: > Updated patch attached. Please disregard 0001-Allow-aggregation-to-happen-in-parallel_2016-03-17.patch. This contained a badly thought through last minute change to how the Gather path is generated and is broken. I played around with ways

Re: [HACKERS] Parallel Aggregate

2016-03-19 Thread David Rowley
gets while I've been working on this, so perhaps not surprising. I've reverted that change now. > 6. > + /* XXX this causes some redundant cost calculation ... */ > + input_target = set_pathtarget_cost_width(root, > input_target); > + return input_target; > > Can'

Re: [HACKERS] Parallel Aggregate

2016-03-19 Thread David Rowley
On 19 March 2016 at 06:15, Robert Haas wrote: > On Fri, Mar 18, 2016 at 9:16 AM, David Rowley > wrote: >> I've attached an updated patch. > > This looks substantially better than earlier versions, although I > haven't studied every detail of it yet. > > + * P

Re: [HACKERS] Combining Aggregates

2016-03-19 Thread David Rowley
On 17 March 2016 at 16:30, Haribabu Kommi wrote: > On Wed, Mar 16, 2016 at 10:08 PM, David Rowley > wrote: >> On 16 March 2016 at 23:54, Haribabu Kommi wrote: >>> On Wed, Mar 16, 2016 at 8:34 AM, David Rowley >>> wrote: >>>> Yes me too, so I spent

Re: [HACKERS] Parallel Aggregate

2016-03-19 Thread David Rowley
fix_combine_agg_expr_mutator(node, &context); > +} > + > +static Node * > +fix_combine_agg_expr_mutator(Node *node, > fix_upper_expr_context *context) > > Don't we want to handle the case of context->subplan_itlist->has_non_vars as > it is handled in fix_upper_

Re: [HACKERS] Combining Aggregates

2016-03-19 Thread David Rowley
On 18 March 2016 at 13:39, Haribabu Kommi wrote: > On Thu, Mar 17, 2016 at 10:59 PM, Tomas Vondra > wrote: >> Hi, >> >> On 03/17/2016 12:53 PM, David Rowley wrote: >>> >> ... >>> >>> >>> I just had a quick skim over the patch

Re: [HACKERS] max_parallel_degree context level

2016-03-20 Thread David Rowley
cases are defects that we should try to > fix. It would only take anything but the CPU to be a bottleneck for this to be highly likely the case. If a non-parallel query is bound on I/O, then adding workers is most likely going to slow it down further. I've seen this when testing parall

Re: [HACKERS] Parallel Aggregate

2016-03-20 Thread David Rowley
On 20 March 2016 at 03:19, Robert Haas wrote: > On Fri, Mar 18, 2016 at 10:32 PM, David Rowley > wrote: >> Updated patch is attached. > > I think this looks structurally correct now, and I think it's doing > the right thing as far as parallelism is concerned. I don&#x

Re: [HACKERS] Choosing parallel_degree

2016-03-20 Thread David Rowley
ast not unless the the page only contained 2 rows. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://

Re: [HACKERS] Parallel Aggregate

2016-03-20 Thread David Rowley
On 21 March 2016 at 09:47, Tomas Vondra wrote: > On 03/20/2016 09:58 AM, David Rowley wrote: >> Thank you for the reviews. The only thing I can think to mention which >> I've not already is that I designed estimate_hashagg_tablesize() to be >> reusable in various pla

Re: [HACKERS] Parallel Aggregate

2016-03-20 Thread David Rowley
On 21 March 2016 at 15:48, Alvaro Herrera wrote: > David Rowley wrote: > >> I've rewritten the comment to become: >> >> /* >> * Providing that the estimated size of the hashtable does not exceed >> * work_mem, we'll generate a HashAgg Path, although i

Re: [HACKERS] Combining Aggregates

2016-03-21 Thread David Rowley
On 20 March 2016 at 16:48, David Rowley wrote: > I've attached another series of patches: > > 0001: This is the latest Parallel Aggregate Patch, not intended for > review here, but is required for the remaining patches. This patch has > changed quite a bit from the previous on

Re: [HACKERS] Parallel Aggregate

2016-03-21 Thread David Rowley
for taking the time to carefully review it and commit it. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://

Re: [HACKERS] parallel aggregation - Numeric is unsupported?

2016-03-21 Thread David Rowley
On 22/03/2016 5:24 pm, "Pavel Stehule" wrote: > > Hi > > is it expected in this moment? > > > Table "public.foo" > Column │ Type │ Modifiers > ╪═╪═══ > a │ integer │ > > postgres=# \d foo2 > Table "public.foo2" > Column │ Type │ Modifiers >

Re: [HACKERS] parallel aggregation - Numeric is unsupported?

2016-03-21 Thread David Rowley
On 22/03/2016 5:24 pm, "Pavel Stehule" wrote: > > Hi > > is it expected in this moment? > > > Table "public.foo" > Column │ Type │ Modifiers > ╪═╪═══ > a │ integer │ > > postgres=# \d foo2 > Table "public.foo2" > Column │ Type │ Modifiers >

Re: [HACKERS] Combining Aggregates

2016-03-24 Thread David Rowley
On 21 January 2016 at 08:06, Robert Haas wrote: > I re-reviewed this and have committed most of it with only minor > kibitizing. A few notes: I realised today that the combinefunc is rather undocumented. I've attached a patch which aims to fix this. Comments are welcome. -- D

Re: [HACKERS] Combining Aggregates

2016-03-24 Thread David Rowley
On 25 March 2016 at 06:17, Robert Haas wrote: > On Mon, Mar 21, 2016 at 2:18 PM, David Rowley > wrote: >> I've attached 2 of the patches which are affected by the changes. > > I think the documentation for 0001 needs some work yet. The > additional paragraph that you&#

Re: [HACKERS] Combining Aggregates

2016-03-25 Thread David Rowley
On 26 March 2016 at 15:07, David Rowley wrote: > Ok, so on further look at this I've decided to make changes and have > it so the serialisation function can be dumb about memory contexts in > the same way as finalize_aggregate() allows the final function to be > dumb... notice

Re: [HACKERS] Can we amend gitignore so git postgresql works with git on windows using Msys/Mingw64

2016-03-26 Thread David Rowley
nse messages like this: Sounds like git is setup to convert the line endings to CRLF. Check: git config --list | grep crlf Likely this will be set to "true" You can change the setting with: git config --global core.autocrlf input drop the --global if you only want to change the curre

Re: [HACKERS] Combining Aggregates

2016-03-29 Thread David Rowley
On 26 March 2016 at 15:07, David Rowley wrote: Many thanks Robert for committing the serialize states portion. > 0005: > Haribabu's patch; no change from last time. Just in case you jump ahead. I just wanted to re-highlight something Haribabu mentioned a while ago about combini

Re: [HACKERS] Combining Aggregates

2016-03-30 Thread David Rowley
On 31 March 2016 at 00:48, Robert Haas wrote: > On Tue, Mar 29, 2016 at 11:14 PM, David Rowley > wrote: >>> 0005: >>> Haribabu's patch; no change from last time. > > So what's the distinction between 0002 and 0005? And what is the > correct author/r

Re: [HACKERS]WIP: Covering + unique indexes.

2016-01-12 Thread David Rowley
On 13 January 2016 at 05:59, Anastasia Lubennikova < a.lubennik...@postgrespro.ru> wrote: > 08.01.2016 00:12, David Rowley: > > On 7 January 2016 at 06:36, Jeff Janes wrote: > >> On Tue, Jan 5, 2016 at 11:55 PM, David Rowley >> wrote: >> > create tabl

Re: [HACKERS]WIP: Covering + unique indexes.

2016-01-12 Thread David Rowley
would make more sense of the omit_opclass_4.0.patch was included together with this. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS]WIP: Covering + unique indexes.

2016-01-13 Thread David Rowley
On 14 January 2016 at 02:58, Anastasia Lubennikova < a.lubennik...@postgrespro.ru> wrote: > 13.01.2016 04:27, David Rowley: > >> I've also done some testing: >> >> create table ab (a int, b int); >> insert into ab select a,b from generate_Series(1,

Re: [HACKERS] Removing Functionally Dependent GROUP BY Columns

2016-01-13 Thread David Rowley
ation, and then we could also make that surplus if we find other Vars which make up the table's primary key. I didn't make these changes as I thought it was a less likely scenario. It wouldn't be too much extra code to add however. I've went and added an XXX comment to explain that

Re: [HACKERS] Removing Functionally Dependent GROUP BY Columns

2016-01-14 Thread David Rowley
two Vars */ > + if (varlist == NULL || list_length(varlist) < 2) > + continue; > > To be perfectly correct, the comment should say "at least two Vars". > > Changed per discussion from you and Geoff > Except the small remaining

Re: [HACKERS] Spelling corrections

2016-01-14 Thread David Rowley
On 15 January 2016 at 17:19, Robert Haas wrote: > On Sun, Jan 10, 2016 at 5:17 AM, David Rowley > wrote: > > On 10 January 2016 at 19:34, Peter Geoghegan wrote: > >> > >> Attached patch fixes a couple of misspellings. > > > > Snap! > > > http:

Re: [HACKERS] Combining Aggregates

2016-01-15 Thread David Rowley
On 16 January 2016 at 03:03, Robert Haas wrote: > On Tue, Dec 29, 2015 at 7:39 PM, David Rowley > wrote: > >> No, the idea I had in mind was to allow it to continue to exist in the > >> expanded format until you really need it in the varlena format, and > >>

Re: [HACKERS]WIP: Covering + unique indexes.

2016-01-17 Thread David Rowley
On 14 January 2016 at 08:24, David Rowley wrote: > I will try to review the omit_opclass_4.0.patch soon. > Hi, as promised, here's my review of the omit_opclass_4.0.patch patch. The following comment needs to be updated: * indexkeys[], indexcollations[], opfamily[], and opcintyp

Re: [HACKERS] Combining Aggregates

2016-01-17 Thread David Rowley
On 18 January 2016 at 14:36, Haribabu Kommi wrote: > On Sat, Jan 16, 2016 at 12:00 PM, David Rowley > wrote: > > On 16 January 2016 at 03:03, Robert Haas wrote: > >> > >> On Tue, Dec 29, 2015 at 7:39 PM, David Rowley > >> wrote: > >> >>

[HACKERS] Compiler warning in pg_am changes

2016-01-17 Thread David Rowley
Hi, I've attached a small patch to fix new compiler warning which is new as of 65c5fcd3 -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services amcostestimate_cast.patch Description: Binary data -- Sent via pgsq

Re: [HACKERS] Combining Aggregates

2016-01-18 Thread David Rowley
On 18 January 2016 at 16:44, Robert Haas wrote: > On Sun, Jan 17, 2016 at 9:26 PM, David Rowley > wrote: > > hmm, so wouldn't that mean that the transition function would need to > (for > > each input tuple): > > > > 1. Parse that StringInfo into token

Re: [HACKERS] Combining Aggregates

2016-01-18 Thread David Rowley
On 19 January 2016 at 02:44, Haribabu Kommi wrote: > On Mon, Jan 18, 2016 at 10:32 PM, David Rowley > wrote: > > I just thought like direct mapping of the structure with text pointer. > something like > the below. > > result = PG_ARGISNULL(0) ? NULL : (text *) PG_GE

Re: [HACKERS] Combining Aggregates

2016-01-18 Thread David Rowley
s coax the planner into a GroupAggregate plan, which is faster, but due to the the hash agg executor code not giving any regard to work_mem. If I set work_mem to 140MB (which is more realistic for this VM), it does cause the same swapping problems to occur. Probably setting aggtransspace for this aggregate to 1024 would help the costing problem, but it would also cause hashagg to be a less chosen option during planning. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Combining Aggregates

2016-01-18 Thread David Rowley
On 19 January 2016 at 18:04, Tomas Vondra wrote: > Hi, > > On 01/19/2016 05:00 AM, David Rowley wrote: > >> On 19 January 2016 at 06:03, Pavel Stehule > <mailto:pavel.steh...@gmail.com>> wrote: >> >> ... > >> >> It is strange, why ha

Re: [HACKERS] Combining Aggregates

2016-01-18 Thread David Rowley
m to. > > > > That's how I did it in my prototype, but the problem with that is that > > spinning up a memory context for every group sucks when there are many > > groups with only a small number of elements each - hence the 50% > > regression that David Rowley obser

Re: [HACKERS] Combining Aggregates

2016-01-19 Thread David Rowley
On 20 January 2016 at 05:56, Robert Haas wrote: > > On Mon, Dec 21, 2015 at 4:02 AM, David Rowley > wrote: > > Now, there has been talk of this previously, on various threads, but I > don't > > believe any final decisions were made on how exactly it should be done.

Re: [HACKERS] Combining Aggregates

2016-01-19 Thread David Rowley
On 20 January 2016 at 05:58, Robert Haas wrote: > > On Mon, Dec 21, 2015 at 4:02 AM, David Rowley > > wrote: > >> Now, there has been talk of this previously, on various threads, but I > don't > >> believe any final decisions were made on how exactly it

Re: [HACKERS] Combining Aggregates

2016-01-20 Thread David Rowley
On 20 January 2016 at 10:54, Robert Haas wrote: > On Tue, Jan 19, 2016 at 4:50 PM, David Rowley > wrote: > >> Oh, one more point: is there any reason why all of this needs to be a > >> single (giant) patch? I feel like the handling of internal states > >> c

Re: [HACKERS] Combining Aggregates

2016-01-20 Thread David Rowley
On 21 January 2016 at 01:44, Robert Haas wrote: > On Wed, Jan 20, 2016 at 7:38 AM, David Rowley > wrote: > >> To my mind, priority #1 ought to be putting this fine new > >> functionality to some use. Expanding it to every aggregate we've got > >> s

Re: [HACKERS] Combining Aggregates

2016-01-20 Thread David Rowley
On 21 January 2016 at 04:59, Robert Haas wrote: > On Wed, Jan 20, 2016 at 7:53 AM, David Rowley > wrote: > > On 21 January 2016 at 01:44, Robert Haas wrote: > >> > >> On Wed, Jan 20, 2016 at 7:38 AM, David Rowley > >> wrote: > >> >> To

Re: [HACKERS] Combining Aggregates

2016-01-20 Thread David Rowley
On 21 January 2016 at 08:06, Robert Haas wrote: > On Wed, Jan 20, 2016 at 7:38 AM, David Rowley > wrote: > > Agreed. So I've attached a version of the patch which does not have any > of > > the serialise/deserialise stuff in it. > > I re-reviewed this and hav

Re: [HACKERS] Combining Aggregates

2016-01-20 Thread David Rowley
tant future) on the serial/deserial part, so would be better to keep this thread for discussion on that. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Parallel Aggregate

2016-01-21 Thread David Rowley
se of partial aggregate was within a single process then you could be quite content with PAT_INTERNAL_ONLY. You'll just need to pull out the logic that checks for serial and deserial functions, since that's not in yet, and just have it return PAT_INTERNAL_ONLY if INTERNAL aggregates are f

Re: [HACKERS] Combining Aggregates

2016-01-21 Thread David Rowley
On 22 January 2016 at 06:56, Robert Haas wrote: > On Wed, Jan 20, 2016 at 8:32 PM, David Rowley > wrote: >> The other two usages which I have thought of are; >> >> 1) Aggregating before UNION ALL, which might be fairly simple after the >> grouping planner changes,

Re: [HACKERS]WIP: Covering + unique indexes.

2016-01-21 Thread David Rowley
On 20 January 2016 at 06:08, Anastasia Lubennikova wrote: > > > > 18.01.2016 01:02, David Rowley пишет: > > On 14 January 2016 at 08:24, David Rowley > wrote: >> >> I will try to review the omit_opclass_4.0.patch soon. > > > Hi, as promised, here'

Re: [HACKERS] Parallel Aggregate

2016-01-22 Thread David Rowley
they propagate through calculations is the subject of an entire branch of mathematics and computer science and will not be discussed here, except for the following points:" [1] [1] http://www.postgresql.org/docs/devel/static/datatype-numeric.html -- David Rowley http://www.

Re: [HACKERS] Performance improvement for joins where outer side is unique

2016-01-22 Thread David Rowley
t. If committer wants that, I can separate these out, but I'll hold off for a response before doing that. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services unique_joins_ba5b9cb_2016-01-23.patch Description: Binary d

Re: [HACKERS] Combining Aggregates

2016-01-22 Thread David Rowley
On 23 January 2016 at 09:17, Jeff Janes wrote: > On Wed, Jan 20, 2016 at 11:06 AM, Robert Haas wrote: >> On Wed, Jan 20, 2016 at 7:38 AM, David Rowley >> wrote: >>> Agreed. So I've attached a version of the patch which does not have any of >>> the seriali

Re: [HACKERS] Combining Aggregates

2016-01-22 Thread David Rowley
On 23 January 2016 at 09:44, David Rowley wrote: > On 23 January 2016 at 09:17, Jeff Janes wrote: >> On Wed, Jan 20, 2016 at 11:06 AM, Robert Haas wrote: >>> On Wed, Jan 20, 2016 at 7:38 AM, David Rowley >>> wrote: >>>> Agreed. So I've attached a ver

Re: [HACKERS] Removing Functionally Dependent GROUP BY Columns

2016-01-23 Thread David Rowley
history behind that is that at one point during developing the patch that test had started failing due to the group by item being removed therefore allowing the join removal conditions to be met. On testing again with the old test query I see this no longer happens, so I've removed the change, althou

Re: [HACKERS] Removing Functionally Dependent GROUP BY Columns

2016-01-23 Thread David Rowley
um_members() is that it's going to loop over each word in the bitmap no matter what, whereas a subset check can abort early in some cases. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via

Re: [HACKERS] Removing Functionally Dependent GROUP BY Columns

2016-01-24 Thread David Rowley
On 24 January 2016 at 08:20, Tom Lane wrote: > David Rowley writes: >> On 23 January 2016 at 12:44, Tom Lane wrote: >>> * What you did to join.sql/join.out seems a bit bizarre. The existing >>> test case that you modified was meant to test something else, and >&g

Re: [HACKERS] Removing Functionally Dependent GROUP BY Columns

2016-01-24 Thread David Rowley
On 25 January 2016 at 10:17, Tom Lane wrote: > David Rowley writes: >> I've looked into why the join is not removed; since the redundant >> GROUP BY columns are removed during planning, and since the outer >> query is planned before the sub query, then when the join rem

  1   2   3   4   5   6   7   8   9   10   >