Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

On Wed, Dec 3, 2014 at 3:23 PM, Kouhei Kaigai wrote: >> On Fri, Nov 14, 2014 at 02:51:32PM +1300, David Rowley wrote: >> > Likely for most aggregates, like count, sum, max, min, bit_and and >> > bit_or the merge function would be the same as the transition >> > function, as the state type is just

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

> On Fri, Nov 14, 2014 at 02:51:32PM +1300, David Rowley wrote: > > Likely for most aggregates, like count, sum, max, min, bit_and and > > bit_or the merge function would be the same as the transition > > function, as the state type is just the same as the input type. It > > would only be aggregate

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

On Fri, Nov 14, 2014 at 02:51:32PM +1300, David Rowley wrote: > Likely for most aggregates, like count, sum, max, min, bit_and and bit_or the > merge function would be the same as the transition function, as the state type > is just the same as the input type. It would only be aggregates like avg()

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

> On 18 November 2014 05:19, Simon Riggs wrote: > > > On 14 November 2014 11:02, Kouhei Kaigai > wrote: > > > I'd like to throw community folks a question. > > Did someone have a discussion to the challenge of aggregate > push-down across > > relations join in the past?

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

On 18 November 2014 05:19, Simon Riggs wrote: > On 14 November 2014 11:02, Kouhei Kaigai wrote: > > > I'd like to throw community folks a question. > > Did someone have a discussion to the challenge of aggregate push-down > across > > relations join in the past? It potentially reduces number of

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

On 17 November 2014 16:01, Robert Haas wrote: > I still don't know what CONTAINS NO SQL means. It's a function marking that would indicate we aren't allowed to take snapshots or run SQL. I think you should publish the scheme for marking functions as safe for parallelism, so we can judge that.

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

On 14 November 2014 11:02, Kouhei Kaigai wrote: > I'd like to throw community folks a question. > Did someone have a discussion to the challenge of aggregate push-down across > relations join in the past? It potentially reduces number of rows to be > joined. > If we already had, I'd like to chec

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

On Thu, Nov 13, 2014 at 7:27 PM, Simon Riggs wrote: > On 12 November 2014 00:54, Robert Haas wrote: >> On Tue, Nov 11, 2014 at 3:29 AM, Simon Riggs wrote: >>> * only functions marked as "CONTAINS NO SQL" >>> We don't really know what proisparallel is, but we do know what >>> CONTAINS NO SQL mean

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

On Fri, Nov 14, 2014 at 1:27 PM, Simon Riggs wrote: > On 12 November 2014 00:54, Robert Haas wrote: > > Interestingly, I have a fairly solid idea of what proisparallel is, > > but I have no clear idea what CONTAINS NO SQL is or why it's relevant. > > I would imagine that srandom() contains no SQ

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

> On 14 November 2014 07:37, Jim Nasby wrote: > > On 11/12/14, 1:54 AM, David Rowley wrote: > >> > >> On Tue, Nov 11, 2014 at 9:29 PM, Simon Riggs >> > wrote: > >> > >> > >> This plan type is widely used in reporting queries, so will hit the > >> mainline of

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

On 14 November 2014 01:51, David Rowley wrote: > When I mentioned this, I didn't mean to appear to be placing a road block.I > was just bringing to the table the information that COUNT(*) + COUNT(*) > works ok for merging COUNT(*)'s "sub totals", but AVG(n) + AVG(n) does not. > > Merge functions

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

On 14 November 2014 07:37, Jim Nasby wrote: > On 11/12/14, 1:54 AM, David Rowley wrote: >> >> On Tue, Nov 11, 2014 at 9:29 PM, Simon Riggs > > wrote: >> >> >> This plan type is widely used in reporting queries, so will hit the >> mainline of BI applications an

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

On 14 November 2014 20:37, Jim Nasby wrote: > On 11/12/14, 1:54 AM, David Rowley wrote: > >> >> We'd also need to add some infrastructure to merge aggregate states >> together for this to work properly. This means that could also work for >> avg() and stddev etc. For max() and min() the merge fun

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

On 11/12/14, 1:54 AM, David Rowley wrote: On Tue, Nov 11, 2014 at 9:29 PM, Simon Riggs mailto:si...@2ndquadrant.com>> wrote: This plan type is widely used in reporting queries, so will hit the mainline of BI applications and many Mat View creations. This will allow SELECT count(*) F

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

> On Fri, Nov 14, 2014 at 2:12 PM, Kouhei Kaigai wrote: > > > > On 12 November 2014 07:54, David Rowley > > > Take int4_avg_accum() for example it does: > > > > > > transdata->count++; > > transdata->sum += newval; > > > > The merge function

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

On Fri, Nov 14, 2014 at 2:12 PM, Kouhei Kaigai wrote: > > On 12 November 2014 07:54, David Rowley > > Take int4_avg_accum() for example it does: > > > > > > transdata->count++; > > transdata->sum += newval; > > > > The merge function would need to perform something like: > > > > transdata-

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

> On 12 November 2014 07:54, David Rowley > wrote: > > On Tue, Nov 11, 2014 at 9:29 PM, Simon Riggs > wrote: > >> > >> > >> This plan type is widely used in reporting queries, so will hit > the > >> mainline of BI applications and many Mat View creations. >

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

On Fri, Nov 14, 2014 at 1:19 PM, Simon Riggs wrote: > On 12 November 2014 07:54, David Rowley wrote: > > On Tue, Nov 11, 2014 at 9:29 PM, Simon Riggs > wrote: > >> > >> > >> This plan type is widely used in reporting queries, so will hit the > >> mainline of BI applications and many Mat View cr

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

On 12 November 2014 00:54, Robert Haas wrote: > On Tue, Nov 11, 2014 at 3:29 AM, Simon Riggs wrote: >> * only functions marked as "CONTAINS NO SQL" >> We don't really know what proisparallel is, but we do know what >> CONTAINS NO SQL means and can easily check for it. >> Plus I already have a pat

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

On 12 November 2014 07:54, David Rowley wrote: > On Tue, Nov 11, 2014 at 9:29 PM, Simon Riggs wrote: >> >> >> This plan type is widely used in reporting queries, so will hit the >> mainline of BI applications and many Mat View creations. >> This will allow SELECT count(*) FROM foo to go faster al

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

On 12 November 2014 00:54, Robert Haas wrote: > I'm going to aim for the simpler: > > Hash Aggregate > -> Parallel Seq Scan > Workers: 4 > > Yeah, I know that won't perform as well as what you're proposing, but > I'm fairly sure it's simpler. Simple is best, so +1. -- Simon Riggs

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

On Wed, Nov 12, 2014 at 9:49 PM, Robert Haas wrote: > On Tue, Nov 11, 2014 at 7:48 PM, Kouhei Kaigai wrote: >> Isn't provolatile = PROVOLATILE_IMMUTABLE sufficient? > > There are certainly things that are parallel-safe that are not > immutable. It might be the case that everything immutable is >

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

On Tue, Nov 11, 2014 at 7:48 PM, Kouhei Kaigai wrote: > Isn't provolatile = PROVOLATILE_IMMUTABLE sufficient? There are certainly things that are parallel-safe that are not immutable. It might be the case that everything immutable is parallel-safe. -- Robert Haas EnterpriseDB: http://www.enter

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

On Wed, Nov 12, 2014 at 1:24 PM, David Rowley wrote: > > On Tue, Nov 11, 2014 at 9:29 PM, Simon Riggs > wrote: > >> >> This plan type is widely used in reporting queries, so will hit the >> mainline of BI applications and many Mat View creations. >> This will allow SELECT count(*) FROM foo to go

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

On Tue, Nov 11, 2014 at 9:29 PM, Simon Riggs wrote: > > This plan type is widely used in reporting queries, so will hit the > mainline of BI applications and many Mat View creations. > This will allow SELECT count(*) FROM foo to go faster also. > > We'd also need to add some infrastructure to mer

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

On Tue, Nov 11, 2014 at 3:29 AM, Simon Riggs wrote: > * only functions marked as "CONTAINS NO SQL" > We don't really know what proisparallel is, but we do know what > CONTAINS NO SQL means and can easily check for it. > Plus I already have a patch for this, slightly bitrotted. Interestingly, I ha

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

> > Given > > where we are with the infrastructure, there would be a number of > > unhandled problems, such as deadlock detection (needs group locking or > > similar), assessment of quals as to parallel-safety (needs > > proisparallel or similar), general waterproofing to make sure that > > pushing

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

On 10 November 2014 15:57, Robert Haas wrote: > On Wed, Oct 15, 2014 at 2:55 PM, Simon Riggs wrote: >> Something usable, with severe restrictions, is actually better than we >> have now. I understand the journey this work represents, so don't be >> embarrassed by submitting things with heuristics

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

ackers@postgresql.org > Subject: Re: [HACKERS] using custom scan nodes to prototype parallel > sequential scan > > On Tue, Nov 11, 2014 at 2:35 PM, Amit Kapila > wrote: > > On Tue, Nov 11, 2014 at 5:30 AM, Haribabu Kommi > > > > wrote: > >> > &

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

On Tue, Nov 11, 2014 at 9:42 AM, Haribabu Kommi wrote: > > On Tue, Nov 11, 2014 at 2:35 PM, Amit Kapila wrote: > > On Tue, Nov 11, 2014 at 5:30 AM, Haribabu Kommi < kommi.harib...@gmail.com> > > wrote: > >> > >> On Tue, Nov 11, 2014 at 10:21 AM, Andres Freund > >> wrote: > >> > On 2014-11-10 10:

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

On Tue, Nov 11, 2014 at 2:35 PM, Amit Kapila wrote: > On Tue, Nov 11, 2014 at 5:30 AM, Haribabu Kommi > wrote: >> >> On Tue, Nov 11, 2014 at 10:21 AM, Andres Freund >> wrote: >> > On 2014-11-10 10:57:16 -0500, Robert Haas wrote: >> >> Does parallelism help at all? >> > >> > I'm pretty damn sure.

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

On Tue, Nov 11, 2014 at 5:30 AM, Haribabu Kommi wrote: > > On Tue, Nov 11, 2014 at 10:21 AM, Andres Freund wrote: > > On 2014-11-10 10:57:16 -0500, Robert Haas wrote: > >> Does parallelism help at all? > > > > I'm pretty damn sure. We can't even make a mildly powerfull storage > > fully busy righ

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

On Tue, Nov 11, 2014 at 10:21 AM, Andres Freund wrote: > On 2014-11-10 10:57:16 -0500, Robert Haas wrote: >> Does parallelism help at all? > > I'm pretty damn sure. We can't even make a mildly powerfull storage > fully busy right now. Heck, I can't make my workstation's storage with a > raid 10 ou

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

Hi Robert, All, On 2014-11-10 10:57:16 -0500, Robert Haas wrote: > On Wed, Oct 15, 2014 at 2:55 PM, Simon Riggs wrote: > > Something usable, with severe restrictions, is actually better than we > > have now. I understand the journey this work represents, so don't be > > embarrassed by submitting

[HACKERS] using custom scan nodes to prototype parallel sequential scan

On Wed, Oct 15, 2014 at 2:55 PM, Simon Riggs wrote: > Something usable, with severe restrictions, is actually better than we > have now. I understand the journey this work represents, so don't be > embarrassed by submitting things with heuristics and good-enoughs in > it. Our mentor, Mr.Lane, achi