Re: [GENERAL] PostgreSQL, OLAP, and Large Clusters

2012-09-27 Thread Chris Travers
On Thu, Sep 27, 2012 at 11:34 AM, Ryan Kelly wrote: > > > > At this time I would try: > > - Postgres-XC > From what I understand, more of a write-scaleable-oriented solution. We > mostly will need read scalability. I also don't think it really handles > redundancy. > >From my understanding it ge

Re: [GENERAL] PostgreSQL, OLAP, and Large Clusters

2012-09-27 Thread Scott Marlowe
On Thu, Sep 27, 2012 at 12:50 PM, Ryan Kelly wrote: > On Wed, Sep 26, 2012 at 03:18:16PM -0600, Scott Marlowe wrote: >> On Wed, Sep 26, 2012 at 5:50 AM, Ryan Kelly wrote: >> > Hi: >> > >> > The size of our database is growing rather rapidly. We're concerned >> > about how well Postgres will scale

Re: [GENERAL] PostgreSQL, OLAP, and Large Clusters

2012-09-27 Thread Ondrej Ivanič
Hi, On 28 September 2012 04:34, Ryan Kelly wrote: > On Thu, Sep 27, 2012 at 08:51:31AM +1000, Ondrej Ivanič wrote: >> - aggregation job ran every 15 minutes and completed under 2 minutes: >> 5mil rows -> aggregation -> 56 tables > 5mil overall, or matching your aggregation query? And is that the

Re: [GENERAL] PostgreSQL, OLAP, and Large Clusters

2012-09-27 Thread Rob Sargent
On 09/27/2012 12:50 PM, Ryan Kelly wrote: What do you mean by "fewer faster cores"? Wouldn't "more faster cores" be better? I believe his point is core does not equal cpu. Too often I've watched cpu performance meters with one core pegged and the other(s) idle, since really it's one cpu. I s

Re: [GENERAL] PostgreSQL, OLAP, and Large Clusters

2012-09-27 Thread Ryan Kelly
On Wed, Sep 26, 2012 at 03:18:16PM -0600, Scott Marlowe wrote: > On Wed, Sep 26, 2012 at 5:50 AM, Ryan Kelly wrote: > > Hi: > > > > The size of our database is growing rather rapidly. We're concerned > > about how well Postgres will scale for OLAP-style queries over terabytes > > of data. Googling

Re: [GENERAL] PostgreSQL, OLAP, and Large Clusters

2012-09-27 Thread Ryan Kelly
On Wed, Sep 26, 2012 at 09:15:35PM -0400, Stephen Frost wrote: > All, > > * Scott Marlowe (scott.marl...@gmail.com) wrote: > > If you want fastish OLAP on postgres you need to do several things. > [...] > > All good suggestions. > > I'd recommend looking at ROLAP approaches and doing aggregation

Re: [GENERAL] PostgreSQL, OLAP, and Large Clusters

2012-09-27 Thread Ryan Kelly
On Thu, Sep 27, 2012 at 08:58:05AM +1200, Gavin Flower wrote: > On 26/09/12 23:50, Ryan Kelly wrote: > >Hi: > > > >The size of our database is growing rather rapidly. We're concerned > >about how well Postgres will scale for OLAP-style queries over terabytes > >of data. Googling around doesn't yiel

Re: [GENERAL] PostgreSQL, OLAP, and Large Clusters

2012-09-27 Thread Ryan Kelly
On Thu, Sep 27, 2012 at 08:51:31AM +1000, Ondrej Ivanič wrote: > Hi, > > On 26 September 2012 21:50, Ryan Kelly wrote: > > The size of our database is growing rather rapidly. We're concerned > > about how well Postgres will scale for OLAP-style queries over terabytes > > of data. Googling around

Re: [GENERAL] PostgreSQL, OLAP, and Large Clusters

2012-09-26 Thread Chris Travers
Obviously OLAP performance depends to some extent on what you want to do with it. My recommendation is to follow Scott's advice first (keep things simple as long as you can), and then look at supplemental approaches when you reach the limit there. The big limit IMO is the lack of intraquery para

Re: [GENERAL] PostgreSQL, OLAP, and Large Clusters

2012-09-26 Thread Stephen Frost
All, * Scott Marlowe (scott.marl...@gmail.com) wrote: > If you want fastish OLAP on postgres you need to do several things. [...] All good suggestions. I'd recommend looking at ROLAP approaches and doing aggregations and materialized views first.. Will depend on exactly what you need/are lookin

Re: [GENERAL] PostgreSQL, OLAP, and Large Clusters

2012-09-26 Thread Ondrej Ivanič
Hi, On 26 September 2012 21:50, Ryan Kelly wrote: > The size of our database is growing rather rapidly. We're concerned > about how well Postgres will scale for OLAP-style queries over terabytes > of data. Googling around doesn't yield great results for vanilla > Postgres in this application, but

Re: [GENERAL] PostgreSQL, OLAP, and Large Clusters

2012-09-26 Thread Gavin Flower
On 26/09/12 23:50, Ryan Kelly wrote: Hi: The size of our database is growing rather rapidly. We're concerned about how well Postgres will scale for OLAP-style queries over terabytes of data. Googling around doesn't yield great results for vanilla Postgres in this application, but generally links

Re: [GENERAL] PostgreSQL, OLAP, and Large Clusters

2012-09-26 Thread Scott Marlowe
On Wed, Sep 26, 2012 at 5:50 AM, Ryan Kelly wrote: > Hi: > > The size of our database is growing rather rapidly. We're concerned > about how well Postgres will scale for OLAP-style queries over terabytes > of data. Googling around doesn't yield great results for vanilla > Postgres in this applicat

[GENERAL] PostgreSQL, OLAP, and Large Clusters

2012-09-26 Thread Ryan Kelly
Hi: The size of our database is growing rather rapidly. We're concerned about how well Postgres will scale for OLAP-style queries over terabytes of data. Googling around doesn't yield great results for vanilla Postgres in this application, but generally links to other software like Greenplum, Nete