Re: [PERFORM] PostgreSQL on ZFS: performance tuning

2016-09-27 Thread Jov
using zfs,you can tune full page write off for pg,which can save wal write io. 2016年7月29日 2:05 PM,"trafdev" 写道: > Hi. > > I have an OLAP-oriented DB (light occasional bulk writes and heavy > aggregated selects over large periods of data) based on Postgres 9.5.3. > > Server is a FreeBSD 10.3 with

Re: [PERFORM] PostgreSQL on ZFS: performance tuning

2016-09-27 Thread Jov
+1 larger record size can increase compression ratio,so reduce the io. Did you set atime off for zfs? 2016年9月28日 6:16 AM,"Karl Denninger" 写道: > On 9/27/2016 16:38, Tomas Vondra wrote: > > On 09/27/2016 06:00 PM, Torsten Zuehlsdorff wrote: > > > > On 29.07.2016 08:30, Tomas Vondra wrote: > > > >

Re: [PERFORM] Unexpected expensive index scan

2016-09-27 Thread Jake Nielsen
On Tue, Sep 27, 2016 at 6:24 PM, Jake Nielsen wrote: > > > On Tue, Sep 27, 2016 at 6:03 PM, Jake Nielsen > wrote: > >> >> On Tue, Sep 27, 2016 at 5:41 PM, Mike Sofen wrote: >> >>> *From:* Jake Nielsen*Sent:* Tuesday, September 27, 2016 5:22 PM >>> >>> >>> the query >>> >>> SELECT * FROM Syn

Re: [PERFORM] Unexpected expensive index scan

2016-09-27 Thread Jake Nielsen
On Tue, Sep 27, 2016 at 6:03 PM, Jake Nielsen wrote: > > On Tue, Sep 27, 2016 at 5:41 PM, Mike Sofen wrote: > >> *From:* Jake Nielsen*Sent:* Tuesday, September 27, 2016 5:22 PM >> >> >> the query >> >> SELECT * FROM SyncerEvent WHERE ID > 12468 AND propogatorId NOT IN >> ('"d8130ab9!-66d0!-4

Re: [PERFORM] Unexpected expensive index scan

2016-09-27 Thread Jake Nielsen
On Tue, Sep 27, 2016 at 5:41 PM, Mike Sofen wrote: > *From:* Jake Nielsen*Sent:* Tuesday, September 27, 2016 5:22 PM > > > the query > > SELECT * FROM SyncerEvent WHERE ID > 12468 AND propogatorId NOT IN > ('"d8130ab9!-66d0!-4f13!-acec!-a9556362f0ad"') AND conflicted != 1 AND > userId = '57dc

Re: [PERFORM] Unexpected expensive index scan

2016-09-27 Thread Mike Sofen
From: Jake NielsenSent: Tuesday, September 27, 2016 5:22 PM the query SELECT * FROM SyncerEvent WHERE ID > 12468 AND propogatorId NOT IN ('"d8130ab9!-66d0!-4f13!-acec!-a9556362f0ad"') AND conflicted != 1 AND userId = '57dc984f1c87461c0967e228' ORDER BY ID LIMIT 4000;^ On Tue, Sep 27, 2

Re: [PERFORM] Unexpected expensive index scan

2016-09-27 Thread Jake Nielsen
Herp, forgot to include the query: SELECT * FROM SyncerEvent WHERE ID > 12468 AND propogatorId NOT IN ('"d8130ab9!-66d0!-4f13!-acec!-a9556362f0ad"') AND conflicted != 1 AND userId = '57dc984f1c87461c0967e228' ORDER BY ID LIMIT 4000;^ On Tue, Sep 27, 2016 at 5:02 PM, Jake Nielsen wrote: > I've g

[PERFORM] Unexpected expensive index scan

2016-09-27 Thread Jake Nielsen
I've got a query that takes a surprisingly long time to run, and I'm having a really rough time trying to figure it out. Before I get started, here are the specifics of the situation: Here is the table that I'm working with (apologies for spammy indices, I've been throwing shit at the wall)

Re: [PERFORM] PostgreSQL on ZFS: performance tuning

2016-09-27 Thread Karl Denninger
On 9/27/2016 16:38, Tomas Vondra wrote: > On 09/27/2016 06:00 PM, Torsten Zuehlsdorff wrote: >> >> >> On 29.07.2016 08:30, Tomas Vondra wrote: >>> >>> >>> On 07/29/2016 08:04 AM, trafdev wrote: Hi. I have an OLAP-oriented DB (light occasional bulk writes and heavy aggregated sel

Re: [PERFORM] PostgreSQL on ZFS: performance tuning

2016-09-27 Thread Tomas Vondra
On 09/27/2016 06:00 PM, Torsten Zuehlsdorff wrote: On 29.07.2016 08:30, Tomas Vondra wrote: On 07/29/2016 08:04 AM, trafdev wrote: Hi. I have an OLAP-oriented DB (light occasional bulk writes and heavy aggregated selects over large periods of data) based on Postgres 9.5.3. Server is a Fre

Re: [PERFORM] Millions of tables

2016-09-27 Thread Greg Spiegelberg
On Tue, Sep 27, 2016 at 10:15 AM, Terry Schmitt wrote: > > > On Sun, Sep 25, 2016 at 7:50 PM, Greg Spiegelberg > wrote: > >> Hey all, >> >> Obviously everyone who's been in PostgreSQL or almost any RDBMS for a >> time has said not to have millions of tables. I too have long believed it >> until

Re: [PERFORM] Millions of tables

2016-09-27 Thread Terry Schmitt
On Sun, Sep 25, 2016 at 7:50 PM, Greg Spiegelberg wrote: > Hey all, > > Obviously everyone who's been in PostgreSQL or almost any RDBMS for a time > has said not to have millions of tables. I too have long believed it until > recently. > > AWS d2.8xlarge instance with 9.5 is my test rig using XF

Re: [PERFORM] PostgreSQL on ZFS: performance tuning

2016-09-27 Thread Torsten Zuehlsdorff
On 29.07.2016 08:30, Tomas Vondra wrote: On 07/29/2016 08:04 AM, trafdev wrote: Hi. I have an OLAP-oriented DB (light occasional bulk writes and heavy aggregated selects over large periods of data) based on Postgres 9.5.3. Server is a FreeBSD 10.3 with 64GB of RAM and 2x500GB SSD (root on

Re: [PERFORM] Millions of tables

2016-09-27 Thread Greg Spiegelberg
On Tue, Sep 27, 2016 at 9:42 AM, Mike Sofen wrote: > *From:* Mike Sofen *Sent:* Tuesday, September 27, 2016 8:10 AM > > *From:* Greg Spiegelberg *Sent:* Monday, September 26, 2016 7:25 AM > I've gotten more responses than anticipated and have answered some > questions and gotten some insight

Re: [PERFORM] Millions of tables

2016-09-27 Thread Greg Spiegelberg
On Tue, Sep 27, 2016 at 8:30 AM, Craig James wrote: > On Sun, Sep 25, 2016 at 7:50 PM, Greg Spiegelberg > wrote: > >> Hey all, >> >> Obviously everyone who's been in PostgreSQL or almost any RDBMS for a >> time has said not to have millions of tables. I too have long believed it >> until recent

Re: [PERFORM] Millions of tables

2016-09-27 Thread Mike Sofen
From: Mike Sofen Sent: Tuesday, September 27, 2016 8:10 AM From: Greg Spiegelberg Sent: Monday, September 26, 2016 7:25 AM I've gotten more responses than anticipated and have answered some questions and gotten some insight but my challenge again is what should I capture along the way to p

Re: [PERFORM] Millions of tables

2016-09-27 Thread Mike Sofen
From: Greg Spiegelberg Sent: Monday, September 26, 2016 7:25 AM I've gotten more responses than anticipated and have answered some questions and gotten some insight but my challenge again is what should I capture along the way to prove or disprove this storage pattern? Alternatives to th

Re: [PERFORM] Millions of tables

2016-09-27 Thread Craig James
On Sun, Sep 25, 2016 at 7:50 PM, Greg Spiegelberg wrote: > Hey all, > > Obviously everyone who's been in PostgreSQL or almost any RDBMS for a time > has said not to have millions of tables. I too have long believed it until > recently. > > AWS d2.8xlarge instance with 9.5 is my test rig using XF