Re: 600 million rows of data. Bad hardware or need partitioning?

2020-05-05 Thread Justin Pryzby
On Tue, May 05, 2020 at 08:31:29PM -0400, Arya F wrote: > On Mon, May 4, 2020 at 5:21 AM Justin Pryzby wrote: > > > I mentioned in February and March that you should plan to set shared_buffers > > to fit the indexes currently being updated. > > The following command gives me > > select pg_size_

Re: 600 million rows of data. Bad hardware or need partitioning?

2020-05-05 Thread Arya F
On Mon, May 4, 2020 at 5:21 AM Justin Pryzby wrote: > I mentioned in February and March that you should plan to set shared_buffers > to fit the indexes currently being updated. > The following command gives me select pg_size_pretty (pg_indexes_size('test_table')); pg_size_pretty --

Re: 600 million rows of data. Bad hardware or need partitioning?

2020-05-05 Thread Arya F
On Mon, May 4, 2020 at 12:44 AM David Rowley wrote: > How long does it take if you first do: > > SET enable_nestloop TO off; I tried this, but it takes much longer Time: 318620.319 ms (05:18.620) Below is the EXPLAIN ANALYZE QUERY PLAN ---

Re: Please help! Query jumps from 1s -> 4m

2020-05-05 Thread Tom Lane
James Thompson writes: > The slowness occurs when the prepared statement changes to a generic plan. > Initial plan: > -> Index Only Scan using table1_typea_include_uniqueid_col16_idx on table1 > table1alias2 (cost=0.56..2549.70 rows=70 width=36) (actual > time=1.901..45.256 rows=65000 loops=1)

Re: Please help! Query jumps from 1s -> 4m

2020-05-05 Thread Justin Pryzby
On Tue, May 05, 2020 at 10:10:18PM +0100, James Thompson wrote: > I've managed to replicate this now with prepared statements. Thanks for all > the guidance so far. > > The slowness occurs when the prepared statement changes to a generic plan. > > Initial plan: > -> Index Only Scan using table1_

Re: Please help! Query jumps from 1s -> 4m

2020-05-05 Thread James Thompson
I've managed to replicate this now with prepared statements. Thanks for all the guidance so far. The slowness occurs when the prepared statement changes to a generic plan. Initial plan: -> Index Only Scan using table1_typea_include_uniqueid_col16_idx on table1 table1alias2 (cost=0.56..2549.70 r

Re: Please help! Query jumps from 1s -> 4m

2020-05-05 Thread Justin Pryzby
On Mon, May 04, 2020 at 02:12:01PM -0500, Justin Pryzby wrote: > On Mon, May 04, 2020 at 08:07:07PM +0100, Jamie Thompson wrote: > > Additionally, the execution plans for the 10th + following queries look > > fine, they have the same structure as if I run the query manually. It's not > > that the q

Re: NUMA settings

2020-05-05 Thread Laurenz Albe
On Tue, 2020-05-05 at 10:11 +0200, Marc Rechté wrote: > > I think that "vm.swappiness" should be 0. > > PostgreSQL does its own memory management, any swapping by the kernel > > would go against that. > > > > Yours, > > Laurenz Albe > > > As said in the post, we wish the idle backends to be swapp

Re: NUMA settings

2020-05-05 Thread Marc Rechté
On Tue, 2020-05-05 at 07:56 +0200, Marc Rechté wrote: Thanks for answers. Further readings make me think that we should *not* start postgres with numactl --interleave=all: this may have counter productive effect on backends anon memory (heap, stack). IMHO, what is important is to use Huge Pages f

Re: Please help! Query jumps from 1s -> 4m

2020-05-05 Thread Laurenz Albe
On Mon, 2020-05-04 at 20:12 +0100, James Thompson wrote: > The change is abrupt, on the 10th execution (but I hadn't spotted it was > always after the > same number of executions until your suggestion - thanks for pointing me in > that direction). > > I don't see any custom configuration on our

Re: NUMA settings

2020-05-05 Thread Laurenz Albe
On Tue, 2020-05-05 at 07:56 +0200, Marc Rechté wrote: > Thanks for answers. Further readings make me think that we should *not* > start postgres with numactl --interleave=all: this may have counter > productive effect on backends anon memory (heap, stack). IMHO, what is > important is to use Hug