Re: [PERFORM] shared_buffers vs Linux file cache

2015-01-15 Thread Roman Konoval
Huan, Residential memory is part of the process memory that is now swapped and is in RAM. This includes also memory shared with other processes so sum of RES for all processes may be greater that total physical memory. I recommend this article http://www.depesz.com/2012/06/09/how-much-ram-is-post

Re: [PERFORM] Strange choice of general index over partial index

2015-01-15 Thread Josh Berkus
> Right, I suspect that bloating is possibly the significant factor then - > can you REINDEX? Believe me, it's on the agenda. Of course, this is on a server with 90% saturated IO, so doing a repack is going to take some finessing. BTW, effective_cache_size is set to 100GB. So I suspect that it

Re: [PERFORM] Strange choice of general index over partial index

2015-01-15 Thread Mark Kirkwood
On 16/01/15 16:28, Josh Berkus wrote: On 01/16/2015 04:17 PM, Mark Kirkwood wrote: On 16/01/15 16:06, Mark Kirkwood wrote: A bit more poking about shows that the major factor (which this fake dataset anyway) is the default for effective_cache_size (changes from 128MB to 4GB in 9.4). Increasing

Re: [PERFORM] shared_buffers vs Linux file cache

2015-01-15 Thread Huan Ruan
Thanks very much, Glyn, Jeff, and Tom. That was very clearly explained. A related case, see the following top dump. The Postgres process is using 87g residential memory, which I thought was the physical memory consumed by a process that can't be shared with others. While, the free+cached is about

Re: [PERFORM] Strange choice of general index over partial index

2015-01-15 Thread Josh Berkus
On 01/16/2015 04:17 PM, Mark Kirkwood wrote: > On 16/01/15 16:06, Mark Kirkwood wrote: > >> A bit more poking about shows that the major factor (which this fake >> dataset anyway) is the default for effective_cache_size (changes from >> 128MB to 4GB in 9.4). Increasing this makes 9.2 start using t

Re: [PERFORM] Strange choice of general index over partial index

2015-01-15 Thread Mark Kirkwood
On 16/01/15 16:06, Mark Kirkwood wrote: A bit more poking about shows that the major factor (which this fake dataset anyway) is the default for effective_cache_size (changes from 128MB to 4GB in 9.4). Increasing this makes 9.2 start using the files_in_flight index in a plain index scan too. A

Re: [PERFORM] Strange choice of general index over partial index

2015-01-15 Thread Mark Kirkwood
On 16/01/15 15:32, Mark Kirkwood wrote: On 16/01/15 13:37, Mark Kirkwood wrote: On 16/01/15 11:30, Josh Berkus wrote: This is an obfuscation and mock up, but: table files ( id serial pk, filename text not null, state varchar(20) not null ... 18 more columns ) index file_state

Re: [PERFORM] Strange choice of general index over partial index

2015-01-15 Thread Tom Lane
Mark Kirkwood writes: > This is with each version loading exactly the same dataset (generated by > the attached scripty). Obviously this is a vast simplification of what > Josh is looking at - but it is (hopefully) interesting that these later > versions are doing so much better... Actually, w

Re: [PERFORM] Strange choice of general index over partial index

2015-01-15 Thread Mark Kirkwood
On 16/01/15 13:37, Mark Kirkwood wrote: On 16/01/15 11:30, Josh Berkus wrote: This is an obfuscation and mock up, but: table files ( id serial pk, filename text not null, state varchar(20) not null ... 18 more columns ) index file_state on (state) (35GB in size) index file_

Re: [PERFORM] Strange choice of general index over partial index

2015-01-15 Thread Tom Lane
Josh Berkus writes: > index file_state on (state) > (35GB in size) > index file_in_flight_state (state) where state in ( > 'waiting','assigning', 'processing' ) > (600MB in size) > ... 10 more indexes > More important facts: > * state = 'done' 95% of the time. thereform the partial i

[PERFORM] Autocompletion with full text search

2015-01-15 Thread Ivan Schneider
Hi, We implemented an autocompletion feature (case and accent insensitive) using PostgreSQL full text search. The query fetches patient ids matching the full text query that belong to a given patient base (rows contain a pg_array with patient_base_ids). Our table grew over time (6.2 million rows n

Re: [PERFORM] Strange choice of general index over partial index

2015-01-15 Thread Mark Kirkwood
On 16/01/15 11:30, Josh Berkus wrote: This is an obfuscation and mock up, but: table files ( id serial pk, filename text not null, state varchar(20) not null ... 18 more columns ) index file_state on (state) (35GB in size) index file_in_flight_state (stat

Re: [PERFORM] Strange choice of general index over partial index

2015-01-15 Thread Jeff Janes
On Thu, Jan 15, 2015 at 2:30 PM, Josh Berkus wrote: > This is an obfuscation and mock up, but: > > table files ( > id serial pk, > filename text not null, > state varchar(20) not null > ... 18 more columns > ) > > index file_state on (state) > (35GB in size

Re: [PERFORM] shared_buffers vs Linux file cache

2015-01-15 Thread Tom Lane
Jeff Janes writes: > On Thu, Jan 15, 2015 at 3:30 AM, Huan Ruan wrote: >> I thought 'shared_buffers' sets how much memory that is dedicated to >> PostgreSQL to use for caching data, therefore not available to other >> applications. > While PostgreSQL has reserves the right to use 32GB, as long a

[PERFORM] Strange choice of general index over partial index

2015-01-15 Thread Josh Berkus
This is an obfuscation and mock up, but: table files ( id serial pk, filename text not null, state varchar(20) not null ... 18 more columns ) index file_state on (state) (35GB in size) index file_in_flight_state (state) where state in ( 'waiting','assigning

Re: [PERFORM] shared_buffers vs Linux file cache

2015-01-15 Thread Jeff Janes
On Thu, Jan 15, 2015 at 3:30 AM, Huan Ruan wrote: > Hi All > > I thought 'shared_buffers' sets how much memory that is dedicated to > PostgreSQL to use for caching data, therefore not available to other > applications. > > However, as shown in the following screenshots, The server (CentOS 6.6 > 6

Re: [PERFORM] shared_buffers vs Linux file cache

2015-01-15 Thread Glyn Astill
> From: Huan Ruan >To: pgsql-performance@postgresql.org >Sent: Thursday, 15 January 2015, 11:30 >Subject: [PERFORM] shared_buffers vs Linux file cache > > > >Hi All > > >I thought 'shared_buffers' sets how much memory that is dedicated to >PostgreSQL to use for caching data, therefore not avail

[PERFORM] shared_buffers vs Linux file cache

2015-01-15 Thread Huan Ruan
Hi All I thought 'shared_buffers' sets how much memory that is dedicated to PostgreSQL to use for caching data, therefore not available to other applications. However, as shown in the following screenshots, The server (CentOS 6.6 64bit) has 64GB of RAM, and 'shared_buffer' is set to 32GB, but the