Re: perf tuning for 28 cores and 252GB RAM

Hi, On 2019-06-18 17:13:20 +0100, Fabio Ugo Venchiarutti wrote: > Does the backend mmap() data files when that's possible? No. That doesn't allow us to control when data is written back to disk, which is crucial for durability/consistency. > I've heard the "use the page cache" suggestion before

Re: perf tuning for 28 cores and 252GB RAM

Thanks to everyone for all the detailed responses so far. The dataset we're working with, in its original form, is roughly a terabyte; I would guess in the database it will come out to about 2TB-ish. It will definitely not fit in RAM, unfortunately, and our access patterns will be quite random, at

Re: perf tuning for 28 cores and 252GB RAM

Hi, On 2019-06-18 12:33:30 -0400, Alvaro Herrera wrote: > On 2019-Jun-18, Andres Freund wrote: > > > On 2019-06-17 19:45:41 -0400, Jeff Janes wrote: > > > If not, I would set the value small (say, 8GB) and let the OS do the > > > heavy lifting of deciding what to keep in cache. > > > > FWIW, in

Re: perf tuning for 28 cores and 252GB RAM

Hello On 2019-Jun-18, Andres Freund wrote: > On 2019-06-17 19:45:41 -0400, Jeff Janes wrote: > > If not, I would set the value small (say, 8GB) and let the OS do the > > heavy lifting of deciding what to keep in cache. > > FWIW, in my opinion this is not a good idea in most cases. E.g. linux's >

Re: perf tuning for 28 cores and 252GB RAM

Hi, On 2019-06-17 19:45:41 -0400, Jeff Janes wrote: > If not, I would set the value small (say, 8GB) and let the OS do the > heavy lifting of deciding what to keep in cache. FWIW, in my opinion this is not a good idea in most cases. E.g. linux's pagecache doesn't scale particularly gracefully to

Re: perf tuning for 28 cores and 252GB RAM

On 18/06/2019 00:45, Jeff Janes wrote: On Mon, Jun 17, 2019 at 4:51 PM Michael Curry > wrote: I am using a Postgres instance in an HPC cluster, where they have generously given me an entire node. This means I have 28 cores and 252GB RAM. I have to assume that

Re: perf tuning for 28 cores and 252GB RAM

On Mon, Jun 17, 2019 at 6:46 PM Jeff Janes wrote: > > On Mon, Jun 17, 2019 at 4:51 PM Michael Curry wrote: >> >> I am using a Postgres instance in an HPC cluster, where they have generously >> given me an entire node. This means I have 28 cores and 252GB RAM. I have to >> assume that the very c

Re: perf tuning for 28 cores and 252GB RAM

> > If your entire database can comfortably fit in RAM, I would make > shared_buffers large enough to hold the entire database. If not, I would > set the value small (say, 8GB) and let the OS do the heavy lifting of > deciding what to keep in cache. If you go with the first option, you > probably

Re: perf tuning for 28 cores and 252GB RAM

On Mon, Jun 17, 2019 at 4:51 PM Michael Curry wrote: > I am using a Postgres instance in an HPC cluster, where they have > generously given me an entire node. This means I have 28 cores and 252GB > RAM. I have to assume that the very conservative default settings for > things like buffers and max