Re: [PERFORM] Working on huge RAM based datasets

2004-07-25 Thread Andy Ballingall
Sorry for the late reply - I've been away. Merlin, I'd like to come back with a few more points! >That's the whole point: memory is a limited resource. If pg is >crawling, then the problem is simple: you need more memory. My posting only relates to the scenario where RAM is not a limiting facto

Re: [PERFORM] Working on huge RAM based datasets

2004-07-20 Thread abhousehunt
Sorry for the late reply - I've been away, and I've had problems posting too :( Merlin, I'd like to come back with a few more points! >That's the whole point: memory is a limited resource. If pg is >crawling, then the problem is simple: you need more memory. My posting only relates to the scena

Re: [PERFORM] Working on huge RAM based datasets

2004-07-13 Thread Andy Ballingall
rom: "J. Andrew Rogers" <[EMAIL PROTECTED]> To: "Andy Ballingall" <[EMAIL PROTECTED]> Sent: Friday, July 09, 2004 10:40 PM Subject: Re: [PERFORM] Working on huge RAM based datasets > On Fri, 2004-07-09 at 02:28, Andy Ballingall wrote: > > After all, we&#x

Re: [PERFORM] Working on huge RAM based datasets

2004-07-12 Thread Jan Wieck
On 7/12/2004 12:38 PM, Josh Berkus wrote: Rond, Chris, > What would be most interesting to see is whether this makes it wise to > increase shared buffer size. It may be more effective to bump down > the cache a little, and bump up sort memory; hard to tell. How do we go about scheduling tests with

Re: [PERFORM] Working on huge RAM based datasets

2004-07-12 Thread Josh Berkus
Rond, Chris, > > What would be most interesting to see is whether this makes it wise to > > increase shared buffer size. It may be more effective to bump down > > the cache a little, and bump up sort memory; hard to tell. > > How do we go about scheduling tests with the OSDL folks? If they could

Re: [PERFORM] Working on huge RAM based datasets

2004-07-12 Thread Rod Taylor
> What would be most interesting to see is whether this makes it wise to > increase shared buffer size. It may be more effective to bump down > the cache a little, and bump up sort memory; hard to tell. How do we go about scheduling tests with the OSDL folks? If they could do 10 runs with buffers

Re: [PERFORM] Working on huge RAM based datasets

2004-07-12 Thread Merlin Moncure
Andy wrote: > Whether the OS caches the data or PG does, you still want it cached. If > your > sorting backends gobble up the pages that otherwise would be filled with > the > database buffers, then your postmaster will crawl, as it'll *really* have > to > wait for stuff from disk. In my scenario,

Re: [PERFORM] Working on huge RAM based datasets

2004-07-12 Thread Merlin Moncure
Jan wrote: > > The disk cache on most operating systems is optimized. Plus, keeping > > shared buffers low gives you more room to bump up the sort memory, which > > will make your big queries run faster. > > Plus, the situation will change dramatically with 7.5 where the disk > cache will have le

Re: [PERFORM] Working on huge RAM based datasets

2004-07-11 Thread Christopher Browne
Martha Stewart called it a Good Thing when [EMAIL PROTECTED] (Jan Wieck) wrote: > On 7/9/2004 10:16 AM, Merlin Moncure wrote: >>> What is it about the buffer cache that makes it so unhappy being >>> able to hold everything? I don't want to be seen as a cache hit >>> fascist, but isn't it just bette

Re: [PERFORM] Working on huge RAM based datasets

2004-07-11 Thread Jan Wieck
On 7/9/2004 10:16 AM, Merlin Moncure wrote: What is it about the buffer cache that makes it so unhappy being able to hold everything? I don't want to be seen as a cache hit fascist, but isn't it just better if the data is just *there*, available in the postmaster's address space ready for each back

Re: [PERFORM] Working on huge RAM based datasets

2004-07-10 Thread Christopher Browne
Quoth [EMAIL PROTECTED] ("Andy Ballingall"): > This is the future, isn't it? Each year, a higher percentage of DB > applications will be able to fit entirely in RAM, and that percentage is > going to be quite significant in just a few years. The disk system gets > relegated to a data preload on sta

Re: [PERFORM] Working on huge RAM based datasets

2004-07-09 Thread Andy Ballingall
Thanks, Chris. > > What is it about the buffer cache that makes it so unhappy being able to > > hold everything? I don't want to be seen as a cache hit fascist, but isn't > > it just better if the data is just *there*, available in the postmaster's > > address space ready for each backend process

Re: [PERFORM] Working on huge RAM based datasets

2004-07-09 Thread Andy Ballingall
>The disk cache on most operating systems is optimized. Plus, keeping shared buffers low gives you more room to bump up the sort memory, which will make your big queries run faster. Thanks merlin, Whether the OS caches the data or PG does, you still want it cached. If your sorting backends gobb

Re: [PERFORM] Working on huge RAM based datasets

2004-07-09 Thread Merlin Moncure
> What is it about the buffer cache that makes it so unhappy being able to > hold everything? I don't want to be seen as a cache hit fascist, but isn't > it just better if the data is just *there*, available in the postmaster's > address space ready for each backend process to access it, rather tha

Re: [PERFORM] Working on huge RAM based datasets

2004-07-08 Thread Christopher Kings-Lynne
What is it about the buffer cache that makes it so unhappy being able to hold everything? I don't want to be seen as a cache hit fascist, but isn't it just better if the data is just *there*, available in the postmaster's address space ready for each backend process to access it, rather than expect

[PERFORM] Working on huge RAM based datasets

2004-07-08 Thread Andy Ballingall
Hi, I'm really stuck and I wonder if any of you could help. I have an application which will be sitting on a quite large database (roughly 8-16GB). The nature of the application is such that, on a second by second basis, the working set of the database is likely to be a substantial portion (e.g.