Re: [HACKERS] bg worker: general purpose requirements

2010-09-27 Thread Markus Wanner
Greg, On 09/25/2010 08:03 PM, Greg Stark wrote: > The dynamic ramp-up is a feature to deal for the default install and > for use case where the system has lots of different users with > different needs. Thanks for sharing this. From that perspective, neither the current min/max nor the timeout co

Re: [HACKERS] bg worker: general purpose requirements

2010-09-25 Thread Greg Stark
On Sat, Sep 18, 2010 at 4:21 AM, Robert Haas wrote: >> (It's exactly what apache pre-fork does, no? Is anybody concerned about the >> idle processes there? Or do they consume much less resources?) > > I don't know whether an idle Apache worker consumes more or less > memory than an idle Postg

Re: [HACKERS] bg worker: general purpose requirements

2010-09-21 Thread Markus Wanner
On 09/21/2010 05:59 PM, Robert Haas wrote: > Oh, wow. Is there another limit on the total number of bgworkers? There currently are three GUCs that control bgworkers: max_background_workers min_spare_background_workers max_spare_background_workers The first replaces the former autovacuum_max_wor

Re: [HACKERS] bg worker: general purpose requirements

2010-09-21 Thread Robert Haas
On Tue, Sep 21, 2010 at 11:31 AM, Markus Wanner wrote: > On 09/21/2010 03:46 PM, Robert Haas wrote: >> Wait, are we in violent agreement here?  An overall limit on the >> number of parallel jobs is exactly what I think *does* make sense. >> It's the other knobs I find odd. > > Note that the max se

Re: [HACKERS] bg worker: general purpose requirements

2010-09-21 Thread Markus Wanner
On 09/21/2010 03:46 PM, Robert Haas wrote: > Wait, are we in violent agreement here? An overall limit on the > number of parallel jobs is exactly what I think *does* make sense. > It's the other knobs I find odd. Note that the max setting I've been talking about here is the maximum amount of *idl

Re: [HACKERS] bg worker: general purpose requirements

2010-09-21 Thread Robert Haas
On Tue, Sep 21, 2010 at 4:23 AM, Markus Wanner wrote: > On 09/21/2010 02:49 AM, Robert Haas wrote: >> OK.  At least for me, what is important is not only how many GUCs >> there are but how likely they are to require tuning and how easy it >> will be to know what the appropriate value is.  It seems

Re: [HACKERS] bg worker: general purpose requirements

2010-09-21 Thread Markus Wanner
On 09/21/2010 02:49 AM, Robert Haas wrote: > OK. At least for me, what is important is not only how many GUCs > there are but how likely they are to require tuning and how easy it > will be to know what the appropriate value is. It seems fairly easy > to tune the maximum number of background work

Re: [HACKERS] bg worker: general purpose requirements

2010-09-20 Thread Robert Haas
On Mon, Sep 20, 2010 at 1:45 PM, Markus Wanner wrote: > Hm.. I see. So in other words, you are saying > min_spare_background_workers isn't flexible enough in case one has > thousands of databases but only uses a few of them frequently. Yes, I think that is true. > I understand that reasoning and

Re: [HACKERS] bg worker: general purpose requirements

2010-09-20 Thread Markus Wanner
Robert, On 09/20/2010 06:57 PM, Robert Haas wrote: > Gee, that doesn't seem slow enough to worry about to me. If we > suppose that you need 2 * CPUs + spindles processes to fully load the > system, that means you should be able to ramp up from zero to > consuming every available system resource i

Re: [HACKERS] bg worker: general purpose requirements

2010-09-20 Thread Robert Haas
On Mon, Sep 20, 2010 at 11:30 AM, Markus Wanner wrote: > Well, Apache pre-forks 5 processes in total (by default, that is, for > high volume webservers a higher MinSpareServers setting is certainly not > out of question). While bgworkers currently needs to fork > min_spare_background_workers proce

Re: [HACKERS] bg worker: general purpose requirements

2010-09-20 Thread Markus Wanner
Hi, On 09/18/2010 05:21 AM, Robert Haas wrote: > Wow, 100 processes??! Really? I guess I don't actually know how large > modern proctables are, but on my MacOS X machine, for example, there > are only 75 processes showing up right now in "ps auxww". My Fedora > 12 machine has 97. That's includi

Re: [HACKERS] bg worker: general purpose requirements

2010-09-20 Thread Markus Wanner
On 09/18/2010 05:43 AM, Tom Lane wrote: > The part of that that would worry me is open files. PG backends don't > have any compunction about holding open hundreds of files. Apiece. > You can dial that down but it'll cost you performance-wise. Last > I checked, most Unix kernels still had limited

Re: [HACKERS] bg worker: general purpose requirements

2010-09-17 Thread tomas
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Fri, Sep 17, 2010 at 11:21:13PM -0400, Robert Haas wrote: [...] > Wow, 100 processes??! Really? I guess I don't actually know how large > modern proctables are, but on my MacOS X machine, for example, there > are only 75 processes showing up righ

Re: [HACKERS] bg worker: general purpose requirements

2010-09-17 Thread Tom Lane
Robert Haas writes: > Wow, 100 processes??! Really? I guess I don't actually know how large > modern proctables are, but on my MacOS X machine, for example, there > are only 75 processes showing up right now in "ps auxww". My Fedora > 12 machine has 97. That's including a PostgreSQL instance in

Re: [HACKERS] bg worker: general purpose requirements

2010-09-17 Thread Robert Haas
On Fri, Sep 17, 2010 at 4:49 PM, Markus Wanner wrote: >> If you're optimizing for >> ability to respond quickly to a sudden load, keeping idle backends >> will probably win even when the number of them you're keeping around >> is fairly high.  If you're optimizing for minimal overall resource >> c

Re: [HACKERS] bg worker: general purpose requirements

2010-09-17 Thread Markus Wanner
Robert, On 09/17/2010 05:52 PM, Robert Haas wrote: Technically, you could start an autonomous transaction from within an autonomous transaction, so I don't think there's a hard maximum of one per normal backend. However, I agree that the expected case is to not have very many. Thanks for poin

Re: [HACKERS] bg worker: general purpose requirements

2010-09-17 Thread Robert Haas
On Fri, Sep 17, 2010 at 11:29 AM, Markus Wanner wrote: > autonomous transactions: max. one per normal backend (correct?), way fewer > should suffice in most cases, only control data to be passed around Technically, you could start an autonomous transaction from within an autonomous transaction, s

Re: [HACKERS] bg worker: general purpose requirements

2010-09-17 Thread Markus Wanner
Hi, On 09/16/2010 07:47 PM, Robert Haas wrote: It would be nice if there were a way to create a general facility here that we could then build various applications on, but I'm not sure whether that's the case. We had some back-and-forth about what is best for replication vs. what is best for va