Re: [HACKERS] WIP - Add ability to constrain backend temporary file space

2011-03-09 Thread Mark Kirkwood
New version: - adds documentation - adds category RESOURCES_DISK temp-files-v2.patch.gz Description: GNU Zip compressed data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WIP - Add ability to constrain backend temporary file space

2011-02-18 Thread Mark Kirkwood
On 19/02/11 11:30, Josh Berkus wrote: Yeah, the disadvantage is that (like statement timeout) it is a 'bottom of the cliff' type of protection. The advantage is there are no false positives... Yeah, just trying to get a handle on the proposed feature. I have no objections; it seems like a harml

Re: [HACKERS] WIP - Add ability to constrain backend temporary file space

2011-02-18 Thread Josh Berkus
> Yeah, the disadvantage is that (like statement timeout) it is a 'bottom > of the cliff' type of protection. The advantage is there are no false > positives... Yeah, just trying to get a handle on the proposed feature. I have no objections; it seems like a harmless limit for most people, and us

Re: [HACKERS] WIP - Add ability to constrain backend temporary file space

2011-02-18 Thread Mark Kirkwood
On 19/02/11 10:38, Josh Berkus wrote: To answer the other question, what happens when the limit is exceeded is modeled on statement timeout, i.e query is canceled and a message says why (exceeded temp files size). When does this happen? When you try to allocate the file, or when it does the o

Re: [HACKERS] WIP - Add ability to constrain backend temporary file space

2011-02-18 Thread Tom Lane
Mark Kirkwood writes: > Added. With respect to the datatype, using int with KB units means the > largest temp size is approx 2047GB - I know that seems like a lot now... > but maybe someone out there wants (say) their temp files limited to > 4096GB :-) [ shrug... ] Sorry, I can't imagine a us

Re: [HACKERS] WIP - Add ability to constrain backend temporary file space

2011-02-18 Thread Josh Berkus
> Obviously you need to do the same sort of arithmetic as you do with > work_mem to decide on a reasonable limit to cope with multiple users > creating temp files. Conservative dbas might want to set it to (free > disk)/max_connections etc. Obviously for ad-hoc systems it is a bit more > challengi

Re: [HACKERS] WIP - Add ability to constrain backend temporary file space

2011-02-18 Thread Mark Kirkwood
On 19/02/11 08:48, Josh Berkus wrote: On 2/18/11 11:44 AM, Robert Haas wrote: On Fri, Feb 18, 2011 at 2:41 PM, Josh Berkus wrote: Second, the main issue with these sorts of macro-counters has generally been their locking effect on concurrent activity. Have you been able to run any tests which

Re: [HACKERS] WIP - Add ability to constrain backend temporary file space

2011-02-18 Thread Mark Kirkwood
On 19/02/11 02:34, Robert Haas wrote: Please add this to the next CommitFest: https://commitfest.postgresql.org/action/commitfest_view/open With respect to the datatype of the GUC, int seems clearly correct. Why would you want to use a float? Added. With respect to the datatype, using int w

Re: [HACKERS] WIP - Add ability to constrain backend temporary file space

2011-02-18 Thread Robert Haas
On Fri, Feb 18, 2011 at 2:48 PM, Josh Berkus wrote: > On 2/18/11 11:44 AM, Robert Haas wrote: >> On Fri, Feb 18, 2011 at 2:41 PM, Josh Berkus wrote: >>> Second, the main issue with these sorts of macro-counters has generally >>> been their locking effect on concurrent activity.  Have you been abl

Re: [HACKERS] WIP - Add ability to constrain backend temporary file space

2011-02-18 Thread Josh Berkus
On 2/18/11 11:44 AM, Robert Haas wrote: > On Fri, Feb 18, 2011 at 2:41 PM, Josh Berkus wrote: >> Second, the main issue with these sorts of macro-counters has generally >> been their locking effect on concurrent activity. Have you been able to >> run any tests which try to run lots of small exter

Re: [HACKERS] WIP - Add ability to constrain backend temporary file space

2011-02-18 Thread Robert Haas
On Fri, Feb 18, 2011 at 2:41 PM, Josh Berkus wrote: > Second, the main issue with these sorts of macro-counters has generally > been their locking effect on concurrent activity.  Have you been able to > run any tests which try to run lots of small externally-sorted queries > at once on a multi-cor

Re: [HACKERS] WIP - Add ability to constrain backend temporary file space

2011-02-18 Thread Josh Berkus
Mark, > I got to wonder how hard this would be to do in Postgres, and attached > is my (WIP) attempt. It provides a guc (max_temp_files_size) to limit > the size of all temp files for a backend and amends fd.c cancel > execution if the total size of temporary files exceeds this. First, are we jus

Re: [HACKERS] WIP - Add ability to constrain backend temporary file space

2011-02-18 Thread Robert Haas
On Thu, Feb 17, 2011 at 10:17 PM, Mark Kirkwood wrote: > This is WIP, it does seem to work ok, but some areas/choices I'm not > entirely clear about are mentioned in the patch itself. Mainly: > > - name of the guc... better suggestions welcome > - datatype for the guc - real would be good, but at