Re: [HACKERS] optimizing CleanupTempFiles

2008-09-18 Thread Alvaro Herrera
I committed this to HEAD. If anything breaks, I request that someone else does the cleanup, as it looks like we're headed for the hospital first thing tomorrow morning to see what newborns look like. -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Co

Re: [HACKERS] optimizing CleanupTempFiles

2008-09-18 Thread Tom Lane
Alvaro Herrera <[EMAIL PROTECTED]> writes: > Don't temp tables use this kind of temp file? I admit I didn't check; I > just assumed they did. No, temp tables go through localbuf.c, which sits atop regular smgr. I don't think fd.c knows any difference from regular tables.

Re: [HACKERS] optimizing CleanupTempFiles

2008-09-18 Thread Alvaro Herrera
Tom Lane wrote: > Heikki Linnakangas <[EMAIL PROTECTED]> writes: > > Tom Lane wrote: > >> [ blink... ] Doesn't look like that should happen. What is your > >> test case? This was simply a CREATE TEMP TABLE ... ON COMMIT DROP. The file stays in place until checkpoint (either a manually invoked o

Re: [HACKERS] optimizing CleanupTempFiles

2008-09-18 Thread Heikki Linnakangas
Tom Lane wrote: Heikki Linnakangas <[EMAIL PROTECTED]> writes: Looks like we didn't make an exception for temporary tables. Although it's harmless, we could put an isTempOrToastNamespace() test in there: Bad, bad idea to have md.c doing any catalog access. isTempOrToastNamespace() doesn't a

Re: [HACKERS] optimizing CleanupTempFiles

2008-09-18 Thread Tom Lane
Heikki Linnakangas <[EMAIL PROTECTED]> writes: > Tom Lane wrote: >> [ blink... ] Doesn't look like that should happen. What is your >> test case? > Hmph, must be because of the patch from last winter to prevent > relfilenode reuse until next checkpoint. Ah. I had misunderstood Alvaro to say t

Re: [HACKERS] optimizing CleanupTempFiles

2008-09-18 Thread Heikki Linnakangas
Simon Riggs wrote: On Thu, 2008-09-18 at 10:19 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: An unfortunate choice of words! Harmless is not how your average DBA would describe it when their disk fills and they are apparently unable to reduce space consumption. So there is still a problem

Re: [HACKERS] optimizing CleanupTempFiles

2008-09-18 Thread Simon Riggs
On Thu, 2008-09-18 at 10:19 +0300, Heikki Linnakangas wrote: > Simon Riggs wrote: > > > > An unfortunate choice of words! Harmless is not how your average DBA > > would describe it when their disk fills and they are apparently unable > > to reduce space consumption. So there is still a problem th

Re: [HACKERS] optimizing CleanupTempFiles

2008-09-18 Thread Heikki Linnakangas
Simon Riggs wrote: On Thu, 2008-09-18 at 09:23 +0300, Heikki Linnakangas wrote: Tom Lane wrote: Alvaro Herrera <[EMAIL PROTECTED]> writes: BTW in testing this patch I was surprised by the fact that temp tables files are removed at checkpoint time, [ blink... ] Doesn't look like that should h

Re: [HACKERS] optimizing CleanupTempFiles

2008-09-18 Thread Simon Riggs
On Thu, 2008-09-18 at 09:23 +0300, Heikki Linnakangas wrote: > Tom Lane wrote: > > Alvaro Herrera <[EMAIL PROTECTED]> writes: > >> BTW in testing this patch I was surprised by the fact that temp tables > >> files are removed at checkpoint time, > > > > [ blink... ] Doesn't look like that should

Re: [HACKERS] optimizing CleanupTempFiles

2008-09-17 Thread Heikki Linnakangas
Tom Lane wrote: Alvaro Herrera <[EMAIL PROTECTED]> writes: BTW in testing this patch I was surprised by the fact that temp tables files are removed at checkpoint time, [ blink... ] Doesn't look like that should happen. What is your test case? Hmph, must be because of the patch from last wi

Re: [HACKERS] optimizing CleanupTempFiles

2008-09-17 Thread Tom Lane
Alvaro Herrera <[EMAIL PROTECTED]> writes: > BTW in testing this patch I was surprised by the fact that temp tables > files are removed at checkpoint time, [ blink... ] Doesn't look like that should happen. What is your test case? regards, tom lane -- Sent via pgsql-ha

Re: [HACKERS] optimizing CleanupTempFiles

2008-09-17 Thread Alvaro Herrera
BTW in testing this patch I was surprised by the fact that temp tables files are removed at checkpoint time, rather than when the transaction ends (at first I thought I had broken the removal of temp files). Is this a recent feature? (I verified that this continues to work fine for WITH HOLD cur

Re: [HACKERS] optimizing CleanupTempFiles

2008-09-17 Thread Alvaro Herrera
Tom Lane wrote: > Alvaro Herrera <[EMAIL PROTECTED]> writes: > > Ah -- like this? > > +1, but there are two kinds of temp files in that module, and only > one of them is relevant here. Call it something like > have_xact_temporary_files to make things clearer. Ok, so that's what I called it. Sim

Re: [HACKERS] optimizing CleanupTempFiles

2008-09-17 Thread Simon Riggs
On Wed, 2008-09-17 at 17:34 -0400, Alvaro Herrera wrote: > Ah -- like this? if test should include || isProcExit so you don't skip non-transactional temp files at proc exit when there weren't any created in the last transaction. -- Simon Riggs www.2ndQuadrant.com PostgreS

Re: [HACKERS] optimizing CleanupTempFiles

2008-09-17 Thread Simon Riggs
On Wed, 2008-09-17 at 17:34 -0400, Alvaro Herrera wrote: > Simon Riggs wrote: > > > > On Wed, 2008-09-17 at 16:25 -0400, Alvaro Herrera wrote: > > > > > We've been profiling a large system (8 CPUs, 64 GB of memory, some > > > dozens of disks) which seems rather more swamped than it should. Part

Re: [HACKERS] optimizing CleanupTempFiles

2008-09-17 Thread Tom Lane
Alvaro Herrera <[EMAIL PROTECTED]> writes: > Ah -- like this? +1, but there are two kinds of temp files in that module, and only one of them is relevant here. Call it something like have_xact_temporary_files to make things clearer. I concur that the explicit test on SizeVfdCache > 0 is a waste o

Re: [HACKERS] optimizing CleanupTempFiles

2008-09-17 Thread Alvaro Herrera
Alvaro Herrera wrote: > Simon Riggs wrote: > > Simple solution is to have a state variable so you can see whether a > > backend has created an temp files in this transaction. Most don't, so I > > think the above two solutions are overkill. If we created any, scan for > > them, if not, don't. Just

Re: [HACKERS] optimizing CleanupTempFiles

2008-09-17 Thread Alvaro Herrera
Simon Riggs wrote: > > On Wed, 2008-09-17 at 16:25 -0400, Alvaro Herrera wrote: > > > We've been profiling a large system (8 CPUs, 64 GB of memory, some > > dozens of disks) which seems rather more swamped than it should. Part > > of the problem seems to come from CleanupTempFiles, the second en

Re: [HACKERS] optimizing CleanupTempFiles

2008-09-17 Thread Simon Riggs
On Wed, 2008-09-17 at 16:25 -0400, Alvaro Herrera wrote: > We've been profiling a large system (8 CPUs, 64 GB of memory, some > dozens of disks) which seems rather more swamped than it should. Part > of the problem seems to come from CleanupTempFiles, the second entry in > oprofile output. I'm

[HACKERS] optimizing CleanupTempFiles

2008-09-17 Thread Alvaro Herrera
Hi, We've been profiling a large system (8 CPUs, 64 GB of memory, some dozens of disks) which seems rather more swamped than it should. Part of the problem seems to come from CleanupTempFiles, the second entry in oprofile output. This is the top 3 symbol report for a 2 minute oprofile run: $ op