Re: [BUGS] Huge number of disk writes after migration to 8.1

2006-02-01 Thread Bruce Momjian
Added to TODO: * Allow statistics collector information to be pulled from the collector process directly, rather than requiring the collector to write a filesystem file twice a second? --- Tom L

Re: [BUGS] Huge number of disk writes after migration to 8.1

2006-01-19 Thread Tom Lane
Alvaro Herrera <[EMAIL PROTECTED]> writes: > Chris Campbell wrote: >> Do the stats need to persist across postmaster restarts (or crashes)? > After a crash we forcibly _delete_ the stat file. I'm not sure why, though --- the writing technique seems fairly safe, and in any case the reader checks

Re: [BUGS] Huge number of disk writes after migration to 8.1

2006-01-19 Thread Magnus Hagander
> > In most cases you're going to see extremely few reads compared to > > writes on pg_stats, right? So why not have the backends > connect to the > > stats process (or perhaps use UDP, or use the pipe, or > whatever) and > > fetch the data when needed. So when nobody fetches any > data, ther

Re: [BUGS] Huge number of disk writes after migration to 8.1

2006-01-19 Thread Tom Lane
"Magnus Hagander" <[EMAIL PROTECTED]> writes: > In most cases you're going to see extremely few reads compared to writes > on pg_stats, right? So why not have the backends connect to the stats > process (or perhaps use UDP, or use the pipe, or whatever) and fetch the > data when needed. So when nob

Re: [BUGS] Huge number of disk writes after migration to 8.1

2006-01-19 Thread Alvaro Herrera
Chris Campbell wrote: > On Jan 19, 2006, at 06:51, Magnus Hagander wrote: > > >In most cases you're going to see extremely few reads compared to > >writes > >on pg_stats, right? So why not have the backends connect to the stats > >process (or perhaps use UDP, or use the pipe, or whatever) and

Re: [BUGS] Huge number of disk writes after migration to 8.1

2006-01-19 Thread Chris Campbell
On Jan 19, 2006, at 06:51, Magnus Hagander wrote: In most cases you're going to see extremely few reads compared to writes on pg_stats, right? So why not have the backends connect to the stats process (or perhaps use UDP, or use the pipe, or whatever) and fetch the data when needed. So when

Re: [BUGS] Huge number of disk writes after migration to 8.1

2006-01-19 Thread Magnus Hagander
> Alvaro Herrera <[EMAIL PROTECTED]> writes: > > Maybe the fact that the stat file is completely rewritten > every 500 ms > > should be reconsidered, if in the future someone chooses to rewrite > > the stat system. We can reconsider this part then, as well. > > Yeah, it's becoming pretty obvio

Re: [BUGS] Huge number of disk writes after migration to 8.1

2006-01-18 Thread Tom Lane
Marcin <[EMAIL PROTECTED]> writes: > Do you think it will be included in one of next 8.1.x releases, or do I > have to wait for 8.2? I'm working on a fix for 8.1.3. regards, tom lane ---(end of broadcast)--- TIP 1: if postin

Re: [BUGS] Huge number of disk writes after migration to 8.1

2006-01-18 Thread Tom Lane
Alvaro Herrera <[EMAIL PROTECTED]> writes: > Maybe the fact that the stat file is completely rewritten every 500 ms > should be reconsidered, if in the future someone chooses to rewrite > the stat system. We can reconsider this part then, as well. Yeah, it's becoming pretty obvious that that desi

Re: [BUGS] Huge number of disk writes after migration to 8.1

2006-01-18 Thread Marcin
Tom Lane wrote: > [ studies code... ] I see the culprit: it's the pgstat_report_vacuum > and pgstat_report_analyze routines that were added in 8.1. Those send > messages unconditionally, meaning that the collector will create table > entries for every table during a database-wide vacuum, even wit

Re: [BUGS] Huge number of disk writes after migration to 8.1

2006-01-18 Thread Alvaro Herrera
Tom Lane wrote: > Alvaro Herrera <[EMAIL PROTECTED]> writes: > > Well, if you issue a manual vacuum then you can argue that it has been > > "modified" (it's seeing some activity). But immediately after a vacuum > > the table will not be vacuumed by autovac if there's no other activity, > > because

Re: [BUGS] Huge number of disk writes after migration to 8.1

2006-01-18 Thread Tom Lane
Alvaro Herrera <[EMAIL PROTECTED]> writes: > Well, if you issue a manual vacuum then you can argue that it has been > "modified" (it's seeing some activity). But immediately after a vacuum > the table will not be vacuumed by autovac if there's no other activity, > because there's no need for it, s

Re: [BUGS] Huge number of disk writes after migration to 8.1

2006-01-18 Thread Alvaro Herrera
Tom Lane wrote: > Alvaro Herrera <[EMAIL PROTECTED]> writes: > > Tom Lane wrote: > >> I'm tempted to change pgstat_recv_vacuum > >> and pgstat_recv_analyze so that they will not create new hash entries, > >> but only insert the row count if the hash entry already exists. I am a > >> bit worried th

Re: [BUGS] Huge number of disk writes after migration to 8.1

2006-01-18 Thread Tom Lane
Alvaro Herrera <[EMAIL PROTECTED]> writes: > Tom Lane wrote: >> I'm tempted to change pgstat_recv_vacuum >> and pgstat_recv_analyze so that they will not create new hash entries, >> but only insert the row count if the hash entry already exists. I am a >> bit worried that I might be missing someth

Re: [BUGS] Huge number of disk writes after migration to 8.1

2006-01-18 Thread Alvaro Herrera
Tom Lane wrote: > Actually, there's another problem here: if we do have row-level stats > turned on, a manual "VACUUM" command will still cause the set of tables > listed in the stats file to grow to include every table in the DB, > whether or not anything interesting is happening to that table.

Re: [BUGS] Huge number of disk writes after migration to 8.1

2006-01-18 Thread Tom Lane
Alvaro Herrera <[EMAIL PROTECTED]> writes: > Tom Lane wrote: >> Given the nature of what's counted, I think that treating these >> messages as "row level" stats would be appropriate. Alvaro, what do >> you think? > Yeah, row level seems appropiate for what we use it. I'll take care of > it, unle

Re: [BUGS] Huge number of disk writes after migration to 8.1

2006-01-18 Thread Tom Lane
Alvaro Herrera <[EMAIL PROTECTED]> writes: > Yeah, row level seems appropiate for what we use it. I'll take care of > it, unless you want to do it. I'll fix it --- I want to put in an immediate tabpurge on drop, too. regards, tom lane ---(end of b

Re: [BUGS] Huge number of disk writes after migration to 8.1

2006-01-18 Thread Alvaro Herrera
Tom Lane wrote: > Stats off and it's still bloating the file?? > > [ studies code... ] I see the culprit: it's the pgstat_report_vacuum > and pgstat_report_analyze routines that were added in 8.1. Those send > messages unconditionally, meaning that the collector will create table > entries for

Re: [BUGS] Huge number of disk writes after migration to 8.1

2006-01-18 Thread Tom Lane
Marcin <[EMAIL PROTECTED]> writes: > On Tue, Jan 17, 2006 at 05:12:09PM -0500, Tom Lane wrote: >> However, that's always been the case, so I don't understand >> why your stats file is so much bigger in 8.1. Have you changed your >> vacuuming strategy at all since the 8.0 installation? Perhaps row

Re: [BUGS] Huge number of disk writes after migration to 8.1

2006-01-18 Thread Marcin
On Tue, Jan 17, 2006 at 05:12:09PM -0500, Tom Lane wrote: > However, that's always been the case, so I don't understand > why your stats file is so much bigger in 8.1. Have you changed your > vacuuming strategy at all since the 8.0 installation? Perhaps row or > block statistics weren't enabled i

Re: [BUGS] Huge number of disk writes after migration to 8.1

2006-01-17 Thread Tom Lane
Alvaro Herrera <[EMAIL PROTECTED]> writes: > Tom Lane wrote: >> Looking at the code, stats entries for dropped tables are cleaned out >> only when a VACUUM command is done; which is something we probably ought >> to change. > I was going to ask if you were confusing pgstat_vacuum_tabstat with a >

Re: [BUGS] Huge number of disk writes after migration to 8.1

2006-01-17 Thread Alvaro Herrera
Tom Lane wrote: > Marcin <[EMAIL PROTECTED]> writes: > > So there's not 81952, BUT, the tables are created (and dropped) quite > > often during the work hours (and they're regular, not TEMPORARY tables). > > I also find out, that there were 11170 tables created (and most of them > > dropped) toda

Re: [BUGS] Huge number of disk writes after migration to 8.1

2006-01-17 Thread Tom Lane
Marcin <[EMAIL PROTECTED]> writes: > Autovacuum is off. > The VACUUM FULL bottom line is: > INFO: free space map contains 59842 pages in 25371 relations > DETAIL: A total of 426720 page slots are in use (including overhead). > 426720 page slots are required to track all free space. > And the qu

Re: [BUGS] Huge number of disk writes after migration to 8.1

2006-01-17 Thread Tom Lane
Marcin <[EMAIL PROTECTED]> writes: > Attached you'll find the archive with data you've asked. Well, breaking down the contents of the stats file I find: 190 backends (218880 bytes) 118 databases (8496 bytes) 81952 tables (8523008 bytes) And the ps listing shows there actual