Re: [PERFORM] Curious about dead rows.

2007-11-16 Thread Kevin Grittner
>>> On Fri, Nov 16, 2007 at 4:01 PM, in message <[EMAIL PROTECTED]>, "Merlin Moncure" <[EMAIL PROTECTED]> wrote: > On Nov 16, 2007 10:56 AM, Brad Nicholson <[EMAIL PROTECTED]> wrote: >> On Wed, 2007-11-14 at 17:46 -0500, Tom Lane wrote: >> > Russell Smith <[EMAIL PROTECTED]> writes: >> > > It is

Re: [PERFORM] Curious about dead rows.

2007-11-16 Thread Merlin Moncure
On Nov 16, 2007 10:56 AM, Brad Nicholson <[EMAIL PROTECTED]> wrote: > On Wed, 2007-11-14 at 17:46 -0500, Tom Lane wrote: > > Russell Smith <[EMAIL PROTECTED]> writes: > > > It is possible that analyze is not getting the number of dead rows right? > > > > Hah, I think you are on to something. ANALY

Re: [PERFORM] Curious about dead rows.

2007-11-16 Thread Jean-David Beyer
Craig James wrote: > Alvaro Herrera wrote: >> To recap: >> >> - your app only does inserts >> - there has been no rollback lately >> - there are no updates >> - there are no deletes >> >> The only other source of dead rows I can think is triggers ... do you >> have any? (Not necessarily on this ta

Re: [PERFORM] Curious about dead rows.

2007-11-16 Thread Brad Nicholson
On Wed, 2007-11-14 at 17:46 -0500, Tom Lane wrote: > Russell Smith <[EMAIL PROTECTED]> writes: > > It is possible that analyze is not getting the number of dead rows right? > > Hah, I think you are on to something. ANALYZE is telling the truth > about how many "dead" rows it saw, but its notion o

Re: [PERFORM] Curious about dead rows.

2007-11-15 Thread Erik Jones
On Nov 14, 2007, at 4:46 PM, Tom Lane wrote: Russell Smith <[EMAIL PROTECTED]> writes: It is possible that analyze is not getting the number of dead rows right? Hah, I think you are on to something. ANALYZE is telling the truth about how many "dead" rows it saw, but its notion of "dead" is

Re: [PERFORM] Curious about dead rows.

2007-11-14 Thread Tom Lane
Russell Smith <[EMAIL PROTECTED]> writes: > It is possible that analyze is not getting the number of dead rows right? Hah, I think you are on to something. ANALYZE is telling the truth about how many "dead" rows it saw, but its notion of "dead" is "not good according to SnapshotNow". Thus, rows

Re: [PERFORM] Curious about dead rows.

2007-11-14 Thread Jean-David Beyer
Tom Lane wrote: > Russell Smith <[EMAIL PROTECTED]> writes: >> It is possible that analyze is not getting the number of dead rows >> right? > > Hah, I think you are on to something. ANALYZE is telling the truth about > how many "dead" rows it saw, but its notion of "dead" is "not good > accordin

Re: [PERFORM] Curious about dead rows.

2007-11-14 Thread Craig James
Alvaro Herrera wrote: To recap: - your app only does inserts - there has been no rollback lately - there are no updates - there are no deletes The only other source of dead rows I can think is triggers ... do you have any? (Not necessarily on this table -- perhaps triggers on other tables can

Re: [PERFORM] Curious about dead rows.

2007-11-14 Thread Jean-David Beyer
Mario Weilguni wrote: > Jean-David Beyer schrieb: >> I am doing lots of INSERTs on a table that starts out empty (I did a >> TRUNCATE on it). I am not, AFAIK, doing DELETEs or UPDATEs. Autovacuum is >> on. I moved logging up to debug2 level to see what was going on, and I >> get >> things like this

Re: [PERFORM] Curious about dead rows.

2007-11-14 Thread Jean-David Beyer
Alvaro Herrera wrote: > Jean-David Beyer wrote: >> Mario Weilguni wrote: > >>> Did you rollback some transactions? It will generate dead rows too - at >>> least I think so. >>> >> No, and the statistics confirm this. > > To recap: > > - your app only does inserts True. > - there has been no ro

Re: [PERFORM] Curious about dead rows.

2007-11-14 Thread Russell Smith
Jean-David Beyer wrote: [snip] 2007-11-14 12:00:31 EST DEBUG: analyzing "public.vl_in" 2007-11-14 12:00:31 EST DEBUG: "vl_in": scanned 2001 of 2001 pages, containing 183983 live rows and 52 dead rows; 3000 rows in sample, 183983 estimated total rows 2007-11-14 12:00:31 EST DEBUG: analyzing "pu

Re: [PERFORM] Curious about dead rows.

2007-11-14 Thread Alvaro Herrera
Jean-David Beyer wrote: > Mario Weilguni wrote: > > Did you rollback some transactions? It will generate dead rows too - at > > least I think so. > > > No, and the statistics confirm this. To recap: - your app only does inserts - there has been no rollback lately - there are no updates - there

Re: [PERFORM] Curious about dead rows.

2007-11-14 Thread Mario Weilguni
Jean-David Beyer schrieb: I am doing lots of INSERTs on a table that starts out empty (I did a TRUNCATE on it). I am not, AFAIK, doing DELETEs or UPDATEs. Autovacuum is on. I moved logging up to debug2 level to see what was going on, and I get things like this: "vl_as": scanned 3000 of 5296 pag

Re: [PERFORM] Curious about dead rows.

2007-11-14 Thread Jean-David Beyer
Andrew Sullivan wrote: > On Wed, Nov 14, 2007 at 07:12:45AM -0500, Jean-David Beyer wrote: >> I know there have been rollbacks but I do a REINDEX, CLUSTER, and >> VACUUM ANALYZE before starting the inserts in question. Do I need to do >> a VACUUM FULL ANALYZE instead? > > I had another idea. As A

Re: [PERFORM] Curious about dead rows.

2007-11-14 Thread Andrew Sullivan
On Wed, Nov 14, 2007 at 11:58:23AM -0500, Andrew Sullivan wrote: > No, every statement in psql is a transaction. Even SELECT. Every statement Err, to be clearer, "Every statement in psql is _somehow_ part of a transaction; if you don't start one explicitly, the statement runs on its own as a tra

Re: [PERFORM] Curious about dead rows.

2007-11-14 Thread Andrew Sullivan
On Wed, Nov 14, 2007 at 11:53:17AM -0500, Jean-David Beyer wrote: > that I run only one at a time, or leaving psql running. But as I understand > it, psql does not bother with transactions, and besides, I normally just do No, every statement in psql is a transaction. Even SELECT. Every statement

Re: [PERFORM] Curious about dead rows.

2007-11-14 Thread Jean-David Beyer
Andrew Sullivan wrote: > On Wed, Nov 14, 2007 at 07:12:45AM -0500, Jean-David Beyer wrote: >> I know there have been rollbacks but I do a REINDEX, CLUSTER, and VACUUM >> ANALYZE before starting the inserts in question. Do I need to do a VACUUM >> FULL ANALYZE instead? > > I had another idea. As A

Re: [PERFORM] Curious about dead rows.

2007-11-14 Thread Andrew Sullivan
On Wed, Nov 14, 2007 at 07:12:45AM -0500, Jean-David Beyer wrote: > > I know there have been rollbacks but I do a REINDEX, CLUSTER, and VACUUM > ANALYZE before starting the inserts in question. Do I need to do a VACUUM > FULL ANALYZE instead? I had another idea. As Alvaro says, CLUSTER will do e

Re: [PERFORM] Curious about dead rows.

2007-11-14 Thread Jean-David Beyer
Alvaro Herrera wrote: > Jean-David Beyer wrote: > >> How do I reset the counters in pg_stat_database and pg_stat_all_tables? >> I tried just restarting postgres, but it seems to be saved in the database, >> not just in the RAM of the server. > > There is a function called pg_stat_reset() or some

Re: [PERFORM] Curious about dead rows.

2007-11-14 Thread Alvaro Herrera
Jean-David Beyer wrote: > How do I reset the counters in pg_stat_database and pg_stat_all_tables? > I tried just restarting postgres, but it seems to be saved in the database, > not just in the RAM of the server. There is a function called pg_stat_reset() or some such. > I suppose that blks_read

Re: [PERFORM] Curious about dead rows.

2007-11-14 Thread Jean-David Beyer
Merlin Moncure wrote: > On Nov 13, 2007 9:26 PM, Jean-David Beyer <[EMAIL PROTECTED]> wrote: >> Merlin Moncure wrote: >>> what does pg_stat_all_tables say (assuming row level stats are on)? >> It says stuff like this: >> >> relname | seq_scan | seq_tup_read | idx_scan | idx_tup_fetch | n_tup_ins

Re: [PERFORM] Curious about dead rows.

2007-11-13 Thread Merlin Moncure
On Nov 13, 2007 9:26 PM, Jean-David Beyer <[EMAIL PROTECTED]> wrote: > Merlin Moncure wrote: > > what does pg_stat_all_tables say (assuming row level stats are on)? > It says stuff like this: > > relname | seq_scan | seq_tup_read | idx_scan | idx_tup_fetch | n_tup_ins | > n_tup_upd | n_tup_del >

Re: [PERFORM] Curious about dead rows.

2007-11-13 Thread Jean-David Beyer
Merlin Moncure wrote: > On Nov 10, 2007 1:38 PM, Jean-David Beyer <[EMAIL PROTECTED]> wrote: >> Tom Lane wrote: >>> Jean-David Beyer <[EMAIL PROTECTED]> writes: I am doing lots of INSERTs on a table that starts out empty (I did a TRUNCATE on it). I am not, AFAIK, doing DELETEs or UPDATEs.

Re: [PERFORM] Curious about dead rows.

2007-11-13 Thread Merlin Moncure
On Nov 10, 2007 1:38 PM, Jean-David Beyer <[EMAIL PROTECTED]> wrote: > Tom Lane wrote: > > Jean-David Beyer <[EMAIL PROTECTED]> writes: > >> I am doing lots of INSERTs on a table that starts out empty (I did a > >> TRUNCATE on it). I am not, AFAIK, doing DELETEs or UPDATEs. Autovacuum is > >> on. I

Re: [PERFORM] Curious about dead rows.

2007-11-13 Thread Ansgar -59cobalt- Wiechers
On 2007-11-13 Trevor Talbot wrote: > On 11/13/07, Alvaro Herrera <[EMAIL PROTECTED]> wrote: >> Jean-David Beyer wrote: >>> Sorry. Most of the lists I send to have ReplyTo set, but a few do >>> not. And then I forget. >> >> If you use "reply to all", it works wonderfully in both cases. > > Then it

Re: [PERFORM] Curious about dead rows.

2007-11-13 Thread Alvaro Herrera
Trevor Talbot escribió: > On 11/13/07, Alvaro Herrera <[EMAIL PROTECTED]> wrote: > > Jean-David Beyer wrote: > > > Andrew Sullivan wrote: > > > > I'm not a private support organisation; please send your replies to the > > > > list, not me. > > > > > > Sorry. Most of the lists I send to have ReplyTo

Re: [PERFORM] Curious about dead rows.

2007-11-13 Thread Trevor Talbot
On 11/13/07, Alvaro Herrera <[EMAIL PROTECTED]> wrote: > Jean-David Beyer wrote: > > Andrew Sullivan wrote: > > > I'm not a private support organisation; please send your replies to the > > > list, not me. > > > > Sorry. Most of the lists I send to have ReplyTo set, but a few do not. > > And then I

Re: [PERFORM] Curious about dead rows.

2007-11-13 Thread Alvaro Herrera
Jean-David Beyer wrote: > Andrew Sullivan wrote: > > I'm not a private support organisation; please send your replies to the > > list, not me. > > Sorry. Most of the lists I send to have ReplyTo set, but a few do not. > And then I forget. If you use "reply to all", it works wonderfully in both ca

Re: [PERFORM] Curious about dead rows.

2007-11-13 Thread Jean-David Beyer
Andrew Sullivan wrote: > I'm not a private support organisation; please send your replies to the > list, not me. Sorry. Most of the lists I send to have ReplyTo set, but a few do not. And then I forget. > > On Tue, Nov 13, 2007 at 04:57:23PM -0500, Jean-David Beyer wrote: >> What is it controlled

Re: [PERFORM] Curious about dead rows.

2007-11-13 Thread Andrew Sullivan
I'm not a private support organisation; please send your replies to the list, not me. On Tue, Nov 13, 2007 at 04:57:23PM -0500, Jean-David Beyer wrote: > What is it controlled by? The following are the non-default values in > postgresql.conf: > > redirect_stderr = on > log_directory = '/srv/dbms/

Re: [PERFORM] Curious about dead rows.

2007-11-13 Thread Andrew Sullivan
On Tue, Nov 13, 2007 at 02:50:59PM -0500, Jean-David Beyer wrote: > > How about ERROR? > > $ grep -i error Tue.log > $ Well, without actually logging into the machine and looking at the application, I confess I am stumped. Oh, wait. You do have the log level high enough that you should see erro

Re: [PERFORM] Curious about dead rows.

2007-11-13 Thread Jean-David Beyer
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Andrew Sullivan wrote: > Please don't drop the list, as someone else may see something. > > On Tue, Nov 13, 2007 at 10:06:13AM -0500, Jean-David Beyer wrote: >> OK. I turned logging from "none" to "mod" and got a gawdawful lot of stuff. > > Yes. > >

Re: [PERFORM] Curious about dead rows.

2007-11-13 Thread Andrew Sullivan
Please don't drop the list, as someone else may see something. On Tue, Nov 13, 2007 at 10:06:13AM -0500, Jean-David Beyer wrote: > OK. I turned logging from "none" to "mod" and got a gawdawful lot of stuff. Yes. > Then I ran it and got all the inserts. Using > grep -i delete file > grep -i updat

Re: [PERFORM] Curious about dead rows.

2007-11-12 Thread Andrew Sullivan
On Sat, Nov 10, 2007 at 09:22:58PM -0500, Jean-David Beyer wrote: > > > > So, there are NO failed inserts, and no updates? Cause that's what > > I'd expect to create the dead rows. > > > So would I. Hence the original question. Foreign keys with cascading deletes or updates? A -- Andrew Sull

Re: [PERFORM] Curious about dead rows.

2007-11-10 Thread Jean-David Beyer
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Scott Marlowe wrote: > On Nov 10, 2007 1:57 PM, Jean-David Beyer <[EMAIL PROTECTED]> wrote: >> Joshua D. Drake wrote: >>> Truncate will not create dead rows. However ROLLBACK will. Are you >>> getting any duplicate key errors or anything like that when

Re: [PERFORM] Curious about dead rows.

2007-11-10 Thread Scott Marlowe
On Nov 10, 2007 1:57 PM, Jean-David Beyer <[EMAIL PROTECTED]> wrote: > > Joshua D. Drake wrote: > > > > Truncate will not create dead rows. However ROLLBACK will. Are you > > getting any duplicate key errors or anything like that when you insert? > > > On the mistaken assumption that TRUNCATE left

Re: [PERFORM] Curious about dead rows.

2007-11-10 Thread Jean-David Beyer
Joshua D. Drake wrote: > On Sat, 10 Nov 2007 13:38:23 -0500 Jean-David Beyer > <[EMAIL PROTECTED]> wrote: > >>> Tom Lane wrote: Jean-David Beyer <[EMAIL PROTECTED]> writes: > I am doing lots of INSERTs on a table that starts out empty (I > did a TRUNCATE on it). I am not, AFAIK, doing

Re: [PERFORM] Curious about dead rows.

2007-11-10 Thread Joshua D. Drake
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Sat, 10 Nov 2007 13:38:23 -0500 Jean-David Beyer <[EMAIL PROTECTED]> wrote: > Tom Lane wrote: > > Jean-David Beyer <[EMAIL PROTECTED]> writes: > >> I am doing lots of INSERTs on a table that starts out empty (I did > >> a TRUNCATE on it). I am not,

Re: [PERFORM] Curious about dead rows.

2007-11-10 Thread Jean-David Beyer
Tom Lane wrote: > Jean-David Beyer <[EMAIL PROTECTED]> writes: >> I am doing lots of INSERTs on a table that starts out empty (I did a >> TRUNCATE on it). I am not, AFAIK, doing DELETEs or UPDATEs. Autovacuum is >> on. I moved logging up to debug2 level to see what was going on, and I get >> things

Re: [PERFORM] Curious about dead rows.

2007-11-10 Thread Tom Lane
Jean-David Beyer <[EMAIL PROTECTED]> writes: > I am doing lots of INSERTs on a table that starts out empty (I did a > TRUNCATE on it). I am not, AFAIK, doing DELETEs or UPDATEs. Autovacuum is > on. I moved logging up to debug2 level to see what was going on, and I get > things like this: > "vl_as

[PERFORM] Curious about dead rows.

2007-11-10 Thread Jean-David Beyer
I am doing lots of INSERTs on a table that starts out empty (I did a TRUNCATE on it). I am not, AFAIK, doing DELETEs or UPDATEs. Autovacuum is on. I moved logging up to debug2 level to see what was going on, and I get things like this: "vl_as": scanned 3000 of 5296 pages, containing 232944 live r