Re: [HACKERS] Surprising dead_tuple_count from pgstattuple

2010-08-10 Thread Tom Lane
Gordon Shannon writes: > - Because my table has no indexes, lazy_scan_heap calls lazy_vacuum_page > directly for each block, and reports the variable tups_vacuumed ("removed > 200 row versions in 2 pages"). However, tups_vacuumed is computed without > counting the 100 LP_DEAD tuples, because per

Re: [HACKERS] Surprising dead_tuple_count from pgstattuple

2010-08-10 Thread Gordon Shannon
After much code reading, testing, and using the extremely handy pageinspect contrib to look at pages, here's what I believe is happening. I am not attempting to describe every possible scenario, only this one test path. Following my short test scenario above... - Inserted rows get line pointers

Re: [HACKERS] Surprising dead_tuple_count from pgstattuple

2010-08-09 Thread Simon Riggs
On Tue, 2010-08-10 at 07:43 +0900, Itagaki Takahiro wrote: > 2010/8/10 Simon Riggs : > > Any SQL statement that reads a block can do HOT pruning, if the block is > > otherwise unlocked. > > We use the term "HOT" for two features: > 1. HOT updates: Avoiding index updates when keys are not modifie

Re: [HACKERS] Surprising dead_tuple_count from pgstattuple

2010-08-09 Thread Itagaki Takahiro
2010/8/10 Simon Riggs : > Any SQL statement that reads a block can do HOT pruning, if the block is > otherwise unlocked. We use the term "HOT" for two features: 1. HOT updates: Avoiding index updates when keys are not modified. 2. HOT pruning: Removing tuple bodies, that works even for indexed

Re: [HACKERS] Surprising dead_tuple_count from pgstattuple

2010-08-09 Thread Robert Haas
On Mon, Aug 9, 2010 at 2:23 PM, Heikki Linnakangas wrote: > On 09/08/10 21:21, Robert Haas wrote: >> >> On Mon, Aug 9, 2010 at 1:50 PM, Simon Riggs  wrote: >>> >>> Any SQL statement that reads a block can do HOT pruning, if the block is >>> otherwise unlocked. >> >> Where does heap_page_prune() ge

Re: [HACKERS] Surprising dead_tuple_count from pgstattuple

2010-08-09 Thread Heikki Linnakangas
On 09/08/10 21:21, Robert Haas wrote: On Mon, Aug 9, 2010 at 1:50 PM, Simon Riggs wrote: Any SQL statement that reads a block can do HOT pruning, if the block is otherwise unlocked. Where does heap_page_prune() get called from in the DELETE path? heapgetpage() -- Heikki Linnakangas Ent

Re: [HACKERS] Surprising dead_tuple_count from pgstattuple

2010-08-09 Thread Robert Haas
On Mon, Aug 9, 2010 at 1:50 PM, Simon Riggs wrote: > On Sat, 2010-08-07 at 16:11 -0700, Gordon Shannon wrote: > >> So, I guess my real question here is, what happened to the "missing" >> 100 items?  If it was HOT prune, can anyone summarize what that does? > > Itagaki already explained that the se

Re: [HACKERS] Surprising dead_tuple_count from pgstattuple

2010-08-09 Thread Simon Riggs
On Sat, 2010-08-07 at 16:11 -0700, Gordon Shannon wrote: > So, I guess my real question here is, what happened to the "missing" > 100 items? If it was HOT prune, can anyone summarize what that does? Itagaki already explained that the second DELETE would have removed the 100 dead rows you conside

Re: [HACKERS] Surprising dead_tuple_count from pgstattuple

2010-08-08 Thread Kevin Grittner
Gordon Shannon wrote: > If it was HOT prune, can anyone summarize what that does? Get a copy of the PostgreSQL source, and read this file: src/backend/access/heap/README.HOT -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription

Re: [HACKERS] Surprising dead_tuple_count from pgstattuple

2010-08-07 Thread Gordon Shannon
I think this simple test highlights the question well. -- create temporary table t(x int) with (autovacuum_enabled=off); insert into t select x from generate_series(1,1,1)x; vacuum verbose t; select dead_tuple_count from pgstattuple('t');--> 0 delete from t where x <= 100;

Re: [HACKERS] Surprising dead_tuple_count from pgstattuple

2010-08-07 Thread Heikki Linnakangas
On 07/08/10 07:43, Gordon Shannon wrote: Regarding HOT prune, I never did any updates, so I think there couldn't be any HOT tuples. Or does HOT prune do more than that? Yes, HOT will also prune away DELETEd tuples. It will leave behind a dead line pointer, so it won't stop the table from grow

Re: [HACKERS] Surprising dead_tuple_count from pgstattuple

2010-08-06 Thread Gordon Shannon
Robert Haas wrote: > > My thought would be "is autovacuum running in the background in > between these commands?". > That's a good thought, but no, autovacuum_vacuum_scale_factor is set to 0.2, meaning that over 1 million dead tuples are necessary for autovacuum. Besides, if autovacuum had ru

Re: [HACKERS] Surprising dead_tuple_count from pgstattuple

2010-08-06 Thread Robert Haas
On Fri, Aug 6, 2010 at 9:11 PM, Itagaki Takahiro wrote: > 2010/8/7 Gordon Shannon : >> 1. I delete 10,000 rows. >> pgstattuple.dead_tuple_count -> 1 >> >> 2. I delete 15,000 more rows. >> pgstattuple.dead_tuple_count -> 15000 ?? >> >> pgstattuple now appears to count the earlier 10K deleted tu

Re: [HACKERS] Surprising dead_tuple_count from pgstattuple

2010-08-06 Thread Itagaki Takahiro
2010/8/7 Gordon Shannon : > 1. I delete 10,000 rows. > pgstattuple.dead_tuple_count -> 1 > > 2. I delete 15,000 more rows. > pgstattuple.dead_tuple_count -> 15000 ?? > > pgstattuple now appears to count the earlier 10K deleted tuples as no longer > dead, but free space. I think it's expected b

[HACKERS] Surprising dead_tuple_count from pgstattuple

2010-08-06 Thread Gordon Shannon
This is an expansion of the question I posed in this thread: http://postgresql.1045698.n5.nabble.com/Need-help-understanding-vacuum-verbose-output-tp2265895p2266912.html I am framing the question here in relation to pgstattuple. Running 8.4.4 on Centos. I have a table T with 5,063,463 rows.