Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2015-01-20 Thread Robert Haas
On Mon, Jan 19, 2015 at 10:53 PM, tim_wilson wrote: > Was slightly optimistic that this comment in the release notes might mean > that my bug with bloat on hot tables might have been fixed in 9.4 > > /Make VACUUM properly report dead but not-yet-removable rows to the > statistics collector (Hari B

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2015-01-19 Thread tim_wilson
Was slightly optimistic that this comment in the release notes might mean that my bug with bloat on hot tables might have been fixed in 9.4 /Make VACUUM properly report dead but not-yet-removable rows to the statistics collector (Hari Babu) Previously these were reported as live rows./ Unfortuna

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-09-10 Thread Bruce Momjian
On Thu, Jun 12, 2014 at 01:40:59PM +0200, Andres Freund wrote: > Hi Tom, > > On 2014-06-06 15:44:25 -0400, Tom Lane wrote: > > I figured it'd be easy enough to get a better estimate by adding another > > counter to count just LIVE and INSERT_IN_PROGRESS tuples (thus effectively > > assuming that i

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-06-25 Thread tim_wilson
Given that this seems to have slipped off the hackers radar (or in too hard basket) I have constructed a horrible solution. I will stop using autovacuum for this relation , I will use our own system to monitor the relation, and I will reset pgclass.reltuples on this relation after vacuum is done t

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-06-12 Thread Andres Freund
Hi Tom, On 2014-06-06 15:44:25 -0400, Tom Lane wrote: > I figured it'd be easy enough to get a better estimate by adding another > counter to count just LIVE and INSERT_IN_PROGRESS tuples (thus effectively > assuming that in-progress inserts and deletes will both commit). Did you plan to backpatc

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-06-11 Thread Kevin Grittner
Andres Freund wrote: > On 2014-06-09 11:24:22 -0700, Kevin Grittner wrote: >> The only way that it could be a problem is if the DELETE is in a >> subtransaction which might get rolled back without rolling back the >> INSERT. > > The way I understand the code in that case the subxid in xmax would

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-06-11 Thread Andres Freund
On 2014-06-09 11:24:22 -0700, Kevin Grittner wrote: > Andres Freund wrote: > > On 2014-06-09 09:45:12 -0700, Kevin Grittner wrote: > > > I am not sure, given predicate.c's coding, how > > HEAPTUPLE_DELETE_IN_PROGRESS could cause problems. Could you elaborate, > > since that's the contentious poin

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-06-09 Thread Kevin Grittner
Tom Lane wrote: > Assuming that Kevin's describing his needs correctly, we could resolve > this by inventing HEAPTUPLE_INSERT_AND_DELETE_IN_PROGRESS.  VACUUM could > assume that that's a probably-dead tuple, while SSI could do something > different. That could work. On the other hand, the old c

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-06-09 Thread Alvaro Herrera
Kevin Grittner wrote: > Andres Freund wrote: > > On 2014-06-09 09:45:12 -0700, Kevin Grittner wrote: > >>> old: > >>> 1) xmin has committed, xmax is in progress, xmax is not just a locker > >>> 2) xmin is in progress, xmax is set and not not just a locker > >>> > >>> Note that the 2) case here ne

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-06-09 Thread Kevin Grittner
Tom Lane wrote: > The reason this stuff is not too carefully spec'd is that when > HTSV was written, there was no expectation that there was any > correctness issue around which of these cases was returned.  I > wonder whether SSI should be using HTSV at all. That's certainly a reasonable questi

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-06-09 Thread Kevin Grittner
Andres Freund wrote: > On 2014-06-09 09:45:12 -0700, Kevin Grittner wrote: > I am not sure, given predicate.c's coding, how > HEAPTUPLE_DELETE_IN_PROGRESS could cause problems. Could you elaborate, > since that's the contentious point with Tom? Since 'both in > progress' > can only happen if xmin

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-06-09 Thread Tom Lane
Andres Freund writes: > On 2014-06-09 09:45:12 -0700, Kevin Grittner wrote: >> For purposes of predicate.c, if the "also deleted" activity might >> be rolled back without rolling back the insert, INSERT_IN_PROGRESS >> is the only correct value. > That's basically the argument for the new behaviou

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-06-09 Thread Andres Freund
On 2014-06-09 09:45:12 -0700, Kevin Grittner wrote: > Andres Freund wrote: > > HEAPTUPLE_INSERT_IN_PROGRESS,    /* inserting xact is still in progress > >*/ > > HEAPTUPLE_DELETE_IN_PROGRESS    /* deleting xact is still in progress */ > > the current code will return INSERT_IN_PROGRESS eve

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-06-09 Thread Kevin Grittner
Andres Freund wrote: > On 2014-06-09 08:00:52 -0700, Kevin Grittner wrote: > I tried to make things a bit clearer there - but I am not sure I've > succeed. I'm certainly willing to explain things further if you can tell > me which are is unclear. Thanks!  IMO, something like this should be inclu

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-06-09 Thread Andres Freund
On 2014-06-09 08:00:52 -0700, Kevin Grittner wrote: > I'm not actually sure yet whether the current state of affairs > causes a problem for the serializable transaction isolation level > implementation. I'd replied in the other thread before noticing you'd replied here... From what I understand ri

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-06-09 Thread Kevin Grittner
Andres Freund wrote: > Well, I think reverting surely wouldn't be the right cure. It > fixes a somewhat nasty bug. I'd certainly be prepared to add the > two lines necessary to make it return DELETE_IN_PROGRESS after > trying to understand Kevin's email about predicate.c and going > through the o

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-06-09 Thread Andres Freund
On 2014-06-09 10:30:43 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2014-06-09 10:14:32 -0400, Robert Haas wrote: > >> I think that would be a good idea for conceptual clarity if nothing > >> else. If callers are OK with it, then they can treat both of those > >> codes alike; but then at

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-06-09 Thread Tom Lane
Andres Freund writes: > On 2014-06-09 10:14:32 -0400, Robert Haas wrote: >> I think that would be a good idea for conceptual clarity if nothing >> else. If callers are OK with it, then they can treat both of those >> codes alike; but then at least there's clear evidence as to the >> author's inte

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-06-09 Thread Andres Freund
On 2014-06-09 10:14:32 -0400, Robert Haas wrote: > On Fri, Jun 6, 2014 at 3:44 PM, Tom Lane wrote: > > It did not use to blow this question off: back around 8.3 you got > > DELETE_IN_PROGRESS if the tuple had a delete pending. I think we need > > less laziness + fuzzy thinking here. Maybe we sho

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-06-09 Thread Robert Haas
On Fri, Jun 6, 2014 at 3:44 PM, Tom Lane wrote: > It did not use to blow this question off: back around 8.3 you got > DELETE_IN_PROGRESS if the tuple had a delete pending. I think we need > less laziness + fuzzy thinking here. Maybe we should have a separate > HEAPTUPLE_INSERT_AND_DELETE_IN_PROG

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-06-08 Thread Amit Kapila
On Sat, Jun 7, 2014 at 1:28 AM, Andres Freund wrote: > > On 2014-06-06 15:44:25 -0400, Tom Lane wrote: > > I figured it'd be easy enough to get a better estimate by adding another > > counter to count just LIVE and INSERT_IN_PROGRESS tuples (thus effectively > > assuming that in-progress inserts a

Re: [HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-06-06 Thread Andres Freund
On 2014-06-06 15:44:25 -0400, Tom Lane wrote: > I figured it'd be easy enough to get a better estimate by adding another > counter to count just LIVE and INSERT_IN_PROGRESS tuples (thus effectively > assuming that in-progress inserts and deletes will both commit). I did > that, and found that it h

[HACKERS] Inaccuracy in VACUUM's tuple count estimates

2014-06-06 Thread Tom Lane
I've been looking at the complaint Tim Wilson posted in pgsql-performance about badly inaccurate reltuples updates coming from VACUUM. There seem to be a number of problems leading to that. The key point is that when VACUUM has scanned only part of the relation, it assumes that the live-tuple den