Re: [HACKERS] Query progress indication - an implementation

2009-07-02 Thread Robert Haas
On Thu, Jul 2, 2009 at 12:48 PM, Euler Taveira de Oliveira wrote: > I know that it didn't solve the estimation problem but ... IMHO the > [under|over]estimation should be treated by an external tool (autoexplain?). > So when we enable the query progress and some node reports a difference > between

Re: [HACKERS] Query progress indication - an implementation

2009-07-02 Thread Greg Stark
On Thu, Jul 2, 2009 at 2:32 AM, Bruce Momjian wrote: > I think the only resonable solution would be to consider the estimated > cost of each node and then compute what percentage complete each node > is. > Well you can do better for some nodes. A sequential scan for example can tell you exactly wh

Re: [HACKERS] Query progress indication - an implementation

2009-07-01 Thread Bruce Momjian
Tom Lane wrote: > Joshua Tolley writes: > > On Mon, Jun 29, 2009 at 02:07:23PM -0400, Tom Lane wrote: > >> I think this is pretty much nonsense --- most queries run all their plan > >> nodes concurrently to some extent. You can't usefully say that a query > >> is "on" some node, nor measure progr

Re: [HACKERS] Query progress indication - an implementation

2009-06-30 Thread Simon Riggs
On Mon, 2009-06-29 at 18:49 -0400, Tom Lane wrote: > Simon Riggs writes: > > On Mon, 2009-06-29 at 14:07 -0400, Tom Lane wrote: > >> I think this is pretty much nonsense --- most queries run all their plan > >> nodes concurrently to some extent. You can't usefully say that a query > >> is "on" s

Re: [HACKERS] Query progress indication - an implementation

2009-06-30 Thread Simon Riggs
On Tue, 2009-06-30 at 07:04 +0200, Dimitri Fontaine wrote: > Le 30 juin 2009 à 01:34, Greg Stark a écrit : > > Basically I disagree that imperfect progress reports annoy users. I > > think we can do better than reporting 250% done or having a percentage > > that goes backward though. It would be

Re: [HACKERS] Query progress indication - an implementation

2009-06-29 Thread Scara Maccai
> +1.  Especially if I run it a few times and I can see > which counters > are still moving. Per-node percentage is easy to do (given the perfect estimates, of course). The problem comes when you want to give an "overall" percentage. I wouldn't know where to put that "explain-like" output though

Re: [HACKERS] Query progress indication - an implementation

2009-06-29 Thread Dimitri Fontaine
Le 30 juin 2009 à 01:34, Greg Stark a écrit : Basically I disagree that imperfect progress reports annoy users. I think we can do better than reporting 250% done or having a percentage that goes backward though. It would be quite tolerable (though perhaps for no logical reason) to have a progres

Re: [HACKERS] Query progress indication - an implementation

2009-06-29 Thread Robert Haas
On Mon, Jun 29, 2009 at 8:15 PM, Ron Mayer wrote: > Greg Stark wrote: >> >> Right, that was why my proposed interface was to dump out the explain >> plan with the number of loops, row counts seen so far, and approximate >> percentage progress. >> >> My thinking was that a human could interpret that

Re: [HACKERS] Query progress indication - an implementation

2009-06-29 Thread Ron Mayer
Greg Stark wrote: Right, that was why my proposed interface was to dump out the explain plan with the number of loops, row counts seen so far, and approximate percentage progress. My thinking was that a human could interpret that to understand where the bottleneck is if, say you're still on the

Re: [HACKERS] Query progress indication - an implementation

2009-06-29 Thread Greg Stark
>> On Mon, 2009-06-29 at 14:07 -0400, Tom Lane wrote: >>> I think this is pretty much nonsense --- most queries run all their plan >>> nodes concurrently to some extent.  You can't usefully say that a query >>> is "on" some node, nor measure progress by whether some node is "done". Right, that was

Re: [HACKERS] Query progress indication - an implementation

2009-06-29 Thread Tom Lane
Simon Riggs writes: > On Mon, 2009-06-29 at 14:07 -0400, Tom Lane wrote: >> I think this is pretty much nonsense --- most queries run all their plan >> nodes concurrently to some extent. You can't usefully say that a query >> is "on" some node, nor measure progress by whether some node is "done".

Re: [HACKERS] Query progress indication - an implementation

2009-06-29 Thread Simon Riggs
On Mon, 2009-06-29 at 14:07 -0400, Tom Lane wrote: > Josh Berkus writes: > > So, while an actual % completed indicator would be perfect, a "query > > steps completed, current step =" would still be very useful and a large > > improvement over what we have now. > > I think this is pretty much n

Re: [HACKERS] Query progress indication - an implementation

2009-06-29 Thread Peter Hunsberger
On Mon, Jun 29, 2009 at 1:07 PM, Tom Lane wrote: > Josh Berkus writes: >> So, while an actual % completed indicator would be perfect, a "query >> steps completed, current step =" would still be very useful and a large >> improvement over what we have now. > > I think this is pretty much nonsense -

Re: [HACKERS] Query progress indication - an implementation

2009-06-29 Thread Tom Lane
Joshua Tolley writes: > On Mon, Jun 29, 2009 at 02:07:23PM -0400, Tom Lane wrote: >> I think this is pretty much nonsense --- most queries run all their plan >> nodes concurrently to some extent. You can't usefully say that a query >> is "on" some node, nor measure progress by whether some node i

Re: [HACKERS] Query progress indication - an implementation

2009-06-29 Thread Joshua Tolley
On Mon, Jun 29, 2009 at 02:07:23PM -0400, Tom Lane wrote: > Josh Berkus writes: > > So, while an actual % completed indicator would be perfect, a "query > > steps completed, current step =" would still be very useful and a large > > improvement over what we have now. > > I think this is pretty

Re: [HACKERS] Query progress indication - an implementation

2009-06-29 Thread Robert Haas
On Mon, Jun 29, 2009 at 11:15 AM, wrote: > So the code that I have right now works "pretty well" for the 10 queries of > my project, but I guess won't work for general queries :( I think that's probably right. >> So, I'm all in favor of what you're trying to conceptually; >> I just >> don't lik

Re: [HACKERS] Query progress indication - an implementation

2009-06-29 Thread Tom Lane
Josh Berkus writes: > So, while an actual % completed indicator would be perfect, a "query > steps completed, current step =" would still be very useful and a large > improvement over what we have now. I think this is pretty much nonsense --- most queries run all their plan nodes concurrently t

Re: [HACKERS] Query progress indication - an implementation

2009-06-29 Thread Josh Berkus
All, Actually, an indicator of even just what step of the query was being executed would be very useful for checking on stuck queries. If a DBA checks once that the query is on "bitmapscan on table_x(index_y)", and it's still on that 15 minutes later, he/she can guess that the query is thras

Re: [HACKERS] Query progress indication - an implementation

2009-06-29 Thread Scara Maccai
> IMO > any > diagnostics you can provide for a low cost are > useful.  The more > detail, the better.  "Step 1 of 10" is good, "80% > complete on step 1 > of 10" is better.  "80% complete on step 1, 10% > complete on 10 steps" > is even better. Well, I guess "Step 1 of 10" would be pretty trivi

Re: [HACKERS] Query progress indication - an implementation

2009-06-29 Thread m_lists
> It's > easy to have > estimates that are off by a factor of two or three, though, > so I think > you'd frequently have situations when the query completed > when the > progress estimater was at 40% or 250%. I thought about implementing a "given perfect estimates" indicator at first then, as a

Re: [HACKERS] Query progress indication - an implementation

2009-06-29 Thread Robert Haas
On Mon, Jun 29, 2009 at 4:47 AM, Scara Maccai wrote: >> As to the content of the patch, I think that what you are doing is >> comparing the actual number of "operations" with the expected number >> of operations.  If that's correct, I'm not sure it's really all that >> useful, because it will only

Re: [HACKERS] Query progress indication - an implementation

2009-06-29 Thread Peter Hunsberger
On Mon, Jun 29, 2009 at 3:47 AM, Scara Maccai wrote: > > Is anyone interested in such a progress indicator??? > I'm relatively new to Postgres and just starting to look at starting to look at what we might do with it for handling large genomic datasets. I've used Toad for Oracle to have a look at

Re: [HACKERS] Query progress indication - an implementation

2009-06-29 Thread Scara Maccai
> You might want to take a look at this: > http://wiki.postgresql.org/wiki/Submitting_a_Patch I will; I'm sorry it wasn't in the proper format. It was just a proof of concept, I guess I should have talked about it before even sending the patch. > As to the content of the patch, I think that wh

Re: [HACKERS] Query progress indication - an implementation

2009-06-28 Thread Peter Eisentraut
On Sunday 28 June 2009 20:38:59 Robert Haas wrote: > The only other use case I can think of for functionality of this type > is some kind of dashboard view on a system with very long-running > queries, where you want to see how far you have yet to go on each one > (maybe to approximate when you can

Re: [HACKERS] Query progress indication - an implementation

2009-06-28 Thread Robert Haas
On Fri, Jun 26, 2009 at 3:37 AM, Scara Maccai wrote: > Hi all, > > following the link in > > http://wiki.postgresql.org/wiki/Query_progress_indication > > but mostly: > > http://www.postech.ac.kr/~swhwang/progress2.pdf [1] > > I'm trying to write an implementation of the "dne" method in postgresql.

[HACKERS] Query progress indication - an implementation

2009-06-26 Thread Scara Maccai
Hi all, following the link in http://wiki.postgresql.org/wiki/Query_progress_indication but mostly: http://www.postech.ac.kr/~swhwang/progress2.pdf [1] I'm trying to write an implementation of the "dne" method in postgresql. I added another column to the pg_stat_get_activity function to repo