Re: [HACKERS] CLUSTER command progress monitor

2019-09-20 Thread Alvaro Herrera
On 2019-Sep-18, Michael Paquier wrote: > On Tue, Sep 17, 2019 at 10:50:22PM -0300, Alvaro Herrera wrote: > > On 2019-Sep-18, Michael Paquier wrote: > >> So, with the clock ticking and the release getting close by, what do > >> we do for this set of issues? REINDEX, CREATE INDEX and CLUSTER all >

Re: [HACKERS] CLUSTER command progress monitor

2019-09-17 Thread Michael Paquier
On Tue, Sep 17, 2019 at 10:50:22PM -0300, Alvaro Herrera wrote: > On 2019-Sep-18, Michael Paquier wrote: >> So, with the clock ticking and the release getting close by, what do >> we do for this set of issues? REINDEX, CREATE INDEX and CLUSTER all >> try to build indexes and the current infrastruc

Re: [HACKERS] CLUSTER command progress monitor

2019-09-17 Thread Alvaro Herrera
On 2019-Sep-18, Michael Paquier wrote: > So, with the clock ticking and the release getting close by, what do > we do for this set of issues? REINDEX, CREATE INDEX and CLUSTER all > try to build indexes and the current infrastructure is not really > adapted to hold all that. Robert, Alvaro and P

Re: [HACKERS] CLUSTER command progress monitor

2019-09-17 Thread Michael Paquier
On Sat, Sep 14, 2019 at 11:45:47AM +0900, Michael Paquier wrote: > I have provided a short summary of the two issues on the open item > page (https://wiki.postgresql.org/wiki/PostgreSQL_12_Open_Items) as > the open item was too much evasive. Here is a copy-paste for the > archives of what I wrote:

Re: [HACKERS] CLUSTER command progress monitor

2019-09-16 Thread Michael Paquier
On Mon, Sep 16, 2019 at 03:26:10PM +0900, Tattsu Yama wrote: > I should have explained the API changes more. I added cmdtype as a given > parameter for all functions (See below). > Therefore, I suppose that my patch is similar to the following fix as you > mentioned on -hackers. Yes, that's an opt

Re: [HACKERS] CLUSTER command progress monitor

2019-09-16 Thread Tatsuro Yamada
Hi Alvaro! Is this fix strictly necessary for pg12, or is this something that we can leave for pg13? Not only me but many DBA needs this progress report feature on PG12, therefore I'm trying to fix the problem. If you send other patch to fix the problem, and it is more elegant than mine, I can

Re: [HACKERS] CLUSTER command progress monitor

2019-09-16 Thread Tatsuro Yamada
Hi Alvaro! Is this fix strictly necessary for pg12, or is this something that we can leave for pg13? Not only me but many DBA needs this progress report feature on PG12, therefore I'm trying to fix the problem. If you send other patch to fix the problem, and it is more elegant than mine, I can

Re: [HACKERS] CLUSTER command progress monitor

2019-09-16 Thread Alvaro Herrera
On 2019-Sep-17, Tatsuro Yamada wrote: > On 2019/09/16 23:12, Alvaro Herrera wrote: > > Is this fix strictly necessary for pg12, or is this something that we > > can leave for pg13? > > Not only me but many DBA needs this progress report feature on PG12, > therefore I'm trying to fix the problem.

Re: [HACKERS] CLUSTER command progress monitor

2019-09-16 Thread Tatsuro Yamada
Hi Alvaro, On 2019/09/16 23:12, Alvaro Herrera wrote: On 2019-Sep-16, Tattsu Yama wrote: I should have explained the API changes more. I added cmdtype as a given parameter for all functions (See below). Therefore, I suppose that my patch is similar to the following fix as you mentioned on -ha

Re: [HACKERS] CLUSTER command progress monitor

2019-09-16 Thread Alvaro Herrera
On 2019-Sep-16, Tattsu Yama wrote: > I should have explained the API changes more. I added cmdtype as a given > parameter for all functions (See below). > Therefore, I suppose that my patch is similar to the following fix as you > mentioned on -hackers. Is this fix strictly necessary for pg12, or

Re: [HACKERS] CLUSTER command progress monitor

2019-09-15 Thread Tattsu Yama
Hi Michael, >> > Attached file is WIP patch.In my patch, I added "command id" to all >> APIs of >> > progress reporting to isolate commands. Therefore, it doesn't allow to >> > cascade updating system views. And my patch is on WIP so it needs >> clean-up >> > and test. >> > I share it anyway. :)

Re: [HACKERS] CLUSTER command progress monitor

2019-09-14 Thread Tattsu Yama
Hi Michael! > Attached file is WIP patch.In my patch, I added "command id" to all APIs > of > > progress reporting to isolate commands. Therefore, it doesn't allow to > > cascade updating system views. And my patch is on WIP so it needs > clean-up > > and test. > > I share it anyway. :) > > +

Re: [HACKERS] CLUSTER command progress monitor

2019-09-13 Thread Michael Paquier
On Sat, Sep 14, 2019 at 01:06:32PM +0900, Tattsu Yama wrote: > Thanks! I can review your patch for fix it. > However, I was starting fixing the problem from the last day of PGConf.Asia > (11 Sep). > Attached file is WIP patch.In my patch, I added "command id" to all APIs of > progress reporting to

Re: [HACKERS] CLUSTER command progress monitor

2019-09-13 Thread Tattsu Yama
Hi Alvaro! > Hello Tatsuro, > On 2019-Aug-13, Tatsuro Yamada wrote: > > On 2019/08/02 3:43, Alvaro Herrera wrote: > > > Hmm, I'm trying this out now and I don't see the index_rebuild_count > > > ever go up. I think it's because the indexes are built using parallel > > > index build ... or maybe

Re: [HACKERS] CLUSTER command progress monitor

2019-09-13 Thread Michael Paquier
On Fri, Sep 13, 2019 at 12:48:40PM -0400, Robert Haas wrote: > On Fri, Sep 13, 2019 at 12:03 PM Alvaro Herrera > wrote: >> Ummm ... I've been operating --in this thread-- under the assumption >> that it is REINDEX to blame for this problem, not CREATE INDEX, because >> my recollection is that I te

Re: [HACKERS] CLUSTER command progress monitor

2019-09-13 Thread Alvaro Herrera
On 2019-Sep-13, Robert Haas wrote: > On Fri, Sep 13, 2019 at 2:49 AM Michael Paquier wrote: > > On Fri, Sep 06, 2019 at 08:10:58AM -0400, Robert Haas wrote: > > > It's fine if things are updated as well -- it's just you need to make > > > sure that those places know whether or not they are suppos

Re: [HACKERS] CLUSTER command progress monitor

2019-09-13 Thread Alvaro Herrera
Hello Tatsuro, On 2019-Aug-13, Tatsuro Yamada wrote: > On 2019/08/02 3:43, Alvaro Herrera wrote: > > Hmm, I'm trying this out now and I don't see the index_rebuild_count > > ever go up. I think it's because the indexes are built using parallel > > index build ... or maybe it was the table AM cha

Re: [HACKERS] CLUSTER command progress monitor

2019-09-13 Thread Robert Haas
On Fri, Sep 13, 2019 at 12:03 PM Alvaro Herrera wrote: > Ummm ... I've been operating --in this thread-- under the assumption > that it is REINDEX to blame for this problem, not CREATE INDEX, because > my recollection is that I tested CREATE INDEX together with CLUSTER and > it worked fine. Has a

Re: [HACKERS] CLUSTER command progress monitor

2019-09-13 Thread Robert Haas
On Fri, Sep 13, 2019 at 2:49 AM Michael Paquier wrote: > On Fri, Sep 06, 2019 at 08:10:58AM -0400, Robert Haas wrote: > > It's fine if things are updated as well -- it's just you need to make > > sure that those places know whether or not they are supposed to be > > doing those updates. Again, why

Re: [HACKERS] CLUSTER command progress monitor

2019-09-12 Thread Michael Paquier
On Fri, Sep 06, 2019 at 08:10:58AM -0400, Robert Haas wrote: > It's fine if things are updated as well -- it's just you need to make > sure that those places know whether or not they are supposed to be > doing those updates. Again, why can't we just pass down a value > telling them "do reindex-styl

Re: [HACKERS] CLUSTER command progress monitor

2019-09-09 Thread Peter Geoghegan
On Fri, Sep 6, 2019 at 5:11 AM Robert Haas wrote: > On Fri, Sep 6, 2019 at 1:44 AM Michael Paquier wrote: > > I don't see exactly why we could not switch to a fixed number of > > slots, say 8, with one code path to start a progress which adds an > > extra report on the stack, one to remove one en

Re: [HACKERS] CLUSTER command progress monitor

2019-09-09 Thread Michael Paquier
On Fri, Sep 06, 2019 at 10:27:02AM -0400, Alvaro Herrera from 2ndQuadrant wrote: > That said, I did spend some time on this type of issue when doing CREATE > INDEX support; you can tell because I defined the columns for block > numbers in a scan separately from CREATE INDEX specific fields, > preci

Re: [HACKERS] CLUSTER command progress monitor

2019-09-06 Thread Alvaro Herrera from 2ndQuadrant
On 2019-Sep-06, Michael Paquier wrote: > Finally comes the question of what do we do for v12? I am adding in > CC Peter, Alvaro being already present, who have been involved in the > commits with CREATE INDEX and REINDEX. It would be sad to revert a > this feature, but well I'd rather do that no

Re: [HACKERS] CLUSTER command progress monitor

2019-09-06 Thread Robert Haas
On Fri, Sep 6, 2019 at 1:44 AM Michael Paquier wrote: > One idea I got was to pass the command ID as an extra argument of the > update routine. I am not completely sure either if we need this level > of complication. I still don't think that's the right approach. > > Those are just weaknesses o

Re: [HACKERS] CLUSTER command progress monitor

2019-09-05 Thread Michael Paquier
On Fri, Sep 06, 2019 at 02:44:18PM +0900, Michael Paquier wrote: > I don't see exactly why we could not switch to a fixed number of > slots, say 8, with one code path to start a progress which adds an > extra report on the stack, one to remove one entry from the stack, and > a new one to reset the

Re: [HACKERS] CLUSTER command progress monitor

2019-09-05 Thread Michael Paquier
On Thu, Sep 05, 2019 at 03:17:51PM -0400, Robert Haas wrote: > Oops. Yeah, that's bogus (as are some of the other things you > mention). I think we're going to have to fix this by passing down > some flags to these functions to tell them what kind of progress > updates to do (or to do none). Or

Re: [HACKERS] CLUSTER command progress monitor

2019-09-05 Thread Robert Haas
On Wed, Sep 4, 2019 at 9:03 PM Michael Paquier wrote: > For CLUSTER, the progress starts and ends in cluster_rel(). CLUSTER > uses its code paths at the beginning, but then things get more > complicated, particularly with finish_heap_swap() which calls directly > reindex_table(). 6f97457 include

Re: [HACKERS] CLUSTER command progress monitor

2019-09-04 Thread Michael Paquier
On Wed, Sep 04, 2019 at 09:18:39AM -0400, Robert Haas wrote: > I think this is all going in the wrong direction. It's the > responsibility of the people who are calling the pgstat_progress_* > functions to only do so when it's appropriate. Having the > pgstat_progress_* functions try to untangle

Re: [HACKERS] CLUSTER command progress monitor

2019-09-04 Thread Robert Haas
On Tue, Sep 3, 2019 at 1:02 AM Masahiko Sawada wrote: > After more thought, even if we don't start a new command progress when > there is another one already started the progress update functions > could be called and these functions don't specify the command type. > Therefore, the progress inform

Re: [HACKERS] CLUSTER command progress monitor

2019-09-04 Thread Masahiko Sawada
On Wed, Sep 4, 2019 at 3:48 PM Michael Paquier wrote: > > On Tue, Sep 03, 2019 at 02:52:28PM +0900, Michael Paquier wrote: > > Indeed, good catch. This is wrong since b6fb647 which has introduced > > the progress reports. I'll fix that one and back-patch if there are > > no objections. > > OK, a

Re: [HACKERS] CLUSTER command progress monitor

2019-09-03 Thread Michael Paquier
On Tue, Sep 03, 2019 at 02:52:28PM +0900, Michael Paquier wrote: > Indeed, good catch. This is wrong since b6fb647 which has introduced > the progress reports. I'll fix that one and back-patch if there are > no objections. OK, applied this part down to 9.6. -- Michael signature.asc Description

Re: [HACKERS] CLUSTER command progress monitor

2019-09-02 Thread Michael Paquier
On Tue, Sep 03, 2019 at 01:59:00PM +0900, Masahiko Sawada wrote: > After more thought, even if we don't start a new command progress when > there is another one already started the progress update functions > could be called and these functions don't specify the command type. > Therefore, the progr

Re: [HACKERS] CLUSTER command progress monitor

2019-09-02 Thread Masahiko Sawada
On Mon, Sep 2, 2019 at 6:32 PM Masahiko Sawada wrote: > > On Mon, Sep 2, 2019 at 4:59 PM Michael Paquier wrote: > > > > On Fri, Aug 30, 2019 at 07:45:57PM +0900, Masahiko Sawada wrote: > > > > I tried 1. and it shown index_rebuild_count, but it also shown > > > > "initializing" phase again and ot

Re: [HACKERS] CLUSTER command progress monitor

2019-09-02 Thread Masahiko Sawada
On Mon, Sep 2, 2019 at 4:59 PM Michael Paquier wrote: > > On Fri, Aug 30, 2019 at 07:45:57PM +0900, Masahiko Sawada wrote: > > > I tried 1. and it shown index_rebuild_count, but it also shown > > > "initializing" phase again and other columns were empty. So, it is > > > not suitable to fix the pro

Re: [HACKERS] CLUSTER command progress monitor

2019-09-02 Thread Michael Paquier
On Fri, Aug 30, 2019 at 07:45:57PM +0900, Masahiko Sawada wrote: > > I tried 1. and it shown index_rebuild_count, but it also shown > > "initializing" phase again and other columns were empty. So, it is > > not suitable to fix the problem. :( > > I'm going to try 2. and 3., but, unfortunately, I ca

Re: [HACKERS] CLUSTER command progress monitor

2019-08-30 Thread Masahiko Sawada
On Thu, Aug 15, 2019 at 12:48 PM Tatsuro Yamada wrote: > > Hi Michael, Alvaro and Robert! > > On 2019/08/14 11:52, Michael Paquier wrote: > > On Wed, Aug 14, 2019 at 11:38:01AM +0900, Tatsuro Yamada wrote: > >> On 2019/08/13 14:40, Tatsuro Yamada wrote: > >>> On 2019/08/02 3:43, Alvaro Herrera wro

Re: [HACKERS] CLUSTER command progress monitor

2019-08-14 Thread Tatsuro Yamada
Hi Michael, Alvaro and Robert! On 2019/08/14 11:52, Michael Paquier wrote: On Wed, Aug 14, 2019 at 11:38:01AM +0900, Tatsuro Yamada wrote: On 2019/08/13 14:40, Tatsuro Yamada wrote: On 2019/08/02 3:43, Alvaro Herrera wrote: Hmm, I'm trying this out now and I don't see the index_rebuild_count

Re: [HACKERS] CLUSTER command progress monitor

2019-08-13 Thread Michael Paquier
On Wed, Aug 14, 2019 at 11:38:01AM +0900, Tatsuro Yamada wrote: > On 2019/08/13 14:40, Tatsuro Yamada wrote: >> On 2019/08/02 3:43, Alvaro Herrera wrote: >>> Hmm, I'm trying this out now and I don't see the index_rebuild_count >>> ever go up.  I think it's because the indexes are built using parall

Re: [HACKERS] CLUSTER command progress monitor

2019-08-13 Thread Tatsuro Yamada
Hi Alvaro and All, On 2019/08/13 14:40, Tatsuro Yamada wrote: Hi Alvaro! On 2019/08/02 3:43, Alvaro Herrera wrote: Hmm, I'm trying this out now and I don't see the index_rebuild_count ever go up.  I think it's because the indexes are built using parallel index build ... or maybe it was the tab

Re: [HACKERS] CLUSTER command progress monitor

2019-08-12 Thread Tatsuro Yamada
Hi Alvaro! On 2019/08/02 3:43, Alvaro Herrera wrote: Hmm, I'm trying this out now and I don't see the index_rebuild_count ever go up. I think it's because the indexes are built using parallel index build ... or maybe it was the table AM changes that moved things around, not sure. There's a per

Re: [HACKERS] CLUSTER command progress monitor

2019-08-01 Thread Alvaro Herrera
Hmm, I'm trying this out now and I don't see the index_rebuild_count ever go up. I think it's because the indexes are built using parallel index build ... or maybe it was the table AM changes that moved things around, not sure. There's a period at the end when the CLUSTER command keeps working, b

Re: [HACKERS] CLUSTER command progress monitor

2019-03-25 Thread Michael Paquier
On Tue, Mar 26, 2019 at 10:04:48AM +0900, Tatsuro Yamada wrote: > Hope this feature will help DBA and user. :) Congrats, Yamada-san. -- Michael signature.asc Description: PGP signature

Re: [HACKERS] CLUSTER command progress monitor

2019-03-25 Thread Tatsuro Yamada
Hi Robert and Reviewers! On 2019/03/26 0:02, Robert Haas wrote: On Sun, Mar 24, 2019 at 9:02 PM Tatsuro Yamada wrote: Please find attached files. :) Committed. Thanks! Thank you! Hope this feature will help DBA and user. :) Regards, Tatsuro Yamada NTT Open Source Software Center

Re: [HACKERS] CLUSTER command progress monitor

2019-03-25 Thread Robert Haas
On Sun, Mar 24, 2019 at 9:02 PM Tatsuro Yamada wrote: > Please find attached files. :) Committed. Thanks! -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: [HACKERS] CLUSTER command progress monitor

2019-03-24 Thread Tatsuro Yamada
Hi Robert! On 2019/03/23 3:31, Robert Haas wrote: On Tue, Mar 19, 2019 at 2:47 PM Robert Haas wrote: how close you were getting to rewriting the entire heap. This is the one thing I found but did not fix; any chance you could make this change and update the documentation to match? Hi, is an

Re: [HACKERS] CLUSTER command progress monitor

2019-03-24 Thread Tattsu Yama
Hi Robert! >On Tue, Mar 19, 2019 at 2:47 PM Robert Haas wrote: >> how close you were getting to rewriting the entire heap. This is the >> one thing I found but did not fix; any chance you could make this >> change and update the documentation to match? > > >Hi, is anybody working on this? Thank yo

Re: [HACKERS] CLUSTER command progress monitor

2019-03-22 Thread Robert Haas
On Tue, Mar 19, 2019 at 2:47 PM Robert Haas wrote: > how close you were getting to rewriting the entire heap. This is the > one thing I found but did not fix; any chance you could make this > change and update the documentation to match? Hi, is anybody working on this? -- Robert Haas Enterpris

Re: [HACKERS] CLUSTER command progress monitor

2019-03-19 Thread Robert Haas
On Mon, Mar 18, 2019 at 10:03 PM Tatsuro Yamada wrote: > Attached patch is a rebased document patch. :) Attached is an updated patch. I went through this patch carefully today, in the hopes of committing it, and I think the attached version is pretty closet to being committable, but there's at l

Re: [HACKERS] CLUSTER command progress monitor

2019-03-19 Thread Kyotaro HORIGUCHI
At Tue, 19 Mar 2019 11:02:57 +0900, Tatsuro Yamada wrote in > On 2019/03/19 10:43, Tatsuro Yamada wrote: > > Hi Rafia! > > On 2019/03/18 20:42, Rafia Sabih wrote: > >> On Fri, 8 Mar 2019 at 09:14, Tatsuro Yamada > >> wrote: > >>> Attached file is rebased patch on current HEAD. > >>> I changed

Re: [HACKERS] CLUSTER command progress monitor

2019-03-18 Thread Tatsuro Yamada
On 2019/03/19 10:43, Tatsuro Yamada wrote: Hi Rafia! On 2019/03/18 20:42, Rafia Sabih wrote: On Fri, 8 Mar 2019 at 09:14, Tatsuro Yamada wrote: Attached file is rebased patch on current HEAD. I changed a status. :) Looks like the patch needs a rebase. I was on the commit fb5806533f9fe04332

Re: [HACKERS] CLUSTER command progress monitor

2019-03-18 Thread Tatsuro Yamada
Hi Rafia! On 2019/03/18 20:42, Rafia Sabih wrote: On Fri, 8 Mar 2019 at 09:14, Tatsuro Yamada wrote: Attached file is rebased patch on current HEAD. I changed a status. :) Looks like the patch needs a rebase. I was on the commit fb5806533f9fe0433290d84c9b019399cd69e9c2 PFA reject file in c

Re: [HACKERS] CLUSTER command progress monitor

2019-03-18 Thread Rafia Sabih
On Fri, 8 Mar 2019 at 09:14, Tatsuro Yamada wrote: > > On 2019/03/06 15:38, Tatsuro Yamada wrote: > > On 2019/03/05 17:56, Tatsuro Yamada wrote: > >> On 2019/03/05 11:35, Robert Haas wrote: > >>> On Mon, Mar 4, 2019 at 5:38 AM Tatsuro Yamada > >>> wrote: > === Current design === > > >>>

Re: [HACKERS] CLUSTER command progress monitor

2019-03-08 Thread Tatsuro Yamada
On 2019/03/06 15:38, Tatsuro Yamada wrote: On 2019/03/05 17:56, Tatsuro Yamada wrote: On 2019/03/05 11:35, Robert Haas wrote: On Mon, Mar 4, 2019 at 5:38 AM Tatsuro Yamada wrote: === Current design === CLUSTER command uses Index Scan or Seq Scan when scanning the heap. Depending on which one

Re: [HACKERS] CLUSTER command progress monitor

2019-03-06 Thread Alvaro Herrera
On 2019-Mar-06, Robert Haas wrote: > On Tue, Mar 5, 2019 at 8:03 PM Alvaro Herrera > wrote: > > One, err, small issue with that idea is that we need the param numbers > > not to conflict for any "progress update providers" that are to be used > > simultaneously by any command. > > Is that reall

Re: [HACKERS] CLUSTER command progress monitor

2019-03-06 Thread Robert Haas
On Tue, Mar 5, 2019 at 8:03 PM Alvaro Herrera wrote: > One, err, small issue with that idea is that we need the param numbers > not to conflict for any "progress update providers" that are to be used > simultaneously by any command. Is that really an issue? I think progress reporting -- at least

Re: [HACKERS] CLUSTER command progress monitor

2019-03-05 Thread Tatsuro Yamada
On 2019/03/05 17:56, Tatsuro Yamada wrote: Hi Robert! On 2019/03/05 11:35, Robert Haas wrote: On Mon, Mar 4, 2019 at 5:38 AM Tatsuro Yamada wrote: === Current design === CLUSTER command uses Index Scan or Seq Scan when scanning the heap. Depending on which one is chosen, the command will pro

Re: [HACKERS] CLUSTER command progress monitor

2019-03-05 Thread Tatsuro Yamada
On 2019/03/06 1:13, Robert Haas wrote: On Tue, Mar 5, 2019 at 3:56 AM Tatsuro Yamada wrote: === Discussion points === - Progress counter for "3. sorting tuples" phase - Should we add pgstat_progress_update_param() in tuplesort.c like a "trace_sort"? Thanks to Peter

Re: [HACKERS] CLUSTER command progress monitor

2019-03-05 Thread Alvaro Herrera
On 2019-Mar-04, Robert Haas wrote: > On Mon, Mar 4, 2019 at 5:38 AM Tatsuro Yamada > wrote: > > === Discussion points === > > > > - Progress counter for "3. sorting tuples" phase > > - Should we add pgstat_progress_update_param() in tuplesort.c like a > >"trace_sort"? > >T

Re: [HACKERS] CLUSTER command progress monitor

2019-03-05 Thread Robert Haas
On Tue, Mar 5, 2019 at 3:56 AM Tatsuro Yamada wrote: > >> === Discussion points === > >> > >>- Progress counter for "3. sorting tuples" phase > >> - Should we add pgstat_progress_update_param() in tuplesort.c like a > >> "trace_sort"? > >> Thanks to Peter Geoghegan for th

Re: [HACKERS] CLUSTER command progress monitor

2019-03-05 Thread Tatsuro Yamada
Hi Robert! On 2019/03/05 11:35, Robert Haas wrote: On Mon, Mar 4, 2019 at 5:38 AM Tatsuro Yamada wrote: === Current design === CLUSTER command uses Index Scan or Seq Scan when scanning the heap. Depending on which one is chosen, the command will proceed in the following sequence of phases:

Re: [HACKERS] CLUSTER command progress monitor

2019-03-05 Thread Tatsuro Yamada
Hi David, On 2019/03/05 17:29, David Steele wrote: On 3/1/19 7:48 AM, Etsuro Fujita wrote: (2019/03/01 14:17), Tatsuro Yamada wrote: Attached patch is wip patch. Is it possible to remove the following patch? Because I registered the patch twice on CF Mar. https://commitfest.postgresql.org/2

Re: Re: [HACKERS] CLUSTER command progress monitor

2019-03-05 Thread David Steele
On 3/1/19 7:48 AM, Etsuro Fujita wrote: (2019/03/01 14:17), Tatsuro Yamada wrote: Attached patch is wip patch. Is it possible to remove the following patch? Because I registered the patch twice on CF Mar. https://commitfest.postgresql.org/22/2049/ Please remove the above and keep this: htt

Re: [HACKERS] CLUSTER command progress monitor

2019-03-04 Thread Robert Haas
On Mon, Mar 4, 2019 at 5:38 AM Tatsuro Yamada wrote: > === Current design === > > CLUSTER command uses Index Scan or Seq Scan when scanning the heap. > Depending on which one is chosen, the command will proceed in the > following sequence of phases: > >* Scan method: Seq Scan > 0. initial

Re: [HACKERS] CLUSTER command progress monitor

2019-03-04 Thread Tatsuro Yamada
On 2019/03/02 4:15, Robert Haas wrote: On Thu, Feb 28, 2019 at 11:54 PM Tatsuro Yamada wrote: Attached patch is wip patch. I rewrote the current design of the progress monitor and also wrote discussion points in the middle of this email. I'd like to get any feedback from -hackers. === Curre

Re: [HACKERS] CLUSTER command progress monitor

2019-03-04 Thread Tatsuro Yamada
On 2019/03/02 4:15, Robert Haas wrote: On Thu, Feb 28, 2019 at 11:54 PM Tatsuro Yamada wrote: Attached patch is wip patch. Thanks for your comments! :) I revised the code and the document. + CLUSTER and VACUUM FULL, showing current progress. and -> or Fixed. + certain comman

Re: [HACKERS] CLUSTER command progress monitor

2019-03-01 Thread Robert Haas
On Thu, Feb 28, 2019 at 11:54 PM Tatsuro Yamada wrote: > Attached patch is wip patch. + CLUSTER and VACUUM FULL, showing current progress. and -> or + certain commands during command execution. Currently, the suppoted + progress reporting commands are VACUUM and CLUSTER. suppoted ->

Re: [HACKERS] CLUSTER command progress monitor

2019-02-28 Thread Etsuro Fujita
(2019/03/01 14:17), Tatsuro Yamada wrote: Attached patch is wip patch. Is it possible to remove the following patch? Because I registered the patch twice on CF Mar. https://commitfest.postgresql.org/22/2049/ Please remove the above and keep this: https://commitfest.postgresql.org/22/1779/

Re: [HACKERS] CLUSTER command progress monitor

2019-02-28 Thread Tatsuro Yamada
Attached patch is wip patch. Is it possible to remove the following patch? Because I registered the patch twice on CF Mar. https://commitfest.postgresql.org/22/2049/ Thanks, Tatsuro Yamada

Re: [HACKERS] CLUSTER command progress monitor

2019-02-28 Thread Tatsuro Yamada
On 2019/02/23 6:02, Robert Haas wrote: On Fri, Dec 28, 2018 at 3:20 AM Tatsuro Yamada wrote: This patch is rebased on HEAD. I'll tackle revising the patch based on feedbacks next month. + Running VACUUM FULL is listed in pg_stat_progress_cluster + view because it uses CLUSTER command inte

Re: [HACKERS] CLUSTER command progress monitor

2019-02-22 Thread Robert Haas
On Fri, Dec 28, 2018 at 3:20 AM Tatsuro Yamada wrote: > This patch is rebased on HEAD. > I'll tackle revising the patch based on feedbacks next month. + Running VACUUM FULL is listed in pg_stat_progress_cluster + view because it uses CLUSTER command internally. See . It's not really true to

Re: [HACKERS] CLUSTER command progress monitor

2018-12-28 Thread Tatsuro Yamada
Hi, Do you have a new version of this patch? If not, do you think you'll have something in time for the upcoming commitfest? Not yet, I'll be able to send only a rebased patch by the end of this month. I mean it has no design change because I can't catch up on how to get a progress from sort

Re: [HACKERS] CLUSTER command progress monitor

2018-12-18 Thread Tatsuro Yamada
Hi Alvaro, On 2018/12/19 2:23, Alvaro Herrera wrote: Hello Yamada-san, On 2018-Dec-03, Tatsuro Yamada wrote: Thank you for managing the CF and Sorry for the late reply. I'll rebase it for the next CF and also I'll clear my head because the patch needs design change to address the feedbacks, I

Re: [HACKERS] CLUSTER command progress monitor

2018-12-18 Thread Peter Geoghegan
On Tue, Dec 18, 2018 at 2:47 PM Alvaro Herrera wrote: > Well, if you think about individual blocks in terms of storage space, > maybe that's true, but I meant in an Heraclitus way of men never > stepping into the same river -- the second time you write the block, > it's not the same block you wrot

Re: [HACKERS] CLUSTER command progress monitor

2018-12-18 Thread Alvaro Herrera
On 2018-Dec-18, Peter Geoghegan wrote: > On Tue, Dec 18, 2018 at 1:02 PM Alvaro Herrera > wrote: > > If we see this in terms of tapes and merges, we can report the total > > number of each of those that we have completed. As far as I understand, > > we write one tape to completion, and only the

Re: [HACKERS] CLUSTER command progress monitor

2018-12-18 Thread Peter Geoghegan
On Tue, Dec 18, 2018 at 1:02 PM Alvaro Herrera wrote: > If we see this in terms of tapes and merges, we can report the total > number of each of those that we have completed. As far as I understand, > we write one tape to completion, and only then start another one, right? > Since there's no way

Re: [HACKERS] CLUSTER command progress monitor

2018-12-18 Thread Alvaro Herrera
On 2017-Nov-21, Peter Geoghegan wrote: > On Mon, Oct 2, 2017 at 6:04 AM, Robert Haas wrote: > > Progress reporting on sorts seems like a tricky problem to me, as I > > said before. In most cases, a sort is going to involve an initial > > stage where it reads all the input tuples and writes out q

Re: [HACKERS] CLUSTER command progress monitor

2018-12-18 Thread Alvaro Herrera
Hello Yamada-san, On 2018-Dec-03, Tatsuro Yamada wrote: > Thank you for managing the CF and Sorry for the late reply. > I'll rebase it for the next CF and also I'll clear my head because the patch > needs design change to address the feedbacks, I guess. Therefore, the status > is > reconsidering

Re: [HACKERS] CLUSTER command progress monitor

2018-12-03 Thread Michael Paquier
On Mon, Dec 03, 2018 at 02:17:25PM -0300, Alvaro Herrera wrote: > I think we should mark it as Returned with Feedback then. +1. -- Michael signature.asc Description: PGP signature

Re: [HACKERS] CLUSTER command progress monitor

2018-12-03 Thread Alvaro Herrera
On 2018-Dec-03, Tatsuro Yamada wrote: > > In the meantime I'm moving it to the next CF. > > Thank you for managing the CF and Sorry for the late reply. > I'll rebase it for the next CF and also I'll clear my head because the patch > needs design change to address the feedbacks, I guess. Therefore

Re: [HACKERS] CLUSTER command progress monitor

2018-12-03 Thread Tatsuro Yamada
On 2018/11/29 21:20, Dmitry Dolgov wrote: On Fri, Aug 24, 2018 at 7:06 AM Tatsuro Yamada wrote: On 2017/11/22 6:07, Peter Geoghegan wrote: On Mon, Oct 2, 2017 at 6:04 AM, Robert Haas wrote: Progress reporting on sorts seems like a tricky problem to me, as I said before. In most cases, a so

Re: [HACKERS] CLUSTER command progress monitor

2018-11-29 Thread Dmitry Dolgov
> On Fri, Aug 24, 2018 at 7:06 AM Tatsuro Yamada > wrote: > > On 2017/11/22 6:07, Peter Geoghegan wrote: > > On Mon, Oct 2, 2017 at 6:04 AM, Robert Haas wrote: > >> Progress reporting on sorts seems like a tricky problem to me, as I > >> said before. In most cases, a sort is going to involve an

Re: [HACKERS] CLUSTER command progress monitor

2018-08-23 Thread Tatsuro Yamada
On 2017/11/22 6:07, Peter Geoghegan wrote: On Mon, Oct 2, 2017 at 6:04 AM, Robert Haas wrote: Progress reporting on sorts seems like a tricky problem to me, as I said before. In most cases, a sort is going to involve an initial stage where it reads all the input tuples and writes out quicksort

Re: [HACKERS] CLUSTER command progress monitor

2017-11-21 Thread Thomas Munro
On Wed, Nov 22, 2017 at 1:53 PM, Michael Paquier wrote: > On Wed, Nov 22, 2017 at 5:55 AM, Robert Haas wrote: >> I have been of the opinion all along that progress monitoring needs to >> report facts, not theories. The number of tuples read thus far is a >> fact, and is fine to report for whatev

Re: [HACKERS] CLUSTER command progress monitor

2017-11-21 Thread Michael Paquier
On Wed, Nov 22, 2017 at 5:55 AM, Robert Haas wrote: > I have been of the opinion all along that progress monitoring needs to > report facts, not theories. The number of tuples read thus far is a > fact, and is fine to report for whatever value it may have to someone. > The number of tuples that w

Re: [HACKERS] CLUSTER command progress monitor

2017-11-21 Thread Peter Geoghegan
On Tue, Nov 21, 2017 at 12:55 PM, Robert Haas wrote: > I agree. > > I have been of the opinion all along that progress monitoring needs to > report facts, not theories. The number of tuples read thus far is a > fact, and is fine to report for whatever value it may have to someone. That makes a l

Re: [HACKERS] CLUSTER command progress monitor

2017-11-21 Thread Peter Geoghegan
On Mon, Oct 2, 2017 at 6:04 AM, Robert Haas wrote: > Progress reporting on sorts seems like a tricky problem to me, as I > said before. In most cases, a sort is going to involve an initial > stage where it reads all the input tuples and writes out quicksorted > runs, and then a merge phase where

Re: [HACKERS] CLUSTER command progress monitor

2017-11-21 Thread Robert Haas
On Mon, Nov 20, 2017 at 12:05 PM, Antonin Houska wrote: > Robert Haas wrote: >> On Wed, Aug 30, 2017 at 10:12 PM, Tatsuro Yamada >> wrote: >> > 1. scanning heap >> > 2. sort tuples >> >> These two phases overlap, though. I believe progress reporting for >> sorts is really hard. In the simpl

Re: [HACKERS] CLUSTER command progress monitor

2017-11-21 Thread Robert Haas
On Mon, Nov 20, 2017 at 12:25 PM, Tom Lane wrote: > Antonin Houska writes: >> Robert Haas wrote: >>> These two phases overlap, though. I believe progress reporting for >>> sorts is really hard. > >> Whatever complexity is hidden in the sort, cost_sort() should have taken it >> into consideration

Re: [HACKERS] CLUSTER command progress monitor

2017-11-21 Thread Antonin Houska
Tom Lane wrote: > Antonin Houska writes: > > Robert Haas wrote: > >> These two phases overlap, though. I believe progress reporting for > >> sorts is really hard. > > > Whatever complexity is hidden in the sort, cost_sort() should have taken it > > into consideration when called via plan_clust

Re: [HACKERS] CLUSTER command progress monitor

2017-11-20 Thread Tom Lane
Antonin Houska writes: > Robert Haas wrote: >> These two phases overlap, though. I believe progress reporting for >> sorts is really hard. > Whatever complexity is hidden in the sort, cost_sort() should have taken it > into consideration when called via plan_cluster_use_sort(). Thus I think that

Re: [HACKERS] CLUSTER command progress monitor

2017-11-20 Thread Antonin Houska
Robert Haas wrote: > On Wed, Aug 30, 2017 at 10:12 PM, Tatsuro Yamada > wrote: > > 1. scanning heap > > 2. sort tuples > > These two phases overlap, though. I believe progress reporting for > sorts is really hard. In the simple case where the data fits in > work_mem, none of the work of th