Re: Improvements and additions to COPY progress reporting

2021-03-15 Thread Michael Paquier
On Mon, Mar 15, 2021 at 12:43:40PM +0100, Matthias van de Meent wrote: > Hmm, does CFBot not run checkout on windows with crlf line endings? I > had expected it to do as such. This is environment-sensitive, so I am not surprised that Appveyor changes the way newlines are handled there. I could se

Re: Improvements and additions to COPY progress reporting

2021-03-15 Thread Matthias van de Meent
On Mon, 15 Mar 2021 at 05:53, Michael Paquier wrote: > > On Wed, Mar 10, 2021 at 09:35:10AM +0100, Matthias van de Meent wrote: > > There are examples in which pg_stat_progress_* -views report > > inaccurate data. I think it is fairly reasonable to at least validate > > some part of the progress r

Re: Improvements and additions to COPY progress reporting

2021-03-14 Thread Michael Paquier
On Wed, Mar 10, 2021 at 09:35:10AM +0100, Matthias van de Meent wrote: > There are examples in which pg_stat_progress_* -views report > inaccurate data. I think it is fairly reasonable to at least validate > some part of the progress reporting, as it is one of the few methods > for administrators t

Re: Improvements and additions to COPY progress reporting

2021-03-10 Thread Matthias van de Meent
On Tue, 9 Mar 2021 at 06:34, Michael Paquier wrote: > > On Mon, Mar 08, 2021 at 05:33:40PM +0100, Matthias van de Meent wrote: > > Seems reasonable. PFA updated patches. I've renamed the previous 0003 > > to 0002 to keep git-format-patch easy. > > Thanks for updating the patch. 0001 has been appl

Re: Improvements and additions to COPY progress reporting

2021-03-09 Thread Josef Šimánek
út 9. 3. 2021 v 6:34 odesílatel Michael Paquier napsal: > > On Mon, Mar 08, 2021 at 05:33:40PM +0100, Matthias van de Meent wrote: > > Seems reasonable. PFA updated patches. I've renamed the previous 0003 > > to 0002 to keep git-format-patch easy. > > Thanks for updating the patch. 0001 has been

Re: Improvements and additions to COPY progress reporting

2021-03-09 Thread Bharath Rupireddy
On Tue, Mar 9, 2021 at 11:04 AM Michael Paquier wrote: > > This is keeping current behaviour of the implementation as committed > > with 8a4f618e, with the rationale of that patch being that this number > > should mirror the number returned by the copy command. > > > > I am not opposed to adding a

Re: Improvements and additions to COPY progress reporting

2021-03-08 Thread Michael Paquier
On Mon, Mar 08, 2021 at 05:33:40PM +0100, Matthias van de Meent wrote: > Seems reasonable. PFA updated patches. I've renamed the previous 0003 > to 0002 to keep git-format-patch easy. Thanks for updating the patch. 0001 has been applied, after tweaking a bit comments, indentation and the docs. >

Re: Improvements and additions to COPY progress reporting

2021-03-08 Thread Matthias van de Meent
On Mon, 8 Mar 2021 at 09:24, Michael Paquier wrote: > > On Sun, Mar 07, 2021 at 04:50:31PM +0530, Bharath Rupireddy wrote: > > Attaching remaining patches 0001 and 0003 from the v11 patch > > set(posted upthread) here to make cfbot happier. > > Looking at patch 0002, the location of each progress

Re: Improvements and additions to COPY progress reporting

2021-03-08 Thread Michael Paquier
On Sun, Mar 07, 2021 at 04:50:31PM +0530, Bharath Rupireddy wrote: > Attaching remaining patches 0001 and 0003 from the v11 patch > set(posted upthread) here to make cfbot happier. Looking at patch 0002, the location of each progress report looks good to me. I have some issues with some of the na

Re: Improvements and additions to COPY progress reporting

2021-03-07 Thread Bharath Rupireddy
On Fri, Mar 5, 2021 at 11:30 AM Michael Paquier wrote: > > On Thu, Mar 04, 2021 at 05:45:50PM +0100, Matthias van de Meent wrote: > > Sure, I'm convinced. PFA the patchset with this change applied. > > 0002 looks fine to me, and in line with the discussion, so applied. Thanks. Attaching remainin

Re: Improvements and additions to COPY progress reporting

2021-03-04 Thread Michael Paquier
On Thu, Mar 04, 2021 at 05:45:50PM +0100, Matthias van de Meent wrote: > Sure, I'm convinced. PFA the patchset with this change applied. 0002 looks fine to me, and in line with the discussion, so applied. -- Michael signature.asc Description: PGP signature

Re: Improvements and additions to COPY progress reporting

2021-03-04 Thread Matthias van de Meent
On Thu, 4 Mar 2021 at 17:29, Justin Pryzby wrote: > > On Thu, Mar 04, 2021 at 05:19:18PM +0100, Matthias van de Meent wrote: > > > > "Backends running [...] report progress to [...] instead" is, > > a.f.a.i.k., correct English. Adding 'will' would indeed still be > > correct, but it would in my op

Re: Improvements and additions to COPY progress reporting

2021-03-04 Thread Justin Pryzby
On Thu, Mar 04, 2021 at 05:19:18PM +0100, Matthias van de Meent wrote: > On Thu, 4 Mar 2021 at 13:36, Bharath Rupireddy > wrote: > > > > + > > +Each backend running VACUUM without the > > +FULL option will report its progress in the > > +pg_stat_progress_vacuum view. Backends runni

Re: Improvements and additions to COPY progress reporting

2021-03-04 Thread Matthias van de Meent
On Thu, 4 Mar 2021 at 13:36, Bharath Rupireddy wrote: > > + > +Each backend running VACUUM without the > +FULL option will report its progress in the > +pg_stat_progress_vacuum view. Backends running > +VACUUM with the FULL option report > +progress in the pg_stat_progress_c

Re: Improvements and additions to COPY progress reporting

2021-03-04 Thread Josef Šimánek
čt 4. 3. 2021 v 12:32 odesílatel Matthias van de Meent napsal: > > On Thu, 4 Mar 2021 at 11:38, Michael Paquier wrote: > > > > On Thu, Mar 04, 2021 at 12:16:17PM +0530, Bharath Rupireddy wrote: > > > IMO, the phrasing proposed by Justin upthread looks good. It's like this: > > > > > > > +Each

Re: Improvements and additions to COPY progress reporting

2021-03-04 Thread Bharath Rupireddy
On Thu, Mar 4, 2021 at 5:02 PM Matthias van de Meent wrote: > > On Thu, 4 Mar 2021 at 11:38, Michael Paquier wrote: > > > > On Thu, Mar 04, 2021 at 12:16:17PM +0530, Bharath Rupireddy wrote: > > > IMO, the phrasing proposed by Justin upthread looks good. It's like this: > > > > > > > +Each ba

Re: Improvements and additions to COPY progress reporting

2021-03-04 Thread Matthias van de Meent
On Thu, 4 Mar 2021 at 11:38, Michael Paquier wrote: > > On Thu, Mar 04, 2021 at 12:16:17PM +0530, Bharath Rupireddy wrote: > > IMO, the phrasing proposed by Justin upthread looks good. It's like this: > > > > > +Each backend running ANALYZE will report its > > > progress in > > > +the pg_

Re: Improvements and additions to COPY progress reporting

2021-03-04 Thread Michael Paquier
On Thu, Mar 04, 2021 at 12:16:17PM +0530, Bharath Rupireddy wrote: > IMO, the phrasing proposed by Justin upthread looks good. It's like this: > > > +Each backend running ANALYZE will report its > > progress in > > +the pg_stat_progress_analyze view. See No objections to just go with tha

Re: Improvements and additions to COPY progress reporting

2021-03-03 Thread Bharath Rupireddy
On Wed, Feb 24, 2021 at 1:23 PM Justin Pryzby wrote: > > On Sun, Feb 21, 2021 at 08:10:09PM +0100, Matthias van de Meent wrote: > > Subject: [PATCH v9 1/3] Add progress-reported components for COPY progress > > reporting > > > /* Increment amount of processed tuples and upda

Re: Improvements and additions to COPY progress reporting

2021-03-03 Thread Bharath Rupireddy
On Thu, Feb 25, 2021 at 1:44 PM Michael Paquier wrote: > > On Wed, Feb 24, 2021 at 01:53:03AM -0600, Justin Pryzby wrote: > > On Sun, Feb 21, 2021 at 08:10:09PM +0100, Matthias van de Meent wrote: > > I think these should say that they report their progress *in* the view (not > > "to"): > > > > >

Re: Improvements and additions to COPY progress reporting

2021-02-25 Thread Michael Paquier
On Wed, Feb 24, 2021 at 01:53:03AM -0600, Justin Pryzby wrote: > On Sun, Feb 21, 2021 at 08:10:09PM +0100, Matthias van de Meent wrote: > I think these should say that they report their progress *in* the view (not > "to"): > > > +Each backend running ANALYZE will report its > > progress to >

Re: Improvements and additions to COPY progress reporting

2021-02-23 Thread Justin Pryzby
On Sun, Feb 21, 2021 at 08:10:09PM +0100, Matthias van de Meent wrote: > Subject: [PATCH v9 1/3] Add progress-reported components for COPY progress > reporting > /* Increment amount of processed tuples and update the > progress */ > /* Increment amount of processed tu

Re: Improvements and additions to COPY progress reporting

2021-02-23 Thread Michael Paquier
On Tue, Feb 23, 2021 at 10:27:24AM +0100, Matthias van de Meent wrote: > Note, I'm happy to be proven wrong here, in which case I don't > disagree, but according to my limited knowledge, these outputs should > be stable. I am planning to look more at 0001 and 0003, but for now I have been looking

Re: Improvements and additions to COPY progress reporting

2021-02-23 Thread Bharath Rupireddy
On Tue, Feb 23, 2021 at 2:57 PM Matthias van de Meent wrote: > On Mon, 22 Feb 2021 at 05:49, Bharath Rupireddy > wrote: > > > > On Mon, Feb 22, 2021 at 12:40 AM Matthias van de Meent > > wrote: > > > > > > On Sat, 20 Feb 2021 at 07:09, Bharath Rupireddy > > > wrote: > > > > > > > > For COPY TO

Re: Improvements and additions to COPY progress reporting

2021-02-23 Thread Matthias van de Meent
On Mon, 22 Feb 2021 at 05:49, Bharath Rupireddy wrote: > > On Mon, Feb 22, 2021 at 12:40 AM Matthias van de Meent > wrote: > > > > On Sat, 20 Feb 2021 at 07:09, Bharath Rupireddy > > wrote: > > > > > > For COPY TO the name "source_type" column and for COPY FROM the name > > > "destination_type"

Re: Improvements and additions to COPY progress reporting

2021-02-21 Thread Bharath Rupireddy
On Mon, Feb 22, 2021 at 12:40 AM Matthias van de Meent wrote: > > On Sat, 20 Feb 2021 at 07:09, Bharath Rupireddy > wrote: > > > > For COPY TO the name "source_type" column and for COPY FROM the name > > "destination_type" makes sense. To have a combined column name for > > both, how about namin

Re: Improvements and additions to COPY progress reporting

2021-02-21 Thread Matthias van de Meent
On Sat, 20 Feb 2021 at 07:09, Bharath Rupireddy wrote: > > For COPY TO the name "source_type" column and for COPY FROM the name > "destination_type" makes sense. To have a combined column name for > both, how about naming that column as "io_type"? Thank you, that's way better! PFA what I believe

Re: Improvements and additions to COPY progress reporting

2021-02-20 Thread Michael Paquier
On Sat, Feb 20, 2021 at 02:29:44PM +0530, Bharath Rupireddy wrote: > Yeah. We could use pgstat_progress_update_multi_param instead of > pgstat_progress_update_param to update multiple params. > > On a quick scan through the code, I found that we can do the following. If > okay, I can start a new t

Re: Improvements and additions to COPY progress reporting

2021-02-20 Thread Josef Šimánek
so 20. 2. 2021 v 7:09 odesílatel Bharath Rupireddy napsal: > > On Fri, Feb 19, 2021 at 2:34 AM Tomas Vondra > wrote: > > > On Mon, 15 Feb 2021 at 17:07, Tomas Vondra > > > wrote: > > >> > > >> - The blocks in copyfrom.cc/copyto.c should be reworked - I don't think > > >> we do this in our codeba

Re: Improvements and additions to COPY progress reporting

2021-02-20 Thread Bharath Rupireddy
On Sat, Feb 20, 2021 at 12:49 PM Michael Paquier wrote: > > On Sat, Feb 20, 2021 at 11:39:22AM +0530, Bharath Rupireddy wrote: > > Actually in the code base the style of that variable declaration and > > usage of pgstat_progress_update_multi_param is a mix. For instance, in > > lazy_scan_heap, Rei

Re: Improvements and additions to COPY progress reporting

2021-02-19 Thread Michael Paquier
On Sat, Feb 20, 2021 at 11:39:22AM +0530, Bharath Rupireddy wrote: > Actually in the code base the style of that variable declaration and > usage of pgstat_progress_update_multi_param is a mix. For instance, in > lazy_scan_heap, ReindexRelationConcurrently, the variables are > declared at the start

Re: Improvements and additions to COPY progress reporting

2021-02-19 Thread Bharath Rupireddy
On Fri, Feb 19, 2021 at 2:34 AM Tomas Vondra wrote: > > On Mon, 15 Feb 2021 at 17:07, Tomas Vondra > > wrote: > >> > >> - The blocks in copyfrom.cc/copyto.c should be reworked - I don't think > >> we do this in our codebase. > > > > I saw this being used in (re)index progress reporting, that's wh

Re: Improvements and additions to COPY progress reporting

2021-02-18 Thread Tomas Vondra
On 2/18/21 4:46 PM, Matthias van de Meent wrote: > Hi, > > Thank you all for the suggestions. PFA version 8 of the patchset, in > which I have applied most of your comments. Unless explicitly named > below, I have applied the suggestions. > Thanks. > > On Mon, 15 Feb 2021 at 17:07, Tomas Vo

Re: Improvements and additions to COPY progress reporting

2021-02-18 Thread Matthias van de Meent
Hi, Thank you all for the suggestions. PFA version 8 of the patchset, in which I have applied most of your comments. Unless explicitly named below, I have applied the suggestions. On Mon, 15 Feb 2021 at 17:07, Tomas Vondra wrote: > > - The blocks in copyfrom.cc/copyto.c should be reworked - I d

Re: Improvements and additions to COPY progress reporting

2021-02-15 Thread Tomas Vondra
Hi, I agree with these changes in general - I have a couple minor comment: 1) 0001 - the SGML docs are missing a couple tags - The blocks in copyfrom.cc/copyto.c should be reworked - I don't think we do this in our codebase. Move the variable declarations to the beginning, get rid of the out bl

Re: Improvements and additions to COPY progress reporting

2021-02-12 Thread Justin Pryzby
--- a/doc/src/sgml/ref/analyze.sgml +++ b/doc/src/sgml/ref/analyze.sgml

Re: Improvements and additions to COPY progress reporting

2021-02-12 Thread Matthias van de Meent
On Fri, 12 Feb 2021 at 13:40, Bharath Rupireddy wrote: > > On Fri, Feb 12, 2021 at 5:40 PM Matthias van de Meent > wrote: > > > > On Fri, 12 Feb 2021 at 12:23, Matthias van de Meent > > wrote: > > > > > > On Thu, 11 Feb 2021 at 15:44, Bharath Rupireddy > > > wrote: > > > > > > > > > > > > On Th

Re: Improvements and additions to COPY progress reporting

2021-02-12 Thread Bharath Rupireddy
On Fri, Feb 12, 2021 at 5:40 PM Matthias van de Meent wrote: > > On Fri, 12 Feb 2021 at 12:23, Matthias van de Meent > wrote: > > > > On Thu, 11 Feb 2021 at 15:44, Bharath Rupireddy > > wrote: > > > > > > > > > On Thu, Feb 11, 2021, 8:08 PM Josef Šimánek > > > wrote: > > >> I have split it sin

Re: Improvements and additions to COPY progress reporting

2021-02-12 Thread Matthias van de Meent
On Fri, 12 Feb 2021 at 12:23, Matthias van de Meent wrote: > > On Thu, 11 Feb 2021 at 15:44, Bharath Rupireddy > wrote: > > > > > > On Thu, Feb 11, 2021, 8:08 PM Josef Šimánek wrote: > >> I have split it since it should be the start of progress reporting > >> testing at all. If you better consid

Re: Improvements and additions to COPY progress reporting

2021-02-12 Thread Matthias van de Meent
On Thu, 11 Feb 2021 at 15:44, Bharath Rupireddy wrote: > > > On Thu, Feb 11, 2021, 8:08 PM Josef Šimánek wrote: >> I have split it since it should be the start of progress reporting >> testing at all. If you better consider this as part of COPY testing, >> feel free to move it to already existing

Re: Improvements and additions to COPY progress reporting

2021-02-11 Thread Bharath Rupireddy
On Thu, Feb 11, 2021, 8:08 PM Josef Šimánek wrote: > čt 11. 2. 2021 v 15:27 odesílatel Matthias van de Meent > napsal: > > > > On Wed, 10 Feb 2021 at 07:43, Bharath Rupireddy > > wrote: > > > > > > On Tue, Feb 9, 2021 at 6:02 PM Matthias van de Meent > > > wrote: > > > > > Also, you can add th

Re: Improvements and additions to COPY progress reporting

2021-02-11 Thread Josef Šimánek
čt 11. 2. 2021 v 15:27 odesílatel Matthias van de Meent napsal: > > On Wed, 10 Feb 2021 at 07:43, Bharath Rupireddy > wrote: > > > > On Tue, Feb 9, 2021 at 6:02 PM Matthias van de Meent > > wrote: > > > > Also, you can add this to the current commitfest. > > > > > > See https://commitfest.postgr

Re: Improvements and additions to COPY progress reporting

2021-02-11 Thread Matthias van de Meent
On Wed, 10 Feb 2021 at 07:43, Bharath Rupireddy wrote: > > On Tue, Feb 9, 2021 at 6:02 PM Matthias van de Meent > wrote: > > > Also, you can add this to the current commitfest. > > > > See https://commitfest.postgresql.org/32/2977/ > > > > On Tue, 9 Feb 2021 at 12:53, Josef Šimánek wrote: > > >

Re: Improvements and additions to COPY progress reporting

2021-02-09 Thread Bharath Rupireddy
On Tue, Feb 9, 2021 at 6:02 PM Matthias van de Meent wrote: > > Also, you can add this to the current commitfest. > > See https://commitfest.postgresql.org/32/2977/ > > On Tue, 9 Feb 2021 at 12:53, Josef Šimánek wrote: > > > > OK, would you mind to integrate my regression test initial patch as >

Re: Improvements and additions to COPY progress reporting

2021-02-09 Thread Matthias van de Meent
On Tue, 9 Feb 2021 at 08:12, Bharath Rupireddy wrote: > > On Tue, Feb 9, 2021 at 12:06 AM Matthias van de Meent > wrote: > > With [0] we got COPY progress reporting. Before the column names of > > this newly added view are effectively set in stone with the release of > > pg14, I propose the follo

Re: Improvements and additions to COPY progress reporting

2021-02-09 Thread 0010203112132233
On Tue, 9 Feb 2021 at 09:32, Josef Šimánek wrote: > > po 8. 2. 2021 v 19:35 odesílatel Matthias van de Meent > napsal: > > Lastly, 0005 adds 'io_target' to the reported information, that is, > > FILE, PROGRAM, STDIO or CALLBACK. Although this can relatively easily > > be determined based on the c

Re: Improvements and additions to COPY progress reporting

2021-02-09 Thread Josef Šimánek
út 9. 2. 2021 v 12:51 odesílatel 0010203112132233 napsal: > > On Tue, 9 Feb 2021 at 09:32, Josef Šimánek wrote: > > > > po 8. 2. 2021 v 19:35 odesílatel Matthias van de Meent > > napsal: > > > Lastly, 0005 adds 'io_target' to the reported information, that is, > > > FILE, PROGRAM, STDIO or CALLB

Re: Improvements and additions to COPY progress reporting

2021-02-09 Thread Josef Šimánek
po 8. 2. 2021 v 19:35 odesílatel Matthias van de Meent napsal: > > Hi, > > With [0] we got COPY progress reporting. Before the column names of > this newly added view are effectively set in stone with the release of > pg14, I propose the following set of relatively small patches. These > are v2, b

Re: Improvements and additions to COPY progress reporting

2021-02-08 Thread Bharath Rupireddy
On Tue, Feb 9, 2021 at 12:06 AM Matthias van de Meent wrote: > With [0] we got COPY progress reporting. Before the column names of > this newly added view are effectively set in stone with the release of > pg14, I propose the following set of relatively small patches. These > are v2, because it is

Re: Improvements and additions to COPY progress reporting

2021-02-08 Thread Josef Šimánek
po 8. 2. 2021 v 19:35 odesílatel Matthias van de Meent napsal: > > Hi, > > With [0] we got COPY progress reporting. Before the column names of > this newly added view are effectively set in stone with the release of > pg14, I propose the following set of relatively small patches. These > are v2, b

Improvements and additions to COPY progress reporting

2021-02-08 Thread Matthias van de Meent
Hi, With [0] we got COPY progress reporting. Before the column names of this newly added view are effectively set in stone with the release of pg14, I propose the following set of relatively small patches. These are v2, because it is a patchset that is based on a set of patches that I previously p