On 2019-04-06 06:40, Alvaro Herrera wrote:
> On 2019-Apr-05, Peter Eisentraut wrote:
>
>> I've reworded the phases a bit. There was a bit of a mixup of waiting
>> for snapshots and waiting for lockers. Perhaps not so important from a
>> user's perspective, but at least now it's more consistent w
On 2019-Apr-05, Peter Eisentraut wrote:
> I've reworded the phases a bit. There was a bit of a mixup of waiting
> for snapshots and waiting for lockers. Perhaps not so important from a
> user's perspective, but at least now it's more consistent with the
> source code comments.
No disagreement w
On 2019-04-05 17:01, Alvaro Herrera wrote:
> Users are going to wonder why the other phases don't appear to complete
> for a long time :-) Keep in mind that the "waiting" phases are very
> confusing to users. I suggest we just define additional phase numbers
> for those phases, then switch the "f
On 2019-Apr-05, Peter Eisentraut wrote:
> It seems we can easily extend this to also monitor REINDEX
> [CONCURRENTLY]. Attached is a quick patch.
That's much easier than I was expecting. I think we should endeavor to
get this done for pg12.
> For the concurrently part, we currently don't have
It seems we can easily extend this to also monitor REINDEX
[CONCURRENTLY]. Attached is a quick patch.
For the concurrently part, we currently don't have any phases defined
for the index swap and drop, but maybe we can just skip that initially.
What happens if we don't have those?
It might be nic
I noticed a couple of inconsistent uses of "heap" which I changed to
"table", and also changed one update_multi_params() call that had only
one param to update_param(). Very minor other tweaks here and there.
And pushed.
Thanks for the reviews!
--
Álvaro Herrerahttps://www.2ndQ
I did this (I should stop c&p'ing this silly little setup code sometime):
create table t (a int) partition by hash (a);
create table t1 partition of t for values with (modulus 10, remainder 0);
create table t2 partition of t for values with (modulus 10, remainder 1);
create table t3 partition of t
On 2019-Apr-02, Rahila Syed wrote:
> 1. FWIW, below results for CIC show that blocks_done does not become equal
> to blocks_total at the end of the phase or it processes last 800 blocks so
> fast that
> the update is not visible in less than 1 secs interval.
Yeah, I noticed this too and decided
Hi,
On Mon, 1 Apr 2019 at 21:40, Alvaro Herrera
wrote:
> Hi Rahila, thanks for reviewing.
>
> On 2019-Mar-25, Rahila Syed wrote:
>
> > Please see few comments below:
> >
> > 1. Makecheck fails currently as view definition of expected rules.out
> does
> > not reflect latest changes in progress me
Hi Rahila, thanks for reviewing.
On 2019-Mar-25, Rahila Syed wrote:
> Please see few comments below:
>
> 1. Makecheck fails currently as view definition of expected rules.out does
> not reflect latest changes in progress metrics numbering.
Ouch ... fixed.
> 2. +
> I think there is a typo
Hi Alvaro,
Please see few comments below:
1. Makecheck fails currently as view definition of expected rules.out does
not reflect latest changes in progress metrics numbering.
2. +
+ When creating an index on a partitioned, this column is set to the
+ total number of partitions
On 2019-Mar-29, Robert Haas wrote:
> On Fri, Mar 29, 2019 at 3:28 PM Alvaro Herrera
> wrote:
> > Maybe we can consider using dynamic shmem for that, and include a
> > pointer to it in the fixed-size chunk. (It's a bit too late to be
> > writing this code, mind; I'm just proposing this for a fu
On Fri, Mar 29, 2019 at 3:28 PM Alvaro Herrera wrote:
> On 2019-Mar-29, Robert Haas wrote:
> > On Fri, Mar 29, 2019 at 2:16 PM Alvaro Herrera
> > wrote:
> > > I think we should consider a new column of an array type, where we could
> > > put things like the list of PIDs to be waited for, the lis
On 2019-Mar-29, Robert Haas wrote:
> On Fri, Mar 29, 2019 at 2:16 PM Alvaro Herrera
> wrote:
> > I think we should consider a new column of an array type, where we could
> > put things like the list of PIDs to be waited for, the list of OIDs of
> > index to rebuild, or the list of partitions to
On Fri, Mar 29, 2019 at 2:16 PM Alvaro Herrera wrote:
> I think we should consider a new column of an array type, where we could
> put things like the list of PIDs to be waited for, the list of OIDs of
> index to rebuild, or the list of partitions to build the index on.
This has to work with a fi
On 2019-Mar-29, Alvaro Herrera wrote:
> So, CLUSTER and ALTER TABLE rewrites only do non-concurrent index
> builds; and REINDEX can reuse pretty much the same wait-for metrics
> columns as CIC. So I think it's okay if I move only the metrics that
> conflict for index_build.
The attached version
On 2019-Mar-29, Andres Freund wrote:
> Hi,
>
> On 2019-03-29 12:02:18 -0300, Alvaro Herrera wrote:
> > I just noticed that the CLUSTER calls index_build, which my patch
> > modifies to include additional progress metrics; this means that during
> > the index build phase, the metrics set by CLUSTE
On 2019-Mar-29, Alvaro Herrera wrote:
> I just noticed that the CLUSTER calls index_build, which my patch
> modifies to include additional progress metrics; this means that during
> the index build phase, the metrics set by CLUSTER will be trashed by the
> ones my patch introduces.
Indeed:
pid
Hi,
On 2019-03-29 12:02:18 -0300, Alvaro Herrera wrote:
> I just noticed that the CLUSTER calls index_build, which my patch
> modifies to include additional progress metrics; this means that during
> the index build phase, the metrics set by CLUSTER will be trashed by the
> ones my patch introduce
I just noticed that the CLUSTER calls index_build, which my patch
modifies to include additional progress metrics; this means that during
the index build phase, the metrics set by CLUSTER will be trashed by the
ones my patch introduces.
--
Álvaro Herrerahttps://www.2ndQuadrant.com
On Thu, Mar 28, 2019 at 12:07 PM Simon Riggs wrote:
> Seems more like our own labelling of the phases is responsible for that,
> rather than it being a specific problem. The numbering should reflect the
> ordinal executed step number. So if a VACUUM has required two sets of index
> scanning, th
On Thu, 28 Mar 2019 at 15:39, Alvaro Herrera
wrote:
> On 2019-Mar-28, Simon Riggs wrote:
>
> > On Thu, 28 Mar 2019 at 14:56, Alvaro Herrera
> > wrote:
> >
> > > I have not reinstated phase numbers; I have Rahila's positive vote for
> > > them. Do I hear any more votes on this issue?
> >
> > If
On 2019-Mar-28, Simon Riggs wrote:
> On Thu, 28 Mar 2019 at 14:56, Alvaro Herrera
> wrote:
>
> > I have not reinstated phase numbers; I have Rahila's positive vote for
> > them. Do I hear any more votes on this issue?
>
> If there is a specific technical issue, I'd like to understand that more
On Thu, 28 Mar 2019 at 14:56, Alvaro Herrera
wrote:
> I have not reinstated phase numbers; I have Rahila's positive vote for
> them. Do I hear any more votes on this issue?
>
If there is a specific technical issue, I'd like to understand that more.
If it is just a usability preference, then I
On 2019-Mar-10, David Fetter wrote:
> Would it be a very large lift to report progress for the rest of the
> index types we support?
Patch v7 I just posted does that. Please give it a look and let me know
what you think.
--
Álvaro Herrerahttps://www.2ndQuadrant.com/
PostgreSQL
Here's v7. This is rebased on top of yesterday's tableam commit
reworking the index build API (thanks Rahila for letting me know it had
already rot). No changes otherwise. Got rid of 0001 because the
tableam changes made that unnecessary. (Each new table AM will have to
include its own progress
On Mon, 25 Mar 2019 at 22:23, Alvaro Herrera
wrote:
> Here's v6 of this patch. I have rebased on top of today's CLUSTER
> monitoring, as well as on table AM commits. The latter caused a bit of
> trouble, as now the number of blocks processed by a scan is not as easy
> to get as before; I added
Hi
On 2019-Mar-25, Andres Freund wrote:
> I've not followed this thread at all, so I might just be out of my depth
> here. From my POV, a later patch in the yet-to-be-applied patchqueue
> moves the main part of cluster below the table AM (as there's enough low
> level details, e.g. dealing with H
Hi,
On 2019-03-25 23:11:00 -0300, Alvaro Herrera wrote:
> On 2019-Mar-25, Alvaro Herrera wrote:
>
> > Here's v6 of this patch. I have rebased on top of today's CLUSTER
> > monitoring, as well as on table AM commits. The latter caused a bit of
> > trouble, as now the number of blocks processed b
On 2019-Mar-25, Alvaro Herrera wrote:
> Here's v6 of this patch. I have rebased on top of today's CLUSTER
> monitoring, as well as on table AM commits. The latter caused a bit of
> trouble, as now the number of blocks processed by a scan is not as easy
> to get as before; I added a new entry poi
On 2019/03/26 1:53, Alvaro Herrera wrote:
> Here's v6 of this patch. I have rebased on top of today's CLUSTER
> monitoring, as well as on table AM commits. The latter caused a bit of
> trouble, as now the number of blocks processed by a scan is not as easy
> to get as before; I added a new entry
Here's v6 of this patch. I have rebased on top of today's CLUSTER
monitoring, as well as on table AM commits. The latter caused a bit of
trouble, as now the number of blocks processed by a scan is not as easy
to get as before; I added a new entry point heapscan_get_blocks_done on
heapam.c to help
On 2019-Mar-11, Robert Haas wrote:
> On Mon, Mar 11, 2019 at 3:43 PM Alvaro Herrera
> wrote:
> > > Huh. Well, that's another option, but then what do we do if the
> > > number of phases is not a constant?
> >
> > Well, why do we care? "Some phases might be skipped".
>
> It seems pretty confus
On Mon, Mar 11, 2019 at 3:43 PM Alvaro Herrera wrote:
> > Huh. Well, that's another option, but then what do we do if the
> > number of phases is not a constant?
>
> Well, why do we care? "Some phases might be skipped".
It seems pretty confusing. I mean, in the case of the CLUSTER patch,
you'r
On 2019-Mar-11, Robert Haas wrote:
> On Mon, Mar 11, 2019 at 3:26 PM Alvaro Herrera
> wrote:
> > Oh. That's easily removed. Though I have to say that other people said
> > that they liked it so much that they would have liked to have it in the
> > original VACUUM one too
> > (5ba2b281-9c84-77
On Mon, Mar 11, 2019 at 3:26 PM Alvaro Herrera wrote:
> Oh. That's easily removed. Though I have to say that other people said
> that they liked it so much that they would have liked to have it in the
> original VACUUM one too (5ba2b281-9c84-772a-cf37-17780d782...@lab.ntt.co.jp).
Huh. Well, th
On 2019-Mar-11, Robert Haas wrote:
> On Mon, Mar 11, 2019 at 3:18 PM Alvaro Herrera
> wrote:
> > On 2019-Mar-11, Robert Haas wrote:
> > > I don't think that I much like this (3 of 8) and (2 of 5) stuff. It's
> > > inconsistent with what we've got already and it doesn't add much.
> > > Someone w
On Mon, Mar 11, 2019 at 3:18 PM Alvaro Herrera wrote:
> On 2019-Mar-11, Robert Haas wrote:
> > I don't think that I much like this (3 of 8) and (2 of 5) stuff. It's
> > inconsistent with what we've got already and it doesn't add much.
> > Someone who wants to know which phase it is can look at th
On 2019-Mar-11, Robert Haas wrote:
> I don't think that I much like this (3 of 8) and (2 of 5) stuff. It's
> inconsistent with what we've got already and it doesn't add much.
> Someone who wants to know which phase it is can look at the underlying
> numbers directly instead of going through the v
On Mon, Mar 11, 2019 at 8:41 AM Alvaro Herrera wrote:
> > I wonder how is the phase 'building index(3 of 8): initializing(1/5)' when
> > the blocks_done count is increasing. Shouldn't it have
> > changed to reflect PROGRESS_BTREE_PHASE_INDEXBUILD_HEAPSCAN as building
> > index(3 of 8): table scan
Hi Alvaro,
On Tue, 5 Mar 2019 at 08:32, Alvaro Herrera
wrote:
> Hi Rahila,
>
> Thanks for looking.
>
> On 2019-Mar-04, Rahila wrote:
>
> > 1. Thank you for incorporating review comments.
> > Can you please rebase the latest 0001-Report-progress-of-
> > CREATE-INDEX-operations.patch on master? Cu
Hi Rahila
On 2019-Mar-11, Rahila Syed wrote:
> On Tue, 5 Mar 2019 at 08:32, Alvaro Herrera
> wrote:
> > +extern char *btbuildphasename(int64 phasenum);
>
> 1. I think int64 is too large a datatype for phasenum.
> Also int32 is used for phasenum in pg_indexam_progress_phasename().
> Can we hav
On Mon, Mar 04, 2019 at 05:46:07PM -0300, Alvaro Herrera wrote:
> Hi Rahila,
>
> Thanks for looking.
>
> On 2019-Mar-04, Rahila wrote:
>
> > 1. Thank you for incorporating review comments.
> > Can you please rebase the latest 0001-Report-progress-of-
> > CREATE-INDEX-operations.patch on master?
Hi Rahila,
Thanks for looking.
On 2019-Mar-04, Rahila wrote:
> 1. Thank you for incorporating review comments.
> Can you please rebase the latest 0001-Report-progress-of-
> CREATE-INDEX-operations.patch on master? Currently it does not apply on
> 754b90f657bd54b482524b73726dae4a9165031c
Hmm, re
Hi Alvaro,
Resending the email as earlier one didn't get sent on pgsql-hackers.
On 2/23/19 3:24 AM, Alvaro Herrera wrote:
On 2019-Feb-13, Amit Langote wrote:
Doesn't the name amphasename sound a bit too generic, given that it can
only describe the phases of ambuild? Maybe ambuildphase?
Hmm,
Hi Alvaro,
> On 2019-Feb-13, Amit Langote wrote:
>
> > Doesn't the name amphasename sound a bit too generic, given that it can
> > only describe the phases of ambuild? Maybe ambuildphase?
>
> Hmm, yeah, maybe it does. I renamed it "ambuildphasename", since it's
> not about reporting the phase i
Hmm, looks like a very bare-bones support for hash indexes does not
require a lot of code, and gives a clear picture (you can sit all night
watching the numbers go up, instead of biting your fingernails wondering
if it'll be completed by dawn.) This part isn't 100% done -- it we
would better to ha
On 2019-Feb-13, Amit Langote wrote:
> Doesn't the name amphasename sound a bit too generic, given that it can
> only describe the phases of ambuild? Maybe ambuildphase?
Hmm, yeah, maybe it does. I renamed it "ambuildphasename", since it's
not about reporting the phase itself -- it's about trans
On 2019/02/13 4:16, Alvaro Herrera wrote:
I added metrics for the validate_index phases; patch attached. This is
still a bit raw, but it looks much better now. Here's a sample
concurrent index build on a 100M tuple table. There are no concurrent
Great!
+ s.pid AS pid, S.datid
Hi Alvaro,
On 2019/02/12 12:18, Alvaro Herrera wrote:
> I ended up defining phases for the index_build phase in the AM itself;
> the code reports a phase number using the regular API, and then the
> pgstat_progress view obtains the name of each phase using a support
> method.
diff --git a/doc/src
On 2019/02/13 11:59, Simon Riggs wrote:
> On Wed, 13 Feb 2019 at 00:46, Alvaro Herrera
> wrote:
>
>> Here's a sample
>> concurrent index build on a 100M tuple table.
>
>
> Cool
+1
Looking at the "([phase] x of x)" in the sample output, I thought
pg_stat_progress_vacuum's output should've had
On Wed, 13 Feb 2019 at 00:46, Alvaro Herrera
wrote:
> Here's a sample
> concurrent index build on a 100M tuple table.
Cool
> There are no concurrent
> transactions, so phases that wait for lockers don't appear.
Would prefer to see them, so we know they are zero.
--
Simon Riggs
I added metrics for the validate_index phases; patch attached. This is
still a bit raw, but it looks much better now. Here's a sample
concurrent index build on a 100M tuple table. There are no concurrent
transactions, so phases that wait for lockers don't appear. I'm not
making any effort to re
Hi Rahila, Pavan,
Thanks for the review. I incorporated some fixes per your comments.
More fixes are needed still. That said, the patch in attachment gives
good insight into how I think this should turn out.
> > index_build
> > ---
> OK.
> I think the main phases in which index_build f
Hi all,
Based on the latest emails exchanged, the patch got some feedback from
Pavan which has not been addressed. So I am marking the patch as
returned with feedback.
--
Michael
signature.asc
Description: PGP signature
On 2019-Jan-09, Pavan Deolasee wrote:
> Looks like it's missing the validate_index blocks-scanned report, which is
> not AM-specific and your original proposal does mention that.
True. That phase is actually 3 phases, which would be reported
separately:
5. index_bulk_delete() scan
6. perform
Hi Alvaro,
The WIP patch needs a rebase. Please see few in-line comments.
On Fri, Dec 21, 2018 at 3:30 AM Alvaro Herrera
wrote:
> Monitoring progress of CREATE INDEX [CONCURRENTLY] is sure to be welcome,
> so here's a proposal.
>
> There are three distinct interesting cases. One is straight CR
On Tue, Jan 1, 2019 at 6:09 AM Alvaro Herrera
wrote:
> For discussion, here's an preliminary patch. This is just a first
> skeleton; needs to grow a lot of flesh yet, per my previous proposal.
> As far as the generic CREATE INDEX stuff goes, I think this is complete;
> it's missing the AM-specif
For discussion, here's an preliminary patch. This is just a first
skeleton; needs to grow a lot of flesh yet, per my previous proposal.
As far as the generic CREATE INDEX stuff goes, I think this is complete;
it's missing the AM-specific bits.
--
Álvaro Herrerahttps://www.2ndQuad
Monitoring progress of CREATE INDEX [CONCURRENTLY] is sure to be welcome,
so here's a proposal.
There are three distinct interesting cases. One is straight CREATE
INDEX of a standalone table; then we have CREATE INDEX CONCURRENTLY;
finally, CREATE INDEX on a partitioned table. Note that there's
60 matches
Mail list logo
