Re: maintenance_work_mem used by Vacuum

2019-10-17 Thread Masahiko Sawada
On Fri, Oct 18, 2019, 11:43 Amit Kapila wrote: > On Thu, Oct 17, 2019 at 3:10 PM Tom Lane wrote: > > > > Amit Kapila writes: > > > Another idea could be each index AM tell whether it uses > > > maintainence_work_mem or not and based on that we can do the > > > computation (divide the maintainen

Re: maintenance_work_mem used by Vacuum

2019-10-17 Thread Amit Kapila
On Thu, Oct 17, 2019 at 3:10 PM Tom Lane wrote: > > Amit Kapila writes: > > Another idea could be each index AM tell whether it uses > > maintainence_work_mem or not and based on that we can do the > > computation (divide the maintainence_work_mem by the number of such > > indexes during parallel

Re: maintenance_work_mem used by Vacuum

2019-10-17 Thread Amit Kapila
On Thu, Oct 17, 2019 at 6:05 PM Masahiko Sawada wrote: > > On Thu, Oct 17, 2019 at 6:13 PM Amit Kapila wrote: > > > > On Wed, Oct 16, 2019 at 5:35 PM Masahiko Sawada > > wrote: > > > > > > On Wed, Oct 16, 2019 at 3:48 PM Amit Kapila > > > wrote: > > > > > > > > > > > > It is not that currentl

Re: maintenance_work_mem used by Vacuum

2019-10-17 Thread Masahiko Sawada
On Thu, Oct 17, 2019 at 6:13 PM Amit Kapila wrote: > > On Wed, Oct 16, 2019 at 5:35 PM Masahiko Sawada wrote: > > > > On Wed, Oct 16, 2019 at 3:48 PM Amit Kapila wrote: > > > > > > > > > It is not that currently, other indexes don't use any additional > > > memory (except for maintainence_work_m

Re: maintenance_work_mem used by Vacuum

2019-10-17 Thread Tom Lane
Amit Kapila writes: > Another idea could be each index AM tell whether it uses > maintainence_work_mem or not and based on that we can do the > computation (divide the maintainence_work_mem by the number of such > indexes during parallel vacuum). FWIW, that seems like a perfectly reasonable API a

Re: maintenance_work_mem used by Vacuum

2019-10-17 Thread Amit Kapila
On Wed, Oct 16, 2019 at 5:35 PM Masahiko Sawada wrote: > > On Wed, Oct 16, 2019 at 3:48 PM Amit Kapila wrote: > > > > > > It is not that currently, other indexes don't use any additional > > memory (except for maintainence_work_mem). For example, Gist index > > can use memory for collecting empt

Re: maintenance_work_mem used by Vacuum

2019-10-16 Thread Amit Kapila
On Thu, Oct 17, 2019 at 6:07 AM Greg Stark wrote: > > It's a bit unfortunate that we're doing the pending list flush while the > vacuum memory is allocated at all. Is there any reason other than the way the > callbacks are defined that gin doesn't do the pending list flush before > vacuum does

Re: maintenance_work_mem used by Vacuum

2019-10-16 Thread Greg Stark
It's a bit unfortunate that we're doing the pending list flush while the vacuum memory is allocated at all. Is there any reason other than the way the callbacks are defined that gin doesn't do the pending list flush before vacuum does the heap scan and before it allocates any memory using maintenan

Re: maintenance_work_mem used by Vacuum

2019-10-16 Thread Masahiko Sawada
On Wed, Oct 16, 2019 at 3:48 PM Amit Kapila wrote: > > On Wed, Oct 16, 2019 at 7:20 AM Masahiko Sawada wrote: > > > > On Sat, Oct 12, 2019 at 8:45 PM Amit Kapila wrote: > > > > > > On Sat, Oct 12, 2019 at 10:49 AM Masahiko Sawada > > > wrote: > > > > > > > > On Fri, Oct 11, 2019 at 5:13 PM Ami

Re: maintenance_work_mem used by Vacuum

2019-10-15 Thread Amit Kapila
On Wed, Oct 16, 2019 at 7:20 AM Masahiko Sawada wrote: > > On Sat, Oct 12, 2019 at 8:45 PM Amit Kapila wrote: > > > > On Sat, Oct 12, 2019 at 10:49 AM Masahiko Sawada > > wrote: > > > > > > On Fri, Oct 11, 2019 at 5:13 PM Amit Kapila > > > wrote: > > > > > > > > That's right, but OTOH, if the

Re: maintenance_work_mem used by Vacuum

2019-10-15 Thread Masahiko Sawada
On Sat, Oct 12, 2019 at 8:45 PM Amit Kapila wrote: > > On Sat, Oct 12, 2019 at 10:49 AM Masahiko Sawada > wrote: > > > > On Fri, Oct 11, 2019 at 5:13 PM Amit Kapila wrote: > > > > > > That's right, but OTOH, if the user specifies gin_pending_list_limit > > > as an option during Create Index wit

Re: maintenance_work_mem used by Vacuum

2019-10-12 Thread Amit Kapila
On Sat, Oct 12, 2019 at 10:49 AM Masahiko Sawada wrote: > > On Fri, Oct 11, 2019 at 5:13 PM Amit Kapila wrote: > > > > That's right, but OTOH, if the user specifies gin_pending_list_limit > > as an option during Create Index with a value greater than GUC > > gin_pending_list_limit, then also we w

Re: maintenance_work_mem used by Vacuum

2019-10-11 Thread Masahiko Sawada
On Fri, Oct 11, 2019 at 5:13 PM Amit Kapila wrote: > > On Fri, Oct 11, 2019 at 7:36 AM Masahiko Sawada wrote: > > > > On Thu, Oct 10, 2019 at 6:38 PM Amit Kapila wrote: > > > > > > > > > It seems you want to say about commit id > > > a1b395b6a26ae80cde17fdfd2def8d351872f399. > > > > Yeah thanks.

Re: maintenance_work_mem used by Vacuum

2019-10-11 Thread Amit Kapila
On Fri, Oct 11, 2019 at 7:36 AM Masahiko Sawada wrote: > > On Thu, Oct 10, 2019 at 6:38 PM Amit Kapila wrote: > > > > > > It seems you want to say about commit id > > a1b395b6a26ae80cde17fdfd2def8d351872f399. > > Yeah thanks. > > > I wonder why they have not > > changed it to gin_penidng_list_li

Re: maintenance_work_mem used by Vacuum

2019-10-10 Thread Masahiko Sawada
On Thu, Oct 10, 2019 at 6:38 PM Amit Kapila wrote: > > On Thu, Oct 10, 2019 at 2:10 PM Masahiko Sawada wrote: > > > > On Thu, Oct 10, 2019 at 3:36 PM Amit Kapila wrote: > > > > > > On Thu, Oct 10, 2019 at 9:58 AM Masahiko Sawada > > > wrote: > > > > > > > > On Wed, Oct 9, 2019 at 7:12 PM Dilip

Re: maintenance_work_mem used by Vacuum

2019-10-10 Thread Amit Kapila
On Thu, Oct 10, 2019 at 2:10 PM Masahiko Sawada wrote: > > On Thu, Oct 10, 2019 at 3:36 PM Amit Kapila wrote: > > > > On Thu, Oct 10, 2019 at 9:58 AM Masahiko Sawada > > wrote: > > > > > > On Wed, Oct 9, 2019 at 7:12 PM Dilip Kumar wrote: > > > > > > > > I think the current situation is not go

Re: maintenance_work_mem used by Vacuum

2019-10-10 Thread Masahiko Sawada
On Thu, Oct 10, 2019 at 3:36 PM Amit Kapila wrote: > > On Thu, Oct 10, 2019 at 9:58 AM Masahiko Sawada wrote: > > > > On Wed, Oct 9, 2019 at 7:12 PM Dilip Kumar wrote: > > > > > > I think the current situation is not good but if we try to cap it to > > > maintenance_work_mem + gin_*_work_mem the

Re: maintenance_work_mem used by Vacuum

2019-10-09 Thread Amit Kapila
On Thu, Oct 10, 2019 at 9:58 AM Masahiko Sawada wrote: > > On Wed, Oct 9, 2019 at 7:12 PM Dilip Kumar wrote: > > > > I think the current situation is not good but if we try to cap it to > > maintenance_work_mem + gin_*_work_mem then also I don't think it will > > make the situation much better.

Re: maintenance_work_mem used by Vacuum

2019-10-09 Thread Masahiko Sawada
On Wed, Oct 9, 2019 at 7:12 PM Dilip Kumar wrote: > > On Wed, Oct 9, 2019 at 2:40 PM Amit Kapila wrote: > > > > On Wed, Oct 9, 2019 at 2:00 PM Dilip Kumar wrote: > > > > > > On Wed, Oct 9, 2019 at 10:22 AM Masahiko Sawada > > > wrote: > > > > > > > > On Tue, Oct 8, 2019 at 2:45 PM Amit Kapila

Re: maintenance_work_mem used by Vacuum

2019-10-09 Thread Dilip Kumar
On Wed, Oct 9, 2019 at 2:40 PM Amit Kapila wrote: > > On Wed, Oct 9, 2019 at 2:00 PM Dilip Kumar wrote: > > > > On Wed, Oct 9, 2019 at 10:22 AM Masahiko Sawada > > wrote: > > > > > > On Tue, Oct 8, 2019 at 2:45 PM Amit Kapila > > > wrote: > > > > > > > > On Tue, Oct 8, 2019 at 1:48 AM Peter G

Re: maintenance_work_mem used by Vacuum

2019-10-09 Thread Amit Kapila
On Wed, Oct 9, 2019 at 2:00 PM Dilip Kumar wrote: > > On Wed, Oct 9, 2019 at 10:22 AM Masahiko Sawada wrote: > > > > On Tue, Oct 8, 2019 at 2:45 PM Amit Kapila wrote: > > > > > > On Tue, Oct 8, 2019 at 1:48 AM Peter Geoghegan wrote: > > > > > > > > ISTM that the use of maintenance_work_mem wasn

Re: maintenance_work_mem used by Vacuum

2019-10-09 Thread Dilip Kumar
On Wed, Oct 9, 2019 at 10:22 AM Masahiko Sawada wrote: > > On Tue, Oct 8, 2019 at 2:45 PM Amit Kapila wrote: > > > > On Tue, Oct 8, 2019 at 1:48 AM Peter Geoghegan wrote: > > > > > > On Mon, Oct 7, 2019 at 12:28 PM Robert Haas wrote: > > > > I would say that sucks, because it makes it harder to

Re: maintenance_work_mem used by Vacuum

2019-10-08 Thread Masahiko Sawada
On Tue, Oct 8, 2019 at 2:45 PM Amit Kapila wrote: > > On Tue, Oct 8, 2019 at 1:48 AM Peter Geoghegan wrote: > > > > On Mon, Oct 7, 2019 at 12:28 PM Robert Haas wrote: > > > I would say that sucks, because it makes it harder to set > > > maintenance_work_mem correctly. Not sure how hard it would

Re: maintenance_work_mem used by Vacuum

2019-10-07 Thread Amit Kapila
On Tue, Oct 8, 2019 at 12:57 AM Robert Haas wrote: > > On Sun, Oct 6, 2019 at 6:55 AM Amit Kapila wrote: > > We wanted to decide how a parallel vacuum should use memory? Can each > > worker consume maintenance_work_mem to clean up the gin Index or all > > workers should use no more than mainte

Re: maintenance_work_mem used by Vacuum

2019-10-07 Thread Amit Kapila
On Tue, Oct 8, 2019 at 1:48 AM Peter Geoghegan wrote: > > On Mon, Oct 7, 2019 at 12:28 PM Robert Haas wrote: > > I would say that sucks, because it makes it harder to set > > maintenance_work_mem correctly. Not sure how hard it would be to fix, > > though. > > ginInsertCleanup() may now be the w

Re: maintenance_work_mem used by Vacuum

2019-10-07 Thread Peter Geoghegan
On Mon, Oct 7, 2019 at 12:28 PM Robert Haas wrote: > I would say that sucks, because it makes it harder to set > maintenance_work_mem correctly. Not sure how hard it would be to fix, > though. ginInsertCleanup() may now be the worst piece of code in the entire tree, so no surprised that it gets

Re: maintenance_work_mem used by Vacuum

2019-10-07 Thread Robert Haas
On Sun, Oct 6, 2019 at 6:55 AM Amit Kapila wrote: > As per docs [1] (see maintenance_work_mem), the maximum amount of memory used > by the Vacuum command must be no more than maintenance_work_mem. However, > during the review/discussion of the "parallel vacuum" patch [2], we observed > that it

maintenance_work_mem used by Vacuum

2019-10-06 Thread Amit Kapila
As per docs [1] (see maintenance_work_mem), the maximum amount of memory used by the Vacuum command must be no more than maintenance_work_mem. However, during the review/discussion of the "parallel vacuum" patch [2], we observed that it is not true. Basically, if there is a gin index defined on a