Re: POC: Parallel processing of indexes in autovacuum

2025-06-18 Thread Daniil Davydov
Hi, On Wed, Jun 18, 2025 at 5:37 AM Masahiko Sawada wrote: > > On Sun, May 25, 2025 at 10:22 AM Daniil Davydov <3daniss...@gmail.com> wrote: > > > > Thanks everybody for feedback! I attach a v4 patch to this letter. > > Main features : > > 1) 'parallel_autovacuum_workers' reloption - integer valu

Re: POC: Parallel processing of indexes in autovacuum

2025-06-17 Thread Masahiko Sawada
On Sun, May 25, 2025 at 10:22 AM Daniil Davydov <3daniss...@gmail.com> wrote: > > Hi, > > On Fri, May 23, 2025 at 6:12 AM Masahiko Sawada wrote: > > > > On Thu, May 22, 2025 at 12:44 AM Daniil Davydov <3daniss...@gmail.com> > > wrote: > > > > > > On Wed, May 21, 2025 at 5:30 AM Masahiko Sawada

Re: POC: Parallel processing of indexes in autovacuum

2025-05-25 Thread Daniil Davydov
Hi, On Fri, May 23, 2025 at 6:12 AM Masahiko Sawada wrote: > > On Thu, May 22, 2025 at 12:44 AM Daniil Davydov <3daniss...@gmail.com> wrote: > > > > On Wed, May 21, 2025 at 5:30 AM Masahiko Sawada > > wrote: > > > > > > I find that the name "autovacuum_reserved_workers_num" is generic. It > > >

Re: POC: Parallel processing of indexes in autovacuum

2025-05-22 Thread Masahiko Sawada
On Thu, May 22, 2025 at 10:48 AM Sami Imseih wrote: > > I started looking at the patch but I have some high level thoughts I would > like to share before looking further. > > > > I find that the name "autovacuum_reserved_workers_num" is generic. It > > > would be better to have a more specific nam

Re: POC: Parallel processing of indexes in autovacuum

2025-05-22 Thread Masahiko Sawada
On Thu, May 22, 2025 at 12:44 AM Daniil Davydov <3daniss...@gmail.com> wrote: > > Hi, > > On Wed, May 21, 2025 at 5:30 AM Masahiko Sawada wrote: > > > > I have some comments on v2-0001 patch > > Thank you for reviewing this patch! > > > + { > > + {"autovacuum_reserved_workers_num", PGC_USE

Re: POC: Parallel processing of indexes in autovacuum

2025-05-22 Thread Sami Imseih
I started looking at the patch but I have some high level thoughts I would like to share before looking further. > > I find that the name "autovacuum_reserved_workers_num" is generic. It > > would be better to have a more specific name for parallel vacuum such > > as autovacuum_max_parallel_worker

Re: POC: Parallel processing of indexes in autovacuum

2025-05-22 Thread Daniil Davydov
Hi, On Wed, May 21, 2025 at 5:30 AM Masahiko Sawada wrote: > > I have some comments on v2-0001 patch Thank you for reviewing this patch! > + { > + {"autovacuum_reserved_workers_num", PGC_USERSET, > RESOURCES_WORKER_PROCESSES, > + gettext_noop("Number of worker processes, reser

Re: POC: Parallel processing of indexes in autovacuum

2025-05-20 Thread Masahiko Sawada
On Thu, May 15, 2025 at 10:10 PM Daniil Davydov <3daniss...@gmail.com> wrote: > > Hi, > > On Fri, May 16, 2025 at 4:06 AM Matheus Alcantara > wrote: > > I've reviewed the v1-0001 patch, the build on MacOS using meson+ninja is > > failing: > > ❯❯❯ ninja -C build install > > ninja: Entering director

Re: POC: Parallel processing of indexes in autovacuum

2025-05-15 Thread Daniil Davydov
Hi, On Fri, May 16, 2025 at 4:06 AM Matheus Alcantara wrote: > I've reviewed the v1-0001 patch, the build on MacOS using meson+ninja is > failing: > ❯❯❯ ninja -C build install > ninja: Entering directory `build' > [1/126] Compiling C object > src/backend/postgres_lib.a.p/utils_misc_guc_tables.c.o

Re: POC: Parallel processing of indexes in autovacuum

2025-05-15 Thread Matheus Alcantara
Hi, On 09/05/25 15:33, Daniil Davydov wrote: > Hi, > As I promised - meet parallel index autovacuum with bgworkers > (Parallel-index-autovacuum-with-bgworkers.patch). This is pretty > simple implementation : > 1) Added new table option `parallel_idx_autovac_enabled` that must be > set to `true` if

Re: POC: Parallel processing of indexes in autovacuum

2025-05-09 Thread Daniil Davydov
Hi, As I promised - meet parallel index autovacuum with bgworkers (Parallel-index-autovacuum-with-bgworkers.patch). This is pretty simple implementation : 1) Added new table option `parallel_idx_autovac_enabled` that must be set to `true` if user wants autovacuum to process table in parallel. 2) Ad

Re: POC: Parallel processing of indexes in autovacuum

2025-05-06 Thread Sami Imseih
> On Mon, May 5, 2025 at 5:21 PM Sami Imseih wrote: > > > > > >> On Sat, May 3, 2025 at 1:10 AM Daniil Davydov <3daniss...@gmail.com> wrote: > >> > > >> > On Sat, May 3, 2025 at 5:28 AM Masahiko Sawada > >> > wrote: > >> > > > >> > > > In current implementation, the leader process sends a signal

Re: POC: Parallel processing of indexes in autovacuum

2025-05-05 Thread Masahiko Sawada
On Mon, May 5, 2025 at 5:21 PM Sami Imseih wrote: > > >> On Sat, May 3, 2025 at 1:10 AM Daniil Davydov <3daniss...@gmail.com> wrote: >> > >> > On Sat, May 3, 2025 at 5:28 AM Masahiko Sawada >> > wrote: >> > > >> > > > In current implementation, the leader process sends a signal to the >> > > > a

Re: POC: Parallel processing of indexes in autovacuum

2025-05-05 Thread Daniil Davydov
On Tue, May 6, 2025 at 7:21 AM Sami Imseih wrote: > > Perhaps we should only provide a reloption, therefore only tables specified > by the user via the reloption can be autovacuumed in parallel? Аfter your comments (earlier in this thread) I decided to do just that. For now we have reloption, so

Re: POC: Parallel processing of indexes in autovacuum

2025-05-05 Thread Daniil Davydov
On Tue, May 6, 2025 at 6:57 AM Masahiko Sawada wrote: > > What I roughly imagined is that we don't need to change the entire > autovacuum scheduling, but would like autovacuum workers to decides > whether or not to use parallel vacuum during its vacuum operation > based on GUC parameters (having a

Re: POC: Parallel processing of indexes in autovacuum

2025-05-05 Thread Sami Imseih
> On Sat, May 3, 2025 at 1:10 AM Daniil Davydov <3daniss...@gmail.com> > wrote: > > > > On Sat, May 3, 2025 at 5:28 AM Masahiko Sawada > wrote: > > > > > > > In current implementation, the leader process sends a signal to the > > > > a/v launcher, and the launcher tries to launch all requested wor

Re: POC: Parallel processing of indexes in autovacuum

2025-05-05 Thread Masahiko Sawada
On Sat, May 3, 2025 at 1:10 AM Daniil Davydov <3daniss...@gmail.com> wrote: > > On Sat, May 3, 2025 at 5:28 AM Masahiko Sawada wrote: > > > > > In current implementation, the leader process sends a signal to the > > > a/v launcher, and the launcher tries to launch all requested workers. > > > But

Re: POC: Parallel processing of indexes in autovacuum

2025-05-03 Thread Daniil Davydov
On Sat, May 3, 2025 at 5:59 AM Sami Imseih wrote: > > > I think it would more make > > sense to maintain the existing autovacuum_max_workers parameter while > > introducing a new parameter that would either control the maximum > > number of parallel vacuum workers per autovacuum worker or set a >

Re: POC: Parallel processing of indexes in autovacuum

2025-05-03 Thread Daniil Davydov
On Sat, May 3, 2025 at 5:28 AM Masahiko Sawada wrote: > > > In current implementation, the leader process sends a signal to the > > a/v launcher, and the launcher tries to launch all requested workers. > > But the number of workers never exceeds `autovacuum_max_workers`. > > Thus, we will never ha

Re: POC: Parallel processing of indexes in autovacuum

2025-05-03 Thread Daniil Davydov
On Sat, May 3, 2025 at 3:17 AM Sami Imseih wrote: > > I think in most cases, the user will want to determine the priority of > a table getting parallel vacuum cycles rather than having the autovacuum > determine the priority. I also see users wanting to stagger > vacuums of large tables with many

Re: POC: Parallel processing of indexes in autovacuum

2025-05-02 Thread Sami Imseih
> I think it would more make > sense to maintain the existing autovacuum_max_workers parameter while > introducing a new parameter that would either control the maximum > number of parallel vacuum workers per autovacuum worker or set a > system-wide cap on the total number of parallel vacuum worker

Re: POC: Parallel processing of indexes in autovacuum

2025-05-02 Thread Masahiko Sawada
On Fri, May 2, 2025 at 11:13 AM Daniil Davydov <3daniss...@gmail.com> wrote: > > On Thu, May 1, 2025 at 8:03 AM Masahiko Sawada wrote: > > > > As I understand it, we initially disabled parallel vacuum for > > autovacuum because their objectives are somewhat contradictory. > > Parallel vacuum aims

Re: POC: Parallel processing of indexes in autovacuum

2025-05-02 Thread Masahiko Sawada
On Fri, May 2, 2025 at 9:58 AM Sami Imseih wrote: > > > Once we have parallel heap vacuum, as discussed in thread[1], it would > > also likely be beneficial to incorporate it into autovacuum during > > aggressive vacuum or failsafe mode. > > IIRC, index cleanup is disabled by failsafe. Yes. My id

Re: POC: Parallel processing of indexes in autovacuum

2025-05-02 Thread Sami Imseih
> On Fri, May 2, 2025 at 11:58 PM Sami Imseih wrote: > > > > I am generally -1 on the idea of autovacuum performing parallel > > index vacuum, because I always felt that the parallel option should > > be employed in a targeted manner for a specific table. if you have a bunch > > of large tables, s

Re: POC: Parallel processing of indexes in autovacuum

2025-05-02 Thread Daniil Davydov
On Fri, May 2, 2025 at 11:58 PM Sami Imseih wrote: > > I am generally -1 on the idea of autovacuum performing parallel > index vacuum, because I always felt that the parallel option should > be employed in a targeted manner for a specific table. if you have a bunch > of large tables, some more imp

Re: POC: Parallel processing of indexes in autovacuum

2025-05-02 Thread Daniil Davydov
On Thu, May 1, 2025 at 8:03 AM Masahiko Sawada wrote: > > As I understand it, we initially disabled parallel vacuum for > autovacuum because their objectives are somewhat contradictory. > Parallel vacuum aims to accelerate the process by utilizing additional > resources, while autovacuum is design

Re: POC: Parallel processing of indexes in autovacuum

2025-05-02 Thread Sami Imseih
Thanks for raising this idea! I am generally -1 on the idea of autovacuum performing parallel index vacuum, because I always felt that the parallel option should be employed in a targeted manner for a specific table. if you have a bunch of large tables, some more important than others, a/c may end

Re: POC: Parallel processing of indexes in autovacuum

2025-04-30 Thread Masahiko Sawada
Hi, On Wed, Apr 16, 2025 at 4:05 AM Maxim Orlov wrote: > > Hi! > > The VACUUM command can be executed with the parallel option. As documentation > states, it will perform index vacuum and index cleanup phases of VACUUM in > parallel using integer background workers. But such an interesting feat

Re: POC: Parallel processing of indexes in autovacuum

2025-04-17 Thread wenhui qiu
HI *Maxim Orlov* Thank you for your working on this ,I like your idea ,but I have a suggestion ,autovacuum_max_workers is not need change requires restart , I think those guc are can like autovacuum_max_workers +#max_parallel_index_autovac_workers = 0 # this feature disabled by default + # (c