Re: CREATE INDEX CONCURRENTLY on partitioned index

2024-10-25 Thread Sergey Sargsyan
I noticed that development on the concurrent index creation for partitioned tables feature seemed to stall a few months ago. The patch looked solid, and there didn’t seem to be any issues with it. Has there been any further progress? This feature would be invaluable, given the prevalence of partiti

Re: CREATE INDEX CONCURRENTLY on partitioned index

2024-07-14 Thread Michael Paquier
On Fri, Jul 12, 2024 at 11:17:25PM +0100, Ilya Gladyshev wrote: > Sure, created a separate thread [1]. Please disregard the second patch in > this thread. Duplicating the last version of the relevant patch here to > avoid any confusion. > > [1] > https://www.postgresql.org/message-id/b72f2d89-820

Re: CREATE INDEX CONCURRENTLY on partitioned index

2024-07-12 Thread Ilya Gladyshev
On 12.07.2024 01:01, Michael Paquier wrote: Please let's move this point to its own thread and deal with it with an independent patch. Hiding that in a thread that's already quite long is not a good idea. This needs proper review, and a separate thread with a good subject to describe the probl

Re: CREATE INDEX CONCURRENTLY on partitioned index

2024-07-11 Thread Michael Paquier
On Thu, Jul 11, 2024 at 09:35:24PM +0100, Ilya Gladyshev wrote: > It is broken in master, I just didn’t want to create a separate > thread, but it can be fixed independently. As I remember, the > problem is that progress is tracked for each table in the hierarchy > as if the table is processed sepa

Re: CREATE INDEX CONCURRENTLY on partitioned index

2024-07-11 Thread Ilya Gladyshev
It is broken in master, I just didn’t want to create a separate thread, but it can be fixed independently. As I remember, the problem is that progress is tracked for each table in the hierarchy as if the table is processed separately, without ever setting partitions_total and partitions_done cou

Re: CREATE INDEX CONCURRENTLY on partitioned index

2024-07-11 Thread Justin Pryzby
On Sat, Jun 15, 2024 at 07:56:38PM +0100, Ilya Gladyshev wrote: > In addition, I noticed that progress tracking is once again broken for > partitioned tables, while looking at REINDEX implementation, attaching the > second patch to fix it. Thanks for the fixes, I started reviewing them but need so

Re: CREATE INDEX CONCURRENTLY on partitioned index

2024-06-20 Thread Ilya Gladyshev
On 15.06.2024 20:40, Justin Pryzby wrote: On Thu, May 23, 2024 at 10:14:57PM +0100, Ilya Gladyshev wrote: Hi, I think it's well worth the effort to revive the patch, so I rebased it on master, updated it and will return it back to the commitfest. Alexander, Justin feel free to add yourselves a

Re: CREATE INDEX CONCURRENTLY on partitioned index

2024-06-15 Thread Justin Pryzby
On Thu, May 23, 2024 at 10:14:57PM +0100, Ilya Gladyshev wrote: > Hi, > > I think it's well worth the effort to revive the patch, so I rebased it on > master, updated it and will return it back to the commitfest. Alexander, > Justin feel free to add yourselves as authors Thanks -- I was intending

Re: CREATE INDEX CONCURRENTLY on partitioned index

2024-06-15 Thread Ilya Gladyshev
On 28.05.2024 07:05, Alexander Pyhalov wrote: Ilya Gladyshev писал(а) 2024-05-28 02:52: Also I'd like to note that in new patch version there's a strange wording in documentation: "This can be very convenient as not only will all existing partitions be  indexed, but any future partitions w

Re: CREATE INDEX CONCURRENTLY on partitioned index

2024-05-27 Thread Alexander Pyhalov
Ilya Gladyshev писал(а) 2024-05-28 02:52: Also I'd like to note that in new patch version there's a strange wording in documentation: "This can be very convenient as not only will all existing partitions be  indexed, but any future partitions will be as well.  CREATE INDEX ... CONCURRENTLY c

Re: CREATE INDEX CONCURRENTLY on partitioned index

2024-05-27 Thread Ilya Gladyshev
On 24.05.2024 10:04, Alexander Pyhalov wrote: Ilya Gladyshev писал(а) 2024-05-24 00:14: Hi, Hi. I think it's well worth the effort to revive the patch, so I rebased it on master, updated it and will return it back to the commitfest. Alexander, Justin feel free to add yourselves as author

Re: CREATE INDEX CONCURRENTLY on partitioned index

2024-05-24 Thread Alexander Pyhalov
Ilya Gladyshev писал(а) 2024-05-24 00:14: Hi, Hi. I think it's well worth the effort to revive the patch, so I rebased it on master, updated it and will return it back to the commitfest. Alexander, Justin feel free to add yourselves as authors On 29.01.2024 12:43, Alexander Pyhalov wrote

Re: CREATE INDEX CONCURRENTLY on partitioned index

2024-05-23 Thread Ilya Gladyshev
Hi, I think it's well worth the effort to revive the patch, so I rebased it on master, updated it and will return it back to the commitfest. Alexander, Justin feel free to add yourselves as authors On 29.01.2024 12:43, Alexander Pyhalov wrote: Hi. I've rebased patch on master and it'seems t

Re: CREATE INDEX CONCURRENTLY on partitioned index

2024-01-29 Thread Alexander Pyhalov
Hi. I've rebased patch on master and it'seems to me there's one more issue - when we call DefineIndexConcurrentInternal() in partitioned case, it waits for transactions, locking tableId, not tabrelid - heaprelid LockRelId is constructed for parent index relation, not for child index relation.

Re: CREATE INDEX CONCURRENTLY on partitioned index

2023-07-12 Thread Alexander Pyhalov
Justin Pryzby писал 2023-07-13 05:27: On Mon, Mar 27, 2023 at 01:28:24PM +0300, Alexander Pyhalov wrote: Justin Pryzby писал 2023-03-26 17:51: > On Sun, Dec 04, 2022 at 01:09:35PM -0600, Justin Pryzby wrote: > > This currently handles partitions with a loop around the whole CIC > > implementatio

Re: CREATE INDEX CONCURRENTLY on partitioned index

2023-07-12 Thread Justin Pryzby
On Mon, Mar 27, 2023 at 01:28:24PM +0300, Alexander Pyhalov wrote: > Justin Pryzby писал 2023-03-26 17:51: > > On Sun, Dec 04, 2022 at 01:09:35PM -0600, Justin Pryzby wrote: > > > This currently handles partitions with a loop around the whole CIC > > > implementation, which means that things like W

Re: CREATE INDEX CONCURRENTLY on partitioned index

2023-03-27 Thread Alexander Pyhalov
Justin Pryzby писал 2023-03-26 17:51: On Sun, Dec 04, 2022 at 01:09:35PM -0600, Justin Pryzby wrote: This currently handles partitions with a loop around the whole CIC implementation, which means that things like WaitForLockers() happen once for each index, the same as REINDEX CONCURRENTLY on a

Re: CREATE INDEX CONCURRENTLY on partitioned index

2023-03-26 Thread Justin Pryzby
On Sun, Dec 04, 2022 at 01:09:35PM -0600, Justin Pryzby wrote: > This currently handles partitions with a loop around the whole CIC > implementation, which means that things like WaitForLockers() happen > once for each index, the same as REINDEX CONCURRENTLY on a partitioned > table. Contrast that

Re: CREATE INDEX CONCURRENTLY on partitioned index

2022-12-09 Thread Ilya Gladyshev
On Sun, 2022-12-04 at 13:09 -0600, Justin Pryzby wrote: > > This beavior is fixed.  I re-factored and re-implented to use > DefineIndex() for building indexes concurrently rather than > reindexing. > That makes the patch smaller, actually, and has the added benefit of > splitting off the "Concurre

Re: CREATE INDEX CONCURRENTLY on partitioned index

2022-12-04 Thread Justin Pryzby
On Sat, Dec 03, 2022 at 07:13:30PM +0400, Ilya Gladyshev wrote: > Hi, > > Thank you Justin and Alexander for working on this, I have reviewed and > tested the latest patch, it works well, the problems mentioned > previously are all fixed. I like the idea of sharing code of reindex > and index, but

Re: CREATE INDEX CONCURRENTLY on partitioned index

2022-12-03 Thread Ilya Gladyshev
Hi, Thank you Justin and Alexander for working on this, I have reviewed and tested the latest patch, it works well, the problems mentioned previously are all fixed. I like the idea of sharing code of reindex and index, but I have noticed some peculiarities as a user.  The reporting is somewhat co

Re: CREATE INDEX CONCURRENTLY on partitioned index

2022-11-21 Thread Alexander Pyhalov
Justin Pryzby писал 2022-11-21 06:00: I finally found time to digest and integrate your changes into my local branch. This fixes the three issues you reported: FORCE_RELEASE, issue with INVALID partitions issue (for which I adapted your patch into an earlier patch in my series), and progress rep

Re: CREATE INDEX CONCURRENTLY on partitioned index

2022-11-20 Thread Justin Pryzby
I finally found time to digest and integrate your changes into my local branch. This fixes the three issues you reported: FORCE_RELEASE, issue with INVALID partitions issue (for which I adapted your patch into an earlier patch in my series), and progress reporting. And rebased. -- Justin >From

Re: CREATE INDEX CONCURRENTLY on partitioned index

2022-06-29 Thread Alexander Pyhalov
Justin Pryzby писал 2022-06-28 21:33: Hi, On Thu, Feb 10, 2022 at 06:07:08PM +0300, Alexander Pyhalov wrote: I've rebased patches and tried to fix issues I've seen. I've fixed reference after table_close() in the first patch (can be seen while building with CPPFLAGS='-DRELCACHE_FORCE_RELEASE'

Re: CREATE INDEX CONCURRENTLY on partitioned index

2022-06-28 Thread Alexander Pyhalov
Justin Pryzby писал 2022-06-28 21:33: Hi, On Thu, Feb 10, 2022 at 06:07:08PM +0300, Alexander Pyhalov wrote: I've rebased patches and tried to fix issues I've seen. I've fixed reference after table_close() in the first patch (can be seen while building with CPPFLAGS='-DRELCACHE_FORCE_RELEASE'

Re: CREATE INDEX CONCURRENTLY on partitioned index

2022-06-28 Thread Justin Pryzby
Hi, On Thu, Feb 10, 2022 at 06:07:08PM +0300, Alexander Pyhalov wrote: > I've rebased patches and tried to fix issues I've seen. I've fixed reference > after table_close() in the first patch (can be seen while building with > CPPFLAGS='-DRELCACHE_FORCE_RELEASE'). Thanks for finding that. The pat

Re: CREATE INDEX CONCURRENTLY on partitioned index

2022-03-25 Thread Justin Pryzby
On Fri, Mar 25, 2022 at 01:05:49AM -0400, Greg Stark wrote: > This patch is marked "waiting on author" in the CF. However the most > recent emails have patches and it's not clear to me what's left from > previous reviews that might not be addressed yet. Should this patch be > marked "Needs Review"?

Re: CREATE INDEX CONCURRENTLY on partitioned index

2022-03-24 Thread Greg Stark
This patch is marked "waiting on author" in the CF. However the most recent emails have patches and it's not clear to me what's left from previous reviews that might not be addressed yet. Should this patch be marked "Needs Review"? Anastasia and Alexander are marked as reviewers. Are you still abl

Re: CREATE INDEX CONCURRENTLY on partitioned index

2022-03-01 Thread Alexander Pyhalov
Hi. I've added 0005-Mark-intermediate-partitioned-indexes-as-valid.patch which fixed the following issues - when partitioned index is created, indexes on intermediate partitioned tables were preserved in invalid state. Also added some more tests. -- Best regards, Alexander Pyhalov, Postgres P

Re: CREATE INDEX CONCURRENTLY on partitioned index

2022-02-10 Thread Alexander Pyhalov
Justin Pryzby писал 2021-02-26 21:20: On Mon, Feb 15, 2021 at 10:07:05PM +0300, Anastasia Lubennikova wrote: 5) Speaking of documentation, I think we need to add a paragraph about CIC on partitioned indexes which will explain that invalid indexes may appear and what user should do to fix them.

CREATE INDEX CONCURRENTLY on partitioned index

2022-02-09 Thread Alexander Pyhalov
Alexander Pyhalov писал 2022-02-09 15:18: Hi. I've looked at patches, introducing CREATE INDEX CONCURRENTLY for partitioned tables - https://www.postgresql.org/message-id/flat/20210226182019.GU20769%40telsasoft.com#da169a0a518bf8121604437d9ab053b3 . The thread didn't have any activity for a year

Re: CREATE INDEX CONCURRENTLY on partitioned index

2021-02-26 Thread Justin Pryzby
On Mon, Feb 15, 2021 at 10:07:05PM +0300, Anastasia Lubennikova wrote: > 5) Speaking of documentation, I think we need to add a paragraph about CIC > on partitioned indexes which will explain that invalid indexes may appear > and what user should do to fix them. I'm not sure about that - it's alre

Re: CREATE INDEX CONCURRENTLY on partitioned index

2021-02-15 Thread Zhihong Yu
Hi, For v13-0006-More-refactoring.patch : + /* It's not a shared catalog, so refuse to move it to shared tablespace */ + if (params->tablespaceOid == GLOBALTABLESPACE_OID && false) + ereport(ERROR, Do you intend to remove the ineffective check ? + else + heapRelation =

Re: CREATE INDEX CONCURRENTLY on partitioned index

2021-02-15 Thread Justin Pryzby
On Mon, Feb 15, 2021 at 10:06:47PM +0300, Anastasia Lubennikova wrote: > On 28.01.2021 17:30, Justin Pryzby wrote: > > On Thu, Jan 28, 2021 at 09:51:51PM +0900, Masahiko Sawada wrote: > > > On Mon, Nov 30, 2020 at 5:22 AM Justin Pryzby > > > wrote: > > > > On Sat, Oct 31, 2020 at 01:31:17AM -0500

Re: CREATE INDEX CONCURRENTLY on partitioned index

2021-02-15 Thread Anastasia Lubennikova
On 28.01.2021 17:30, Justin Pryzby wrote: On Thu, Jan 28, 2021 at 09:51:51PM +0900, Masahiko Sawada wrote: On Mon, Nov 30, 2020 at 5:22 AM Justin Pryzby wrote: On Sat, Oct 31, 2020 at 01:31:17AM -0500, Justin Pryzby wrote: Forking this thread, since the existing CFs have been closed. https://

Re: CREATE INDEX CONCURRENTLY on partitioned index

2021-02-15 Thread Anastasia Lubennikova
On 28.01.2021 17:30, Justin Pryzby wrote: On Thu, Jan 28, 2021 at 09:51:51PM +0900, Masahiko Sawada wrote: On Mon, Nov 30, 2020 at 5:22 AM Justin Pryzby wrote: On Sat, Oct 31, 2020 at 01:31:17AM -0500, Justin Pryzby wrote: Forking this thread, since the existing CFs have been closed. https://

Re: CREATE INDEX CONCURRENTLY on partitioned index

2021-01-28 Thread Justin Pryzby
On Thu, Jan 28, 2021 at 09:51:51PM +0900, Masahiko Sawada wrote: > On Mon, Nov 30, 2020 at 5:22 AM Justin Pryzby wrote: > > On Sat, Oct 31, 2020 at 01:31:17AM -0500, Justin Pryzby wrote: > > > Forking this thread, since the existing CFs have been closed. > > > https://www.postgresql.org/message-id

Re: CREATE INDEX CONCURRENTLY on partitioned index

2021-01-28 Thread Masahiko Sawada
On Mon, Nov 30, 2020 at 5:22 AM Justin Pryzby wrote: > > On Sat, Oct 31, 2020 at 01:31:17AM -0500, Justin Pryzby wrote: > > Forking this thread, since the existing CFs have been closed. > > https://www.postgresql.org/message-id/flat/20200914143102.GX18552%40telsasoft.com#58b1056488451f8594b0f0ba40

Re: CREATE INDEX CONCURRENTLY on partitioned index

2020-11-29 Thread Justin Pryzby
On Sat, Oct 31, 2020 at 01:31:17AM -0500, Justin Pryzby wrote: > Forking this thread, since the existing CFs have been closed. > https://www.postgresql.org/message-id/flat/20200914143102.GX18552%40telsasoft.com#58b1056488451f8594b0f0ba40996afd > > The strategy is to create catalog entries for all

CREATE INDEX CONCURRENTLY on partitioned index

2020-10-30 Thread Justin Pryzby
Forking this thread, since the existing CFs have been closed. https://www.postgresql.org/message-id/flat/20200914143102.GX18552%40telsasoft.com#58b1056488451f8594b0f0ba40996afd The strategy is to create catalog entries for all tables with indisvalid=false, and then process them like REINDEX CONCUR