Re: generic plans and "initial" pruning

2025-02-24 Thread Amit Langote
On Sun, Feb 23, 2025 at 9:46 PM Tender Wang wrote: > Amit Langote 于2025年2月23日周日 16:36写道: >> On Sun, Feb 23, 2025 at 2:03 AM Tender Wang wrote: >> > Alexander Lakhin 于2025年2月22日周六 23:00写道: >> >> Please look at new error, produced by the following script, >> >> starting from 525392d57: >> >> CREA

Re: generic plans and "initial" pruning

2025-02-23 Thread Tender Wang
Amit Langote 于2025年2月23日周日 16:36写道: > On Sun, Feb 23, 2025 at 2:03 AM Tender Wang wrote: > > Alexander Lakhin 于2025年2月22日周六 23:00写道: > >> 21.02.2025 05:40, Amit Langote wrote: > >> > >> I pushed the final piece yesterday. > >> > >> > >> Please look at new error, produced by the following script

Re: generic plans and "initial" pruning

2025-02-23 Thread Amit Langote
On Sun, Feb 23, 2025 at 2:03 AM Tender Wang wrote: > Alexander Lakhin 于2025年2月22日周六 23:00写道: >> 21.02.2025 05:40, Amit Langote wrote: >> >> I pushed the final piece yesterday. >> >> >> Please look at new error, produced by the following script, >> starting from 525392d57: >> CREATE TABLE t(id int

Re: generic plans and "initial" pruning

2025-02-22 Thread Tender Wang
Alexander Lakhin 于2025年2月22日周六 23:00写道: > Hello Amit, > > 21.02.2025 05:40, Amit Langote wrote: > > I pushed the final piece yesterday. > > > Please look at new error, produced by the following script, > starting from 525392d57: > CREATE TABLE t(id int) PARTITION BY RANGE (id); > CREATE INDEX idx

Re: generic plans and "initial" pruning

2025-02-22 Thread Alexander Lakhin
Hello Amit, 21.02.2025 05:40, Amit Langote wrote: I pushed the final piece yesterday. Please look at new error, produced by the following script, starting from 525392d57: CREATE TABLE t(id int) PARTITION BY RANGE (id); CREATE INDEX idx on t(id); CREATE TABLE tp_1 PARTITION OF t FOR VALUES FROM

Re: generic plans and "initial" pruning

2025-02-21 Thread Amit Langote
On Sat, Feb 22, 2025 at 11:13 AM Amit Langote wrote: > On Sat, Feb 22, 2025 at 12:55 AM Tom Lane wrote: > > Amit Langote writes: > > > The short of it is that the cached-plan-inval test in the > > > delay_execution suite can never be made to work under > > > CLOBBER_CACHE_ALWAYS. The test assume

Re: generic plans and "initial" pruning

2025-02-21 Thread Amit Langote
On Sat, Feb 22, 2025 at 12:55 AM Tom Lane wrote: > Amit Langote writes: > > The short of it is that the cached-plan-inval test in the > > delay_execution suite can never be made to work under > > CLOBBER_CACHE_ALWAYS. The test assumes that locks on partitions for a > > reused generic plan are not

Re: generic plans and "initial" pruning

2025-02-21 Thread Tom Lane
Amit Langote writes: > The short of it is that the cached-plan-inval test in the > delay_execution suite can never be made to work under > CLOBBER_CACHE_ALWAYS. The test assumes that locks on partitions for a > reused generic plan are not taken until InitPlan(). However, under > CLOBBER_CACHE_ALWA

Re: generic plans and "initial" pruning

2025-02-21 Thread Amit Langote
On Fri, Feb 21, 2025 at 3:36 PM Amit Langote wrote: > On Fri, Feb 21, 2025 at 3:04 PM Tom Lane wrote: > > > > Amit Langote writes: > > > I pushed the final piece yesterday. > > > > trilobite reports that this fails under -DCLOBBER_CACHE_ALWAYS: > > > > https://buildfarm.postgresql.org/cgi-bin/sh

Re: generic plans and "initial" pruning

2025-02-20 Thread Tom Lane
Amit Langote writes: > I pushed the final piece yesterday. trilobite reports that this fails under -DCLOBBER_CACHE_ALWAYS: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=trilobite&dt=2025-02-20%2019%3A37%3A12 regards, tom lane

Re: generic plans and "initial" pruning

2025-02-20 Thread Amit Langote
On Fri, Feb 21, 2025 at 3:04 PM Tom Lane wrote: > > Amit Langote writes: > > I pushed the final piece yesterday. > > trilobite reports that this fails under -DCLOBBER_CACHE_ALWAYS: > > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=trilobite&dt=2025-02-20%2019%3A37%3A12 Looking, thanks

Re: generic plans and "initial" pruning

2025-02-20 Thread Amit Langote
On Wed, Feb 12, 2025 at 8:53 PM Amit Langote wrote: > On Thu, Feb 6, 2025 at 11:35 AM Amit Langote wrote: > > Per cfbot-ci, the new test case output in 0002 needed to be updated. > > > > I plan to push 0001 tomorrow, barring any objections. > > I pushed that last Friday. With bb3ec16e, d47cbf47,

Re: generic plans and "initial" pruning

2025-02-16 Thread Amit Langote
Hi Junwang, On Sun, Feb 16, 2025 at 1:37 PM Junwang Zhao wrote: > On Sat, Feb 15, 2025 at 3:51 PM Amit Langote wrote: > > Thanks! It looks like I missed updating the MERGE-related lists in > > ModifyTable. > > > > I've attached a fix with a test added based on your example. I plan to > > push t

Re: generic plans and "initial" pruning

2025-02-15 Thread Junwang Zhao
Hi Amit, On Sat, Feb 15, 2025 at 3:51 PM Amit Langote wrote: > > Hi Alexander, > > On Sat, Feb 15, 2025 at 6:00 AM Alexander Lakhin wrote: > > > > Hello Amit, > > > > 06.02.2025 04:35, Amit Langote wrote: > > > > I plan to push 0001 tomorrow, barring any objections. > > > > > > Please try the fo

Re: generic plans and "initial" pruning

2025-02-14 Thread Amit Langote
Hi Alexander, On Sat, Feb 15, 2025 at 6:00 AM Alexander Lakhin wrote: > > Hello Amit, > > 06.02.2025 04:35, Amit Langote wrote: > > I plan to push 0001 tomorrow, barring any objections. > > > Please try the following script: > CREATE TABLE pt (a int, b int) PARTITION BY range (a); > CREATE TABLE

Re: generic plans and "initial" pruning

2025-02-14 Thread Alexander Lakhin
Hello Amit, 06.02.2025 04:35, Amit Langote wrote: I plan to push 0001 tomorrow, barring any objections. Please try the following script: CREATE TABLE pt (a int, b int) PARTITION BY range (a); CREATE TABLE tp1 PARTITION OF pt FOR VALUES FROM (1) TO (2); CREATE TABLE tp2 PARTITION OF pt FOR VAL

Re: generic plans and "initial" pruning

2025-02-12 Thread Amit Langote
On Thu, Feb 6, 2025 at 11:35 AM Amit Langote wrote: > Per cfbot-ci, the new test case output in 0002 needed to be updated. > > I plan to push 0001 tomorrow, barring any objections. I pushed that last Friday. With bb3ec16e, d47cbf47, and cbc12791 now in: * Pruning information is now stored separa

Re: generic plans and "initial" pruning

2024-12-09 Thread Amit Langote
On Fri, Dec 6, 2024 at 5:18 PM Amit Langote wrote: > On Thu, Dec 5, 2024 at 11:07 PM Tomas Vondra wrote: > > On 12/5/24 12:28, Amit Langote wrote: > > > On Thu, Dec 5, 2024 at 3:53 PM Amit Langote > > > wrote: > > >> On Thu, Dec 5, 2024 at 2:20 AM Tomas Vondra wrote: > > >>> Sure, changing the

Re: generic plans and "initial" pruning

2024-12-06 Thread Amit Langote
On Thu, Dec 5, 2024 at 10:53 PM Tomas Vondra wrote: > On 12/5/24 07:53, Amit Langote wrote: > > On Thu, Dec 5, 2024 at 2:20 AM Tomas Vondra wrote: > >> ... > >> > What if an > extension doesn't do that? What weirdness will happen? > >>> > >>> The QueryDesc.planstate won't contain a Plan

Re: generic plans and "initial" pruning

2024-12-06 Thread Amit Langote
On Thu, Dec 5, 2024 at 11:07 PM Tomas Vondra wrote: > On 12/5/24 12:28, Amit Langote wrote: > > On Thu, Dec 5, 2024 at 3:53 PM Amit Langote wrote: > >> On Thu, Dec 5, 2024 at 2:20 AM Tomas Vondra wrote: > >>> Sure, changing the APIs is allowed, I'm just wondering if maybe there > >>> might be a

Re: generic plans and "initial" pruning

2024-12-05 Thread Tomas Vondra
On 12/5/24 12:28, Amit Langote wrote: > On Thu, Dec 5, 2024 at 3:53 PM Amit Langote wrote: >> On Thu, Dec 5, 2024 at 2:20 AM Tomas Vondra wrote: >>> Sure, changing the APIs is allowed, I'm just wondering if maybe there >>> might be a way to not have this issue, or at least notice the missing >

Re: generic plans and "initial" pruning

2024-12-05 Thread Tomas Vondra
On 12/5/24 07:53, Amit Langote wrote: > On Thu, Dec 5, 2024 at 2:20 AM Tomas Vondra wrote: >> ... >> What if an extension doesn't do that? What weirdness will happen? >>> >>> The QueryDesc.planstate won't contain a PlanState tree for starters >>> and other state information that InitP

Re: generic plans and "initial" pruning

2024-12-05 Thread Amit Langote
On Thu, Dec 5, 2024 at 3:53 PM Amit Langote wrote: > On Thu, Dec 5, 2024 at 2:20 AM Tomas Vondra wrote: > > Sure, changing the APIs is allowed, I'm just wondering if maybe there > > might be a way to not have this issue, or at least notice the missing > > call early. > > > > I haven't tried, woul

Re: generic plans and "initial" pruning

2024-12-04 Thread Amit Langote
On Thu, Dec 5, 2024 at 2:20 AM Tomas Vondra wrote: > On 12/4/24 14:34, Amit Langote wrote: > > On Mon, Dec 2, 2024 at 3:36 AM Tomas Vondra wrote: > >> 0001 > >> > >> > >> 1) But if we don't expect this error to actually happen, do we really > >> need to make it ereport()? Maybe it should be

Re: generic plans and "initial" pruning

2024-12-04 Thread Amit Langote
On Thu, Dec 5, 2024 at 2:32 AM Tom Lane wrote: > Tomas Vondra writes: > > I'm not forcing you to do elog, if you think ereport() is better. I'm > > only asking because AFAIK the "policy" is that ereport is for cases that > > think can happen (and thus get translated), while elog(ERROR) is for > >

Re: generic plans and "initial" pruning

2024-12-04 Thread Tom Lane
Tomas Vondra writes: > I'm not forcing you to do elog, if you think ereport() is better. I'm > only asking because AFAIK the "policy" is that ereport is for cases that > think can happen (and thus get translated), while elog(ERROR) is for > cases that we believe shouldn't happen. The proposed cod

Re: generic plans and "initial" pruning

2024-12-04 Thread Tomas Vondra
On 12/4/24 14:34, Amit Langote wrote: > Hi Tomas, > > On Mon, Dec 2, 2024 at 3:36 AM Tomas Vondra wrote: >> Hi, >> >> I took a look at this patch, mostly to familiarize myself with the >> pruning etc. I have a bunch of comments, but all of that is minor, >> perhaps even nitpicking - with prior

Re: generic plans and "initial" pruning

2024-12-04 Thread Amit Langote
Hi Tomas, On Mon, Dec 2, 2024 at 3:36 AM Tomas Vondra wrote: > Hi, > > I took a look at this patch, mostly to familiarize myself with the > pruning etc. I have a bunch of comments, but all of that is minor, > perhaps even nitpicking - with prior feedback from David, Tom and > Robert, I can't real

Re: generic plans and "initial" pruning

2024-10-15 Thread Tom Lane
Robert Haas writes: > On Fri, Oct 11, 2024 at 3:30 AM Amit Langote wrote: >> Maybe just "relids" suffices with a comment updated like this: >> >> * relids RelOptInfo.relids of the parent plan node (e.g. Append >> * or MergeAppend) to which his PartitionPruneInf

Re: generic plans and "initial" pruning

2024-10-15 Thread Robert Haas
On Fri, Oct 11, 2024 at 3:30 AM Amit Langote wrote: > Maybe just "relids" suffices with a comment updated like this: > > * relids RelOptInfo.relids of the parent plan node (e.g. Append > * or MergeAppend) to which his PartitionPruneInfo node > *

Re: generic plans and "initial" pruning

2024-10-11 Thread Amit Langote
Robert, On Fri, Oct 11, 2024 at 5:15 AM Robert Haas wrote: > > Hi Amit, > > This is not a full review (sorry!) but here are a few comments. Thank you for taking a look. > In general, I don't have a problem with this direction. I thought > Tom's previous proposal of abandoning ExecInitNode() in

Re: generic plans and "initial" pruning

2024-10-10 Thread Robert Haas
Hi Amit, This is not a full review (sorry!) but here are a few comments. In general, I don't have a problem with this direction. I thought Tom's previous proposal of abandoning ExecInitNode() in medias res if we discover that we need to replan was doable and I still think that, but ISTM that this

Re: generic plans and "initial" pruning

2024-09-02 Thread Amit Langote
On Sat, Aug 31, 2024 at 9:30 PM Junwang Zhao wrote: > @@ -1241,7 +1244,7 @@ GetCachedPlan(CachedPlanSource *plansource, > ParamListInfo boundParams, > if (customplan) > { > /* Build a custom plan */ > - plan = BuildCachedPlan(plansource, qlist, boundParams, queryEnv); > + plan = BuildCachedP

Re: generic plans and "initial" pruning

2024-08-31 Thread Junwang Zhao
Hi, On Thu, Aug 29, 2024 at 9:34 PM Amit Langote wrote: > > On Fri, Aug 23, 2024 at 9:48 PM Amit Langote wrote: > > On Wed, Aug 21, 2024 at 10:10 PM Robert Haas wrote: > > > On Wed, Aug 21, 2024 at 8:45 AM Amit Langote > > > wrote: > > > > * The replanning aspect of the lock-in-the-executor d

Re: generic plans and "initial" pruning

2024-08-23 Thread Amit Langote
On Wed, Aug 21, 2024 at 10:10 PM Robert Haas wrote: > On Wed, Aug 21, 2024 at 8:45 AM Amit Langote wrote: > > * The replanning aspect of the lock-in-the-executor design would be > > simpler if a CachedPlan contained the plan for a single query rather > > than a list of queries, as previously ment

Re: generic plans and "initial" pruning

2024-08-21 Thread Robert Haas
On Wed, Aug 21, 2024 at 8:45 AM Amit Langote wrote: > * The replanning aspect of the lock-in-the-executor design would be > simpler if a CachedPlan contained the plan for a single query rather > than a list of queries, as previously mentioned. This is particularly > due to the requirements of the

Re: generic plans and "initial" pruning

2024-08-21 Thread Amit Langote
On Tue, Aug 20, 2024 at 11:53 PM Robert Haas wrote: > On Tue, Aug 20, 2024 at 9:00 AM Amit Langote wrote: > > I think we'd modify plancache.c to postpone the locking of only > > prunable relations (i.e., partitions), so we're looking at only a > > handful of concurrent modifications that are goin

Re: generic plans and "initial" pruning

2024-08-20 Thread Robert Haas
On Tue, Aug 20, 2024 at 9:00 AM Amit Langote wrote: > I think we'd modify plancache.c to postpone the locking of only > prunable relations (i.e., partitions), so we're looking at only a > handful of concurrent modifications that are going to cause execution > errors. That's because we disallow ma

Re: generic plans and "initial" pruning

2024-08-20 Thread Amit Langote
On Tue, Aug 20, 2024 at 3:21 AM Robert Haas wrote: > On Mon, Aug 19, 2024 at 1:52 PM Tom Lane wrote: > > Robert Haas writes: > > > But that seems somewhat incidental to what this thread is about. > > > > Perhaps. But if we're running into issues related to that, it might > > be good to set asid

Re: generic plans and "initial" pruning

2024-08-20 Thread Amit Langote
On Tue, Aug 20, 2024 at 1:39 AM Robert Haas wrote: > On Fri, Aug 16, 2024 at 8:36 AM Amit Langote wrote: > > So it is possible for the executor to try to run a plan that has > > become invalid since it was created, so... > > I'm not sure what the "so what" here is. I meant that if the executor h

Re: generic plans and "initial" pruning

2024-08-19 Thread Robert Haas
On Mon, Aug 19, 2024 at 1:52 PM Tom Lane wrote: > Robert Haas writes: > > But that seems somewhat incidental to what this thread is about. > > Perhaps. But if we're running into issues related to that, it might > be good to set aside the long-term goal for a bit and come up with > a cleaner answ

Re: generic plans and "initial" pruning

2024-08-19 Thread Tom Lane
Robert Haas writes: > But that seems somewhat incidental to what this thread is about. Perhaps. But if we're running into issues related to that, it might be good to set aside the long-term goal for a bit and come up with a cleaner answer for intra-session locking. That could allow the pruning

Re: generic plans and "initial" pruning

2024-08-19 Thread Robert Haas
On Mon, Aug 19, 2024 at 12:54 PM Tom Lane wrote: > What the examples here are showing is that AcquireExecutorLocks > is incomplete because it only provides defenses against DDL > initiated by other sessions, not by our own session. We have > CheckTableNotInUse but I'm not sure if it could be appl

Re: generic plans and "initial" pruning

2024-08-19 Thread Tom Lane
Robert Haas writes: > On Fri, Aug 16, 2024 at 8:36 AM Amit Langote wrote: >> So it is possible for the executor to try to run a plan that has >> become invalid since it was created, so... > I'm not sure what the "so what" here is. The fact that there are holes in our protections against that do

Re: generic plans and "initial" pruning

2024-08-19 Thread Robert Haas
On Fri, Aug 16, 2024 at 8:36 AM Amit Langote wrote: > So it is possible for the executor to try to run a plan that has > become invalid since it was created, so... I'm not sure what the "so what" here is. > One perhaps crazy idea [1]: > > What if we remove AcquireExecutorLocks() and move the res

Re: generic plans and "initial" pruning

2024-08-16 Thread Amit Langote
On Fri, Aug 16, 2024 at 12:35 AM Robert Haas wrote: > On Thu, Aug 15, 2024 at 8:57 AM Amit Langote wrote: > > TBH, it's more of a hunch that people who are not involved in this > > development might find the new reality, whereby the execution is not > > racefree until ExecutorRun(), hard to reaso

Re: generic plans and "initial" pruning

2024-08-15 Thread Robert Haas
On Thu, Aug 15, 2024 at 8:57 AM Amit Langote wrote: > TBH, it's more of a hunch that people who are not involved in this > development might find the new reality, whereby the execution is not > racefree until ExecutorRun(), hard to reason about. I'm confused by what you mean here by "racefree". A

Re: generic plans and "initial" pruning

2024-08-14 Thread Robert Haas
On Mon, Aug 12, 2024 at 8:54 AM Amit Langote wrote: > 1. I went through many iterations of the changes to ExecInitNode() to > return a partially initialized PlanState tree when it detects that the > CachedPlan was invalidated after locking a child table and to > ExecEndNode() to account for the Pl

Re: generic plans and "initial" pruning

2024-08-12 Thread Amit Langote
On Thu, Jun 20, 2024 at 2:09 AM Alvaro Herrera wrote: > I hope we can get this new executor code in 18. Thanks for doing the benchmark, Alvaro, and sorry for the late reply. Yes, I'm hoping to get *some* version of this into v18. I've been thinking how to move this forward and I'm starting to t

Re: generic plans and "initial" pruning

2024-06-19 Thread Alvaro Herrera
I had occasion to run the same benchmark you described in the initial email in this thread. To do so I applied patch series v49 on top of 07cb29737a4e, which is just one that happened to have the same date as v49. I then used a script like this (against a server having plan_cache_mode=force_gener

Re: generic plans and "initial" pruning

2024-05-20 Thread Amit Langote
On Sun, May 19, 2024 at 9:39 AM David Rowley wrote: > For #1, the locks taken for SELECT queries are less likely to conflict > with other locks obtained by PostgreSQL, but at least at the moment if > someone is getting deadlocks with a DDL type operation, they can > change their query or DDL scrip

Re: generic plans and "initial" pruning

2024-05-18 Thread David Rowley
On Sun, 19 May 2024 at 13:27, Tom Lane wrote: > > David Rowley writes: > > 1. No ability to control the order that the locks are obtained. The > > order in which the locks are taken will be at the mercy of the plan > > the planner chooses. > > I do not think I buy this argument, because plancache

Re: generic plans and "initial" pruning

2024-05-18 Thread Tom Lane
David Rowley writes: > With the caveat of not yet having looked at the latest patch, my > thoughts are that having the executor startup responsible for taking > locks is a bad idea and I don't think we should go down this path. OK, it's certainly still up for argument, but ... > 1. No ability to

Re: generic plans and "initial" pruning

2024-05-18 Thread David Rowley
On Fri, 20 Jan 2023 at 08:39, Tom Lane wrote: > I spent some time re-reading this whole thread, and the more I read > the less happy I got. We are adding a lot of complexity and introducing > coding hazards that will surely bite somebody someday. And after awhile > I had what felt like an epipha

Re: generic plans and "initial" pruning

2024-04-08 Thread Amit Langote
Hi Andrey, On Sun, Mar 31, 2024 at 2:03 PM Andrey M. Borodin wrote: > > On 6 Dec 2023, at 23:52, Robert Haas wrote: > > > > I hope that it's at least somewhat useful. > > > On 5 Jan 2024, at 15:46, vignesh C wrote: > > > > There is a leak reported > > Hi Amit, > > this is a kind reminder that

Re: generic plans and "initial" pruning

2024-03-30 Thread Andrey M. Borodin
> On 6 Dec 2023, at 23:52, Robert Haas wrote: > > I hope that it's at least somewhat useful. > > On 5 Jan 2024, at 15:46, vignesh C wrote: > > There is a leak reported Hi Amit, this is a kind reminder that some feedback on your patch[0] is waiting for your reply. Thank you for your w

Re: generic plans and "initial" pruning

2024-01-05 Thread vignesh C
On Mon, 20 Nov 2023 at 10:00, Amit Langote wrote: > > On Thu, Sep 28, 2023 at 5:26 PM Amit Langote wrote: > > On Tue, Sep 26, 2023 at 10:06 PM Amit Langote > > wrote: > > > After sleeping on this, I think we do need the checks after all the > > > ExecInitNode() calls too, because we have many i

Re: generic plans and "initial" pruning

2023-12-06 Thread Robert Haas
Reviewing 0001: Perhaps ExecEndCteScan needs an adjustment. What if node->leader was never set? Other than that, I think this is in good shape. Maybe there are other things we'd want to adjust here, or maybe there aren't, but there doesn't seem to be any good reason to bundle more changes into th

Re: generic plans and "initial" pruning

2023-09-25 Thread Amit Langote
On Wed, Sep 6, 2023 at 11:20 PM Robert Haas wrote: > On Wed, Sep 6, 2023 at 5:12 AM Amit Langote wrote: > > Attached updated patches. Thanks for the review. > > I think 0001 looks ready to commit. I'm not sure that the commit > message needs to mention future patches here, since this code cleanu

Re: generic plans and "initial" pruning

2023-09-06 Thread Robert Haas
On Wed, Sep 6, 2023 at 5:12 AM Amit Langote wrote: > Attached updated patches. Thanks for the review. I think 0001 looks ready to commit. I'm not sure that the commit message needs to mention future patches here, since this code cleanup seems like a good idea regardless, but if you feel otherwis

Re: generic plans and "initial" pruning

2023-09-05 Thread Robert Haas
On Tue, Sep 5, 2023 at 3:13 AM Amit Langote wrote: > Attached 0001 removes unnecessary cleanup calls from ExecEnd*() routines. It also adds a few random Assert()s to verify that unrelated pointers are not NULL. I suggest that it shouldn't do that. The commit message doesn't mention the removal o

Re: generic plans and "initial" pruning

2023-08-28 Thread Robert Haas
On Fri, Aug 11, 2023 at 9:50 AM Amit Langote wrote: > After removing the unnecessary cleanup code from most node types’ ExecEnd* > functions, one thing I’m tempted to do is remove the functions that do > nothing else but recurse to close the outerPlan, innerPlan child nodes. We > could instead

Re: generic plans and "initial" pruning

2023-08-11 Thread Amit Langote
On Fri, Aug 11, 2023 at 14:31 Amit Langote wrote: > On Wed, Aug 9, 2023 at 1:05 AM Robert Haas wrote: > > On Tue, Aug 8, 2023 at 10:32 AM Amit Langote > wrote: > > > But should ExecInitNode() subroutines return the partially initialized > > > PlanState node or NULL on detecting invalidation? I

Re: generic plans and "initial" pruning

2023-08-08 Thread Robert Haas
On Tue, Aug 8, 2023 at 10:32 AM Amit Langote wrote: > But should ExecInitNode() subroutines return the partially initialized > PlanState node or NULL on detecting invalidation? If I'm > understanding how you think this should be working correctly, I think > you mean the former, because if it were

Re: generic plans and "initial" pruning

2023-08-08 Thread Amit Langote
On Tue, Aug 8, 2023 at 12:36 AM Robert Haas wrote: > On Thu, Aug 3, 2023 at 4:37 AM Amit Langote wrote: > > Here's a patch set where the refactoring to move the ExecutorStart() > > calls to be closer to GetCachedPlan() (for the call sites that use a > > CachedPlan) is extracted into a separate pa

Re: generic plans and "initial" pruning

2023-08-07 Thread Robert Haas
On Mon, Aug 7, 2023 at 11:44 AM Tom Lane wrote: > Right, I doubt that changing that is going to work out well. > Hash joins might have issues with it too. I thought about the case, because Hash and Hash Join are such closely intertwined nodes, but I don't see any problem there. It doesn't really

Re: generic plans and "initial" pruning

2023-08-07 Thread Tom Lane
Robert Haas writes: > Second, I wondered whether the ordering of cleanup operations could be > an issue. Right now, a node can position cleanup code before, after, > or both before and after recursing to child nodes, whereas with this > design change, the cleanup code will always be run before rec

Re: generic plans and "initial" pruning

2023-08-07 Thread Robert Haas
On Thu, Aug 3, 2023 at 4:37 AM Amit Langote wrote: > Here's a patch set where the refactoring to move the ExecutorStart() > calls to be closer to GetCachedPlan() (for the call sites that use a > CachedPlan) is extracted into a separate patch, 0002. Its commit > message notes an aspect of this ref

Re: generic plans and "initial" pruning

2023-07-18 Thread Thom Brown
On Tue, 18 Jul 2023, 08:26 Amit Langote, wrote: > Hi Thom, > > On Tue, Jul 18, 2023 at 1:33 AM Thom Brown wrote: > > On Thu, 13 Jul 2023 at 13:59, Amit Langote > wrote: > > > In an absolutely brown-paper-bag moment, I realized that I had not > > > updated src/backend/executor/README to reflect

Re: generic plans and "initial" pruning

2023-07-17 Thread Thom Brown
On Thu, 13 Jul 2023 at 13:59, Amit Langote wrote: > In an absolutely brown-paper-bag moment, I realized that I had not > updated src/backend/executor/README to reflect the changes to the > executor's control flow that this patch makes. That is, after > scrapping the old design back in January wh

Re: generic plans and "initial" pruning

2023-07-03 Thread Daniel Gustafsson
> On 8 Jun 2023, at 16:23, Amit Langote wrote: > > Here is a new version. The local planstate variable in the hunk below is shadowing the function parameter planstate which cause a compiler warning: @@ -1495,18 +1556,15 @@ ExecEndPlan(PlanState *planstate, EState *estate) ListCell *l;

Re: generic plans and "initial" pruning

2023-04-05 Thread Amit Langote
On Tue, Apr 4, 2023 at 10:29 PM Amit Langote wrote: > On Tue, Apr 4, 2023 at 6:41 AM Tom Lane wrote: > > A few concrete thoughts: > > > > * I understand that your plan now is to acquire locks on all the > > originally-named tables, then do permissions checks (which will > > involve only those tab

Re: generic plans and "initial" pruning

2023-04-04 Thread Amit Langote
On Tue, Apr 4, 2023 at 6:41 AM Tom Lane wrote: > Amit Langote writes: > > [ v38 patchset ] > > I spent a little bit of time looking through this, and concluded that > it's not something I will be wanting to push into v16 at this stage. > The patch doesn't seem very close to being committable on i

Re: generic plans and "initial" pruning

2023-04-03 Thread Tom Lane
Amit Langote writes: > [ v38 patchset ] I spent a little bit of time looking through this, and concluded that it's not something I will be wanting to push into v16 at this stage. The patch doesn't seem very close to being committable on its own terms, and even if it was now is not a great time in

Re: generic plans and "initial" pruning

2023-03-02 Thread Amit Langote
On Wed, Feb 8, 2023 at 7:31 PM Amit Langote wrote: > On Tue, Feb 7, 2023 at 23:38 Andres Freund wrote: >> The tests seem to frequently hang on freebsd: >> https://cirrus-ci.com/github/postgresql-cfbot/postgresql/commitfest%2F42%2F3478 > > Thanks for the heads up. I’ve noticed this one too, thoug

Re: generic plans and "initial" pruning

2023-02-08 Thread Amit Langote
On Tue, Feb 7, 2023 at 23:38 Andres Freund wrote: > Hi, > > On 2023-02-03 22:01:09 +0900, Amit Langote wrote: > > I've added a test case under src/modules/delay_execution by adding a > > new ExecutorStart_hook that works similarly as > > delay_execution_planner(). The test works by allowing a co

Re: generic plans and "initial" pruning

2023-02-07 Thread Andres Freund
Hi, On 2023-02-03 22:01:09 +0900, Amit Langote wrote: > I've added a test case under src/modules/delay_execution by adding a > new ExecutorStart_hook that works similarly as > delay_execution_planner(). The test works by allowing a concurrent > session to drop an object being referenced in a cach

Re: generic plans and "initial" pruning

2023-02-03 Thread Amit Langote
On Thu, Feb 2, 2023 at 11:49 PM Amit Langote wrote: > On Fri, Jan 27, 2023 at 4:01 PM Amit Langote wrote: > > I didn't actually go with calling the plancache on every lock taken on > > a relation, that is, in ExecGetRangeTableRelation(). One thing about > > doing it that way that I didn't quite

Re: generic plans and "initial" pruning

2023-02-02 Thread Amit Langote
On Fri, Jan 27, 2023 at 4:01 PM Amit Langote wrote: > On Fri, Jan 20, 2023 at 12:52 PM Amit Langote wrote: > > Alright, I'll try to get something out early next week. Thanks for > > all the pointers. > > Sorry for the delay. Attached is what I've come up with so far. > > I didn't actually go wi

Re: generic plans and "initial" pruning

2023-01-19 Thread Amit Langote
On Fri, Jan 20, 2023 at 12:58 PM Tom Lane wrote: > Amit Langote writes: > > On Fri, Jan 20, 2023 at 12:31 PM Tom Lane wrote: > >> It might be possible to incorporate this pointer into PlannedStmt > >> instead of passing it separately. > > > Yeah, that would be less churn. Though, I wonder if yo

Re: generic plans and "initial" pruning

2023-01-19 Thread Tom Lane
Amit Langote writes: > On Fri, Jan 20, 2023 at 12:31 PM Tom Lane wrote: >> It might be possible to incorporate this pointer into PlannedStmt >> instead of passing it separately. > Yeah, that would be less churn. Though, I wonder if you still hold > that PlannedStmt should not be scribbled upon

Re: generic plans and "initial" pruning

2023-01-19 Thread Amit Langote
On Fri, Jan 20, 2023 at 12:31 PM Tom Lane wrote: > Amit Langote writes: > > On Fri, Jan 20, 2023 at 4:39 AM Tom Lane wrote: > >> I had what felt like an epiphany: the whole problem arises because the > >> system is wrongly factored. We should get rid of AcquireExecutorLocks > >> altogether, all

Re: generic plans and "initial" pruning

2023-01-19 Thread Tom Lane
Amit Langote writes: > On Fri, Jan 20, 2023 at 4:39 AM Tom Lane wrote: >> I had what felt like an epiphany: the whole problem arises because the >> system is wrongly factored. We should get rid of AcquireExecutorLocks >> altogether, allowing the plancache to hand back a generic plan that >> it's

Re: generic plans and "initial" pruning

2023-01-19 Thread Amit Langote
On Fri, Jan 20, 2023 at 4:39 AM Tom Lane wrote: > I spent some time re-reading this whole thread, and the more I read > the less happy I got. Thanks a lot for your time on this. > We are adding a lot of complexity and introducing > coding hazards that will surely bite somebody someday. And aft

Re: generic plans and "initial" pruning

2023-01-19 Thread Tom Lane
I spent some time re-reading this whole thread, and the more I read the less happy I got. We are adding a lot of complexity and introducing coding hazards that will surely bite somebody someday. And after awhile I had what felt like an epiphany: the whole problem arises because the system is wron

Re: generic plans and "initial" pruning

2022-12-21 Thread Tom Lane
Alvaro Herrera writes: > This version of the patch looks not entirely unreasonable to me. I'll > set this as Ready for Committer in case David or Tom or someone else > want to have a look and potentially commit it. I will have a look during the January CF. regards, tom l

Re: generic plans and "initial" pruning

2022-12-21 Thread Amit Langote
On Wed, Dec 21, 2022 at 7:18 PM Alvaro Herrera wrote: > This version of the patch looks not entirely unreasonable to me. I'll > set this as Ready for Committer in case David or Tom or someone else > want to have a look and potentially commit it. Thank you, Alvaro. -- Thanks, Amit Langote EDB:

Re: generic plans and "initial" pruning

2022-12-21 Thread Alvaro Herrera
This version of the patch looks not entirely unreasonable to me. I'll set this as Ready for Committer in case David or Tom or someone else want to have a look and potentially commit it. -- Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/

Re: generic plans and "initial" pruning

2022-12-15 Thread Amit Langote
On Wed, Dec 14, 2022 at 5:35 PM Amit Langote wrote: > I have moved the original functionality of GetCachedPlan() to > GetCachedPlanInternal(), turning the former into a sort of controller > as described shortly. The latter's CheckCachedPlan() part now only > locks the "minimal" set of, non-prunab

Re: generic plans and "initial" pruning

2022-12-14 Thread Amit Langote
On Tue, Dec 13, 2022 at 2:24 AM Alvaro Herrera wrote: > On 2022-Dec-12, Amit Langote wrote: > > I started feeling like putting all the new logic being added > > by this patch into plancache.c at the heart of GetCachedPlan() and > > tweaking its API in kind of unintuitive ways may not have been suc

Re: generic plans and "initial" pruning

2022-12-12 Thread Alvaro Herrera
On 2022-Dec-12, Amit Langote wrote: > I started feeling like putting all the new logic being added > by this patch into plancache.c at the heart of GetCachedPlan() and > tweaking its API in kind of unintuitive ways may not have been such a > good idea to begin with. So I started thinking again ab

Re: generic plans and "initial" pruning

2022-12-12 Thread Amit Langote
On Fri, Dec 9, 2022 at 8:37 PM Alvaro Herrera wrote: > On 2022-Dec-09, Amit Langote wrote: > > > Pruning will be done afresh on every fetch of a given cached plan when > > CheckCachedPlan() is called on it, so the part_prune_results_list part > > will be discarded and rebuilt as many times as the

Re: generic plans and "initial" pruning

2022-12-09 Thread Alvaro Herrera
On 2022-Dec-09, Amit Langote wrote: > Pruning will be done afresh on every fetch of a given cached plan when > CheckCachedPlan() is called on it, so the part_prune_results_list part > will be discarded and rebuilt as many times as the plan is executed. > You'll find a description around CachedPlan

Re: generic plans and "initial" pruning

2022-12-09 Thread Amit Langote
On Fri, Dec 9, 2022 at 7:49 PM Alvaro Herrera wrote: > On 2022-Dec-09, Amit Langote wrote: > > On Fri, Dec 9, 2022 at 6:52 PM Alvaro Herrera > > wrote: > > > Remind me again why is part_prune_results_list not part of struct > > > CachedPlan then? I tried to understand that based on comments upt

Re: generic plans and "initial" pruning

2022-12-09 Thread Alvaro Herrera
On 2022-Dec-09, Amit Langote wrote: > On Fri, Dec 9, 2022 at 6:52 PM Alvaro Herrera wrote: > > Remind me again why is part_prune_results_list not part of struct > > CachedPlan then? I tried to understand that based on comments upthread, > > but I was unable to find anything. > > It used to be

Re: generic plans and "initial" pruning

2022-12-09 Thread Amit Langote
On Fri, Dec 9, 2022 at 6:52 PM Alvaro Herrera wrote: > On 2022-Dec-09, Amit Langote wrote: > > On Wed, Dec 7, 2022 at 4:00 AM Alvaro Herrera > > wrote: > > > I find the API of GetCachedPlans a little weird after this patch. > > > David, in his Apr 7 reply on this thread, also sounded to suggest

Re: generic plans and "initial" pruning

2022-12-09 Thread Alvaro Herrera
On 2022-Dec-09, Amit Langote wrote: > On Wed, Dec 7, 2022 at 4:00 AM Alvaro Herrera wrote: > > I find the API of GetCachedPlans a little weird after this patch. > David, in his Apr 7 reply on this thread, also sounded to suggest > something similar. > > Hmm, I was / am not so sure if GetCachedP

Re: generic plans and "initial" pruning

2022-12-09 Thread Amit Langote
Thanks for the review. On Wed, Dec 7, 2022 at 4:00 AM Alvaro Herrera wrote: > I find the API of GetCachedPlans a little weird after this patch. I > think it may be better to have it return a pointer of a new struct -- > one that contains both the CachedPlan pointer and the list of pruning > resu

Re: generic plans and "initial" pruning

2022-12-06 Thread Alvaro Herrera
I find the API of GetCachedPlans a little weird after this patch. I think it may be better to have it return a pointer of a new struct -- one that contains both the CachedPlan pointer and the list of pruning results. (As I understand, the sole caller that isn't interested in the pruning results,

Re: generic plans and "initial" pruning

2022-12-04 Thread Amit Langote
On Mon, Dec 5, 2022 at 12:00 PM Amit Langote wrote: > On Fri, Dec 2, 2022 at 7:40 PM Amit Langote wrote: > > Thought it might be good for PartitionPruneResult to also have > > root_parent_relids that matches with the corresponding > > PartitionPruneInfo. ExecInitPartitionPruning() does a sanity

  1   2   >