Re: RFC: Logging plan of the running query

2025-04-30 Thread torikoshia
Hi, On 2025-04-28 08:55, Hannu Krosing wrote: Have you also checked out https://github.com/postgrespro/pg_query_state which logs running query plan AND collected counts and timings as a response to a signal? Yes. For example, see the discussion: https://www.postgresql.org/message-id/d68c3ae316

Re: RFC: Logging plan of the running query

2025-04-27 Thread Hannu Krosing
Have you also checked out https://github.com/postgrespro/pg_query_state which logs running query plan AND collected counts and timings as a response to a signal? Has this ever been discussed for inclusion in core ? On Thu, Apr 24, 2025 at 2:49 PM torikoshia wrote: > > Hi, > > Attached a rebased

Re: RFC: Logging plan of the running query

2025-04-24 Thread torikoshia
Hi, Attached a rebased version of the patch. -- Regards, -- Atsushi Torikoshi Seconded from NTT DATA GROUP CORPORATION to SRA OSS K.K.From 112467238f585bb3398d86ac6e1a71caa6549fb4 Mon Sep 17 00:00:00 2001 From: Atsushi Torikoshi Date: Thu, 24 Apr 2025 20:50:41 +0900 Subject: [PATCH v44] Add fu

Re: RFC: Logging plan of the running query

2025-04-05 Thread Sami Imseih
> > FWIW, I have been thinking about auto_explain for another task, > > remote plans for fdw [0], and perhaps there are now other good > > reasons, some that you mention, that can be simplified if "auto_explain" > > becomes a core feature. This could be a proposal taken up in 19. > > For what we're

Re: RFC: Logging plan of the running query

2025-04-05 Thread Sami Imseih
> > 2/ > > It should be noted that the plan will not print to the log until > > the plan begins executing the next plan node? depending on the > > operation, that could take some time ( i.e.long seq scan of a table, > > etc.) > > Does this behavior need to be called out in docs? > > Seems reasonabl

Re: RFC: Logging plan of the running query

2025-04-05 Thread Sadeq Dousti
Hi, Sergey and I reviewed this patch and have a few comments, listed below. > BTW the patch adds about 400 lines to explain.c and it may be better > to split the file as well as 9173e8b6046, but I leave it as it is for > now. 1. As rightfully described by the OP above, explain.c has grown too bi

Re: RFC: Logging plan of the running query

2025-04-05 Thread Robert Haas
On Mon, Mar 31, 2025 at 9:43 PM torikoshia wrote: > Previously, Rafael proposed a patch in this thread that added execution > progress tracking. However, I felt that expanding the scope could make > it harder to get the patch reviewed or committed. So, I suggested first > completing a feature that

Re: RFC: Logging plan of the running query

2025-04-04 Thread Atsushi Torikoshi
On Thu, Apr 3, 2025 at 11:10 PM Robert Haas wrote: > Looking at ExplainAssembleLogOutput() is making me realize that > auto_explain is in serious need of some cleanup. That's not really the > fault of this patch, but the hack whereby we overwrite the [] that > would have surrounded the JSON output

Re: RFC: Logging plan of the running query

2025-04-04 Thread torikoshia
On 2025-04-02 03:52, Robert Haas wrote: Thank you for review! On Fri, Mar 21, 2025 at 8:40 AM torikoshia wrote: Rebased it again. On 2025-03-10 14:10, torikoshia wrote: > BTW the patch adds about 400 lines to explain.c and it may be better > to split the file as well as 9173e8b6046, but I leav

Re: RFC: Logging plan of the running query

2025-04-03 Thread Robert Haas
On Thu, Apr 3, 2025 at 1:32 AM torikoshia wrote: > I think tracking execution progress involves more challenges to solve > compared to simply outputting the plan. > For this reason, I believe an incremental approach -- first completing > the basic plan output functionality in this thread and then

Re: RFC: Logging plan of the running query

2025-04-02 Thread torikoshia
On 2025-04-03 01:22, Sami Imseih wrote: > FWIW, I have been thinking about auto_explain for another task, > remote plans for fdw [0], and perhaps there are now other good > reasons, some that you mention, that can be simplified if "auto_explain" > becomes a core feature. This could be a proposal

Re: RFC: Logging plan of the running query

2025-04-02 Thread torikoshia
On 2025-04-01 03:51, Sadeq Dousti wrote: Hi, Sergey and I reviewed this patch and have a few comments, listed below. Thanks for your review! BTW the patch adds about 400 lines to explain.c and it may be better to split the file as well as 9173e8b6046, but I leave it as it is for now. 1. A

Re: RFC: Logging plan of the running query

2025-04-01 Thread Robert Haas
On Tue, Apr 1, 2025 at 3:05 PM Sami Imseih wrote: > > Looking at ExplainAssembleLogOutput() is making me realize that > > auto_explain is in serious need of some cleanup. That's not really the > > fault of this patch, but the hack whereby we overwrite the [] that > > would have surrounded the JSON

Re: RFC: Logging plan of the running query

2025-04-01 Thread Sami Imseih
> Looking at ExplainAssembleLogOutput() is making me realize that > auto_explain is in serious need of some cleanup. That's not really the > fault of this patch, but the hack whereby we overwrite the [] that > would have surrounded the JSON output with {} is not very nice. I also > think that the a

Re: RFC: Logging plan of the running query

2025-04-01 Thread Robert Haas
On Fri, Mar 21, 2025 at 8:40 AM torikoshia wrote: > Rebased it again. > > On 2025-03-10 14:10, torikoshia wrote: > > BTW the patch adds about 400 lines to explain.c and it may be better > > to split the file as well as 9173e8b6046, but I leave it as it is for > > now. > > This part remains unchang

Re: RFC: Logging plan of the running query

2025-04-01 Thread torikoshia
On 2025-04-01 21:32, Robert Haas wrote: Thanks for your comment. On Mon, Mar 31, 2025 at 9:43 PM torikoshia wrote: Previously, Rafael proposed a patch in this thread that added execution progress tracking. However, I felt that expanding the scope could make it harder to get the patch reviewed

Re: RFC: Logging plan of the running query

2025-03-31 Thread torikoshia
On 2025-04-01 04:24, Robert Haas wrote: On Fri, Mar 21, 2025 at 8:40 AM torikoshia wrote: Rebased it again. Hi, I apologize for not having noticed this thread sooner. Thank you for your checking! No worries. I just became aware of it as a result of a discussion in the hacking Discord ser

Re: RFC: Logging plan of the running query

2025-03-31 Thread Robert Haas
On Fri, Mar 21, 2025 at 8:40 AM torikoshia wrote: > Rebased it again. Hi, I apologize for not having noticed this thread sooner. I just became aware of it as a result of a discussion in the hacking Discord server. I think this has got a lot over overlap with the progressive EXPLAIN patch from Ra

Re: RFC: Logging plan of the running query

2025-03-21 Thread torikoshia
Hi, Rebased it again. On 2025-03-10 14:10, torikoshia wrote: BTW the patch adds about 400 lines to explain.c and it may be better to split the file as well as 9173e8b6046, but I leave it as it is for now. This part remains unchanged. -- Regards, -- Atsushi Torikoshi Seconded from NTT DATA G

Re: RFC: Logging plan of the running query

2025-03-09 Thread torikoshia
On 2025-03-09 00:42, Akshat Jaimini wrote: Hi, I think there is still some problem with the patch. I am not able to apply it to the master branch. Can you please take another look at it? Thanks for pointing it out! Modified it. BTW the patch adds about 400 lines to explain.c and it may be bet

Re: RFC: Logging plan of the running query

2025-03-08 Thread Akshat Jaimini
Hi, I think there is still some problem with the patch. I am not able to apply it to the master branch. Can you please take another look at it? Thanks, Akshat Jaimini

Re: RFC: Logging plan of the running query

2025-02-03 Thread torikoshia
Hi, Rebased the patch since it couldn't be applied anymore. -- Regards, -- Atsushi Torikoshi Seconded from NTT DATA GROUP CORPORATION to SRA OSS K.K.From 3f580c17214595eb8d6013674f5f054ec352ab7a Mon Sep 17 00:00:00 2001 From: Atsushi Torikoshi Date: Mon, 3 Feb 2025 21:22:40 +0900 Subject: [PAT

Re: RFC: Logging plan of the running query

2024-10-28 Thread torikoshia
Hi I've recently resumed work on this proposal. I was revising tests that use the injection point, but given that there was a recent talk at PGConf.EU about further features based on this proposal [1], I thought it might be of interest to some. Therefore, I'll share a patch that removes the i

Re: RFC: Logging plan of the running query

2024-03-25 Thread Andres Freund
Hi, On 2024-03-13 15:33:02 -0400, Robert Haas wrote: > But also ... having to wrap the entire plan tree like this seems > pretty awful. I don't really like the idea of a large-scan plan > modification like this in the middle of the query. It's not great. But I also don't really see an alternative

Re: RFC: Logging plan of the running query

2024-03-17 Thread torikoshia
On 2024-03-14 04:33, Robert Haas wrote: Thanks for your review! On Wed, Mar 13, 2024 at 1:28 AM torikoshia wrote: - I saw no way to find the next node to be executed from the planstate tree, so the patch wraps all the ExecProcNode of the planstate tree at CHECK_FOR_INTERRUPTS(). I don't thin

Re: RFC: Logging plan of the running query

2024-03-13 Thread jian he
On Wed, Mar 13, 2024 at 1:28 PM torikoshia wrote: > > On Fri, Feb 16, 2024 at 11:42 PM torikoshia > wrote: > > I'm not so sure about the implementation now, i.e. finding the next > > node > > to be executed from the planstate tree, but I'm going to try this > > approach. > > Attached a patch whic

Re: RFC: Logging plan of the running query

2024-03-13 Thread Robert Haas
On Wed, Mar 13, 2024 at 1:28 AM torikoshia wrote: > - I saw no way to find the next node to be executed from the planstate > tree, so the patch wraps all the ExecProcNode of the planstate tree at > CHECK_FOR_INTERRUPTS(). I don't think it does this correctly, because some node types have children

Re: RFC: Logging plan of the running query

2024-03-12 Thread torikoshia
On Fri, Feb 16, 2024 at 11:42 PM torikoshia wrote: I'm not so sure about the implementation now, i.e. finding the next node to be executed from the planstate tree, but I'm going to try this approach. Attached a patch which takes this approach. - I saw no way to find the next node to be execu

Re: RFC: Logging plan of the running query

2024-03-04 Thread Robert Haas
On Sat, Mar 2, 2024 at 10:46 AM James Coleman wrote: > If I can rephrase this idea: it's basically "delay this interrupt > until inline to the next ExecProcNode execution". Yes, but it's not just that. It also means that the code which would handle the interrupt doesn't need to be called at every

Re: RFC: Logging plan of the running query

2024-03-02 Thread James Coleman
On Wed, Feb 28, 2024 at 1:18 AM Robert Haas wrote: > > On Mon, Feb 26, 2024 at 5:31 PM torikoshia wrote: > > It would be nice if there was a place accessed once every few seconds or > > so.. > > I think this comment earlier from Andres deserves close attention: > > # If we went with something lik

Re: RFC: Logging plan of the running query

2024-02-27 Thread Robert Haas
On Mon, Feb 26, 2024 at 5:31 PM torikoshia wrote: > It would be nice if there was a place accessed once every few seconds or > so.. I think this comment earlier from Andres deserves close attention: # If we went with something like tht approach, I think we'd have to do something # like redirecti

Re: RFC: Logging plan of the running query

2024-02-26 Thread Julien Rouhaud
On Mon, Feb 26, 2024 at 01:56:44PM +0530, Robert Haas wrote: > On Sun, Feb 25, 2024 at 5:00 PM Julien Rouhaud wrote: > > Yeah, trying to find a generalized solution seems like worth investing some > > time to avoid having a bunch of CHECK_FOR_XXX() calls scattered in the code > > a > > few years

Re: RFC: Logging plan of the running query

2024-02-26 Thread torikoshia
On 2024-02-24 00:23, Robert Haas wrote: On Fri, Feb 23, 2024 at 7:50 PM Julien Rouhaud wrote: On Fri, Feb 23, 2024 at 10:22:32AM +0530, Robert Haas wrote: > On Thu, Feb 22, 2024 at 6:25 AM James Coleman wrote: > > This is potentially a bit of a wild idea, but I wonder if having some > > kind o

Re: RFC: Logging plan of the running query

2024-02-26 Thread Robert Haas
On Sun, Feb 25, 2024 at 5:00 PM Julien Rouhaud wrote: > Yeah, trying to find a generalized solution seems like worth investing some > time to avoid having a bunch of CHECK_FOR_XXX() calls scattered in the code a > few years down the road. I just don't really see how to do it. I suspect that every

Re: RFC: Logging plan of the running query

2024-02-26 Thread Ashutosh Bapat
On Mon, Feb 26, 2024 at 1:25 PM Julien Rouhaud wrote: > > > > > > > > #define CHECK_FOR_INTERRUPTS_X(x, f, CFI_IMPL, ...) CFI_IMPL > > > > > > #define CHECK_FOR_INTERRUPTS(...) \ > > > CHECK_FOR_INTERRUPTS_X(, ##__VA_ARGS__, \ > > >

Re: RFC: Logging plan of the running query

2024-02-25 Thread Julien Rouhaud
On Mon, Feb 26, 2024 at 12:19:42PM +0530, Ashutosh Bapat wrote: > On Sun, Feb 25, 2024 at 5:00 PM Julien Rouhaud wrote: > > > > > > On Fri, Feb 23, 2024 at 7:50 PM Julien Rouhaud > > > > wrote: > > > > > > > > > > But it doesn't have to be all or nothing right? I mean each call > > > > > could

Re: RFC: Logging plan of the running query

2024-02-25 Thread Ashutosh Bapat
On Sun, Feb 25, 2024 at 5:00 PM Julien Rouhaud wrote: > > On Sat, Feb 24, 2024 at 08:56:41AM -0500, James Coleman wrote: > > On Fri, Feb 23, 2024 at 10:23 AM Robert Haas wrote: > > > > > > On Fri, Feb 23, 2024 at 7:50 PM Julien Rouhaud wrote: > > > > > > > > But it doesn't have to be all or noth

Re: RFC: Logging plan of the running query

2024-02-25 Thread Julien Rouhaud
On Sat, Feb 24, 2024 at 08:56:41AM -0500, James Coleman wrote: > On Fri, Feb 23, 2024 at 10:23 AM Robert Haas wrote: > > > > On Fri, Feb 23, 2024 at 7:50 PM Julien Rouhaud wrote: > > > > > > But it doesn't have to be all or nothing right? I mean each call could > > > say > > > what the situatio

Re: RFC: Logging plan of the running query

2024-02-24 Thread James Coleman
On Fri, Feb 23, 2024 at 10:23 AM Robert Haas wrote: > > On Fri, Feb 23, 2024 at 7:50 PM Julien Rouhaud wrote: > > On Fri, Feb 23, 2024 at 10:22:32AM +0530, Robert Haas wrote: > > > On Thu, Feb 22, 2024 at 6:25 AM James Coleman wrote: > > > > This is potentially a bit of a wild idea, but I wonder

Re: RFC: Logging plan of the running query

2024-02-23 Thread Robert Haas
On Fri, Feb 23, 2024 at 7:50 PM Julien Rouhaud wrote: > On Fri, Feb 23, 2024 at 10:22:32AM +0530, Robert Haas wrote: > > On Thu, Feb 22, 2024 at 6:25 AM James Coleman wrote: > > > This is potentially a bit of a wild idea, but I wonder if having some > > > kind of argument to CHECK_FOR_INTERRUPTS(

Re: RFC: Logging plan of the running query

2024-02-23 Thread Julien Rouhaud
Hi, On Fri, Feb 23, 2024 at 10:22:32AM +0530, Robert Haas wrote: > On Thu, Feb 22, 2024 at 6:25 AM James Coleman wrote: > > This is potentially a bit of a wild idea, but I wonder if having some > > kind of argument to CHECK_FOR_INTERRUPTS() signifying we're in > > "normal" as opposed to "critical

Re: RFC: Logging plan of the running query

2024-02-22 Thread Robert Haas
On Thu, Feb 22, 2024 at 6:25 AM James Coleman wrote: > This is potentially a bit of a wild idea, but I wonder if having some > kind of argument to CHECK_FOR_INTERRUPTS() signifying we're in > "normal" as opposed to "critical" (using that word differently than > the existing critical sections) woul

Re: RFC: Logging plan of the running query

2024-02-21 Thread Ashutosh Bapat
On Thu, Feb 22, 2024 at 6:25 AM James Coleman wrote: > > > > ...I think the current approach is just plain dead, because of this > > issue. We can't take an approach that creates an unbounded number of > > unclear reentrancy issues and then insert hacks one by one to cure > > them (or hack around

Re: RFC: Logging plan of the running query

2024-02-21 Thread James Coleman
On Mon, Feb 19, 2024 at 11:53 PM Robert Haas wrote: > > On Fri, Feb 16, 2024 at 12:29 AM Andres Freund wrote: > > If we went with something like tht approach, I think we'd have to do > > something > > like redirecting node->ExecProcNode to a wrapper, presumably from within a > > CFI. That wrappe

Re: RFC: Logging plan of the running query

2024-02-19 Thread Robert Haas
On Fri, Feb 16, 2024 at 12:29 AM Andres Freund wrote: > If we went with something like tht approach, I think we'd have to do something > like redirecting node->ExecProcNode to a wrapper, presumably from within a > CFI. That wrapper could then implement the explain support, without slowing > down t

Re: RFC: Logging plan of the running query

2024-02-16 Thread torikoshia
On Thu, Feb 15, 2024 at 6:12 PM Robert Haas wrote: Hi, I've just been catching up on this thread. + if (MyProc->heldLocks) + { + ereport(LOG_SERVER_ONLY, + errmsg("ignored request for logging query plan due to lock conflicts"), + errdetail("You can try again in a moment.")); + return; + } I

Re: RFC: Logging plan of the running query

2024-02-15 Thread Andres Freund
Hi, On 2024-02-15 14:42:11 +0530, Robert Haas wrote: > I think the issue is very general. We have lots of subsystems that > both (a) use global variables and (b) contain CHECK_FOR_INTERRUPTS(). > If we process an interrupt while that code is in the middle of > manipulating its global variables and

Re: RFC: Logging plan of the running query

2024-02-15 Thread Robert Haas
Hi, I've just been catching up on this thread. + if (MyProc->heldLocks) + { + ereport(LOG_SERVER_ONLY, + errmsg("ignored request for logging query plan due to lock conflicts"), + errdetail("You can try again in a moment.")); + return; + } I don't like this for several reasons. First, I think it

Re: RFC: Logging plan of the running query

2024-02-13 Thread torikoshia
On 2024-02-13 11:30, torikoshia wrote: I'll update the patch including other points such as removing ensuring no page lock code. Updated the patch. Using injection point support we should be able to add tests for testing pg_log_query_plan behaviour when there are page locks held or when auto

Re: RFC: Logging plan of the running query

2024-02-12 Thread torikoshia
On 2024-02-12 09:00, jian he wrote: Thanks for you comments. On Mon, Jan 29, 2024 at 9:02 PM torikoshia wrote: Hi, Updated the patch to fix typos and move ProcessLogQueryPlanInterruptActive from errfinish() to AbortTransaction. + + + + pg_log_query_plan +

Re: RFC: Logging plan of the running query

2024-02-12 Thread torikoshia
On 2024-02-07 19:14, torikoshia wrote: On 2024-02-07 13:58, Ashutosh Bapat wrote: The prologue refers to a very populated lock hash table. I think that will happen if thousands of tables are queried in a single query OR a query runs on a partitioned table with thousands of partitions. May be w

Re: RFC: Logging plan of the running query

2024-02-11 Thread jian he
On Mon, Feb 12, 2024 at 12:42 PM Ashutosh Bapat wrote: > > On Mon, Feb 12, 2024 at 5:31 AM jian he wrote: > > > > On Wed, Feb 7, 2024 at 12:58 PM Ashutosh Bapat > > wrote: > > > > > > > > > > > > */ > > > > > How bad this performance could be. Let's assume that a query is taking > > > > > time

Re: RFC: Logging plan of the running query

2024-02-11 Thread Ashutosh Bapat
On Mon, Feb 12, 2024 at 5:31 AM jian he wrote: > > On Wed, Feb 7, 2024 at 12:58 PM Ashutosh Bapat > wrote: > > > > > > > > > */ > > > > How bad this performance could be. Let's assume that a query is taking > > > > time and pg_log_query_plan() is invoked to examine the plan of this > > > > query

Re: RFC: Logging plan of the running query

2024-02-11 Thread Ashutosh Bapat
On Mon, Feb 12, 2024 at 5:31 AM jian he wrote: > > I found a bug: > src8=# select *, pg_sleep(10) from tenk1 for update; > 2024-02-11 15:54:17.944 CST [48602] LOG: query plan running on > backend with PID 48602 is: > Query Text: select *, pg_sleep(10) from tenk1 for update; > Lo

Re: RFC: Logging plan of the running query

2024-02-11 Thread jian he
On Wed, Feb 7, 2024 at 12:58 PM Ashutosh Bapat wrote: > > > > > > */ > > > How bad this performance could be. Let's assume that a query is taking > > > time and pg_log_query_plan() is invoked to examine the plan of this > > > query. Is it possible that the looping over all the locks itself takes

Re: RFC: Logging plan of the running query

2024-02-11 Thread jian he
On Mon, Jan 29, 2024 at 9:02 PM torikoshia wrote: > > Hi, > > Updated the patch to fix typos and move > ProcessLogQueryPlanInterruptActive from errfinish() to AbortTransaction. > + + + + pg_log_query_plan + +pg_log_query_plan ( pid integer ) +b

Re: RFC: Logging plan of the running query

2024-02-07 Thread torikoshia
On 2024-02-07 13:58, Ashutosh Bapat wrote: On Wed, Feb 7, 2024 at 9:38 AM torikoshia wrote: Hi Ashutosh, On 2024-02-06 19:51, Ashutosh Bapat wrote: > Thanks for the summary. It is helpful. I think patch is also getting > better. > > I have a few questions and suggestions Thanks for your com

Re: RFC: Logging plan of the running query

2024-02-06 Thread Ashutosh Bapat
On Wed, Feb 7, 2024 at 9:38 AM torikoshia wrote: > > Hi Ashutosh, > > On 2024-02-06 19:51, Ashutosh Bapat wrote: > > > Thanks for the summary. It is helpful. I think patch is also getting > > better. > > > > I have a few questions and suggestions > > Thanks for your comments. > > > 1. Prologue of

Re: RFC: Logging plan of the running query

2024-02-06 Thread torikoshia
Hi Ashutosh, On 2024-02-06 19:51, Ashutosh Bapat wrote: Thanks for the summary. It is helpful. I think patch is also getting better. I have a few questions and suggestions Thanks for your comments. 1. Prologue of GetLockMethodLocalHash() mentions * NOTE: When there are many entries in Lo

Re: RFC: Logging plan of the running query

2024-02-06 Thread Ashutosh Bapat
Hi Atsushi, On Mon, Jan 29, 2024 at 6:32 PM torikoshia wrote: > > Hi, > > Updated the patch to fix typos and move > ProcessLogQueryPlanInterruptActive from errfinish() to AbortTransaction. > > > BTW since the thread is getting long, I list the some points of the > discussion so far: > > # Safety

Re: RFC: Logging plan of the running query

2024-01-29 Thread torikoshia
Hi, Updated the patch to fix typos and move ProcessLogQueryPlanInterruptActive from errfinish() to AbortTransaction. BTW since the thread is getting long, I list the some points of the discussion so far: # Safety concern ## Catalog access inside CFI - it seems safe if the CFI call is insid

Re: RFC: Logging plan of the running query

2023-12-10 Thread torikoshia
On 2023-12-07 08:33, Rafael Thofehrn Castro wrote: Hello hackers, Last Saturday I submitted a patch to the pgsql-hackers list with the title "Proposal: In-flight explain logging" with a patch proposing a feature very similar to the one being worked on in this thread. I should have done a better

Re: RFC: Logging plan of the running query

2023-12-06 Thread Rafael Thofehrn Castro
Hello hackers, Last Saturday I submitted a patch to the pgsql-hackers list with the title "Proposal: In-flight explain logging" with a patch proposing a feature very similar to the one being worked on in this thread. I should have done a better search in the commitfest before implementing somethin

Re: RFC: Logging plan of the running query

2023-11-09 Thread Ashutosh Bapat
On Thu, Nov 9, 2023 at 4:56 PM torikoshia wrote: > > On 2023-11-09 16:11, Ashutosh Bapat wrote: > > On Thu, Nov 9, 2023 at 12:03 PM torikoshia > > wrote: > >> >> > >> >> 1. When a backend is running nested queries, we will see the plan of > >> >> the innermost query. That query may not be the act

Re: RFC: Logging plan of the running query

2023-11-09 Thread torikoshia
On 2023-11-09 16:11, Ashutosh Bapat wrote: On Thu, Nov 9, 2023 at 12:03 PM torikoshia wrote: >> >> 1. When a backend is running nested queries, we will see the plan of >> the innermost query. That query may not be the actual culprit if the >> user query is running slowly. E.g a query being run

Re: RFC: Logging plan of the running query

2023-11-08 Thread Ashutosh Bapat
On Thu, Nov 9, 2023 at 12:03 PM torikoshia wrote: > >> > >> 1. When a backend is running nested queries, we will see the plan of > >> the innermost query. That query may not be the actual culprit if the > >> user query is running slowly. E.g a query being run as part of inner > >> side nested loop

Re: RFC: Logging plan of the running query

2023-11-08 Thread torikoshia
On 2023-11-06 15:32, Ashutosh Bapat wrote: Thanks for the suggestion and example. On Fri, Nov 3, 2023 at 7:31 PM Ashutosh Bapat wrote: I have following questions related to the functionality. (Please point me to the relevant discussion if this has been already discussed.) 1. When a backend

Re: RFC: Logging plan of the running query

2023-11-05 Thread Ashutosh Bapat
On Fri, Nov 3, 2023 at 7:31 PM Ashutosh Bapat wrote: > > I have following questions related to the functionality. (Please point > me to the relevant discussion if this has been already discussed.) > > 1. When a backend is running nested queries, we will see the plan of > the innermost query. That

Re: RFC: Logging plan of the running query

2023-11-03 Thread Ashutosh Bapat
I have following questions related to the functionality. (Please point me to the relevant discussion if this has been already discussed.) 1. When a backend is running nested queries, we will see the plan of the innermost query. That query may not be the actual culprit if the user query is running

Re: RFC: Logging plan of the running query

2023-10-27 Thread Étienne BERSAC
> Hi, > > If we use client log message, that message is shown on the target > process whose pid is specified by the parameter of > pg_log_query_plan(): > >    (pid:1000)=# select pg_sleep(60); >    (pid:1001)=# select pg_log_query_plan(1000); >    (pid:1000)=# LOG:  query plan running on backen

Re: RFC: Logging plan of the running query

2023-10-27 Thread torikoshia
On 2023-10-27 16:06, Étienne BERSAC wrote: Hi Torikoshia, If so, we once tried to implement such function for getting memory contexts. However, this attempt didn't succeed because it could lead dead lock situation[1]. Thanks for the pointer. Why not use client log message to allow client to g

Re: RFC: Logging plan of the running query

2023-10-27 Thread Étienne BERSAC
Hi Torikoshia, > If so, we once tried to implement such function for getting memory > contexts. > However, this attempt didn't succeed because it could lead dead lock > situation[1]. Thanks for the pointer. Why not use client log message to allow client to get plan without locking backend mem

Re: RFC: Logging plan of the running query

2023-10-25 Thread torikoshia
On 2023-10-25 12:40, Ashutosh Bapat wrote: On Wed, Oct 18, 2023 at 10:04 PM James Coleman wrote: While that decision as regards auto_explain has long since been made (and there would be work to undo it), I don't believe that we should repeat that choice here. I'm -10 on moving this into auto_e

Re: RFC: Logging plan of the running query

2023-10-24 Thread Ashutosh Bapat
On Wed, Oct 18, 2023 at 10:04 PM James Coleman wrote: > > While that decision as regards auto_explain has long since been made > (and there would be work to undo it), I don't believe that we should > repeat that choice here. I'm -10 on moving this into auto_explain. > Right. > However perhaps th

Re: RFC: Logging plan of the running query

2023-10-24 Thread torikoshia
On 2023-10-24 16:12, Étienne BERSAC wrote: Hi, Yeah, and when we have a situation where we want to run pg_log_query_plan(), we can run it in any environment as long as it is bundled with the core. Is it possible to get the plan of a backend programmatically without access to the logs? Somethi

Re: RFC: Logging plan of the running query

2023-10-24 Thread Étienne BERSAC
> Hi, > > Yeah, and when we have a situation where we want to run > pg_log_query_plan(), we can run it in any environment as long as it > is bundled with the core. Is it possible to get the plan of a backend programmatically without access to the logs? Something like pg_get_query_plan(pid, forma

Re: RFC: Logging plan of the running query

2023-10-18 Thread James Coleman
On Wed, Oct 18, 2023 at 2:09 AM torikoshia wrote: > > On 2023-10-16 18:46, Ashutosh Bapat wrote: > > On Thu, Oct 12, 2023 at 6:51 PM torikoshia > > wrote: > >> > >> On 2023-10-11 16:22, Ashutosh Bapat wrote: > >> > > >> > Considering the similarity with auto_explain I wondered whether this > >> >

Re: RFC: Logging plan of the running query

2023-10-18 Thread Ashutosh Bapat
On Wed, Oct 18, 2023 at 11:39 AM torikoshia wrote: > > > > I am more interested in avoiding the duplication of code, esp. the > > first comment in my reply > > If there are no objections, I will try porting it to auto_explain and > see its feasibility. If you want it in core, you will need to por

Re: RFC: Logging plan of the running query

2023-10-17 Thread torikoshia
On 2023-10-16 18:46, Ashutosh Bapat wrote: On Thu, Oct 12, 2023 at 6:51 PM torikoshia wrote: On 2023-10-11 16:22, Ashutosh Bapat wrote: > > Considering the similarity with auto_explain I wondered whether this > function should be part of auto_explain contrib module itself? If we > do that user

Re: RFC: Logging plan of the running query

2023-10-16 Thread Ashutosh Bapat
On Thu, Oct 12, 2023 at 6:51 PM torikoshia wrote: > > On 2023-10-11 16:22, Ashutosh Bapat wrote: > > > > Considering the similarity with auto_explain I wondered whether this > > function should be part of auto_explain contrib module itself? If we > > do that users will need to load auto_explain ex

Re: RFC: Logging plan of the running query

2023-10-12 Thread torikoshia
On 2023-10-11 16:22, Ashutosh Bapat wrote: Like many others I think this feature is useful to debug a long running query. Sorry for jumping late into this. I have a few of high level comments Thanks for your comments! There is a lot of similarity between what this feature does and what au

Re: RFC: Logging plan of the running query

2023-10-11 Thread Ashutosh Bapat
Hi, On Tue, Oct 10, 2023 at 7:00 PM torikoshia wrote: > Thanks for your comments! > > I tested v30 patch with > v28-0002-Testing-attempt-logging-plan-on-ever-CFI-call.patch which makes > CFI() call ProcessLogQueryPlanInterrupt() internally, and confirmed that > very few logging queries failed wi

Re: RFC: Logging plan of the running query

2023-10-10 Thread torikoshia
On 2023-10-04 03:00, James Coleman wrote: and I think what we need to do is explicitly disallow running this code any time we are inside of lock acquisition code. Updated patch to check if any locks have already been acquired by examining MyProc->heldLocks. I'm not sure this change can "disa

Re: RFC: Logging plan of the running query

2023-10-06 Thread Andres Freund
Hi, On 2023-10-06 21:58:46 +0900, torikoshia wrote: > Yeah, I think it's a straightforward workaround. > And I'm also concerned that the condition of being in the process > of acquiring some kind of lock is too strict and will make it almost > impossible to output a running plan. How so? We shoul

Re: RFC: Logging plan of the running query

2023-10-06 Thread James Coleman
On Fri, Oct 6, 2023 at 8:58 AM torikoshia wrote: > > On 2023-10-04 03:00, James Coleman wrote: > > On Thu, Sep 7, 2023 at 2:09 AM torikoshia > > wrote: > >> > >> On 2023-09-06 11:17, James Coleman wrote: > >> > >> >> > I've never been able to reproduce it (haven't tested the new version, > >> >>

Re: RFC: Logging plan of the running query

2023-10-06 Thread torikoshia
On 2023-10-04 03:00, James Coleman wrote: On Thu, Sep 7, 2023 at 2:09 AM torikoshia wrote: On 2023-09-06 11:17, James Coleman wrote: >> > I've never been able to reproduce it (haven't tested the new version, >> > but v28 at least) on my M1 Mac; where I've reproduced it is on Debian >> > (firs

Re: RFC: Logging plan of the running query

2023-10-03 Thread James Coleman
On Thu, Sep 7, 2023 at 2:09 AM torikoshia wrote: > > On 2023-09-06 11:17, James Coleman wrote: > > >> > I've never been able to reproduce it (haven't tested the new version, > >> > but v28 at least) on my M1 Mac; where I've reproduced it is on Debian > >> > (first buster and now bullseye). > >> >

Re: RFC: Logging plan of the running query

2023-09-27 Thread Andrey Lepikhov
On 28/9/2023 09:04, torikoshia wrote: On 2023-09-25 18:49, Andrey Lepikhov wrote: On 25/9/2023 14:21, torikoshia wrote: On 2023-09-20 14:39, Lepikhov Andrei wrote: Hmm, as a test, I made sure to call ProcessLogQueryPlanInterrupt() on all CFI using v28-0002-Testing-attempt-logging-plan-on-ever

Re: RFC: Logging plan of the running query

2023-09-27 Thread torikoshia
On 2023-09-25 18:49, Andrey Lepikhov wrote: On 25/9/2023 14:21, torikoshia wrote: On 2023-09-20 14:39, Lepikhov Andrei wrote: Hmm, as a test, I made sure to call ProcessLogQueryPlanInterrupt() on all CFI using v28-0002-Testing-attempt-logging-plan-on-ever-CFI-call.patch, and then ran the foll

Re: RFC: Logging plan of the running query

2023-09-25 Thread Andrey Lepikhov
On 25/9/2023 14:21, torikoshia wrote: On 2023-09-20 14:39, Lepikhov Andrei wrote: Hmm, as a test, I made sure to call ProcessLogQueryPlanInterrupt() on all CFI using v28-0002-Testing-attempt-logging-plan-on-ever-CFI-call.patch, and then ran the following query but did not cause any problems.

Re: RFC: Logging plan of the running query

2023-09-25 Thread torikoshia
On 2023-09-20 14:39, Lepikhov Andrei wrote: Thanks for your reply. On Tue, Sep 19, 2023, at 8:39 PM, torikoshia wrote: On 2023-09-15 15:21, Lepikhov Andrei wrote: On Thu, Sep 7, 2023, at 1:09 PM, torikoshia wrote: I have explored this patch and, by and large, agree with the code. But some qu

Re: RFC: Logging plan of the running query

2023-09-19 Thread Lepikhov Andrei
On Tue, Sep 19, 2023, at 8:39 PM, torikoshia wrote: > On 2023-09-15 15:21, Lepikhov Andrei wrote: >> On Thu, Sep 7, 2023, at 1:09 PM, torikoshia wrote: >> I have explored this patch and, by and large, agree with the code. But >> some questions I want to discuss: >> 1. Maybe add a hook to give a cha

Re: RFC: Logging plan of the running query

2023-09-19 Thread torikoshia
On 2023-09-15 15:21, Lepikhov Andrei wrote: On Thu, Sep 7, 2023, at 1:09 PM, torikoshia wrote: On 2023-09-06 11:17, James Coleman wrote: It seems that we can know what queries were running from the stack traces you shared. As described above, I suspect a lock which was acquired prior to ProcessL

Re: RFC: Logging plan of the running query

2023-09-14 Thread Lepikhov Andrei
On Thu, Sep 7, 2023, at 1:09 PM, torikoshia wrote: > On 2023-09-06 11:17, James Coleman wrote: > It seems that we can know what queries were running from the stack > traces you shared. > As described above, I suspect a lock which was acquired prior to > ProcessLogQueryPlanInterrupt() caused the i

Re: RFC: Logging plan of the running query

2023-09-06 Thread torikoshia
On 2023-09-06 11:17, James Coleman wrote: > I've never been able to reproduce it (haven't tested the new version, > but v28 at least) on my M1 Mac; where I've reproduced it is on Debian > (first buster and now bullseye). > > I'm attaching several stacktraces in the hope that they provide some >

Re: RFC: Logging plan of the running query

2023-09-05 Thread James Coleman
On Tue, Sep 5, 2023 at 9:59 AM torikoshia wrote: > > On 2023-08-28 22:47, James Coleman wrote: > > On Mon, Aug 28, 2023 at 3:01 AM torikoshia > > wrote: > >> > >> On 2023-08-26 21:03, James Coleman wrote: > >> > On Fri, Aug 25, 2023 at 7:43 AM James Coleman wrote: > >> >> > >> >> On Thu, Aug 17,

Re: RFC: Logging plan of the running query

2023-09-05 Thread torikoshia
On 2023-08-28 22:47, James Coleman wrote: On Mon, Aug 28, 2023 at 3:01 AM torikoshia wrote: On 2023-08-26 21:03, James Coleman wrote: > On Fri, Aug 25, 2023 at 7:43 AM James Coleman wrote: >> >> On Thu, Aug 17, 2023 at 10:02 AM torikoshia >> wrote: >> > >> > On 2023-06-16 01:34, James Colema

Re: RFC: Logging plan of the running query

2023-08-28 Thread James Coleman
On Mon, Aug 28, 2023 at 3:01 AM torikoshia wrote: > > On 2023-08-26 21:03, James Coleman wrote: > > On Fri, Aug 25, 2023 at 7:43 AM James Coleman wrote: > >> > >> On Thu, Aug 17, 2023 at 10:02 AM torikoshia > >> wrote: > >> > > >> > On 2023-06-16 01:34, James Coleman wrote: > >> > > Attached is

Re: RFC: Logging plan of the running query

2023-08-28 Thread torikoshia
On 2023-08-26 21:03, James Coleman wrote: On Fri, Aug 25, 2023 at 7:43 AM James Coleman wrote: On Thu, Aug 17, 2023 at 10:02 AM torikoshia wrote: > > On 2023-06-16 01:34, James Coleman wrote: > > Attached is v28 > > which sets ProcessLogQueryPlanInterruptActive to false in errfinish > > whe

Re: RFC: Logging plan of the running query

2023-08-26 Thread James Coleman
On Fri, Aug 25, 2023 at 7:43 AM James Coleman wrote: > > On Thu, Aug 17, 2023 at 10:02 AM torikoshia > wrote: > > > > On 2023-06-16 01:34, James Coleman wrote: > > > Attached is v28 > > > which sets ProcessLogQueryPlanInterruptActive to false in errfinish > > > when necessary. Once built with th

  1   2   3   >