Re: Improve explicit cursor handling in pg_stat_statements

2025-06-05 Thread Michael Paquier
On Thu, Jun 05, 2025 at 07:42:48AM -0500, Sami Imseih wrote: > v5 attached. I'm OK with this version, so switching that as ready for committer. -- Michael signature.asc Description: PGP signature

Re: track generic and custom plans in pg_stat_statements

2025-06-05 Thread Sami Imseih
> Thanks for your work on this. > > Since we've changed the placement of these parameters > between parallel_workers and stats_since, we should also update > the documentation to reflect their new location. Absolutely. My miss. Here is v9 with the doc updated. Thanks! -- Sami v9-0001-Add-plan_

Re: track generic and custom plans in pg_stat_statements

2025-06-05 Thread Ilia Evdokimov
On 03.06.2025 06:31, Sami Imseih wrote: See v8 with the field names reorganized. Apologies if you received two identical emails. Since we've changed the placement of these parameters between parallel_workers and stats_since, we should also update the documentation to reflect their new loca

Re: track generic and custom plans in pg_stat_statements

2025-06-05 Thread Ilia Evdokimov
On 03.06.2025 06:31, Sami Imseih wrote: See v8 with the field names reorganized. Thanks for your work on this. Since we've changed the placement of these parameters between parallel_workers and stats_since, we should also update the documentation to reflect their new location. -- Best re

Re: Improve explicit cursor handling in pg_stat_statements

2025-06-05 Thread Sami Imseih
> + * If an direction_keyword (i.e., FETCH FORWARD) is used, set this field > + * to distinguish it from its numeric counterpart (i.e., FETCH 1). This > + * value is set only within gram.y. > > One nitpick comment here is that I would have mentioned that this > matters for query jumbling. Done

Re: Improve explicit cursor handling in pg_stat_statements

2025-06-04 Thread Michael Paquier
On Wed, Jun 04, 2025 at 11:51:56AM -0500, Sami Imseih wrote: >> An enum would shine here IMO, because the value could be >> self-documented and one would not need to guess what each integer >> value means. That could be something like a direction_keyword, filled >> with FETCH_KEYWORD_LAST, FETCH_K

Re: Improve explicit cursor handling in pg_stat_statements

2025-06-04 Thread Sami Imseih
> Hmm. I was not sure if we'd really need to get down to that, as most > of the grammar keywords have the same parsed meaning, but there's a > good point with LAST for example that uses a negative value for > howMany. If we silence the number, LAST would map with everything > else that has FETCH_

Re: Improve explicit cursor handling in pg_stat_statements

2025-06-03 Thread Michael Paquier
On Tue, Jun 03, 2025 at 01:04:36PM -0500, Sami Imseih wrote: > v3 is what I'm thinking: In FetchStmt, introduce a new field called > explicit_direction (maybe there's a better name?), which is an int that > we can assign a value to in gram.c. The value will be 0 if no explicit > direction is used.

Re: Improve explicit cursor handling in pg_stat_statements

2025-06-03 Thread Sami Imseih
> Hmm, we could do that to differentiate the keyword ALL. I had a thought > earlier about differentiating the other keywords as well: FIRST, LAST, > BACKWARD, FORWARD, and ABSOLUTE. Initially, I thought it might > be a bit too much, but I do see the merit in this approach, as these are > syntactica

Re: Improve explicit cursor handling in pg_stat_statements

2025-06-03 Thread Sami Imseih
> Should we offer something more consistent with DeallocateStmt, where > we have a boolean flag that would be set when ALL is specified, > included in the jumbling? This would mean two separate entries: one > for the constants and one for ALL. Hmm, we could do that to differentiate the keyword AL

Re: Improve explicit cursor handling in pg_stat_statements

2025-06-03 Thread Michael Paquier
On Mon, Jun 02, 2025 at 02:44:36PM -0500, Sami Imseih wrote: > Since the FETCH case is clear-cut, here is a patch that normalizes variable > fetch sizes in a FETCH command. At a minimum, we can apply this patch. > I’ve also added tests in pg_stat_statements utility.sql to demonstr

Re: track generic and custom plans in pg_stat_statements

2025-06-02 Thread Sami Imseih
> Now, one thing I don't like is the fact that the columns stats_since > and minmax_stats_since > are in between counters all of the sudden. I think we either need to > move them to > the front of the view, after the query field or within this patch > move them after the > new generic/custom plan

Re: Improve explicit cursor handling in pg_stat_statements

2025-06-02 Thread Sami Imseih
> > The FETCH and CLOSE are already not clear to what underlying SQL > > they are referring to, and there is not much chance to actually > > improve that unless > > we track a cursor queryId in pg_stat_statements ( at that point we can show > > that > > F

Re: track generic and custom plans in pg_stat_statements

2025-06-02 Thread Sami Imseih
> I reviewed v6: > > - applies to master cleanly, builds, tests pass and all works as expected > - overall, the patch looks great and I found no major issues > - tests and docs look good overall Thanks for the valuable feedback! > - in docs, one minor comment: > > "Total number of statements

Re: track generic and custom plans in pg_stat_statements

2025-05-31 Thread Nikolay Samokhvalov
On Sat, May 31, 2025 at 5:06 PM Nikolay Samokhvalov wrote: > On Thu, May 29, 2025 at 11:56 AM Sami Imseih wrote: > >> It turns out that 1722d5eb05d8e reverted 525392d5727f, which >> made CachedPlan available in QueryDesc and thus >> available to pgss_ExecutorEnd. >> >> So now we have to make Cac

Re: track generic and custom plans in pg_stat_statements

2025-05-31 Thread Nikolay Samokhvalov
On Thu, May 29, 2025 at 11:56 AM Sami Imseih wrote: > It turns out that 1722d5eb05d8e reverted 525392d5727f, which > made CachedPlan available in QueryDesc and thus > available to pgss_ExecutorEnd. > > So now we have to make CachedPlan available to QueryDesc as > part of this change. The reason t

Re: Add comment explaining why queryid is int64 in pg_stat_statements

2025-05-30 Thread Michael Paquier
On Fri, May 30, 2025 at 11:09:23PM +1200, David Rowley wrote: > Now done. Cool, thanks. Just did the same for the query ID, closing the open item. -- Michael signature.asc Description: PGP signature

Re: Add comment explaining why queryid is int64 in pg_stat_statements

2025-05-30 Thread Shaik Mohammad Mujeeb
Thanks for making the appropriate changes, David. Now everything make sense. :) On Fri, 30 May 2025 16:39:23 +0530 David Rowley wrote --- On Fri, 30 May 2025 at 11:51, David Rowley < mailto:dgrowle...@gmail.com > wrote: > > On Fri, 30 May 2025 at 11:35, Michael Paquier < ma

Re: Add comment explaining why queryid is int64 in pg_stat_statements

2025-05-30 Thread David Rowley
On Fri, 30 May 2025 at 11:51, David Rowley wrote: > > On Fri, 30 May 2025 at 11:35, Michael Paquier wrote: > > Thanks, Nathan. Let's proceed with the change then. David, would you > > prefer handling the patch you have written by yourself for the query > > ID part? > > Yes. I'll look at that ag

Re: Add comment explaining why queryid is int64 in pg_stat_statements

2025-05-30 Thread David Rowley
Thanks for the review. On Wed, 21 May 2025 at 03:35, Sami Imseih wrote: > 2/ Should "DatumGetInt64(hash_any_extended(.." be turned into a > macro since it's used in > several places? Also, we can add a comment in the macro such as > " > Output the queryId as an int64 rather than a uint64, to

Re: Add comment explaining why queryid is int64 in pg_stat_statements

2025-05-29 Thread David Rowley
On Fri, 30 May 2025 at 11:35, Michael Paquier wrote: > Thanks, Nathan. Let's proceed with the change then. David, would you > prefer handling the patch you have written by yourself for the query > ID part? Yes. I'll look at that again shortly. David

Re: Add comment explaining why queryid is int64 in pg_stat_statements

2025-05-29 Thread Michael Paquier
On Thu, May 29, 2025 at 09:28:35AM -0500, Nathan Bossart wrote: > On Thu, May 29, 2025 at 01:53:07PM +0900, Michael Paquier wrote: >> Now, I don't really want to take a leap of faith without the RMT being >> OK with that now that we are in beta1. > > After reading through this thread and the lates

Re: Improve explicit cursor handling in pg_stat_statements

2025-05-29 Thread Sami Imseih
> > postgres_fdw, as an example, in which cursor name get reused > > for different queries. Notice below "c1" and "c2" is reused for different > > queries, so now what underlying sql is FETCH, i.e. FETCH 100 FROM c1 > > referring > > to? v2-0001 does not help us with the FETCH problem > > because

Re: track generic and custom plans in pg_stat_statements

2025-05-29 Thread Sami Imseih
It turns out that 1722d5eb05d8e reverted 525392d5727f, which made CachedPlan available in QueryDesc and thus available to pgss_ExecutorEnd. So now we have to make CachedPlan available to QueryDesc as part of this change. The reason the patch was reverted is related to a memory leak [0] in the Buil

Re: track generic and custom plans in pg_stat_statements

2025-05-29 Thread Sami Imseih
> It turns out that 1722d5eb05d8e reverted 525392d5727f, which > made CachedPlan available in QueryDesc and thus > available to pgss_ExecutorEnd. I also forgot to mention, the revert also removes the generic plan is_reused logic: ``` boolis_reused; /* is it a reused gener

Re: Add comment explaining why queryid is int64 in pg_stat_statements

2025-05-29 Thread Nathan Bossart
On Thu, May 29, 2025 at 01:53:07PM +0900, Michael Paquier wrote: > On Thu, May 22, 2025 at 01:01:14PM +0900, Michael Paquier wrote: >> I have added an open item about the plan ID part as it applies to v18, >> adding the RMT in CC to get an opinion. If we cannot get a consensus >> on all that, lett

Re: Add comment explaining why queryid is int64 in pg_stat_statements

2025-05-28 Thread Michael Paquier
On Thu, May 22, 2025 at 01:01:14PM +0900, Michael Paquier wrote: > I have added an open item about the plan ID part as it applies to v18, > adding the RMT in CC to get an opinion. If we cannot get a consensus > on all that, letting things as they are is still logically correct, > even with the -Ww

Re: Add comment explaining why queryid is int64 in pg_stat_statements

2025-05-21 Thread Michael Paquier
On Thu, May 22, 2025 at 02:36:38PM +1200, David Rowley wrote: > You could argue that if it reduces the locations that need to be > changed by using a typedef, then it's a win. But there are also > negative aspects to typedefs that need to be considered. For me, those > are the added level of indire

Re: Add comment explaining why queryid is int64 in pg_stat_statements

2025-05-21 Thread David Rowley
On Thu, 22 May 2025 at 02:58, Peter Eisentraut wrote: > > On 20.05.25 08:38, Michael Paquier wrote: > > On Tue, May 20, 2025 at 05:51:51PM +1200, David Rowley wrote: > >> Given the planId stuff is new and has the same issue, I think that > >> pushes me towards thinking now is better than later for

Re: Add comment explaining why queryid is int64 in pg_stat_statements

2025-05-21 Thread Peter Eisentraut
On 20.05.25 08:38, Michael Paquier wrote: On Tue, May 20, 2025 at 05:51:51PM +1200, David Rowley wrote: Given the planId stuff is new and has the same issue, I think that pushes me towards thinking now is better than later for fixing both. I'm happy to adjust my patch unless you've started work

Re: Add comment explaining why queryid is int64 in pg_stat_statements

2025-05-21 Thread Julien Rouhaud
On Wed, May 21, 2025 at 09:34:02AM +0900, Michael Paquier wrote: > On Tue, May 20, 2025 at 11:28:17PM +1200, David Rowley wrote: > > Certainly, a bit late, yes. It basically requires implementing > > unsigned types on the SQL level. I believe there are a few sunken > > ships along that coastline, a

Re: Add comment explaining why queryid is int64 in pg_stat_statements

2025-05-20 Thread Michael Paquier
On Tue, May 20, 2025 at 10:35:39AM -0500, Sami Imseih wrote: > 1/ this was missed in pg_stat_statements > ./contrib/pg_stat_statements/pg_stat_statements.c:uint64 > saved_queryId = pstmt->queryId; Right. Some greps based on "queryid" and "query_id" point th

Re: Add comment explaining why queryid is int64 in pg_stat_statements

2025-05-20 Thread Michael Paquier
On Tue, May 20, 2025 at 11:28:17PM +1200, David Rowley wrote: > Certainly, a bit late, yes. It basically requires implementing > unsigned types on the SQL level. I believe there are a few sunken > ships along that coastline, and plenty of history in the archives if > you want to understand some of

Re: Add comment explaining why queryid is int64 in pg_stat_statements

2025-05-20 Thread Sami Imseih
nable and cleaner. A few comments on the patches: 1/ this was missed in pg_stat_statements ./contrib/pg_stat_statements/pg_stat_statements.c:uint64 saved_queryId = pstmt->queryId; 2/ Should "DatumGetInt64(hash_any_extended(.." be turned into a macro since it's used in s

Re: Add comment explaining why queryid is int64 in pg_stat_statements

2025-05-20 Thread David Rowley
On Tue, 20 May 2025 at 18:12, Julien Rouhaud wrote: > I don't think it was discussed, but why not go the other way, keep everything > as uint64 and expose an uint64 datatype at the SQL level instead? That's > something I actually want pretty often so I would be happy to have that > natively, whet

Re: Add comment explaining why queryid is int64 in pg_stat_statements

2025-05-19 Thread Michael Paquier
e/rewriteHandler.c | 2 +- src/backend/tcop/postgres.c | 4 +-- src/backend/utils/activity/backend_status.c | 12 +++ src/backend/utils/adt/pgstatfuncs.c | 4 +-- .../pg_stat_statements/pg_stat_statements.c | 36 +-- 16 files changed, 67 inse

Re: Add comment explaining why queryid is int64 in pg_stat_statements

2025-05-19 Thread Julien Rouhaud
On Tue, May 20, 2025 at 02:09:13PM +0900, Michael Paquier wrote: > On Mon, May 19, 2025 at 08:43:25PM -0700, Lukas Fittl wrote: > > Yeah, +1 to making this consistent across both query ID and the new plan > > ID, and changing both to int64 internally seems best. > > Any thoughts from others? I'm O

Re: Add comment explaining why queryid is int64 in pg_stat_statements

2025-05-19 Thread David Rowley
On Tue, 20 May 2025 at 17:09, Michael Paquier wrote: > > On Mon, May 19, 2025 at 08:43:25PM -0700, Lukas Fittl wrote: > > Yeah, +1 to making this consistent across both query ID and the new plan > > ID, and changing both to int64 internally seems best. > > Any thoughts from others? I'm OK to take

Re: Add comment explaining why queryid is int64 in pg_stat_statements

2025-05-19 Thread Michael Paquier
On Mon, May 19, 2025 at 08:43:25PM -0700, Lukas Fittl wrote: > Yeah, +1 to making this consistent across both query ID and the new plan > ID, and changing both to int64 internally seems best. Any thoughts from others? I'm OK to take the extra step of switching both fields on HEAD and write a patc

Re: Add comment explaining why queryid is int64 in pg_stat_statements

2025-05-19 Thread Lukas Fittl
On Mon, May 19, 2025 at 8:12 PM Michael Paquier wrote: > On Tue, May 20, 2025 at 02:03:37PM +1200, David Rowley wrote: > > Aside from the struct field types changing for Query.queryId, > > PlannedStmt.queryId and PgBackendStatus.st_query_id, the > > external-facing changes are limited to: > > > >

Re: Add comment explaining why queryid is int64 in pg_stat_statements

2025-05-19 Thread Michael Paquier
On Tue, May 20, 2025 at 02:03:37PM +1200, David Rowley wrote: > Aside from the struct field types changing for Query.queryId, > PlannedStmt.queryId and PgBackendStatus.st_query_id, the > external-facing changes are limited to: > > 1. pgstat_report_query_id() now returns int64 instead of uint64 > 2

Re: Add comment explaining why queryid is int64 in pg_stat_statements

2025-05-19 Thread David Rowley
On Tue, 20 May 2025 at 03:05, Peter Eisentraut wrote: > Or why not store query IDs as int64_t > internally, too? I had the same thought. Changing to int64 seems like a good and less bug-prone tidy-up. I expected we ended up with uint64 as the previous type was uint32, and uint64 is the natural se

Re: Add comment explaining why queryid is int64 in pg_stat_statements

2025-05-19 Thread Sami Imseih
FWIW, all the hash function SQL interfaces, \df hash*, have this behavior in which the result is a signed (int/bigint), but the internal representation of the hash is an unsigned (int/bigint). I am not sure why a comment is needed specifically for pg_stat_statements -- Sami Imseih Amazon Web

Re: Add comment explaining why queryid is int64 in pg_stat_statements

2025-05-19 Thread Peter Eisentraut
On 17.05.25 14:49, Michael Paquier wrote: On Fri, May 16, 2025 at 04:05:01PM +0530, Shaik Mohammad Mujeeb wrote: This conversion is intentional - most likely to match the bigint type of the queryid column in pg_stat_statements. However, without an explicit comment, this can be misleading. A

Re: Add comment explaining why queryid is int64 in pg_stat_statements

2025-05-18 Thread Junwang Zhao
t; > > > Thanks and Regards, > Shaik Mohammad Mujeeb > Member Technical Staff > Zoho Corp > > > > On Sat, 17 May 2025 18:19:23 +0530 *Michael Paquier > >* wrote --- > > On Fri, May 16, 2025 at 04:05:01PM +0530, Shaik Mohammad Mujeeb wrote: > > T

Re: Add comment explaining why queryid is int64 in pg_stat_statements

2025-05-17 Thread Shaik Mohammad Mujeeb
Member Technical Staff Zoho Corp On Sat, 17 May 2025 18:19:23 +0530 Michael Paquier wrote --- On Fri, May 16, 2025 at 04:05:01PM +0530, Shaik Mohammad Mujeeb wrote: > This conversion is intentional - most likely to match the bigint > type of the queryid column in pg_stat_state

Re: Add comment explaining why queryid is int64 in pg_stat_statements

2025-05-17 Thread Michael Paquier
On Fri, May 16, 2025 at 04:05:01PM +0530, Shaik Mohammad Mujeeb wrote: > This conversion is intentional - most likely to match the bigint > type of the queryid column in pg_stat_statements. However, without > an explicit comment, this can be misleading. A beginner reading this > might

Re: Add comment explaining why queryid is int64 in pg_stat_statements

2025-05-16 Thread wenhui qiu
fter assigning it to *queryid* (which is an *int64*), it becomes *-5699455397997599811* *due to overflow*. > This conversion is intentional - most likely to match the *bigint* type of the *queryid* column in *pg_stat_statements*. However, > without an explicit comment, this can be misleading.

Re: Add comment explaining why queryid is int64 in pg_stat_statements

2025-05-16 Thread Shaik Mohammad Mujeeb
gning it to queryid (which is an int64), it becomes -5699455397997599811 due to overflow. This conversion is intentional - most likely to match the bigint type of the queryid column in pg_stat_statements. However, without an explicit comment, this can be misleading. A beginner reading th

Re: Add comment explaining why queryid is int64 in pg_stat_statements

2025-05-16 Thread Ilia Evdokimov
think this is intentional - the queryid is cast to /int64/ *to match the bigint type of the queryid column in the pg_stat_statements view*. Please find the attached patch, which adds a clarifying comment to make the rationale explicit and avoid potential confusion for future readers

Add comment explaining why queryid is int64 in pg_stat_statements

2025-05-15 Thread Shaik Mohammad Mujeeb
match the bigint type of the queryid column in the pg_stat_statements view. Please find the attached patch, which adds a clarifying comment to make the rationale explicit and avoid potential confusion for future readers. Thanks and Regards, Shaik Mohammad Mujeeb Member Technical Staff Zoho

[PATCH v2] Re: Cancel problems of query to pg_stat_statements

2025-05-12 Thread Roman Khapov
e the query execution stuck on PGSS_TEXT_FILE file >> reading in function qtext_load_file, which doesn't have CHECK_FOR_INTERRUPTS >> in the read cycle. In addition to our case with large PGSS_TEXT_FILE (and >> maybe the problems with virtual disk i/o) that can exp

Re: Improve explicit cursor handling in pg_stat_statements

2025-05-07 Thread Michael Paquier
nd there is not much chance to actually > improve that unless > we track a cursor queryId in pg_stat_statements ( at that point we can show > that > FETCH or CLOSE refer to this specific cursor statement ). I don't really have an issue for FETCH with the number as the name is s

Re: Improve explicit cursor handling in pg_stat_statements

2025-05-02 Thread Sami Imseih
1; postgres=# select * from t2, t ; id | id + 1 | 1 (1 row) postgres=# select * from t, t2 ; id | id + 1 | 1 (1 row) ``` on the remote side ``` postgres=# select calls, query from pg_stat_statements where query like '% c%';

Re: Improve explicit cursor handling in pg_stat_statements

2025-04-30 Thread Michael Paquier
On Wed, Apr 30, 2025 at 02:43:41PM -0500, Sami Imseih wrote: > I also want to add that the decision to not normalize the cursor name in > the FETCH command is because it would not make sense to combine > FETCH commands for various cursors into the same entry. - calls | rows |

Improve explicit cursor handling in pg_stat_statements

2025-04-30 Thread Sami Imseih
Hi hackers, I recently looked into a workload that makes heavy use of explicit cursors, and I found that pg_stat_statements can become a bottleneck. The application in question declares hundreds of cursors, and for each one, performs many FETCH and MOVE operations with varying fetch sizes. As a

Re: Improve explicit cursor handling in pg_stat_statements

2025-04-30 Thread Sami Imseih
I forgot to add the proper tests to the Normalize cursor utility statements. Reattaching v2. I also want to add that the decision to not normalize the cursor name in the FETCH command is because it would not make sense to combine FETCH commands for various cursors into the same entry. Regards, -

Re: Cancel problems of query to pg_stat_statements

2025-04-27 Thread Andrey Borodin
> in the read cycle. In addition to our case with large PGSS_TEXT_FILE (and > maybe the problems with virtual disk i/o) that can explain uncancellable > pg_stat_statements queries. I'm afraid it might be not so easy to add CHECK_FOR_INTERRUPTS there. Most probably you was holding a

Cancel problems of query to pg_stat_statements

2025-04-24 Thread Roman Khapov
ancellable pg_stat_statements queries. Also, the reading block size can be reduced from 1GB to 32MB in order to increase the frequency of CHECK_FOR_INTERRUPTS calls without qtext_load_file performance degradation. To check that I did some little testing with fio like: fio --name=readtest --filename=./random-bytes

Re: track generic and custom plans in pg_stat_statements

2025-04-10 Thread Sami Imseih
e explain, and we may not even need it at all since generic plans are already displayed with $1, $2 parameters. Let me know if you have other comments for v5, the pg_stat_statements enhancements. — Sami Imseih Amazon Web Services (AWS)

Re: track generic and custom plans in pg_stat_statements

2025-04-10 Thread Sami Imseih
rebased in the attached v5. -- Sami Imseih Amazon Web Services (AWS) v5-0001-Add-plan_cache-counters-to-pg_stat_statements.patch Description: Binary data

Re: Sample rate added to pg_stat_statements

2025-04-09 Thread Ilia Evdokimov
Since at the moment these changes are not a priority, and it's more important to address [0] for pg_stat_statements, I'm marking this patch as "Returned with Feedback". [0]: https://www.postgresql.org/message-id/Zz0MFPq1s4WFxxpL%40paquier.xyz -- Best regards, Ilia Evdokimov, Tantor Labs LLC.

Re: track generic and custom plans in pg_stat_statements

2025-04-08 Thread Ilia Evdokimov
After the introduction of pg_overexplain extension and Robert's comment [0], I'm starting to have doubts about whether it's still appropriate to add this information to EXPLAIN itself. If there are compelling reasons that showing the plan type would be broadly useful to users in EXPLAIN, I’m ha

Re: pg_stat_statements: improve loading and saving routines for the dump file

2025-04-05 Thread m . litsarev
Hi! Thank you for detailed explanations. Respectfully, Mikhail Litsarev, Postgres Professional: https://postgrespro.com

Re: Sample rate added to pg_stat_statements

2025-04-04 Thread Ilia Evdokimov
Hi, I attached rebased patch v20. -- Best regards, Ilia Evdokimov, Tantor Labs LLC. From accee46b077e1debdc3db61555923b2c11e18d5e Mon Sep 17 00:00:00 2001 From: Ilia Evdokimov Date: Fri, 21 Mar 2025 17:37:08 +0300 Subject: [PATCH v20] Allow setting sample rate for pg_stat_statements New

Re: pg_stat_statements: improve loading and saving routines for the dump file

2025-04-03 Thread Tom Lane
m.litsa...@postgrespro.ru writes: >> What does this patch give on aglobal scale? Does it save much memory or >> increase performance? How many times? > This patch makes the code semantically more correct and we don't lose > anything. It is obviously not about performance or memory optimisation.

Re: pg_stat_statements and "IN" conditions

2025-03-22 Thread Sami Imseih
> Sure, it's important and I'm planning to tackle this next. If you want, > we can collaborate on a patch for that. I spent some time looking some more at this, and I believe all that needs to be done is check for a PRAM node with a type of PARAM_EXTERN. During planning the planner turns the Para

Re: pg_stat_statements and "IN" conditions

2025-03-19 Thread Dmitry Dolgov
> On Tue, Mar 18, 2025 at 02:54:18PM GMT, Sami Imseih wrote: > I want to mention that the current patch does not deal > with external parameters ( prepared statements ) [0] [1]. This could be a > follow-up, as it may need some further discussion. It is important to > address this case, IMO. Sure,

Re: pg_stat_statements and "IN" conditions

2025-03-19 Thread Dmitry Dolgov
> On Tue, Mar 18, 2025 at 07:33:25PM GMT, Álvaro Herrera wrote: > > By the way, I'm still open to adding the 'powers' mechanism that was > discussed earlier and other simple additions on top of what's there now, > if you have some spare energy to spend on this. (For full disclosure: > the "powers"

Re: pg_stat_statements and "IN" conditions

2025-03-18 Thread Álvaro Herrera
On 2025-Mar-18, Sami Imseih wrote: > I want to mention that the current patch does not deal > with external parameters ( prepared statements ) [0] [1]. This could be a > follow-up, as it may need some further discussion. It is important to > address this case, IMO. Yes, I realize that. Feel free

Re: pg_stat_statements and "IN" conditions

2025-03-18 Thread Álvaro Herrera
st that has zero elements. Anyway, something to play with. BTW, it's fun to execute a query that's literally select col from tab where col in (1 /*, ... */); and then select col from tab where col in (1, 2); because now you have two entries in pg_stat_statements with visibly the same quer

Re: pg_stat_statements and "IN" conditions

2025-03-18 Thread Sami Imseih
I want to mention that the current patch does not deal with external parameters ( prepared statements ) [0] [1]. This could be a follow-up, as it may need some further discussion. It is important to address this case, IMO. [0] https://www.postgresql.org/message-id/CAA5RZ0uGfxXyzhp9N5nfsS%2BZqF5ng

Re: pg_stat_statements and "IN" conditions

2025-03-18 Thread Dmitry Dolgov
> On Mon, Mar 17, 2025 at 08:14:16PM GMT, Álvaro Herrera wrote: > On 2025-Mar-17, Álvaro Herrera wrote: > > > You can see my patch on top of yours here: > > https://github.com/alvherre/postgres/commits/query_id_squash_values/ > > and the CI run here: > > https://cirrus-ci.com/build/5660053472018432

Re: pg_stat_statements and "IN" conditions

2025-03-17 Thread Álvaro Herrera
On 2025-Mar-17, Álvaro Herrera wrote: > You can see my patch on top of yours here: > https://github.com/alvherre/postgres/commits/query_id_squash_values/ > and the CI run here: > https://cirrus-ci.com/build/5660053472018432 Heh, this blew up on bogus SGML markup :-( Fixed and running again: http

Re: pg_stat_statements and "IN" conditions

2025-03-17 Thread Álvaro Herrera
data ++-- +(0 rows) + +SELECT query, calls FROM pg_stat_statements ORDER BY query COLLATE "C"; + query| calls ++--- + SELECT * FROM test_squash_cast WHERE data IN +| 1 +

Re: pg_stat_statements and "IN" conditions

2025-03-17 Thread vignesh C
On Mon, 17 Mar 2025 at 13:42, Dmitry Dolgov <9erthali...@gmail.com> wrote: > > > On Mon, Mar 17, 2025 at 12:07:44PM GMT, vignesh C wrote: > > > > I noticed that the feedback from Sami at [1] has not yet been > > addressed, I have changed the status to Waiting on Author, kindly > > address them and

Re: pg_stat_statements and "IN" conditions

2025-03-17 Thread Álvaro Herrera
On 2025-Mar-17, Dmitry Dolgov wrote: > I'm afraid there is a disagreement about this part of the feedback. I'm > not yet convinced about the idea suggested over there (treating mutable > functions in the same way as constants) and not planning to change > anything, at least not in the current vers

Re: pg_stat_statements and "IN" conditions

2025-03-17 Thread Dmitry Dolgov
> On Mon, Mar 17, 2025 at 12:07:44PM GMT, vignesh C wrote: > > I noticed that the feedback from Sami at [1] has not yet been > addressed, I have changed the status to Waiting on Author, kindly > address them and update the status to Needs review. > [1] - > https://www.postgresql.org/message-id/CAA

Re: pg_stat_statements and "IN" conditions

2025-03-16 Thread vignesh C
On Mon, 3 Mar 2025 at 17:26, Álvaro Herrera wrote: > > On 2025-Feb-18, Sami Imseih wrote: > > > > It's not a question about whether it's possible to implement this, > > > but about whether it makes sense. In case of plain constants it's > > > straightforward -- they will not change anything meanin

Re: track generic and custom plans in pg_stat_statements

2025-03-15 Thread Sami Imseih
> Thank you for your patch. It is really useful for tracking the history > of generic and custom plan usage. Thanks for the review! > 1. Is there any reason for the double check of cplan != NULL? It seems > unnecessary, and we could simplify it to: > > -if (cplan && cplan->status == PLAN_CACHE_ST

Re: track generic and custom plans in pg_stat_statements

2025-03-10 Thread Sami Imseih
> I don't quite understand why do we need to differentiate between > PLAN_CACHE_STATUS_GENERIC_PLAN_BUILD and > PLAN_CACHE_STATUS_GENERIC_PLAN_REUSE? > We could simply keep PLAN_CACHE_STATUS_GENERIC_PLAN_REUSE. > I don't think users would see much of a difference in either

Re: track generic and custom plans in pg_stat_statements

2025-03-10 Thread Ilia Evdokimov
to differentiate between PLAN_CACHE_STATUS_GENERIC_PLAN_BUILD and PLAN_CACHE_STATUS_GENERIC_PLAN_REUSE? We could simply keep PLAN_CACHE_STATUS_GENERIC_PLAN_REUSE. I don't think users would see much of a difference in either pg_stat_statements or EXPLAIN. As for EXPLAIN, maybe we should include this in VERBOSE mode? -

Re: track generic and custom plans in pg_stat_statements

2025-03-05 Thread Ilia Evdokimov
Hi, Thank you for your patch. It is really useful for tracking the history of generic and custom plan usage. At first glance, I have the following suggestions for improvement: 1. Is there any reason for the double check of cplan != NULL? It seems unnecessary, and we could simplify it to: -

Re: track generic and custom plans in pg_stat_statements

2025-03-05 Thread Sami Imseih
> > Please see v2 > oops, had to fix a typo in meson.build. Please see v3. -- Sami v3-0001-add-plan_cache-counters-to-pg_stat_statements.patch Description: Binary data

Re: track generic and custom plans in pg_stat_statements

2025-03-05 Thread Sami Imseih
trong feelings about this. > FILE: contrib/pg_stat_statements/expected/plan_cache.out > > These tests seem very verbose. Why not just prepare a simple query: > > prepare foo as select $1 > 0; > execute foo(1); > ...then tweak things via plan_cache_mode to ensure we test the ri

Re: track generic and custom plans in pg_stat_statements

2025-03-05 Thread Greg Sabino Mullane
being somewhat useful (especially for non-interactive things like auto_explain), so a weak +1 on that. Definitely not useful for pg_stat_database IMHO. Some quick comments on the patch: FILE: contrib/pg_stat_statements/expected/plan_cache.out These tests seem very verbose. Why not just prepare a

track generic and custom plans in pg_stat_statements

2025-03-05 Thread Sami Imseih
information cumulatively and over time. [0] had intentions to add these counters to pg_stat_statements, but was withdrawn due to timing with the commitfest at the time [0] and never picked back up again. I think it's useful to add these counters. Therefore, the attached patch adds two new colum

Re: pg_stat_statements and "IN" conditions

2025-03-03 Thread Sami Imseih
> > > It's not a question about whether it's possible to implement this, > > > but about whether it makes sense. In case of plain constants it's > > > straightforward -- they will not change anything meaningfully and > > > hence could be squashed from the query. Now for a function, that > > > might

Re: pg_stat_statements and "IN" conditions

2025-03-03 Thread Dmitry Dolgov
> On Mon, Mar 03, 2025 at 12:56:24PM GMT, Álvaro Herrera wrote: > In the meantime, here's v28 which is Dmitry's v27 plus pgindent. No > other changes. Dmitry, were you planning to submit a new version? Nope, there was nothing I wanted to add so far.

Re: Sample rate added to pg_stat_statements

2025-03-03 Thread Ilia Evdokimov
On 20.02.2025 04:04, Sami Imseih wrote: In my opinion, sample rate is a better fit for pg_stat_statements, since the queries that you care about the most are usually the most frequently executed. Sampling them will still provide enough good data without the risk of not capturing statistics

Re: pg_stat_statements and "IN" conditions

2025-03-03 Thread Álvaro Herrera
5e8d7a960ebaf12e1a5798fb1a8941b55771fcc Mon Sep 17 00:00:00 2001 From: Dmitrii Dolgov <9erthali...@gmail.com> Date: Tue, 3 Dec 2024 14:55:45 +0100 Subject: [PATCH v28] Prevent jumbling of every element in ArrayExpr pg_stat_statements produces multiple entries for queries like SELECT

Re: Sample rate added to pg_stat_statements

2025-02-26 Thread Ilia Evdokimov
On 20.02.2025 03:32, Sami Imseih wrote: As you can see, the query has not been normalized. Is it a bug, expected or undefined behavior? No, this behavior is expected. The ability to normalize a statement is only available during post_parse_analyze (pgss_post_parse_analyze), as this is when we

Re: Redact user password on pg_stat_statements

2025-02-25 Thread Sami Imseih
> > Well sure, but best effort is better than no effort at all. Preventing > CREATE/ALTER will catch 99% of items, and as I advocated, there really is no > reason for them to be in pg_stat_statements in the first place. > >> >> The clients that set passwords could si

Re: Redact user password on pg_stat_statements

2025-02-25 Thread Greg Sabino Mullane
On Tue, Feb 25, 2025 at 10:12 AM Sami Imseih wrote: > > What about a more general solution, such as a flag to turn off logging > of ALTER ROLE statements completely? > > IMO, flags for a specific type of utility statement seems way too much for > pg_stat_statements, and

Re: Redact user password on pg_stat_statements

2025-02-25 Thread Tom Lane
Sami Imseih writes: >> What about a more general solution, such as a flag to turn off logging of >> ALTER ROLE statements completely? > IMO, flags for a specific type of utility statement seems way too much > for pg_stat_statements, and this will also not completely prevent le

Re: Redact user password on pg_stat_statements

2025-02-25 Thread Sami Imseih
> What about a more general solution, such as a flag to turn off logging of > ALTER ROLE statements completely? IMO, flags for a specific type of utility statement seems way too much for pg_stat_statements, and this will also not completely prevent leaking plain text passwords from all way

Re: Redact user password on pg_stat_statements

2025-02-25 Thread Greg Sabino Mullane
What about a more general solution, such as a flag to turn off logging of ALTER ROLE statements completely? Does anyone really need to know the standard deviation of the timings for "ALTER ROLE alice SET work_mem='50MB'"? Let's be honest, there are a lot of things that g

Re: Redact user password on pg_stat_statements

2025-02-25 Thread Andrew Dunstan
On 2025-02-24 Mo 11:04 AM, Peter Eisentraut wrote: On 21.02.25 17:38, Andrew Dunstan wrote: I don't think this is such a terrible kluge. I think it's different from the server log case, which after all requires access to the server file system to exploit. To me, the mechanism by which this

Re: Redact user password on pg_stat_statements

2025-02-25 Thread Matheus Alcantara
Thanks for all the comments on this folks! I probably underestimated this change. Thanks all. -- Matheus Alcantara

Re: Redact user password on pg_stat_statements

2025-02-24 Thread Peter Eisentraut
On 21.02.25 17:38, Andrew Dunstan wrote: I don't think this is such a terrible kluge. I think it's different from the server log case, which after all requires access to the server file system to exploit. To me, the mechanism by which this patch works is completely nonobvious and coincidental

  1   2   3   4   5   6   7   8   9   10   >