Re: add function for creating/attaching hash table in DSM registry

2025-06-12 Thread Sami Imseih
> Okay, I've done this in the attached patch. Thanks! v7 LGTM. -- Sami

Re: queryId constant squashing does not support prepared statements

2025-06-12 Thread Sami Imseih
On Thu, Jun 12, 2025 at 11:32 AM Álvaro Herrera wrote: > > Hello, > > I have pushed that now, thanks! > and here's a rebase of patch 0003 to add support > for PARAM_EXTERN. I'm not really sure about this one yet ... see v11. I added a missing test to show how external param normalization behav

Re: pg_get_multixact_members not documented

2025-06-11 Thread Sami Imseih
> Attached patch removing extra comma. Otherwise LGTM. Thanks! For v4, the final comma in the list is grammatically correct, and it’s the style we use throughout the docs. I marked the patch RFC. -- Sami

Re: add function for creating/attaching hash table in DSM registry

2025-06-11 Thread Sami Imseih
I tested v6 and I think GetNamedDSA is a good addition. I did not find any issues with the code. However, I am still convinced that GetNamedDSMHash should not append " Hash" to the tranche name of the dshash [0]. I am ok with " DSA" because the DSA tranche is created implicitly by the API. Also,

Re: pg_get_multixact_members not documented

2025-06-11 Thread Sami Imseih
> It's not clear, without some effort, which lock mode go with which row lock > in that description. > For example, "keysh" does not have "for" in it but "fornokeyupd" does. How > about rephrasing this > as "The lock modes "keysh", "sh", fornokeyupd", and "forupd" correspond to > FOR KEY SHARE, F

Re: [PATCH] Re: Proposal to Enable/Disable Index using ALTER INDEX

2025-06-11 Thread Sami Imseih
> IMO, having this GUC to force the use of invisible indexes is quite > strange. In my view, it detracts from the guarantees that you're meant > to get from disabling indexes. What if some connection has > use_invisible_index set to true? The DBA might assume all is well > after having seen nobody

Re: add function for creating/attaching hash table in DSM registry

2025-06-10 Thread Sami Imseih
> So, if we were adding named LWLocks today, I suspect we might do it > differently. The first thing that comes to mind is that we could store a > shared LWLockTrancheNames table. +1 > and stop requiring each backend to register them individually. which will prevent odd behavior when a backend

Re: add function for creating/attaching hash table in DSM registry

2025-06-10 Thread Sami Imseih
There is also that dynamic tranche named are stored in local backend look-up table, so if you have some backends that attached some dynamic hash table and others that did not, only the ones that registered would be able to resolve the tranche id to its name. This is the case which I encountered ye

Re: add function for creating/attaching hash table in DSM registry

2025-06-10 Thread Sami Imseih
> I'm not quite following your uneasiness with the tranche names. For the > dshash table, we'll need a tranche for the DSA and one for the hash table, > so presumably any wait events for those locks should be named accordingly, > right? I may be alone in this opinion, but I prefer the suffixless

Re: add function for creating/attaching hash table in DSM registry

2025-06-09 Thread Sami Imseih
> It is not expected behavior IMO, and I still need to debug this a bit more, > but it may be something outside the scope of this patch that the patch just > surfaced. It seems I got it backward. If the tranch is registered, then the wait event name is the one of the tranche, in that case we will

Re: add function for creating/attaching hash table in DSM registry

2025-06-09 Thread Sami Imseih
> On Mon, Jun 09, 2025 at 03:10:30PM -0500, Sami Imseih wrote: > > One thing I noticed, and I´m not too fond of, is that the wait_event name > > shows > > up with the _dsh suffix: > > > > ``` > > postgres=# select query, wait_event, wait_event_type from pg

Re: add function for creating/attaching hash table in DSM registry

2025-06-09 Thread Sami Imseih
Thanks, I tested v2 a bit more and did a quick hack of pg_stat_statements just to get a feel for what it would take to use the new API to move the hash table from static to dynamic. One thing I noticed, and I’m not too fond of, is that the wait_event name shows up with the _dsh suffix: ``` postgr

Re: queryId constant squashing does not support prepared statements

2025-06-09 Thread Sami Imseih
> I've spent a bunch of time looking at this series and here's my take on > the second one. Thanks! > I realized that the whole in_expr production in gram.y is pointless, and > the whole private struct that was added was unnecessary. A much simpler > solution is to remove in_expr, expand its use

Re: [PATCH] Re: Proposal to Enable/Disable Index using ALTER INDEX

2025-06-07 Thread Sami Imseih
> In that type of environment, the GUC-only method enables you to > control changes at very precise levels, so you can do things like: > - run it ad-hoc at the session level to confirm that the explain plans > you get in production match your expectations. > - you can stay ad-hoc at the session lev

Re: [PATCH] Re: Proposal to Enable/Disable Index using ALTER INDEX

2025-06-06 Thread Sami Imseih
> Don't get me wrong, it would be an improvement to have some type of > mechanism that can move you from almost 100% to 100%, but the real > problem is how do you SAFELY get to almost 100% in the first place? This big use case is precisely the "almost 100% to 100%" confidence problem. Usually, us

Re: [PATCH] Re: Proposal to Enable/Disable Index using ALTER INDEX

2025-06-05 Thread Sami Imseih
derstand, it works only at the EXPLAIN level. It is purely an experimentation tool. However, the proposed GUC can also be used in more places, including, pg_hint_plan ( at least with the SET hint without any changes to pg_hint_plan). > > P.S. I really do want to thank Shayon for sticking with this; > +1 +1 -- Sami Imseih Amazon Web Services (AWS)

Re: pg_get_multixact_members not documented

2025-06-05 Thread Sami Imseih
Thanks for the review! > +1. PFA diff of some edits. Please incorporate them in > your patch if you find them correct. sure, the diff looks fine to me. will fix. > We developers may understand the mode text "sh", "keysh" etc. But it may not > be user friendly. yes, I can see that being a point

Re: track generic and custom plans in pg_stat_statements

2025-06-05 Thread Sami Imseih
> Thanks for your work on this. > > Since we've changed the placement of these parameters > between parallel_workers and stats_since, we should also update > the documentation to reflect their new location. Absolutely. My miss. Here is v9 with the doc updated. Thanks! -- Sami v9-0001-Add-plan_

Re: add function for creating/attaching hash table in DSM registry

2025-06-05 Thread Sami Imseih
h_tranche_name); 3/ It will be good to "Assert(dsh)" before "return dsh;" for safety? MemoryContextSwitchTo(oldcontext); LWLockRelease(DSMRegistryLock); return dsh; } -- Sami Imseih Amazon Web Services (AWS)

Re: Improve explicit cursor handling in pg_stat_statements

2025-06-05 Thread Sami Imseih
> + * If an direction_keyword (i.e., FETCH FORWARD) is used, set this field > + * to distinguish it from its numeric counterpart (i.e., FETCH 1). This > + * value is set only within gram.y. > > One nitpick comment here is that I would have mentioned that this > matters for query jumbling. Done

Re: queryId constant squashing does not support prepared statements

2025-06-04 Thread Sami Imseih
I realized this thread did not have a CF entry, so here it is https://commitfest.postgresql.org/patch/5801/ -- Sami

Re: pg_get_multixact_members not documented

2025-06-04 Thread Sami Imseih
s-postgresql-multixact-member-exhaustion-incidents-may-2025 -- Sami Imseih Amazon Web Services (AWS) v2-0001-Document-pg_get_multixact_members.patch Description: Binary data

Re: Add log_autovacuum_{vacuum|analyze}_min_duration

2025-06-04 Thread Sami Imseih
ging (or vice versa), but there may be, and this will be more flexible. -- Sami Imseih

Re: Improve explicit cursor handling in pg_stat_statements

2025-06-04 Thread Sami Imseih
> Hmm. I was not sure if we'd really need to get down to that, as most > of the grammar keywords have the same parsed meaning, but there's a > good point with LAST for example that uses a negative value for > howMany. If we silence the number, LAST would map with everything > else that has FETCH_

Re: Add log_autovacuum_{vacuum|analyze}_min_duration

2025-06-03 Thread Sami Imseih
y I suggest we keep its current behavior, which is to control logging for both autoanalyze and autovacuum, and instead introduce only one new GUC, log_autovacuum_analyze_min_duration, which defaults to -1? For workloads that require different logging for autoanalyze, this new setting can be enabled. -- Sami Imseih Amazon Web Services (AWS)

Re: Improve explicit cursor handling in pg_stat_statements

2025-06-03 Thread Sami Imseih
ues for n->explicit_direction are nearby each other. It does not change the behavior. Also, modified the tests to match and some additional code comments. -- Sami Imseih Amazon Web Services (AWS) v3-0001-Normalize-variable-fetch-sizes-in-a-FETCH-command.patch Description: Binary data

Re: Improve explicit cursor handling in pg_stat_statements

2025-06-03 Thread Sami Imseih
> Should we offer something more consistent with DeallocateStmt, where > we have a boolean flag that would be set when ALL is specified, > included in the jumbling? This would mean two separate entries: one > for the constants and one for ALL. Hmm, we could do that to differentiate the keyword AL

Re: track generic and custom plans in pg_stat_statements

2025-06-02 Thread Sami Imseih
, but that could still be taken up after. See v8 with the field names reorganized. -- Sami Imseih Amazon Web Services (AWS) v8-0001-Add-plan_cache-counters-to-pg_stat_statements.patch Description: Binary data

Re: pg_get_multixact_members not documented

2025-06-02 Thread Sami Imseih
-- Sami Imseih Amazon Web Services (AWS) On Mon, Jun 2, 2025 at 3:03 PM Nathan Bossart wrote: > On Mon, Jun 02, 2025 at 12:46:51PM -0500, Sami Imseih wrote: > > v1-0001 is the documentation only patch. I improved upon the description > > suggested in [0] > > Your patch

Re: Improve explicit cursor handling in pg_stat_statements

2025-06-02 Thread Sami Imseih
d tests in pg_stat_statements utility.sql to demonstrate how queryIds are grouped for the variants of the FETCH statement. -- Sami Imseih Amazon Web Services (AWS) v2-0001-Normalize-variable-fetch-sizes-in-a-FETCH-command.patch Description: Binary data

Re: pg_get_multixact_members not documented

2025-06-02 Thread Sami Imseih
); begin; select from t for update ; savepoint s1; update t set v = v; select pg_get_multixact_members(a.relminmxid), a.relminmxid from (select relminmxid from pg_class where relname = 't') a; commit; ``` Thoughts? -- Sami Imseih Amazon Web Services (AWS) [0] https://www.postgresql.or

Re: track generic and custom plans in pg_stat_statements

2025-06-02 Thread Sami Imseih
c/custom plan counters. I prefer the former since we do it once in a major version and do not have to worry about it once new counters are added. It just feels odd that they sit in between the counters as they have a high level purpose. Thanks! Sami Imseih Amazon Web Services (AWS) v7-0001-Add-plan_cache-counters-to-pg_stat_statements.patch Description: Binary data

Re: pg_get_multixact_members not documented

2025-05-30 Thread Sami Imseih
Thanks! > blog posts that recommend it. In any case, I can't think of a reason it > ought to remain undocumented. I agree, especially with blogs referencing it. > Want to put together a patch? Yes, will do — Sami

pg_get_multixact_members not documented

2025-05-30 Thread Sami Imseih
...@postgresql.org -- Sami Imseih Amazon Web Services (AWS)

Re: queryId constant squashing does not support prepared statements

2025-05-30 Thread Sami Imseih
ested for "-- Unique query IDs with parameter numbers switched." and what is tested for "-- Two groups of two queries with the same query ID." I also added a comment for `` bool extern_param; ``` v8 addresses the above. -- Sami Imseih Amazon Web Services (AWS) v8-0003-Support-Squashing-of-External-Parameters.patch Description: Binary data v8-0002-Fix-Normalization-for-squashed-query-texts.patch Description: Binary data v8-0001-Enhanced-query-jumbling-squashing-tests.patch Description: Binary data

Re: Improve explicit cursor handling in pg_stat_statements

2025-05-29 Thread Sami Imseih
to have a pg_stat_statements.track_cursor_utility GUC that can toggle the reporting of utility statements related to explicit cursors, while still tracking the underlying statements. I would even suggest making 'on' the default to maintain the current behavior. I don’t like that we have to introduce a new GUC for this, but I can't think of a better alternative. Thoughts? -- Sami Imseih Amazon Web Services (AWS)

Re: track generic and custom plans in pg_stat_statements

2025-05-29 Thread Sami Imseih
BuildCachedPlan code and is not related to the part that made CachedPlan available to QueryDesc. See v6 for the rebase of the patch and addition of testing for EXPLAIN and EXPLAIN ANALYZE which was missing from v5. [0] https://www.postgresql.org/message-id/605328.1747710...@sss.pgh.pa.us -- Sami

Re: track generic and custom plans in pg_stat_statements

2025-05-29 Thread Sami Imseih
a reused generic plan? */ ``` which we had to account for up until v5. So this simplifies the tracking a bit more as the only states to track are "generic plan" or "custom plan" only. -- Sami Imseih Amazon Web Services (AWS)

Re: queryId constant squashing does not support prepared statements

2025-05-28 Thread Sami Imseih
>> * 0001: > You have mentioned the addition of tests, but v6-0001 includes nothing > of the kind. Am I missing something? How much coverage did you > intend to add here? These seem to be included in squashing.sql in > patch v6-0002, but IMO this should be moved somewhere else to work > with th

Re: queryId constant squashing does not support prepared statements

2025-05-27 Thread Sami Imseih
> I've spent a bit of time looking at this, and I want to > propose the following patchset. Sorry about this, but I missed to add a comment in one of the test cases for 0004 that describes the behavior of parameters and constants that live outside of the squashed list. The following 2 cases will

Re: queryId constant squashing does not support prepared statements

2025-05-27 Thread Sami Imseih
> > therefore, a user supplied query like this: > > ``` > > select where $5 in ($1, $2, $3) and $6 = $4 and 1 = 2 > > ``` > > > > will be normalized to: > > ``` > > select where $1 in ($2 /*...*/) and $3 = $4 and $5 = $6 > > ``` > > Hmm, interesting. > > I think this renumbering should not be a pro

Re: Relstats after VACUUM FULL and CLUSTER

2025-05-24 Thread Sami Imseih
> If you subtract recently dead from that number within the heap > implementation, then it will no longer > reflect non-removable tuples and the log message in the cluster > function "found %.0f removable, %.0f nonremovable row versions" will no > longer be correct. Yes, that's correct. I did no

Re: queryId constant squashing does not support prepared statements

2025-05-24 Thread Sami Imseih
> In v17, we are a bit smarter with the numbering, with a normalization > giving the following, starting at $1: > select where $5 in ($1, $2, $3) and $6 = $4 > > So your argument about the $n parameters is kind of true, but I think > the numbering logic in v17 to start at $1 is a less-confusing res

Re: queryId constant squashing does not support prepared statements

2025-05-23 Thread Sami Imseih
> On Fri, May 23, 2025 at 04:29:45PM +0200, Dmitry Dolgov wrote: > > I think it's better to recursively call IsSquashableConst on the nested > > expression (arg or args for FuncExpr). Something like that was done in > > the original patch version and was concidered too much at that time, but > > si

Re: queryId constant squashing does not support prepared statements

2025-05-23 Thread Sami Imseih
> > On Thu, May 22, 2025 at 10:23:31PM GMT, Sami Imseih wrote: > > > This does not get squashed: > > > Q: select where 2 in (1, 4) and > > > 1 in (5, cast(7 as int), 6, (cast(8 as int)), 9, 10, (cast(8 as > > > text))::int); > > > R: select where

Re: queryId constant squashing does not support prepared statements

2025-05-22 Thread Sami Imseih
> For example with bind queries like that: > select where $1 in ($3, $2) and 1 in ($4, cast($5 as int)) > \bind 0 1 2 3 4 > > Should we have a bit more coverage, where we use multiple IN and/or > ARRAY lists with constants and/or external parameters? I will add more test coverage. All the tests we

Re: Relstats after VACUUM FULL and CLUSTER

2025-05-22 Thread Sami Imseih
s the value set in > pg_class.reltuples ) > after we process all the tuples, which looks like the best fix to me. something like the attached. -- Sami Imseih Amazon Web Services (AWS) v1-0001-Correct-reltuples-count-after-a-VACUUM-CLUSTER-op.patch Description: Binary data

Re: queryId constant squashing does not support prepared statements

2025-05-22 Thread Sami Imseih
> IMO adding a struct as suggested is okay, especially if it reduces the > overall code complexity. But we don't want a node, just a bare struct. > Adding a node would be more troublesome. In v4, a new private struct is added in gram.y, but we are also adding additional fields to track the expres

Re: Relstats after VACUUM FULL and CLUSTER

2025-05-22 Thread Sami Imseih
acting tups_recently_dead from num_tuples ( which is the value set in pg_class.reltuples ) after we process all the tuples, which looks like the best fix to me. -- Sami Imseih Amazon Web Services (AWS)

Re: POC: Parallel processing of indexes in autovacuum

2025-05-22 Thread Sami Imseih
ance_worker' to something like 8. If it so happens that all 3 tables are auto-vacuumed at the same time, there may not be enough parallel workers, so one table will be a loser and be vacuumed in serial. That is acceptable, and a/v logging ( and perhaps other stat views ) should display this behavior: workers planned vs workers launched. thoughts? -- Sami Imseih Amazon Web Services (AWS)

Re: Relstats after VACUUM FULL and CLUSTER

2025-05-22 Thread Sami Imseih
elect reltuples::int from pg_class where relname = 't'; select n_dead_tup, n_live_tup from pg_stat_all_tables where relname = 't'; reltuples --- 99 (1 row) postgres=# select n_dead_tup, n_live_tup from pg_stat_all_tables where relname = 't'; n_dead_tup | n_

Re: vacuum_multixact_failsafe_age doesn't account for MultiXact member exhaustion

2025-05-21 Thread Sami Imseih
is the lack of visibility into multixact members. [0] https://www.postgresql.org/message-id/flat/CALdSSPi3Gh08NtcCn44uVeUAYGOT74sU6uei_06qUTa5rMK43g%40mail.gmail.com#bfd9ae766ef42f7599258183aa8ddb3b -- Sami Imseih Amazon Web Services (AWS)

Re: queryId constant squashing does not support prepared statements

2025-05-21 Thread Sami Imseih
> > > At the same time AFAICT there isn't much more code paths > > > to worry about in case of a LocationExpr as a node > > > > I can imagine there are others like value expressions, > > row expressions, json array expressions, etc. that we may > > want to also normalize. > Exactly. When using a n

Re: queryId constant squashing does not support prepared statements

2025-05-20 Thread Sami Imseih
ion ParseLoc in A_ArrayExpr and A_Expr. Doing it this will keep changes to the parse_expr.c code to a minimum, only the IN transformation will need to set the values of the A_Expr into the final A_ArrayExpr. -- Sami Imseih Amazon Web Services (AWS) diff --git a/src/backend/parser/gram.y b/src/backend

Re: Add comment explaining why queryid is int64 in pg_stat_statements

2025-05-20 Thread Sami Imseih
likely(jumble_len >= JUMBLE_SIZE)) { - uint64 start_hash; + int64 start_hash; - start_hash = DatumGetUInt64(hash_any_extended(jumble, - JUMBLE_SIZE, 0)); + start_hash = DatumGetInt64(hash_any_extended(jumble, + JUMBLE_SIZE, 0)); memcpy(jumble, &start_hash, sizeof(start_hash)); jumble_len = sizeof(start_hash); -- Sami Imseih Amazon Web Services (AWS)

Re: Regression in statement locations

2025-05-20 Thread Sami Imseih
Tested the patch and it looks good to me. Not that I thought it would fail, but I also confirmed the pgaudit case works as expected. ``` LOG: AUDIT: SESSION,10,2,DDL,CREATE TABLE,,,"CREATE TABLE do_table (""weird name"" INT)", LOG: AUDIT: SESSION,10,3,DDL,DROP TABLE,,,DROP table do_table, DO ``

Re: Regression in statement locations

2025-05-19 Thread Sami Imseih
ows) I am still not sure why this is the case, but wanted to share this for now. -- Sami Imseih Amazon Web Services (AWS)

Re: Add comment explaining why queryid is int64 in pg_stat_statements

2025-05-19 Thread Sami Imseih
FWIW, all the hash function SQL interfaces, \df hash*, have this behavior in which the result is a signed (int/bigint), but the internal representation of the hash is an unsigned (int/bigint). I am not sure why a comment is needed specifically for pg_stat_statements -- Sami Imseih Amazon Web

Re: Possible regression in PG18 beta1

2025-05-18 Thread Sami Imseih
> I'm now thinking maybe the compilation options for PG 17.5 have been > different. I'm using the default version that comes with Homebrew, but for > PG 18, I compiled it myself. Here are the results for `select version();` > on both: > > PostgreSQL 17.5 (Homebrew) on aarch64-apple-darwin24.4.0, >

Re: Possible regression in PG18 beta1

2025-05-18 Thread Sami Imseih
On Sat, May 17, 2025 at 2:00 PM Sadeq Dousti wrote: >> >> thanks. I don't see regression for a normal table, at least for this test. > > > No, there isn't. I just added them as per your request ;) > > >> In terms of your original test, I tried it out on my Ubuntu machine >> >> and with your test a

Re: Possible regression in PG18 beta1

2025-05-17 Thread Sami Imseih
>> I'll be curious about tests with a normal table as well with a >> sufficiently large shared_buffers. > Here are results for a normal table with default shared_buffers (128 MB) and > large shared_buffers (4GB): thanks. I don't see regression for a normal table, at least for this test. In term

Re: Possible regression in PG18 beta1

2025-05-17 Thread Sami Imseih
both versions use >> > "Parallel Seq Scan", with PG18 being ~25% faster. (190ms vs. 150ms). I'll be curious about tests with a normal table as well with a sufficiently large shared_buffers. -- Sami Imseih Amazon Web Services (AWS)

Re: queryId constant squashing does not support prepared statements

2025-05-12 Thread Sami Imseih
> > On Fri, May 09, 2025 at 12:47:19PM GMT, Sami Imseih wrote: > > So, I think we can create a new parse node ( parsenode.h ) that will only be > > used in parsing (and gram.c only ) to track the start/end locations > > and List and > > based on this node we can crea

Re: queryId constant squashing does not support prepared statements

2025-05-09 Thread Sami Imseih
> > To clarify, I had in mind something like in the attached patch. The > > idea is to make start/end location capturing relatively independent from > > the constants squashing. The new parsing node conveys the location > > information, which is then getting transformed to be a part of an > > Array

Re: Prolonged truncation phase during vacuum on toast table with repeated interruptions by lock waiters and a proposed POC patch

2025-05-08 Thread Sami Imseih
even try at all? [0] https://postgr.es/m/Z2DE4lDX4tHqNGZt%40dev.null -- Sami Imseih Amazon Web Services (AWS)

Re: queryId constant squashing does not support prepared statements

2025-05-08 Thread Sami Imseih
On Thu, May 8, 2025 at 2:36 PM Dmitry Dolgov <9erthali...@gmail.com> wrote: > > > On Thu, May 08, 2025 at 02:22:00PM GMT, Michael Paquier wrote: > > On Wed, May 07, 2025 at 10:41:22AM +0200, Dmitry Dolgov wrote: > > > Ah, I see what you mean. I think the idea is fine, it will simplify > > > certain

Re: Prolonged truncation phase during vacuum on toast table with repeated interruptions by lock waiters and a proposed POC patch

2025-05-08 Thread Sami Imseih
> actually determine you are in this situation, or how bad the situation > was, in order to know that setting truncate off would help? To that Adding counters for this area is not a bad idea in general, as this hits customers particularly hard on hot standbys when the truncate does actually occur

Re: queryId constant squashing does not support prepared statements

2025-05-06 Thread Sami Imseih
query| calls --++--- t | select where $1 in ($2 /*, ... */) | 2 (1 row) ``` What do you think? -- Sami Imseih From 15f1313ef66e964e588b0bf19ede676437ea5a42 Mon Sep 17 00:00:00 2001 From: Sami Imseih Date: Tue, 6 May 2

Re: POC: Parallel processing of indexes in autovacuum

2025-05-06 Thread Sami Imseih
> On Mon, May 5, 2025 at 5:21 PM Sami Imseih wrote: > > > > > >> On Sat, May 3, 2025 at 1:10 AM Daniil Davydov <3daniss...@gmail.com> wrote: > >> > > >> > On Sat, May 3, 2025 at 5:28 AM Masahiko Sawada > >> > wrote: > &g

Re: queryId constant squashing does not support prepared statements

2025-05-06 Thread Sami Imseih
| calls --+-+--- t| select where $1 in ($2 /*, ... */) and $3 in ($4 /*, ... */($5 as int)) | 1 (1 row) """ Without properly accounting for the boundaries of the list of expressions, i.e., the star

Re: POC: Parallel processing of indexes in autovacuum

2025-05-05 Thread Sami Imseih
> On Sat, May 3, 2025 at 1:10 AM Daniil Davydov <3daniss...@gmail.com> > wrote: > > > > On Sat, May 3, 2025 at 5:28 AM Masahiko Sawada > wrote: > > > > > > > In current implementation, the leader process sends a signal to the > > > > a/v launcher, and the launcher tries to launch all requested wor

Re: POC: Parallel processing of indexes in autovacuum

2025-05-02 Thread Sami Imseih
> I think it would more make > sense to maintain the existing autovacuum_max_workers parameter while > introducing a new parameter that would either control the maximum > number of parallel vacuum workers per autovacuum worker or set a > system-wide cap on the total number of parallel vacuum worker

Re: Improve explicit cursor handling in pg_stat_statements

2025-05-02 Thread Sami Imseih
> Hmm. What are the workloads that you have seen as problematic? Do > these involve cursor names generated randomly, where most of them are > similar with a random factor for the name? postgres_fdw, as an example, in which cursor name get reused for different queries. Notice below "c1" and "c2"

Re: POC: Parallel processing of indexes in autovacuum

2025-05-02 Thread Sami Imseih
> On Fri, May 2, 2025 at 11:58 PM Sami Imseih wrote: > > > > I am generally -1 on the idea of autovacuum performing parallel > > index vacuum, because I always felt that the parallel option should > > be employed in a targeted manner for a specific table. if you have

Re: POC: Parallel processing of indexes in autovacuum

2025-05-02 Thread Sami Imseih
in thread[1], it would > also likely be beneficial to incorporate it into autovacuum during > aggressive vacuum or failsafe mode. IIRC, index cleanup is disabled by failsafe. -- Sami Imseih Amazon Web Services (AWS)

Re: queryId constant squashing does not support prepared statements

2025-05-01 Thread Sami Imseih
I spent a few hours looking into this today and to your points below: > > I agree that the current solution we have in the tree feels incomplete > > because we are not taking into account the most common cases that > > users would care about. Now, allowing PARAM_EXTERN means that we > > allow any

queryId constant squashing does not support prepared statements

2025-04-30 Thread Sami Imseih
ich will be very disappointing. Thoughts? Sami Imseih Amazon Web Services (AWS) v1-0001-Allow-query-jumble-to-squash-a-list-external-para.patch Description: Binary data

Improve explicit cursor handling in pg_stat_statements

2025-04-30 Thread Sami Imseih
also was not happy with the approach I took in [0]. Looking forward to feedback! Regards, -- Sami Imseih Amazon Web Services (AWS) [0] https://www.postgresql.org/message-id/flat/203CFCF7-176E-4AFC-A48E-B2CECFECD6AA%40amazon.com v1-0001-Improve-cursor-handling-in-pg_stat_statements.patch Descript

Re: Improve explicit cursor handling in pg_stat_statements

2025-04-30 Thread Sami Imseih
, -- Sami Imseih Amazon Web Services (AWS) v2-0002-Improve-cursor-handling-in-pg_stat_statements.patch Description: Binary data v2-0001-Normalize-cursor-utility-statements.patch Description: Binary data

Re: Introduce some randomness to autovacuum

2025-04-30 Thread Sami Imseih
or autovacuum to auto-tune this setting to give more workers at times when it's needed. Not sure what that looks like, but it is more possible now that this setting does not require a restart. -- Sami Imseih Amazon Web Services (AWS)

Re: Introduce some randomness to autovacuum

2025-04-30 Thread Sami Imseih
oblems, this will not solve the "spinning" problem because if the most common reason for this is a long-running transaction, etc., all your tables will eventually end up with wasted vacuum cycles because the xmin horizon is not advancing. -- Sami Imseih

Re: Disallow redundant indexes

2025-04-29 Thread Sami Imseih
> On Mon, Apr 28, 2025 at 01:12:18PM -0500, Sami Imseih wrote: > > If I had a dollar for every time I asked a user to check for duplicate > > indexes :) So > > I think it's a good idea to provide some way to detect these ( besides > > a query in the > > Wiki

Re: Disallow redundant indexes

2025-04-28 Thread Sami Imseih
tect these ( besides a query in the Wiki ), but I don't think we should prevent it. While a WARNING will be a good to have, it could easily go unnoticed, but it's still good to have. I also think we should either provide a psql shortcut to detect these indexes or to add annotation to an index in the \d command, or perhaps both of these things. -- Sami Imseih Amazon Web Services (AWS)

Re: [BUG] temporary file usage report with extended protocol and unnamed portals

2025-04-26 Thread Sami Imseih
logging is the CLOSE command: i.e. ``` 2025-04-26 18:46:38.084 UTC [10415] LOG: temporary file: path "base/pgsql_tmp/pgsql_tmp10415.0", size 1014030336 2025-04-26 18:46:38.084 UTC [10415] STATEMENT: close mycursor_1; ``` I don't think there is much we can do there, or should we.

Re: [BUG] temporary file usage report with extended protocol and unnamed portals

2025-04-25 Thread Sami Imseih
Thanks for testing. I also tested it a bit more today with other patterns like different fetch sizes, named portal, etc. and I can't find an issue with this, but I could be missing something. I will go ahead and attach this change in patch form. -- Sami Imseih Amazon Web Services (AWS)

Re: Conflicting updates of command progress

2025-04-24 Thread Sami Imseih
> pgstat_progress_start_command() is called twice: First with > cmdtype=PROGRESS_COMMAND_CLUSTER, second with > PROGRESS_COMMAND_CREATE_INDEX. The first happens in cluster_rel() the second > in cluster_rel() -> rebuild_relation() -> finish_heap_swap() -> > reindex_relation() -> reindex_index(). > >

Re: Conflicting updates of command progress

2025-04-23 Thread Sami Imseih
d by a top-level command, such as the COPY case above. 2/ a top-level command triggered some other progress code implicitly, such as CLUSTER triggering CREATE INDEX code. I also like the shared memory approach because we can then not have to use a message like the one introduced in f1889729

Re: [BUG] temporary file usage report with extended protocol and unnamed portals

2025-04-23 Thread Sami Imseih
; ?column? -- 1 (1 row) postgres=*# commit; COMMIT ``` ``` 2025-04-23 11:11:47.777 CDT [67362] LOG: temporary file: path "base/pgsql_tmp/pgsql_tmp67362.0", size 1009983488 2025-04-23 11:11:47.777 CDT [67362] STATEMENT: select from foo order by a ; ``` thoughts? -- Sami Imseih Amazon Web Services (AWS)

Re: [BUG] temporary file usage report with extended protocol and unnamed portals

2025-04-22 Thread Sami Imseih
_files is a better alternative and a statement can be logged with other existing mechanisms like log_min_duration_statement. -- Sami Imseih Amazon Web Services (AWS)

Re: [BUG] temporary file usage report with extended protocol and unnamed portals

2025-04-22 Thread Sami Imseih
e statement to blame for the temp files and will cover all cases? [0] https://www.postgresql.org/message-id/CAA5RZ0ssqRTz_9T0Gy74SiTViiX3V0rSFxc4N_4GNcbEBK9wow%40mail.gmail.com -- Sami Imseih Amazon Web Services (AWS)

Re: [BUG] temporary file usage report with extended protocol and unnamed portals

2025-04-19 Thread Sami Imseih
> Sami Imseih writes: > > I think the solution proposed by Frédéric seems reasonable: to switch > > the debug_query_string > > inside PortalDrop. However, I do not like the way the > > debug_query_string changes values > > after the CreatePortal call inside exec_b

Re: [BUG] temporary file usage report with extended protocol and unnamed portals

2025-04-19 Thread Sami Imseih
CreatePortal call inside exec_bind_message; that seems incorrect. So, I believe we should temporarily switch the debug_query_string value only while running PortalDrop. Attached is what I think could be safer to do. What do you think? -- Sami Imseih Amazon Web Services (AWS) v2-0001-Fix-race

Re: n_ins_since_vacuum stats for aborted transactions

2025-04-18 Thread Sami Imseih
> I can see it being confusing. What about this wording to make it more > clear when the field is > updated? here are both of the changes in v4. -- Sami Imseih Amazon Web Services (AWS) v4-0001-Clarify-when-aborted-rows-are-tracked-for-tuple-r.patch Description: Binary data

Re: n_ins_since_vacuum stats for aborted transactions

2025-04-18 Thread Sami Imseih
On Thu, Apr 17, 2025 at 11:13 PM David Rowley wrote: > > On Sat, 12 Apr 2025 at 07:33, Sami Imseih wrote: > > What do you think of the attached? > > I looked at the v3 patch and I'm having trouble getting excited about it. > > I'd say this part is

Re: [BUG] temporary file usage report with extended protocol and unnamed portals

2025-04-18 Thread Sami Imseih
care of the problem because it ensures that the drop portal, when it occurs, is referencing the correct sql. I am not yet sure if the patch the right solution yet, it maybe the best answer. I don't have a better answer, but wanted to share this research as well. -- Sami Imseih Amazon Web Services (AWS) Test.java Description: Binary data

Re: Conflicting updates of command progress

2025-04-14 Thread Sami Imseih
herefore, only the top level command is updating progress. what do you think? [1] https://commitfest.postgresql.org/patch/5117/ -- Sami Imseih Amazon Web Services (AWS)

Re: stats.sql fails during installcheck on mac

2025-04-11 Thread Sami Imseih
> > what do you think of this? I think we should set fsync = on > > at least for the part of the test that proceeds the 2 checkpoints and > > set if back to off at the end of the tests for fsync stats. It is concerning > > the tests for the fsync stats are not being exercised in > > the buildfarm.

Re: n_ins_since_vacuum stats for aborted transactions

2025-04-11 Thread Sami Imseih
> WFM. Though is there a reason to avoid adding the "why" of the exception for > n_mod_since_analyze? I did think about that. I thought it will be understood that since this is a field that deals with ANALYZE, it will be understood that only committed changes matter here, and not worth adding mo

Re: n_ins_since_vacuum stats for aborted transactions

2025-04-11 Thread Sami Imseih
think the current explanation is clear enough, I am also not too thrilled about the "...to facilitate performance monitoring." since the cumulative stats system as a whole is known to be used to facilitate perf monitoring. What do you think of the attached? -- Sami Imseih Amazon Web Servi

Re: stats.sql fails during installcheck on mac

2025-04-11 Thread Sami Imseih
e it runs > with fsync=off, as I mentioned at the top of the thread. what do you think of this? I think we should set fsync = on at least for the part of the test that proceeds the 2 checkpoints and set if back to off at the end of the tests for fsync stats. It is concerning the tests for the f

  1   2   3   4   >