Re: Accept recovery conflict interrupt on blocked writing

2025-01-21 Thread Anthonin Bonnefoy
ocessRecoveryConflictInterrupt that could very probably be simplified, but I wanted to validate the approach first. It would also be probably worth adding more tests to cover all conflicts while there's a blocked write, though I did add the RAISE case in the tests. Regards, Anthonin Bonnefoy v04-0002-Accept-recovery-conflict-interrupt-on-blocked-wr.patch Description: Binary data v04-0001-Add-PgProto-test-module-to-send-message-on-a-raw.patch Description: Binary data

Re: Accept recovery conflict interrupt on blocked writing

2025-01-17 Thread Anthonin Bonnefoy
I've tested on a PG16, the issue is indeed triggered with the replication blocked while the conflicting query is stuck in ClientWrite. I've cleaned up the tests: I've created a dedicated PgProto (definitely open to suggestions for a better name...) module containing all the helpers to send and rec

Re: Accept recovery conflict interrupt on blocked writing

2025-01-16 Thread Anthonin Bonnefoy
Bonjour Thomas, On Wed, Jan 15, 2025 at 6:21 AM Thomas Munro wrote: > Right. Before commit 0da096d7 in v17, the recovery conflict code > running in a signal handler would have set ProcDiePending, so this > looks like an unintended regression due to that commit. The issue is happening on instanc

Re: Add Pipelining support in psql

2025-01-14 Thread Anthonin Bonnefoy
During my tests, I've noticed I didn't handle query cancellation, this is now fixed. I've also added additional comments related to available_results to make it clearer that it depends on what the server has flushed to the client. v07-0001-Add-pipelining-support-in-psql.patch Description: Binary

Accept recovery conflict interrupt on blocked writing

2025-01-13 Thread Anthonin Bonnefoy
to block recovery for an extended period of time and it is fine to be aggressive when the standby delay is exceeded. [1]: https://pracucci.com/linux-tcp-rto-min-max-and-tcp-retries2.html [2]: https://commitfest.postgresql.org/51/5407/ Regards, Anthonin Bonnefoy v01-0001-Accept-recovery-conflict-

Re: Add Pipelining support in psql

2025-01-10 Thread Anthonin Bonnefoy
> I feel there's a large overlap between \flush and \flushrequest. On > the other hand, if people want to test the behaviour of pushing data > with and without a flush request message, then \flush can be useful. I've been looking into some issues related to a backend process stuck in ClientWrite s

Re: Add Pipelining support in psql

2024-12-12 Thread Anthonin Bonnefoy
Thanks for the review! On Thu, Dec 12, 2024 at 12:53 AM Jelte Fennema-Nio wrote: > I think that new prompt is super useful, so useful in fact that I'd > suggest linking to it from the \startpipeline docs. Good point, I've added a paragraph with the link to the %P prompt. > I do think that > the

Re: Add Pipelining support in psql

2024-12-10 Thread Anthonin Bonnefoy
An improved version with simplifications and refinements. num_queries (2nd element in the pipeline status prompt) is now used to track queued queries that were not flushed (with a flush request or sync) to the server. It used to count both unflushed queries and flushed queries. Code in ExecQueryA

Re: Add Pipelining support in psql

2024-12-04 Thread Anthonin Bonnefoy
An updated version of the patch, still a bit rough around the edges. I've added \flushrequest, \flush and \getrequests meta-commands. \flushrequest and \getresults look interesting as they add an additional protocol message to test, but it also introduces the possibility to be in the middle of an

Re: Add Pipelining support in psql

2024-11-28 Thread Anthonin Bonnefoy
On Wed, Nov 27, 2024 at 11:46 AM Kirill Reshke wrote: > I'm very doubtful about the \syncpipeline . Maybe we should instead > support \sync meta-command in psql? This will be a useful contribution > itself. \syncpipeline is useful to cover regression tests involving implicit transaction blocks wi

Re: Consider pipeline implicit transaction as a transaction block

2024-11-27 Thread Anthonin Bonnefoy
On Thu, Nov 28, 2024 at 12:26 AM Michael Paquier wrote: > I don't mind being more careful here based on your concerns, so I'll > go remove that in the stable branches. Sorry about that. I didn't have a strong need for this to be backpatched and should have made this clearer.

Add Pipelining support in psql

2024-11-27 Thread Anthonin Bonnefoy
Hi, With \bind, \parse, \bind_named and \close, it is possible to issue queries from psql using the extended protocol. However, it wasn't possible to send those queries using pipelining and the only way to test pipelined queries was through pgbench's tap tests. The attached patch adds pipelining

Re: Consider pipeline implicit transaction as a transaction block

2024-11-26 Thread Anthonin Bonnefoy
On Wed, Nov 27, 2024 at 1:42 AM Michael Paquier wrote: > I've edited the whole, added this extra test based on \syncpipeline in > 17~, kept the remaining tests in 14~ where pgbench is able to handle > them, and backpatched that down to 13. Let's see now what we can do > with the psql bits. Thank

Re: Consider pipeline implicit transaction as a transaction block

2024-11-25 Thread Anthonin Bonnefoy
Thanks for the review! On Mon, Nov 25, 2024 at 7:39 AM Michael Paquier wrote: > > This breaks an existing property of psql. One example: \edit where we > should keep the existing query buffer rather than discard it if a > failure happens. This patch forcibly removes the contents of the > query

Re: Consider pipeline implicit transaction as a transaction block

2024-11-22 Thread Anthonin Bonnefoy
Some minor changes: I forgot to add the new pipeline meta-commands to psql's help output, this is now added in 0002. I've also reworded the doc a bit. v04-0001-Reset-query-buffer-on-a-backslash-error.patch Description: Binary data v04-0002-Add-pipeline-support-in-psql.patch Description: Binary

Re: Consider pipeline implicit transaction as a transaction block

2024-11-20 Thread Anthonin Bonnefoy
On Tue, Nov 5, 2024 at 6:50 AM Michael Paquier wrote: > It would be nice to document all these behaviors with regression > tests in pgbench as it is the only place where we can control that > with error pattern checks. It's not the first time I wanted to be able to do pipelining in psql as relying

Re: Segfault in jit tuple deforming on arm64 due to LLVM issue

2024-11-05 Thread Anthonin Bonnefoy
On Tue, Nov 5, 2024 at 9:00 AM Thomas Munro wrote: > Reasoning: Old LLVM required C++11. LLVM 9 switched to C++14. LLVM > 14 switched C++17. Pretty soon they'll flip to C++20 or C++23, they > don't mess around. The corresponding -std=c++XX flag finishes up in > our compile lines, because llvm

Re: Consider pipeline implicit transaction as a transaction block

2024-11-04 Thread Anthonin Bonnefoy
On Sat, Nov 2, 2024 at 4:11 AM Michael Paquier wrote: > Now, here is a fancy case: SAVEPOINT and its two brothers. An error > would be reported on HEAD if attempting a SAVEPOINT, complaining that > we are not in a transaction block. The patch causes a different, more > confusing, failure: > FATA

Re: Segfault in jit tuple deforming on arm64 due to LLVM issue

2024-10-31 Thread Anthonin Bonnefoy
On Thu, Oct 31, 2024 at 6:49 AM Thomas Munro wrote: > There are a couple of cases of dual-licensed code in our tree where we > explicitly used the Boost alternative instead of Apache 2. I plead > complete ignorance of this topic and defer to those who know about > such things: can we actually do

Consider pipeline implicit transaction as a transaction block

2024-10-30 Thread Anthonin Bonnefoy
Hi, When using pipelining with implicit transaction, a transaction will start from the first command and be committed with the Sync message. Functions like IsInTransactionBlock and PreventInTransactionBlock already assimilate this implicit transaction as a transaction block, relying on the XACT_FL

Re: Set query_id for query contained in utility statement

2024-10-23 Thread Anthonin Bonnefoy
On Wed, Oct 23, 2024 at 8:10 AM Michael Paquier wrote: > > I have some more minor comments. > > -if (@$ < 0) /* see comments for YYLLOC_DEFAULT */ > -@$ = @2; > > With 14e5680eee19 now in the tree (interesting timing as this did not > exist until yesterday), it looks li

Re: Set query_id for query contained in utility statement

2024-10-22 Thread Anthonin Bonnefoy
On Tue, Oct 22, 2024 at 7:06 AM Michael Paquier wrote: > -Query * > -transformTopLevelStmt(ParseState *pstate, RawStmt *parseTree) > > What's the advantage of removing this routine? Is that because you're > pushing the initialization of stmt_location and stmt_len into > transformOptionalSelectInt

Re: Segfault in jit tuple deforming on arm64 due to LLVM issue

2024-10-17 Thread Anthonin Bonnefoy
I've run some additional tests, mostly pgbench with options=-cjit_above_cost=0 for an extended duration on an instance that was impacted. I haven't seen any issues nor performance regressions compared to the unpatched version. I will switch the commitfest entry to Ready for Committer if there's no

Re: Set query_id for query contained in utility statement

2024-10-17 Thread Anthonin Bonnefoy
Hi Jian, On Thu, Oct 17, 2024 at 5:59 AM jian he wrote: > in [3] I mentioned adding "ParseLoc location" to ExplainStmt, then you > found some problems at [4] with multi statements, > now I found a way to resolve it. > I also add "ParseLoc location;" to typedef struct CopyStmt. > copy (select 1) t

Re: Set query_id for query contained in utility statement

2024-10-16 Thread Anthonin Bonnefoy
On Tue, Oct 15, 2024 at 3:23 PM jian he wrote: > explain(verbose) SELECT 1, 2, 3\; explain SELECT 1, 2, 3, 4; > will transformed to > explain(verbose) SELECT 1, 2, 3; explain SELECT 1, 2, 3, 4; > > it seems to me your patch care about following position. > explain(verbose

Re: Converting tab-complete.c's else-if chain to a switch

2024-10-10 Thread Anthonin Bonnefoy
Hi, bd1276a3c9 seems to have introduced a segfault when trying to complete a word that doesn't have any match. For example, 'postgres=# z\t' will yield the following backtrace: #0: psql`pg_strcasecmp(s1="", s2="ACCESS METHOD") at pgstrcasecmp.c:40:39 #1: psql`psql_completion(text=":pgss-", start=

Re: Set query_id for query contained in utility statement

2024-10-08 Thread Anthonin Bonnefoy
On Mon, Oct 7, 2024 at 7:39 AM Michael Paquier wrote: > GOod point, this is confusing. The point is that having only > stmt_location is not enough to detect where the element in the query > you want to track is because it only points at its start location in > the full query string. In an ideal

Re: Set query_id for query contained in utility statement

2024-10-04 Thread Anthonin Bonnefoy
On Wed, Oct 2, 2024 at 6:39 AM Michael Paquier wrote: > FWIW, I've always found this case with EXPLAIN with two entries > confusing, so what's the advantage in trying to apply this rule for > the rest? We know that EXPLAIN, DECLARE and CTAS run a query attached > to their DDL, hence isn't it suff

Re: Set query_id for query contained in utility statement

2024-10-01 Thread Anthonin Bonnefoy
On Mon, Sep 30, 2024 at 5:14 PM jian he wrote: > Is it possible to normalize top level utilities explain query, make > these two have the same queryid? > explain(verbose) EXECUTE test_prepare_pgss1(1, 2); > explain(verbose) EXECUTE test_prepare_pgss1(1, 3); > > I guess this is a corner case. This

Re: [PATCH] Add additional extended protocol commands to psql: \parse and \bind

2024-09-18 Thread Anthonin Bonnefoy
On Tue, Sep 17, 2024 at 5:00 PM Alexander Lakhin wrote: > > Please look at an assertion failure, caused by \bind_named: > regression=# SELECT $1 \parse s > \bind_named s > > regression=# \bind_named > \bind_named: missing required argument > regression=# 1 \g > psql: common.c:1501: ExecQueryAndPro

Re: Segfault in jit tuple deforming on arm64 due to LLVM issue

2024-08-30 Thread Anthonin Bonnefoy
I created a commitfest entry[1] to have the CI test the patch. There was a failure in headerscheck and cpluspluscheck when the include of SectionMemoryManager.h is checked[2] In file included from /usr/include/llvm/ADT/SmallVector.h:18, from /tmp/cirrus-ci-build/src/include/jit/SectionMemoryManage

Re: Set query_id for query contained in utility statement

2024-08-30 Thread Anthonin Bonnefoy
On Tue, Aug 27, 2024 at 11:14 AM jian he wrote: > also it's ok to use passed (ParseState *pstate) for > { > estate = CreateExecutorState(); > estate->es_param_list_info = params; > paramLI = EvaluateParams(pstate, entry, execstmt->params, estate); > } > ? > I really don't k

Re: Segfault in jit tuple deforming on arm64 due to LLVM issue

2024-08-29 Thread Anthonin Bonnefoy
On Wed, Aug 28, 2024 at 12:24 AM Thomas Munro wrote: > 2. I tested against LLVM 10-18, and found that 10 and 11 lack some > needed symbols. So I just hid this code from them. Even though our > stable branches support those and even older versions, I am not sure > if it's worth trying to do some

Re: Segfault in jit tuple deforming on arm64 due to LLVM issue

2024-08-27 Thread Anthonin Bonnefoy
On Tue, Aug 27, 2024 at 12:01 PM Thomas Munro wrote: > > Thanks! And that's great news. Do you want to report this experience > to the PR, in support of committing it? That'd make it seem easier to > consider shipping a back-ported copy... Yes, I will do that.

Re: Segfault in jit tuple deforming on arm64 due to LLVM issue

2024-08-27 Thread Anthonin Bonnefoy
On Tue, Aug 27, 2024 at 1:33 AM Thomas Munro wrote: > I am sure this requires changes for various LLVM versions. I tested > it with LLVM 14 on a Mac where I've never managed to reproduce the > original complaint, but ... h, this might be exacerbated by ASLR, > and macOS only has a small ALSR

Re: Segfault in jit tuple deforming on arm64 due to LLVM issue

2024-08-26 Thread Anthonin Bonnefoy
On Mon, Aug 26, 2024 at 4:33 AM Thomas Munro wrote: > IIUC this one is a random and rare crash depending on malloc() and > perhaps also the working size of your virtual memory dart board. > (Annoyingly, I had tried to reproduce this quite a few times on small ARM > systems when earlier reports cam

Re: Set query_id for query contained in utility statement

2024-08-26 Thread Anthonin Bonnefoy
On Mon, Aug 26, 2024 at 5:26 AM jian he wrote: >queryid| left > | plans | calls | rows > --+--+---+---+-- > 2800308901962295548 | EXPLAIN (verbose

Re: Segfault in jit tuple deforming on arm64 due to LLVM issue

2024-08-23 Thread Anthonin Bonnefoy
On Thu, Aug 22, 2024 at 12:33 PM Thomas Munro wrote: > I fear that back-porting, for the LLVM project, would mean "we fix it > in main/20.x, and also back-port it to 19.x". Do distros back-port > further? That's also my fear, I'm not familiar with distros back-port policy but eyeballing ubuntu p

Segfault in jit tuple deforming on arm64 due to LLVM issue

2024-08-22 Thread Anthonin Bonnefoy
essage-id/flat/CABa%2BnRvwZy_5t1QF9NJNGwAf03tv_PO_Sg1FsN1%2B-3Odb1XgBA%40mail.gmail.com [11] https://www.postgresql.org/message-id/flat/CADAf1kavcN-kY%3DvEm3MYxhUa%2BrtGFs7tym5d7Ee6Ni2cwwxGqQ%40mail.gmail.com Regards, Anthonin Bonnefoy

Re: [PATCH] Add additional extended protocol commands to psql: \parse and \bind

2024-08-21 Thread Anthonin Bonnefoy
On Wed, Aug 21, 2024 at 12:00 AM Jelte Fennema-Nio wrote: > @Anthonin are you planning to update the patch accordingly? Here's the patch with \bindx renamed to \bind_named. I've also made a small change to Michael's refactoring in 0002 by initialising success to false in ExecQueryAndProcessResul

Re: Set query_id for query contained in utility statement

2024-08-05 Thread Anthonin Bonnefoy
IN (verbose) create table test_t as select 1; QUERY PLAN -- Result (cost=0.00..0.01 rows=1 width=4) Output: 1 Query Identifier: 2800308901962295548 Regards, Anthonin Bonnefoy v2-0001-Set-query_id-for-queries-contained-in-utility-sta.

Set query_id for query contained in utility statement

2024-08-02 Thread Anthonin Bonnefoy
utility statements during post_parse, setting the query_id for those contained queries and removing the need for ExplainQuery to do it for the Explain statements. Regards, Anthonin Bonnefoy v1-0001-Set-query_id-for-queries-contained-in-utility-sta.patch Description: Binary data

Re: Use pgBufferUsage for block reporting in analyze

2024-07-31 Thread Anthonin Bonnefoy
UM VERBOSE output. But as you said, it's not really providing valuable information so it's probably better to keep the noise down and drop it. Regards, Anthonin Bonnefoy

Re: Use pgBufferUsage for block reporting in analyze

2024-07-31 Thread Anthonin Bonnefoy
On Tue, Jul 30, 2024 at 9:21 AM Anthonin Bonnefoy wrote: > A possible change would be to pass an inh flag when an acquirefunc is > called from acquire_inherited_sample_rows. The acquirefunc could then > use an alternative log format to append to logbuf. This way, we could > have a

Re: Use pgBufferUsage for block reporting in analyze

2024-07-30 Thread Anthonin Bonnefoy
Hi, On Tue, Jul 30, 2024 at 1:13 AM Masahiko Sawada wrote: > I have one comment on 0001 patch: > The comment should also be updated or removed. Ok, I've removed the comment. > However, as the above comment says, delay_in_ms can have a duration up > to 25 days. I guess it would not be a problem

Re: Use pgBufferUsage for block reporting in analyze

2024-07-29 Thread Anthonin Bonnefoy
On Sat, Jul 27, 2024 at 12:35 AM Masahiko Sawada wrote: > An alternative idea would > be to pass StringInfo to AcquireSampleRowsFunc() so that callback can > write its messages there. This is somewhat similar to what we do in > the EXPLAIN command (cf, ExplainPropertyText() etc). It could be too >

Add ALL_CANDIDATES option to EXPLAIN

2024-07-26 Thread Anthonin Bonnefoy
Hi, I have a prototype for an ALL_CANDIDATES option for EXPLAIN. The goal of this option is to print all plan candidates instead of only the cheapest plan. It will output the plans from the most expensive at the top to the cheapest. Here's an example: explain (all_candidates) select * from pgbenc

Re: query_id, pg_stat_activity, extended query protocol

2024-07-26 Thread Anthonin Bonnefoy
On Thu, Jul 18, 2024 at 10:56 AM Michael Paquier wrote: > Please attach things to your emails, if your repository disappears for > a reason or another we would lose knowledge in the archives of the > community lists. Noted and thanks for the reminder, I'm still learning about mailing list etiquet

Re: [PATCH] Add additional extended protocol commands to psql: \parse and \bind

2024-07-24 Thread Anthonin Bonnefoy
On Wed, Jul 24, 2024 at 05:33:07PM +0200, Peter Eisentraut wrote: > This commit message confused me, because I don't think this is what the > \bindx command actually does. AFAICT, it only binds, it does not execute. > At least that is what the documentation in the content of the patch appears > to

Re: Use pgBufferUsage for block reporting in analyze

2024-07-24 Thread Anthonin Bonnefoy
On Mon, Jul 22, 2024 at 10:59 PM Masahiko Sawada wrote: > The first line would vary depending on whether an autovacuum worker or > not. And the above suggestion includes a change of term "row" to > "tuple" for better consistency with VACUUM VERBOSE outputs. I think it > would be great if autoanaly

Re: Possible incorrect row estimation for Gather paths

2024-07-22 Thread Anthonin Bonnefoy
On Wed, Jul 17, 2024 at 3:59 AM Richard Guo wrote: > > I can reproduce this problem with the query below. > > explain (costs on) select * from tenk1 order by twenty; >QUERY PLAN > -

Correctly propagate queryId for utility stmt in function

2024-07-18 Thread Anthonin Bonnefoy
ility statements within function calls. This patch fixes the issue by correctly propagating queryId from the cached queryTree to the plannedStmt. Regards, Anthonin From b70163d1bd1279ca3f9b29166431cc64d25ca586 Mon Sep 17 00:00:00 2001 From: Anthonin Bonnefoy Date: Thu, 18 Jul 2024 11:50:37 +0200 Su

Re: query_id, pg_stat_activity, extended query protocol

2024-07-17 Thread Anthonin Bonnefoy
Hi, Wouldn't it be enough to call pgstat_report_query_id in ExecutorRun and ProcessUtility? With those changes [1], both normal statements and utility statements called through extended protocol will correctly report the query_id. -- Test utility statement with extended protocol show all \bind \g

Re: Possible incorrect row estimation for Gather paths

2024-07-16 Thread Anthonin Bonnefoy
Hi Rafia, Thanks for reviewing. On Wed, Jul 10, 2024 at 4:54 PM Rafia Sabih wrote: > But I don't quite understood the purpose of this, > + total_groups = input_rel->rows; > + > + /* > + * If the number of rows is unknown, fallback to gather rows > + * estimation > + */ > + if (total_groups == 0)

Re: Use pgBufferUsage for block reporting in analyze

2024-07-08 Thread Anthonin Bonnefoy
Hi, Thanks for the review, I've updated the patches with the suggestions: - moved renaming of misses to reads to the first patch - added intermediate variables for total blks usage I've also done some additional tests using the provided vacuum_analyze_buffer_usage.sql script. It relies on pg_stat

Re: Fix possible overflow of pg_stat DSA's refcnt

2024-06-25 Thread Anthonin Bonnefoy
On Wed, Jun 26, 2024 at 7:40 AM Michael Paquier wrote: > > Very good catch! It looks like you have seen that in the field, then. > Sad face. Yeah, this happened last week on one of our replicas (version 15.5) last week that had 134 days uptime. We are doing a lot of parallel queries on this clus

Fix possible overflow of pg_stat DSA's refcnt

2024-06-25 Thread Anthonin Bonnefoy
Hi, During backend initialisation, pgStat DSA is attached using dsa_attach_in_place with a NULL segment. The NULL segment means that there's no callback to release the DSA when the process exits. pgstat_detach_shmem only calls dsa_detach which, as mentioned in the function's comment, doesn't inclu

Re: Remove dependency on VacuumPage(Hit/Miss/Dirty) counters in do_analyze_rel

2024-06-06 Thread Anthonin Bonnefoy
Hi, I sent a similar patch for this in https://www.postgresql.org/message-id/flat/cao6_xqr__kttclkftqs0qscm-j7_xbrg3ge2rwhucxqjmjh...@mail.gmail.com Regards, Anthonin On Thu, Jun 6, 2024 at 11:10 AM Dilip Kumar wrote: > As part of commit 5cd72cc0c5017a9d4de8b5d465a75946da5abd1d, the > dependen

Re: Use pgBufferUsage for block reporting in analyze

2024-05-28 Thread Anthonin Bonnefoy
Thanks for having a look. On Fri, May 10, 2024 at 12:40 PM Michael Paquier wrote: > This needs some runtime check to make sure that the calculations > are consistent before and after the fact (cannot do that now). > Yeah, testing this is also a bit painful as buffer usage of analyze is only disp

Possible incorrect row estimation for Gather paths

2024-05-24 Thread Anthonin Bonnefoy
Hi, While experimenting on an explain option to display all plan candidates (very rough prototype here [1]), I've noticed some discrepancies in some generated plans. EXPLAIN (ALL_CANDIDATES) SELECT * FROM pgbench_accounts order by aid; Plan 1 -> Gather Merge (cost=11108.32..22505.38 rows=1

Use pgBufferUsage for block reporting in analyze

2024-05-10 Thread Anthonin Bonnefoy
Hi, Analyze logs within autovacuum uses specific variables VacuumPage{Hit,Miss,Dirty} to track the buffer usage count. However, pgBufferUsage already provides block usage tracking and handles more cases (temporary tables, parallel workers...). Those variables were only used in two places, block u

Re: Fix parallel vacuum buffer usage reporting

2024-05-02 Thread Anthonin Bonnefoy
On Wed, May 1, 2024 at 5:37 AM Masahiko Sawada wrote: > Thank you for further testing! I've pushed the patch. Thanks! Here is the rebased version for the follow-up patch removing VacuumPage variables. Though I'm not sure if I should create a dedicated mail thread since the bug was fixed and the

Re: Fix parallel vacuum buffer usage reporting

2024-04-29 Thread Anthonin Bonnefoy
I've done some additional tests to validate the reported numbers. Using pg_statio, it's possible to get the minimum number of block hits (Full script attached). -- Save block hits before vacuum SELECT pg_stat_force_next_flush(); SELECT heap_blks_hit, idx_blks_hit FROM pg_statio_all_tables where re

Re: Fix parallel vacuum buffer usage reporting

2024-04-25 Thread Anthonin Bonnefoy
On Wed, Apr 24, 2024 at 4:01 PM Alena Rybakina wrote: > I tested the main postgres branch with and without your fix using a script > that was written by me. It consists of five scenarios and I made a > comparison in the logs between the original version of the master branch > and the master branc

Re: Fix parallel vacuum buffer usage reporting

2024-04-24 Thread Anthonin Bonnefoy
Thanks for the review! > I think that if the anayze command doesn't have the same issue, we > don't need to change it. Good point, I've wrongly assumed that analyze was also impacted but there's no parallel analyze so the block count is correct. > (a) make lazy vacuum use BufferUsage instead o

Re: Fix parallel vacuum buffer usage reporting

2024-04-22 Thread Anthonin Bonnefoy
On Sat, Apr 20, 2024 at 2:00 PM Alena Rybakina wrote: > Hi, thank you for your work with this subject. > > While I was reviewing your code, I noticed that your patch conflicts with > another patch [0] that been committed. > > [0] > https://www.postgresql.org/message-id/flat/CA%2BhUKGJkOiOCa%2Bmag

Re: Fix parallel vacuum buffer usage reporting

2024-03-28 Thread Anthonin Bonnefoy
Hi, Thanks for the review. On Thu, Mar 28, 2024 at 4:07 AM Masahiko Sawada wrote: > As for the proposed patch, the following part should handle the temp > tables too: > True, I've missed the local blocks. I've updated the patch: - read_rate and write_rate now include local block usage - I've a

Re: POC: Extension for adding distributed tracing - pg_tracing

2024-03-12 Thread Anthonin Bonnefoy
Hi all! Thanks for the reviews and comments. > - pg_tracing.c should include postgres.h as the first thing Will do. > afaict none of the use of volatile is required, spinlocks have been barriers > for a long time now Got it, I will remove them. I've been mimicking what was done in pg_stat_statem

Fix expecteddir argument in pg_regress

2024-03-11 Thread Anthonin Bonnefoy
Hi all! pg_regress accepts the expecteddir argument. However, it is never used and outputdir is used instead to get the expected files paths. This patch fixes this and uses the expecteddir argument as expected. Regards, Anthonin v01-0001-pg_regress-Use-expecteddir-for-the-expected-file.patch D

Fix parallel vacuum buffer usage reporting

2024-02-09 Thread Anthonin Bonnefoy
Hi, With a db setup with pgbench, we add an additional index: CREATE INDEX ON pgbench_accounts(abalance) And trigger several updates and vacuum to reach a stable amount of dirtied pages: UPDATE pgbench_accounts set abalance = abalance + 1 WHERE aid=1; CHECKPOINT; VACUUM (VERBOSE, INDEX_CLEANUP ON

Re: POC: Extension for adding distributed tracing - pg_tracing

2024-02-06 Thread Anthonin Bonnefoy
Hi! On Fri, Jan 26, 2024 at 1:43 PM Nikita Malakhov wrote: > It's a good idea to split a big patch into several smaller ones. > But you've already implemented these features - you could provide them > as sequential small patches (i.e. v13-0002-guc-context-propagation.patch and > so on) Keeping

Re: A performance issue with Memoize

2024-01-31 Thread Anthonin Bonnefoy
Hi, I've seen a similar issue with the following query (tested on the current head): EXPLAIN ANALYZE SELECT * FROM tenk1 t1 LEFT JOIN LATERAL (SELECT t1.two, tenk2.hundred, tenk2.two FROM tenk2) t2 ON t1.hundred = t2.hundred WHERE t1.hundred < 5;

Re: Add \syncpipeline command to pgbench

2024-01-22 Thread Anthonin Bonnefoy
bench script is reached while there's still an ongoing pipeline. 0002 adds the \syncpipeline command (original patch with an additional test case). Regards, Anthonin On Mon, Jan 22, 2024 at 7:16 AM Michael Paquier wrote: > > On Fri, Jan 19, 2024 at 08:55:31AM +0100, Anthonin Bonnefoy

Re: Add \syncpipeline command to pgbench

2024-01-18 Thread Anthonin Bonnefoy
On Fri, Jan 19, 2024 at 5:08 AM Michael Paquier wrote: > The logic looks sound, but I have a > comment about the docs: could it be better to group \syncpipeline with > \startpipeline and \endpipeline? \syncpipeline requires a pipeline to > work. I've updated the doc to group the commands. It doe

Add \syncpipeline command to pgbench

2024-01-18 Thread Anthonin Bonnefoy
Hi, PQsendSyncMessage() was added in https://commitfest.postgresql.org/46/4262/. It allows users to add a Sync message without flushing the buffer. As a follow-up, this change adds an additional meta-command to pgbench, \syncpipeline, which will call PQsendSyncMessage(). This will make it possibl

Re: [PATCH] Add additional extended protocol commands to psql: \parse and \bindx

2024-01-18 Thread Anthonin Bonnefoy
> Hmm. So it does not lead to any user-visible changes, right? >From what I can tell, there's no change in the behaviour. All paths would eventually go through HandleSlashCmds's cleaning logic. This is also mentioned in ignore_slash_options's comment. * Read and discard "normal" slash command op

Re: [PATCH] Add additional extended protocol commands to psql: \parse and \bindx

2024-01-17 Thread Anthonin Bonnefoy
> I do realize the same is true for plain \bind, but it seems > like a bug there too. The unscanned bind's parameters are discarded later in the HandleSlashCmds functions. So adding the ignore_slash_options() for inactive branches scans and discards them earlier. I will add it to match what's done

Re: [PATCH] Add additional extended protocol commands to psql: \parse and \bindx

2024-01-16 Thread Anthonin Bonnefoy
so a tiny nitpick: stmt_name should be replaced with STMT_NAME in > this line of the help message. Fixed On Sat, Jan 13, 2024 at 3:37 PM Jelte Fennema-Nio wrote: > > On Thu, 2 Nov 2023 at 10:52, Anthonin Bonnefoy > wrote: > > The main goal is to provide more ways to test extend

Re: POC: Extension for adding distributed tracing - pg_tracing

2024-01-04 Thread Anthonin Bonnefoy
Hi, > This approach avoids the need to rewrite SQL or give special meaning to SQL > comments. SQLCommenter already has a good amount of support and is referenced in opentelemetry https://github.com/open-telemetry/opentelemetry-sqlcommenter So the goal was more to leverage the existing trace prop

Re: Possible segfault when sending notification within a ProcessUtility hook

2023-12-06 Thread Anthonin Bonnefoy
> On Tue, Dec 5, 2023 at 9:03 PM Tom Lane wrote: > Why should we regard that as anything other than a bug in the > ProcessUtility hook? A failed transaction should not send any > notifies. Fair point. That was also my initial assumption but I thought that the transaction state was not available

Possible segfault when sending notification within a ProcessUtility hook

2023-12-05 Thread Anthonin Bonnefoy
Hi, I've encountered the following segfault: #0: 0x000104e821a8 postgres`list_head(l=0x7f7f7f7f7f7f7f7f) at pg_list.h:130:17 #1: 0x000104e81c9c postgres`PreCommit_Notify at async.c:932:16 #2: 0x000104dd02f8 postgres`CommitTransaction at xact.c:2236:2 #3: 0x000104dcfc24 postgres`Co

Re: Add PQsendSyncMessage() to libpq

2023-11-13 Thread Anthonin Bonnefoy
Hi, I've played a bit with the patch on my side. One thing that would be great would be to make this available in pgbench through a \syncpipeline meta command. That would make it easier for users to test whether there's a positive impact with their queries or not. I've wrote a patch to add it to

[PATCH] Add additional extended protocol commands to psql: \parse and \bindx

2023-11-02 Thread Anthonin Bonnefoy
Hi all! Currently, only unnamed prepared statements are supported by psql with the \bind command and it's not possible to create or use named prepared statements through extended protocol. This patch introduces 2 additional commands: \parse and \bindx. \parse allows us to issue a Parse message to

Re: POC: Extension for adding distributed tracing - pg_tracing

2023-07-28 Thread Anthonin Bonnefoy
> What do you think about using INSTR_TIME_SET_CURRENT, INSTR_TIME_SUBTRACT and INSTR_TIME_GET_MILLISEC > macros for timing calculations? If you're talking of the two instances where I'm modifying the instr_time's ticks, it's because I can't use the macros there. The first case is for the parse spa

Re: POC: Extension for adding distributed tracing - pg_tracing

2023-07-28 Thread Anthonin Bonnefoy
Hi, > I'd keep Autotools build up to date Definitely. The patch is still in a rough state and updating Autotools fell through the cracks. > I'm currently playing with the patch and > reviewing sources, if you need any cooperation - please let us know. The goal for me was to get validation on the

Re: POC: Extension for adding distributed tracing - pg_tracing

2023-07-27 Thread Anthonin Bonnefoy
> Agree, something goes wrong when using Autotools (but not Meson) on > both Linux and MacOS. I didn't investigate the issue though. I was only using meson and forgot to keep Automake files up to date when I've split pg_tracing.c in multiple files (span.c, explain.c...). On Fri, Jul 28, 2023 at 8:

Re: POC: Extension for adding distributed tracing - pg_tracing

2023-07-26 Thread Anthonin Bonnefoy
I've initially thought of sending the spans from PostgreSQL since this is the usual behavior of tracing libraries. However, this created a lot potential issues: - Protocol support and differences between trace collectors. OpenTelemetry seems to use gRPC, others are using http and those will requi

Re: Flush SLRU counters in checkpointer process

2023-07-18 Thread Anthonin Bonnefoy
://github.com/postgres/postgres/blob/c8e1ba736b2b9e8c98d37a5b77c4ed31baf94147/src/backend/utils/activity/pgstat_slru.c#L161-L162 ). I will try to get a new patch with improved test stability. On Mon, Jul 3, 2023 at 3:18 PM Daniel Gustafsson wrote: > > On 3 Mar 2023, at 09:06

[PATCH] Add statement_timeout in pg_stat_activity

2023-04-04 Thread Anthonin Bonnefoy
Hello hackers. This patch adds the backend's statement_timeout value to pg_stat_activity. This would provide some insights on clients that are disabling a default statement timeout or overriding it through a pgbouncer, messing with other sessions. pg_stat_activity seemed like the best place to h

Re: Flush SLRU counters in checkpointer process

2023-03-03 Thread Anthonin Bonnefoy
Here's the patch rebased with Andres' suggestions. Happy to update it if there's any additionalj change required. On Wed, Mar 1, 2023 at 8:46 PM Gregory Stark (as CFM) wrote: > On Thu, 12 Jan 2023 at 03:46, Anthonin Bonnefoy > wrote: > > > > > > That wou

Re: Flush SLRU counters in checkpointer process

2023-01-12 Thread Anthonin Bonnefoy
On Wed, Jan 11, 2023 at 5:33 PM Andres Freund wrote: > Hi, > > On 2023-01-11 10:29:06 +0100, Anthonin Bonnefoy wrote: > > Currently, the Checkpointer process only reports SLRU statistics at > server > > shutdown, leading to delayed statistics for SLRU flushes. This pat

Flush SLRU counters in checkpointer process

2023-01-11 Thread Anthonin Bonnefoy
Hello hackers, Currently, the Checkpointer process only reports SLRU statistics at server shutdown, leading to delayed statistics for SLRU flushes. This patch adds a flush of SLRU stats to the end of checkpoints. Best regards, Anthonin flush-slru-counters.patch Description: Binary data