Re: logical decoding bug: segfault in ReorderBufferToastReplace()

2019-12-11 Thread Drouvot, Bertrand
On 12/9/19, 10:10 AM, "Tomas Vondra" wrote: >On Wed, Dec 04, 2019 at 05:36:16PM -0800, Jeremy Schneider wrote: >>On 9/8/19 14:01, Tom Lane wrote: >>> Fix RelationIdGetRelation calls that weren't bothering with error checks. >>> >>> ... >>> >>> Details >>> ---

Re: Change xl_hash_vacuum_one_page.ntuples from int to uint16

2023-02-27 Thread Drouvot, Bertrand
Hi, On 2/27/23 6:27 AM, Amit Kapila wrote: On Fri, Feb 17, 2023 at 7:44 PM Drouvot, Bertrand wrote: On 2/16/23 1:26 PM, Drouvot, Bertrand wrote: Hi, On 2/16/23 12:00 PM, Amit Kapila wrote: BTW, feel free to create the second patch (to align the types for variables/arguments) as that

Re: Track IO times in pg_stat_io

2023-02-28 Thread Drouvot, Bertrand
Hi, On 2/26/23 5:03 PM, Melanie Plageman wrote: Hi, As suggested in [1], the attached patch adds IO times to pg_stat_io; Thanks for the patch! I started to have a look at it and figured out that a tiny rebase was needed (due to 728560db7d and b9f0e54bc9), so please find the rebase (aka V2)

Re: Add shared buffer hits to pg_stat_io

2023-02-28 Thread Drouvot, Bertrand
Hi, On 2/25/23 9:16 PM, Melanie Plageman wrote: Hi, As suggested in [1], the attached patch adds shared buffer hits to pg_stat_io. Thanks for the patch! BufferDesc * LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, -bool *foundP

Re: Minimal logical decoding on standbys

2023-03-01 Thread Drouvot, Bertrand
Hi, On 3/1/23 1:48 AM, Jeff Davis wrote: On Mon, 2023-02-27 at 09:40 +0100, Drouvot, Bertrand wrote: Please find attached V51 tiny rebase due to a6cd1fc692 (for 0001) and 8a8661828a (for 0005). [ Jumping into this thread late, so I apologize if these comments have already been covered

Re: Normalization of utility queries in pg_stat_statements

2023-03-01 Thread Drouvot, Bertrand
Hi, On 3/1/23 5:47 AM, Michael Paquier wrote: On Mon, Feb 20, 2023 at 11:34:59AM +0900, Michael Paquier wrote: With the patches.. Attached is an updated patch set, where I have done more refactoring work for the regression tests of pg_stat_statements, splitting pg_stat_statments.sql into the

Re: Generate pg_stat_get_xact*() functions with Macros

2023-03-01 Thread Drouvot, Bertrand
Hi, On 3/1/23 8:54 PM, Gregory Stark (as CFM) wrote: Looks like you have a path forward on this and it's not ready to commit yet? In which case I'll mark it Waiting on Author? Yeah, there is some dependencies around this one. [1]: depends on it Current one depends of [2], [3] and [4] Waitin

Re: Minimal logical decoding on standbys

2023-03-02 Thread Drouvot, Bertrand
Hi, On 3/2/23 1:40 AM, Jeff Davis wrote: On Wed, 2023-03-01 at 11:51 +0100, Drouvot, Bertrand wrote: Why not "simply" call ConditionVariablePrepareToSleep() without any call to ConditionVariableTimedSleep() later? ConditionVariableSleep() re-inserts itself into the queue

Re: Fix comments in gistxlogDelete, xl_heap_freeze_page and xl_btree_delete

2023-03-02 Thread Drouvot, Bertrand
Hi, On 1/6/23 11:05 AM, Drouvot, Bertrand wrote: Hi hackers, Please find attached a patch to $SUBJECT. The wrong comments have been discovered by Robert in [1]. Submitting this here as a separate thread so it does not get lost in the logical decoding on standby thread. [1]: https

Re: Minimal logical decoding on standbys

2023-03-03 Thread Drouvot, Bertrand
Hi, On 3/2/23 8:45 PM, Jeff Davis wrote: On Thu, 2023-03-02 at 10:20 +0100, Drouvot, Bertrand wrote: Right, but in our case, right after the wakeup (the one due to the CV broadcast, aka the one that will remove it from the wait queue) we'll exit the loop due to: " /* che

Re: Minimal logical decoding on standbys

2023-03-03 Thread Drouvot, Bertrand
Hi, On 3/3/23 8:58 AM, Jeff Davis wrote: On Thu, 2023-03-02 at 11:45 -0800, Jeff Davis wrote: In this case it looks easier to add the right API than to be sure about whether it's needed or not. I attached a sketch of one approach. Oh, that's very cool, thanks a lot! I'm not very confiden

Re: Fix comments in gistxlogDelete, xl_heap_freeze_page and xl_btree_delete

2023-03-03 Thread Drouvot, Bertrand
Hi, On 3/3/23 12:30 PM, Amit Kapila wrote: On Thu, Mar 2, 2023 at 6:35 PM Drouvot, Bertrand wrote: On 1/6/23 11:05 AM, Drouvot, Bertrand wrote: Hi hackers, Please find attached a patch to $SUBJECT. The wrong comments have been discovered by Robert in [1]. Submitting this here as a

Re: Fix comments in gistxlogDelete, xl_heap_freeze_page and xl_btree_delete

2023-03-04 Thread Drouvot, Bertrand
Hi, On 3/3/23 6:53 PM, Robert Haas wrote: On Fri, Mar 3, 2023 at 11:28 AM Drouvot, Bertrand wrote: Thanks for having looked at it! +1. Committed. Thanks! Not a big deal, but the commit message that has been used is not 100% accurate. Indeed, for gistxlogDelete, that's the othe

Re: Reconcile stats in find_tabstat_entry() and get rid of PgStat_BackendFunctionEntry

2023-03-05 Thread Drouvot, Bertrand
Hi, On 2/16/23 10:21 PM, Andres Freund wrote: Hi, On 2023-02-15 09:21:48 +0100, Drouvot, Bertrand wrote: diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c index f793ac1516..b26e2a5a7a 100644 --- a/src/backend/utils/activity

Re: Add shared buffer hits to pg_stat_io

2023-03-07 Thread Drouvot, Bertrand
Hi, On 3/6/23 4:38 PM, Melanie Plageman wrote: Thanks for the review! On Tue, Feb 28, 2023 at 7:36 AM Drouvot, Bertrand wrote: BufferDesc * LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, -bool *foundPtr, IOContext

Re: Track IO times in pg_stat_io

2023-03-07 Thread Drouvot, Bertrand
Hi, On 3/6/23 5:30 PM, Melanie Plageman wrote: Thanks for the review! On Tue, Feb 28, 2023 at 4:49 AM Drouvot, Bertrand wrote: On 2/26/23 5:03 PM, Melanie Plageman wrote: The timings will only be non-zero when track_io_timing is on That could lead to incorrect interpretation if one wants

Re: Minimal logical decoding on standbys

2023-03-08 Thread Drouvot, Bertrand
Hi, On 3/3/23 5:26 PM, Drouvot, Bertrand wrote: Hi, On 3/3/23 8:58 AM, Jeff Davis wrote: On Thu, 2023-03-02 at 11:45 -0800, Jeff Davis wrote: In this case it looks easier to add the right API than to be sure about whether it's needed or not. I attached a sketch of one approach.

Re: Track IO times in pg_stat_io

2023-03-08 Thread Drouvot, Bertrand
Hi, On 3/7/23 7:47 PM, Andres Freund wrote: On 2023-03-07 13:43:28 -0500, Melanie Plageman wrote: Now I've a second thought: what do you think about resetting the related number of operations and *_time fields when enabling/disabling track_io_timing? (And mention it in the doc). That way it'd

Re: Track IO times in pg_stat_io

2023-03-09 Thread Drouvot, Bertrand
Hi, On 3/9/23 1:34 AM, Andres Freund wrote: Hi, On 2023-03-08 12:55:34 +0100, Drouvot, Bertrand wrote: On 3/7/23 7:47 PM, Andres Freund wrote: On 2023-03-07 13:43:28 -0500, Melanie Plageman wrote: No, I don't think we can do that. It can be enabled on a per-session basis. Oh right. So

Re: Avoid multiple SetLatch() calls in procsignal_sigusr1_handler()

2023-03-09 Thread Drouvot, Bertrand
Hi, On 2/28/23 4:30 PM, Bharath Rupireddy wrote: Hi, Most of the multiplexed SIGUSR1 handlers are setting latch explicitly when the procsignal_sigusr1_handler() can do that for them at the end. Right. These multiplexed handlers are currently being used as SIGUSR1 handlers, not as independen

Re: Add shared buffer hits to pg_stat_io

2023-03-09 Thread Drouvot, Bertrand
Hi, On 3/9/23 2:23 PM, Melanie Plageman wrote: On Wed, Mar 8, 2023 at 2:23 PM Andres Freund wrote: On 2023-03-08 13:44:32 -0500, Melanie Plageman wrote: However, I am concerned that, while unlikely, this could be flakey. Something could happen to force all of those blocks out of shared buffe

Re: Minimal logical decoding on standbys

2023-03-10 Thread Drouvot, Bertrand
Hi, On 3/8/23 11:25 AM, Drouvot, Bertrand wrote: Hi, On 3/3/23 5:26 PM, Drouvot, Bertrand wrote: Hi, On 3/3/23 8:58 AM, Jeff Davis wrote: On Thu, 2023-03-02 at 11:45 -0800, Jeff Davis wrote: In this case it looks easier to add the right API than to be sure about whether it's needed o

Re: [BUG] pg_stat_statements and extended query protocol

2023-03-13 Thread Drouvot, Bertrand
Hi, On 3/2/23 8:27 AM, Michael Paquier wrote: On Wed, Jan 25, 2023 at 11:22:04PM +, Imseih (AWS), Sami wrote: Doing some work with extended query protocol, I encountered the same issue that was discussed in [1]. It appears when a client is using extended query protocol and sends an Execute

Re: Get rid of PgStat_BackendFunctionEntry

2023-03-16 Thread Drouvot, Bertrand
Hi, On 3/16/23 6:25 AM, Michael Paquier wrote: On Wed, Mar 08, 2023 at 01:38:38PM -0800, Nathan Bossart wrote: Looks reasonable to me. I have been catching up with this thread and the other thread, and indeed it looks like this is going to help in refactoring pgstatfuncs.c to have more macros

Re: Reconcile stats in find_tabstat_entry() and get rid of PgStat_BackendFunctionEntry

2023-03-16 Thread Drouvot, Bertrand
Hi, On 3/16/23 7:29 AM, Michael Paquier wrote: On Mon, Mar 06, 2023 at 08:33:15AM +0100, Drouvot, Bertrand wrote: Thanks for having looked at it! Looking at that, I have a few comments. +tabentry = (PgStat_TableStatus *) entry_ref->pending; +tablestatus = palloc(siz

Re: Speed-up shared buffers prewarming

2023-03-16 Thread Drouvot, Bertrand
Hi, On 3/15/23 10:40 PM, Matthias van de Meent wrote: On Wed, 15 Mar 2023 at 21:38, Konstantin Knizhnik wrote: Hi hackers, It is well known fact that queries using sequential scan can not be used to prewarm cache, because them are using ring buffer even if shared buffers are almost empty. I

Re: Reconcile stats in find_tabstat_entry() and get rid of PgStat_BackendFunctionEntry

2023-03-16 Thread Drouvot, Bertrand
On 3/16/23 12:46 PM, Michael Paquier wrote: On Thu, Mar 16, 2023 at 11:32:56AM +0100, Drouvot, Bertrand wrote: On 3/16/23 7:29 AM, Michael Paquier wrote: From what I get with this change, the number of tuples changed by DMLs have their computations done a bit earlier, Thanks for looking at

Re: Remove nonmeaningful prefixes in PgStat_* fields

2023-03-20 Thread Drouvot, Bertrand
Hi, On 3/20/23 8:32 AM, Michael Paquier wrote: /* Total time previously charged to function, as of function start */ - instr_time save_f_total_time; + instr_time save_total_time; I have something to say about this one, though.. I find this change a bit confusing.

Re: Reconcile stats in find_tabstat_entry() and get rid of PgStat_BackendFunctionEntry

2023-03-20 Thread Drouvot, Bertrand
Hi, On 3/20/23 12:43 AM, Michael Paquier wrote: At the end, documenting both still sounds like the best move to me. Agree. Please find attached v1-0001-pg_stat_get_xact_blocks_fetched-and_hit-doc.patch doing so. I did not put the exact same wording as the one being removed in ddfc2d9, as:

Re: Reconcile stats in find_tabstat_entry() and get rid of PgStat_BackendFunctionEntry

2023-03-21 Thread Drouvot, Bertrand
Hi, On 3/22/23 5:45 AM, Michael Paquier wrote: On Wed, Mar 22, 2023 at 11:37:03AM +0900, Kyotaro Horiguchi wrote: In the original description, "buffer fetches" appears to be a plural form of a compound noun and correct, similar to "buffer hits" mentioned later. If we reword it, I think it shoul

Re: Reconcile stats in find_tabstat_entry() and get rid of PgStat_BackendFunctionEntry

2023-03-22 Thread Drouvot, Bertrand
Hi, On 3/22/23 7:44 AM, Drouvot, Bertrand wrote: Hi, On 3/22/23 5:45 AM, Michael Paquier wrote: On Wed, Mar 22, 2023 at 11:37:03AM +0900, Kyotaro Horiguchi wrote: In the original description, "buffer fetches" appears to be a plural form of a compound noun and correct, similar to &q

Re: [BUG] pg_stat_statements and extended query protocol

2023-03-22 Thread Drouvot, Bertrand
Hi, On 3/21/23 2:16 PM, Imseih (AWS), Sami wrote: This indeed feels a bit more natural seen from here, after looking at the code paths using an Instrumentation in the executor and explain, for example. At least, this stresses me much less than adding 16 bytes to EState for something restricted t

Re: Remove nonmeaningful prefixes in PgStat_* fields

2023-03-22 Thread Drouvot, Bertrand
Hi, On 3/23/23 1:09 AM, Michael Paquier wrote: On Wed, Mar 22, 2023 at 02:52:23PM -0400, Melanie Plageman wrote: This comment still has the t_ prefix as does the one for tuples_updated and deleted. otherwise, LGTM. Good catch. pgstat_count_heap_update() has a t_tuples_hot_updated, and pgsta

Re: [BUG] pg_stat_statements and extended query protocol

2023-03-23 Thread Drouvot, Bertrand
Hi, On 3/22/23 10:35 PM, Imseih (AWS), Sami wrote: What about using an uint64 for calls? That seems more appropriate to me (even if queryDesc->totaltime->calls will be passed (which is int64), but that's already also the case for the "rows" argument and queryDesc->totaltime->rows_processed) Th

Re: Generate pg_stat_get_xact*() functions with Macros

2023-03-23 Thread Drouvot, Bertrand
Hi, On 3/24/23 1:04 AM, Michael Paquier wrote: On Thu, Mar 02, 2023 at 08:39:14AM +0100, Drouvot, Bertrand wrote: Yeah, there is some dependencies around this one. [1]: depends on it Current one depends of [2], [3] and [4] Waiting on Author is then the right state, thanks for having moved it

Re: Orphaned wait event

2023-03-23 Thread Drouvot, Bertrand
Hi, On 3/23/23 11:00 PM, Thomas Munro wrote: I think if we want proper automation here we should look into a way to define wait events in a central file similar to what we do for src/backend/storage/lmgr/lwlocknames.txt. It could give the enum name, the display name, and the documentation sente

Re: Checksum errors in pg_stat_database

2022-12-11 Thread Drouvot, Bertrand
On 12/12/22 12:40 AM, Michael Paquier wrote: On Sun, Dec 11, 2022 at 09:18:42PM +0100, Magnus Hagander wrote: It would be less of a concern yes, but I think it still would be a concern. If you have a large amount of corruption you could quickly get to millions of rows to keep track of which w

Re: Checksum errors in pg_stat_database

2022-12-11 Thread Drouvot, Bertrand
On 12/12/22 5:09 AM, Michael Paquier wrote: On Sun, Dec 11, 2022 at 08:48:15PM -0500, Tom Lane wrote: I think a stats table indexed solely by relfilenode wouldn't be a great idea, because you'd lose all the stats about a table as soon as you do vacuum full/cluster/rewriting-alter-table/etc.

Re: Checksum errors in pg_stat_database

2022-12-12 Thread Drouvot, Bertrand
On 12/12/22 8:15 AM, Drouvot, Bertrand wrote: On 12/12/22 5:09 AM, Michael Paquier wrote: On Sun, Dec 11, 2022 at 08:48:15PM -0500, Tom Lane wrote: I think a stats table indexed solely by relfilenode wouldn't be a great idea, because you'd lose all the stats about a table as s

Re: Minimal logical decoding on standbys

2022-12-13 Thread Drouvot, Bertrand
Hi, On 12/12/22 6:41 PM, Robert Haas wrote: On Sat, Dec 10, 2022 at 3:09 AM Drouvot, Bertrand wrote: Attaching V30, mandatory rebase due to 66dcb09246. It's a shame that this hasn't gotten more attention, because the topic is important, but I'm as guilty of being too busy to

Re: Minimal logical decoding on standbys

2022-12-13 Thread Drouvot, Bertrand
Hi, On 12/13/22 2:50 PM, Robert Haas wrote: On Tue, Dec 13, 2022 at 5:49 AM Drouvot, Bertrand wrote: I think the real problem here is that RelationIsAccessibleInLogicalDecoding is returning *the wrong answer* when the relation is a user-catalog table. It does so because it relies on

Re: Minimal logical decoding on standbys

2022-12-13 Thread Drouvot, Bertrand
On 12/13/22 5:42 PM, Robert Haas wrote: On Tue, Dec 13, 2022 at 11:37 AM Drouvot, Bertrand wrote: It seems kind of unfortunate to have to add payload to a whole bevy of record types for this feature. I think it's worth it, both because the feature is extremely important, Agree and I

Re: Minimal logical decoding on standbys

2022-12-13 Thread Drouvot, Bertrand
On 12/13/22 5:49 PM, Robert Haas wrote: On Tue, Dec 13, 2022 at 11:46 AM Drouvot, Bertrand wrote: Agree and I don't think that there is other option than adding some payload in some WAL records (at the very beginning the proposal was to periodically log a new record that announce

Re: Minimal logical decoding on standbys

2022-12-14 Thread Drouvot, Bertrand
Hi, On 12/13/22 5:37 PM, Drouvot, Bertrand wrote: Hi, On 12/13/22 2:50 PM, Robert Haas wrote: On Tue, Dec 13, 2022 at 5:49 AM Drouvot, Bertrand It seems kind of unfortunate to have to add payload to a whole bevy of record types for this feature. I think it's worth it, both becaus

Re: Minimal logical decoding on standbys

2022-12-15 Thread Drouvot, Bertrand
Hi, On 12/14/22 6:48 PM, Robert Haas wrote: On Wed, Dec 14, 2022 at 12:35 PM Andres Freund wrote: typedef struct xl_heap_prune I think this is unsafe on alignment-picky machines. I think it will cause the offset numbers to be aligned at an odd address. heap_xlog_prune() doesn't copy the dat

Re: Minimal logical decoding on standbys

2022-12-15 Thread Drouvot, Bertrand
Hi, On 12/14/22 4:55 PM, Robert Haas wrote: On Wed, Dec 14, 2022 at 8:06 AM Drouvot, Bertrand> Other comments: +if RelationIsAccessibleInLogicalDecoding(rel) +xlrec.flags |= VISIBILITYMAP_ON_CATALOG_ACCESSIBLE_IN_LOGICAL_DECODING; This is a few parentheses short of where

Re: Minimal logical decoding on standbys

2022-12-16 Thread Drouvot, Bertrand
Hi, On 12/15/22 11:31 AM, Drouvot, Bertrand wrote: Hi, On 12/14/22 4:55 PM, Robert Haas wrote:   On Wed, Dec 14, 2022 at 8:06 AM Drouvot, Bertrand> Other comments: +    if RelationIsAccessibleInLogicalDecoding(rel) +    xlrec.fl

Re: Minimal logical decoding on standbys

2022-12-16 Thread Drouvot, Bertrand
Hi, On 12/16/22 2:51 PM, Andres Freund wrote: Hi, On 2022-12-16 11:33:50 +0100, Drouvot, Bertrand wrote: @@ -235,13 +236,14 @@ typedef struct xl_btree_delete TransactionId snapshotConflictHorizon; uint16 ndeleted; uint16 nupdated; + bool

Re: Minimal logical decoding on standbys

2022-12-16 Thread Drouvot, Bertrand
Hi, On 12/16/22 5:38 PM, Robert Haas wrote: On Fri, Dec 16, 2022 at 10:08 AM Drouvot, Bertrand wrote: After 1489b1ce728 the name mayConflictInLogicalDecoding seems odd. Seems it should be a riff on snapshotConflictHorizon? Gotcha, what about logicalSnapshotConflictThreat

Re: Minimal logical decoding on standbys

2022-12-20 Thread Drouvot, Bertrand
Hi, On 12/16/22 6:24 PM, Drouvot, Bertrand wrote: Hi, On 12/16/22 5:38 PM, Robert Haas wrote: On Fri, Dec 16, 2022 at 10:08 AM Drouvot, Bertrand wrote: After 1489b1ce728 the name mayConflictInLogicalDecoding seems odd. Seems it should be a riff on snapshotConflictHorizon? Gotcha, what

Re: Minimal logical decoding on standbys

2022-12-20 Thread Drouvot, Bertrand
Hi, On 12/20/22 7:31 PM, Robert Haas wrote: On Tue, Dec 20, 2022 at 1:19 PM Andres Freund wrote: I don't understand what the "may*" or "*Possible" really are about. snapshotConflictHorizon is a conflict with a certain xid - there commonly won't be anything to conflict with. If there's a confli

Re: Minimal logical decoding on standbys

2022-12-21 Thread Drouvot, Bertrand
Hi, On 12/20/22 10:41 PM, Robert Haas wrote: On Tue, Dec 20, 2022 at 3:39 PM Robert Haas wrote: I think this might be the only WAL record type where there's a problem, but I haven't fully confirmed that yet. It's not. GIST has the same issue. The same test case demonstrates the problem there

Re: Minimal logical decoding on standbys

2022-12-21 Thread Drouvot, Bertrand
Hi, On 12/21/22 10:06 AM, Drouvot, Bertrand wrote: Hi, On 12/20/22 10:41 PM, Robert Haas wrote: On Tue, Dec 20, 2022 at 3:39 PM Robert Haas wrote: I guess whatever else we do here, we should fix the comments. Agree, please find attached a patch proposal doing so. Bottom line is that I

Re: Add index scan progress to pg_stat_progress_vacuum

2023-01-04 Thread Drouvot, Bertrand
Hi, On 1/3/23 5:46 PM, Imseih (AWS), Sami wrote: cirrus-ci.com/task/4557389261701120 I earlier compiled without building with --enable-cassert, which is why the compilation errors did not produce on my buid. Fixed in v19. Thanks Thanks for the updated patch! Some comments about it: + +

Re: Add a new pg_walinspect function to extract FPIs from WAL records

2023-01-04 Thread Drouvot, Bertrand
Hi, On 12/27/22 12:48 PM, Bharath Rupireddy wrote: Hi, Here's a patch that implements the idea of extracting full page images from WAL records [1] [2] with a function in pg_walinspect. This new function accepts start and end lsn and returns full page image info such as WAL record lsn, tablespac

Re: Split index and table statistics into different types of stats

2023-01-05 Thread Drouvot, Bertrand
Hi, On 1/5/23 1:27 AM, Andres Freund wrote: Hi, On 2023-01-03 15:19:18 +0100, Drouvot, Bertrand wrote: diff --git a/src/backend/access/common/relation.c b/src/backend/access/common/relation.c index 4017e175e3..fca166a063 100644 --- a/src/backend/access/common/relation.c +++ b/src/backend

Generate pg_stat_get_xact*() functions with Macros

2023-01-05 Thread Drouvot, Bertrand
Hi hackers, Please find attached a patch proposal to $SUBJECT. This is the same kind of work that has been done in 83a1a1b566 and 8018ffbf58 but this time for the pg_stat_get_xact*() functions (as suggested by Andres in [1]). The function names remain the same, but some fields have to be renam

Re: Generate pg_stat_get_xact*() functions with Macros

2023-01-06 Thread Drouvot, Bertrand
Hi, On 1/5/23 9:19 PM, Corey Huinker wrote: I like code cleanups like this. It makes sense, it results in less code, and anyone doing a `git grep pg_stat_get_live_tuples` will quickly find the macro definition. Unsurprisingly, it passes `make check-world`. So I think it's good to go as-is.

Re: Generate pg_stat_get_xact*() functions with Macros

2023-01-06 Thread Drouvot, Bertrand
Hi, On 1/6/23 12:21 AM, Andres Freund wrote: Hi, On 2023-01-05 15:19:54 -0500, Corey Huinker wrote: It does get me wondering, however, if we reordered the three typedefs to group like-typed registers together, we could make them an array with the names becoming defined constant index values (o

Re: Minimal logical decoding on standbys

2023-01-06 Thread Drouvot, Bertrand
Hi, On 1/6/23 4:40 AM, Andres Freund wrote: Hi, Thomas, there's one point at the bottom wrt ConditionVariables that'd be interesting for you to comment on. On 2023-01-05 16:15:39 -0500, Robert Haas wrote: On Tue, Jan 3, 2023 at 2:42 AM Drouvot, Bertrand wrote: Please find at

Fix comments in gistxlogDelete, xl_heap_freeze_page and xl_btree_delete

2023-01-06 Thread Drouvot, Bertrand
Hi hackers, Please find attached a patch to $SUBJECT. The wrong comments have been discovered by Robert in [1]. Submitting this here as a separate thread so it does not get lost in the logical decoding on standby thread. [1]: https://www.postgresql.org/message-id/CA%2BTgmoYTTsxP8y6uknZvCBNCR

Re: Add a new pg_walinspect function to extract FPIs from WAL records

2023-01-10 Thread Drouvot, Bertrand
Hi, On 1/6/23 6:41 PM, Bharath Rupireddy wrote: On Fri, Jan 6, 2023 at 11:47 AM Bharath Rupireddy wrote: On Thu, Jan 5, 2023 at 6:51 PM Bharath Rupireddy wrote: I'm also wondering if it would make sense to extend the test coverage of it (and pg_waldump) to "validate" that both extracted

Change xl_hash_vacuum_one_page.ntuples from int to uint16

2023-01-10 Thread Drouvot, Bertrand
Hi hackers, While working on [1], I noticed that xl_hash_vacuum_one_page.ntuples is an int. Unless I'm missing something, It seems to me that it would make more sense to use an uint16 (like this is done for gistxlogDelete.ntodelete for example). Please find attached a patch proposal to do so.

Re: Strengthen pg_waldump's --save-fullpage tests

2023-01-10 Thread Drouvot, Bertrand
Hi, On 1/10/23 2:22 AM, Michael Paquier wrote: On Mon, Jan 09, 2023 at 08:30:00AM +0530, Bharath Rupireddy wrote: A recent commit [1] added --save-fullpage option to pg_waldump to extract full page images (FPI) from WAL records and save them into files (one file per FPI) under a specified direc

Re: Strengthen pg_waldump's --save-fullpage tests

2023-01-11 Thread Drouvot, Bertrand
Hi, On 1/11/23 5:17 AM, Bharath Rupireddy wrote: On Wed, Jan 11, 2023 at 6:32 AM Michael Paquier wrote: On Tue, Jan 10, 2023 at 05:25:44PM +0100, Drouvot, Bertrand wrote: I like the idea of comparing the full page (and not just the LSN) but I'm not sure that adding the pagein

Re: Minimal logical decoding on standbys

2023-01-11 Thread Drouvot, Bertrand
Hi, On 1/11/23 8:32 AM, Bharath Rupireddy wrote: On Tue, Jan 10, 2023 at 2:03 PM Drouvot, Bertrand wrote: Please find attached, V37 taking care of: Thanks. I started to digest the design specified in the commit message and these patches. Thanks for looking at it! Here are some quick

Re: Strengthen pg_waldump's --save-fullpage tests

2023-01-11 Thread Drouvot, Bertrand
Hi, On 1/12/23 5:44 AM, Michael Paquier wrote: On Wed, Jan 11, 2023 at 07:17:47PM +0530, Bharath Rupireddy wrote: Note that the raw page on the table might differ not just in page LSN but also in other fields, for instance see heap_mask for instance. It masks lsn, checksum, hint bits, unused sp

Re: Generate pg_stat_get_xact*() functions with Macros

2023-01-11 Thread Drouvot, Bertrand
Hi, On 1/11/23 11:59 PM, Andres Freund wrote: Hi, Michael, CCing you because of the point about PG_STAT_GET_DBENTRY_FLOAT8 below. On 2023-01-05 14:48:39 +0100, Drouvot, Bertrand wrote: While at it, I also took the opportunity to create the macros for pg_stat_get_xact_function_total_time

Remove nonmeaningful prefixes in PgStat_* fields

2023-01-12 Thread Drouvot, Bertrand
Hi hackers, Please find attached a patch to $SUBJECT. It is a preliminary patch for [1]. The main ideas are: 1) to have consistent naming between the pg_stat_get*() functions and their associated counters and 2) to define the new macros in [1] the same way as it has been done in 8018ffbf58 (a

Re: Minimal logical decoding on standbys

2023-01-13 Thread Drouvot, Bertrand
Hi, On 1/11/23 9:27 PM, Andres Freund wrote: Hi, On 2023-01-06 10:52:06 +0100, Drouvot, Bertrand wrote: The problem I have with that is that I saw a lot of flakiness in the tests due to the race condition. So introducing them in that order just doesn't make a whole lot of sense to me.

Re: Generate pg_stat_get_xact*() functions with Macros

2023-01-13 Thread Drouvot, Bertrand
Hi, On 1/12/23 7:24 PM, Andres Freund wrote: Hi, On 2023-01-12 08:38:57 +0100, Drouvot, Bertrand wrote: On 1/11/23 11:59 PM, Andres Freund wrote: Now that this patch renames some fields I don't mind renaming the fields - the prefixes really don't provide anything useful. But it&#

Helper functions for wait_for_catchup() in Cluster.pm

2023-01-17 Thread Drouvot, Bertrand
Hi hackers, please find attached a patch proposal to define $SUBJECT. The idea has been raised in [1], where we are adding more calls to wait_for_catchup() in 'replay' mode. The current code already has 25 of those, so the proposed patch is defining a new wait_for_replay_catchup() function.

Re: Helper functions for wait_for_catchup() in Cluster.pm

2023-01-17 Thread Drouvot, Bertrand
Hi, On 1/17/23 12:23 PM, Alvaro Herrera wrote: On 2023-Jan-17, Drouvot, Bertrand wrote: The idea has been raised in [1], where we are adding more calls to wait_for_catchup() in 'replay' mode. This seems mostly useless as presented. Maybe if you're able to reduce the nois

Re: Minimal logical decoding on standbys

2023-01-18 Thread Drouvot, Bertrand
Hi, On 1/6/23 4:40 AM, Andres Freund wrote: Hi, 0004: @@ -3037,6 +3037,43 @@ $SIG{TERM} = $SIG{INT} = sub { =pod +=item $node->create_logical_slot_on_standby(self, master, slot_name, dbname) + +Create logical replication slot on given standby + +=cut + +sub create_logical_slot_on_standby +

Re: Helper functions for wait_for_catchup() in Cluster.pm

2023-01-19 Thread Drouvot, Bertrand
Hi, On 1/18/23 10:59 AM, Alvaro Herrera wrote: On 2023-Jan-18, Drouvot, Bertrand wrote: The current calls are done that way: wait_for_replay_catchup called: - 8 times with write LSN as an argument - 1 time with insert LSN as an argument - 16 times with flush LSN as an argument So it looks

Re: Minimal logical decoding on standbys

2023-01-19 Thread Drouvot, Bertrand
Hi, On 1/19/23 3:46 AM, Andres Freund wrote: Hi, On 2023-01-18 11:24:19 +0100, Drouvot, Bertrand wrote: On 1/6/23 4:40 AM, Andres Freund wrote: Hm, that's quite expensive. Perhaps worth adding a C helper that can do that for us instead? This will likely also be needed in real applica

Re: Generate pg_stat_get_xact*() functions with Macros

2023-03-26 Thread Drouvot, Bertrand
Hi, On 3/27/23 3:20 AM, Michael Paquier wrote: On Sat, Mar 25, 2023 at 11:50:50AM +0900, Michael Paquier wrote: On Fri, Mar 24, 2023 at 06:58:30AM +0100, Drouvot, Bertrand wrote: - Does not include the refactoring for pg_stat_get_xact_function_total_time(), pg_stat_get_xact_function_self_time

Re: Generate pg_stat_get_xact*() functions with Macros

2023-03-26 Thread Drouvot, Bertrand
On 3/27/23 8:40 AM, Michael Paquier wrote: On Mon, Mar 27, 2023 at 07:45:26AM +0200, Drouvot, Bertrand wrote: Thanks! LGTM, but what about also taking care of pg_stat_get_xact_function_total_time() and pg_stat_get_xact_function_self_time() while at it? With a macro that uses

Re: Generate pg_stat_get_xact*() functions with Macros

2023-03-27 Thread Drouvot, Bertrand
On 3/27/23 9:23 AM, Michael Paquier wrote: On Mon, Mar 27, 2023 at 08:54:13AM +0200, Drouvot, Bertrand wrote: Yes, something like V1 up-thread was doing. I think it can be added with your current proposal. Sure, I can write that. Or perhaps you'd prefer write something yourself? P

Re: Generate pg_stat_get_xact*() functions with Macros

2023-03-27 Thread Drouvot, Bertrand
On 3/28/23 12:41 AM, Michael Paquier wrote: On Mon, Mar 27, 2023 at 07:08:51PM +0900, Michael Paquier wrote: The patch has one mistake: PG_STAT_GET_XACT_FUNCENTRY_FLOAT8_MS does not need a slash on its last line or it would include the next, empty line. This could lead to mistakes (no need t

Re: Reconcile stats in find_tabstat_entry() and get rid of PgStat_BackendFunctionEntry

2023-03-27 Thread Drouvot, Bertrand
Hi, On 3/28/23 7:23 AM, Michael Paquier wrote: On Tue, Mar 28, 2023 at 12:36:15PM +0900, Kyotaro Horiguchi wrote: I found that commit ddfc2d9a37 removed the descriptions for pg_stat_get_blocks_fetched and pg_stat_get_blocks_hit. Right before that commit, monitoring.sgml had these lines: -

Re: Reconcile stats in find_tabstat_entry() and get rid of PgStat_BackendFunctionEntry

2023-03-28 Thread Drouvot, Bertrand
Hi, On 3/29/23 2:09 AM, Michael Paquier wrote: On Tue, Mar 28, 2023 at 05:43:26PM +0900, Kyotaro Horiguchi wrote: No. Fine by me, except that "block read requests" seems to suggest kernel read() calls, maybe because it's not clear whether "block" refers to our buffer blocks or file blocks to me

Autogenerate some wait events code and documentation

2023-03-29 Thread Drouvot, Bertrand
Hi hackers, In another thread [1], Thomas had the idea to $SUBJECT in a similar way to what is currently done with src/backend/storage/lmgr/lwlocknames.txt. Doing so, like in the attached patch proposal, would help to avoid: - wait event without documentation like observed in [2] - orphaned wai

Re: Autogenerate some wait events code and documentation

2023-03-29 Thread Drouvot, Bertrand
Hi, On 3/29/23 11:44 AM, Drouvot, Bertrand wrote: Looking forward to your feedback, Just realized that more polishing was needed. Done in V2 attached. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.comFrom

Re: Minimal logical decoding on standbys

2023-03-31 Thread Drouvot, Bertrand
Hi, On 3/31/23 1:58 PM, Amit Kapila wrote: On Fri, Mar 31, 2023 at 4:17 PM Drouvot, Bertrand wrote: + * This is needed for logical decoding on standby. Indeed the "problem" is that + * WalSndWaitForWal() waits for the *replay* LSN to increase, but gets woken up + * by walreceive

Re: regression coverage gaps for gist and hash indexes

2023-03-31 Thread Drouvot, Bertrand
Hi, On 4/1/23 1:13 AM, Andres Freund wrote: Hi, On 2023-03-31 17:00:00 +0300, Alexander Lakhin wrote: 31.03.2023 15:55, Tom Lane wrote: See also the thread about bug #16329 [1]. Alexander promised to look into improving the test coverage in this area, maybe he can keep an eye on the WAL logi

Re: Minimal logical decoding on standbys

2023-04-02 Thread Drouvot, Bertrand
Hi, On 4/1/23 6:50 AM, Amit Kapila wrote: On Fri, Mar 31, 2023 at 7:14 PM Drouvot, Bertrand wrote: That sounds like a good idea. We could imagine creating a LogicalWalSndWakeup() doing the Walsender(s) triage based on a new variable (as you suggest). But, it looks to me that we: - would

Pass heaprel to GlobalVisTestFor() in vacuumRedirectAndPlaceholder()

2023-04-02 Thread Drouvot, Bertrand
hi hackers, now that the heap relation is passed down to vacuumRedirectAndPlaceholder() thanks to 61b313e47e, we can also pass it down to GlobalVisTestFor() too (to allow more pruning). Please find attached a tiny patch to do so. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Op

Re: Minimal logical decoding on standbys

2023-04-02 Thread Drouvot, Bertrand
Hi, On 4/3/23 7:20 AM, Amit Kapila wrote: On Mon, Apr 3, 2023 at 1:31 AM Jeff Davis wrote: On Sun, 2023-04-02 at 10:11 +0200, Drouvot, Bertrand wrote: I was thinking that, if a new LogicalWalSndWakeup() replaces "ConditionVariableBroadcast(&XLogRecoveryCtl->replayedCV);"

Re: Minimal logical decoding on standbys

2023-04-02 Thread Drouvot, Bertrand
Hi, On 4/3/23 7:35 AM, Amit Kapila wrote: On Mon, Apr 3, 2023 at 4:26 AM Jeff Davis wrote: On Fri, 2023-03-31 at 02:50 -0700, Jeff Davis wrote: But if the ConditionVariableEventSleep() API is added, then I think we should change the non-recovery case to use a CV as well for consistency, and

Re: Pass heaprel to GlobalVisTestFor() in vacuumRedirectAndPlaceholder()

2023-04-03 Thread Drouvot, Bertrand
Hi, On 4/3/23 12:30 AM, Andres Freund wrote: Hi, On 2023-04-02 10:22:18 -0700, Peter Geoghegan wrote: On Sun, Apr 2, 2023 at 10:18 AM Peter Geoghegan wrote: Making nbtree page deletion more efficient when logical decoding is in use seems well worthwhile. There is an "XXX" comment about this

Re: Minimal logical decoding on standbys

2023-04-03 Thread Drouvot, Bertrand
27:45 +0200, Drouvot, Bertrand wrote: 5.3% doc/src/sgml/ 6.2% src/backend/access/transam/ 4.6% src/backend/replication/logical/ 55.6% src/backend/replication/ 4.4% src/backend/storage/ipc/ 6.9% src/backend/tcop/ 5.3% src/backend/ 3.8% src/include/catalog/ 5.3

Re: Minimal logical decoding on standbys

2023-04-03 Thread Drouvot, Bertrand
Hi, On 4/3/23 8:10 AM, Drouvot, Bertrand wrote: Hi, On 4/3/23 7:35 AM, Amit Kapila wrote: On Mon, Apr 3, 2023 at 4:26 AM Jeff Davis wrote: Agreed, even Bertrand and myself discussed the same approach few emails above. BTW, if we have this selective logic to wake physical/logical walsenders

Re: Minimal logical decoding on standbys

2023-04-04 Thread Drouvot, Bertrand
Hi, On 4/4/23 7:57 AM, Amit Kapila wrote: On Mon, Apr 3, 2023 at 8:51 PM Drouvot, Bertrand wrote: I made it as simple as: /* * If the slot is already invalid or is a non conflicting slot, we don't * need to do anything. */ islo

Re: Minimal logical decoding on standbys

2023-04-04 Thread Drouvot, Bertrand
Hi, On 4/4/23 9:48 AM, Masahiko Sawada wrote: On Tue, Apr 4, 2023 at 10:55 AM Masahiko Sawada wrote: Regarding 0004 patch: @@ -2626,6 +2626,12 @@ InitWalSenderSlot(void) walsnd->sync_standby_priority = 0; walsnd->latch = &MyProc->procLatch;

Re: Split index and table statistics into different types of stats

2023-04-04 Thread Drouvot, Bertrand
Hi, On 4/3/23 11:47 PM, Gregory Stark (as CFM) wrote: On Thu, 16 Mar 2023 at 05:25, Drouvot, Bertrand wrote: My plan was to get [1] done before resuming working on the "Split index and table statistics into different types of stats" one. [1]: https://commitfest.postgresql.o

Re: Patch proposal: New hooks in the connection path

2023-04-04 Thread Drouvot, Bertrand
Hi, On 4/4/23 12:08 AM, Gregory Stark (as CFM) wrote: This looks like it was a good discussion -- last summer. But it doesn't seem to be a patch under active development now. It sounds like there were some design constraints that still need some new ideas to solve and a new patch will be needed

Re: Minimal logical decoding on standbys

2023-04-04 Thread Drouvot, Bertrand
Hi, On 4/4/23 1:43 PM, Amit Kapila wrote: On Tue, Apr 4, 2023 at 3:14 PM Drouvot, Bertrand wrote: +static inline bool +LogicalReplicationSlotXidsConflict(ReplicationSlot *s, TransactionId xid) +{ + TransactionId slot_xmin; + TransactionId slot_catalog_xmin; + + slot_xmin = s->data.x

Re: Minimal logical decoding on standbys

2023-04-04 Thread Drouvot, Bertrand
Hi, On 4/4/23 3:43 PM, Amit Kapila wrote: On Tue, Apr 4, 2023 at 6:05 PM Drouvot, Bertrand I think we might want to consider slot's effective_xmin instead of data.xmin as we use that to store xmin_horizon when we build the full snapshot. Oh, I did not know about the 'effective_xmi

Re: Minimal logical decoding on standbys

2023-04-04 Thread Drouvot, Bertrand
Hi, On 4/4/23 1:21 PM, Alvaro Herrera wrote: Hi, On 2023-Apr-03, Andres Freund wrote: Hm? That's what the _'s do. We build strings in parts in other places too. No, what _() does is mark each piece for translation separately. But a translation cannot be done on string pieces, and later hav

  1   2   3   4   5   >