Re: Getting better results from valgrind leak tracking

2021-03-17 Thread Andres Freund
Hi, (really need to fix my mobile phone mail program to keep the CC list...) On 2021-03-17 08:15:43 -0700, Andres Freund wrote: > On Wed, Mar 17, 2021, at 07:16, Tom Lane wrote: > > Andres Freund writes: > > > On 2021-03-16 20:50:17 -0700, Andres Freund wrote: > > Meanwh

Re: [HACKERS] Custom compression methods

2021-03-17 Thread Andres Freund
_tuple(). I think that might be easier for people looking at > this code in the future to understand what's happening. OTOH heap_form_flattened_tuple() has the advantage that we can optimize it further (e.g. to do the conversion to flattened values in fill_val()) without changing the outside API. Greetings, Andres Freund

Re: [HACKERS] Custom compression methods

2021-03-17 Thread Andres Freund
ied from the tableam logic, without a clear need. But if it's useful, then ... Greetings, Andres Freund

replication slot stats memory bug

2021-03-17 Thread Andres Freund
shouldn't allocate it dynamically at all? max_replication_slots doesn't change during postmaster lifetime, so it seems like it should just be allocated once? Greetings, Andres Freund

Re: replication slot stats memory bug

2021-03-17 Thread Andres Freund
Hi, On 2021-03-17 16:04:47 -0700, Andres Freund wrote: > I'll push the minimal fix of forcing the allocation to happen in > pgStatLocalContext and setting it to NULL in pgstat_clear_snapshot(). Done: https://git.postgresql.org/gitweb/?p=postgresql.git

Re: Getting better results from valgrind leak tracking

2021-03-17 Thread Andres Freund
Hi, On 2021-03-17 18:07:36 -0400, Tom Lane wrote: > Andres Freund writes: > >> I found a way around that late last night. Need to mark the context > >> itself as an allocation. But I made a mess on the way to that and need > >> to clean the patch up before s

Re: Permission failures with WAL files in 13~ on Windows

2021-03-17 Thread Andres Freund
ing creation of new segments, or timeline files or whatnot. But I still don't see what the whole thing is supposed to protect us against realistically. Greetings, Andres Freund

Re: replication slot stats memory bug

2021-03-17 Thread Andres Freund
red my perception that it's better to just have a statically sized memory allocation for this. Noteworthy that SLRU stats are done in a fixed size allocation as well... Greetings, Andres Freund

Re: Getting better results from valgrind leak tracking

2021-03-17 Thread Andres Freund
Hi, On 2021-03-17 21:30:48 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2021-03-17 18:07:36 -0400, Tom Lane wrote: > >> Huh, interesting. I wonder why that makes the ident problem go away? > > > I spent a bit of time looking at valgrind, and it looks t

Re: Permission failures with WAL files in 13~ on Windows

2021-03-17 Thread Andres Freund
cycling, opening them, closing them, maybe?). Plus error reporting for unlink failures, of course. Greetings, Andres Freund

Re: Getting better results from valgrind leak tracking

2021-03-17 Thread Andres Freund
t I think it might make sense to add a flag indicating contexts that shouldn't be used for non-transient data. Seems like we fairly regularly have "bugs" around this? Greetings, Andres Freund

Re: Getting better results from valgrind leak tracking

2021-03-17 Thread Andres Freund
increasing memory usage for compiled plpgsql functions unnecessarily, of course. The latter would be good to fix, but the former seems like it might be a practical issue for poolers and the like? So I think we should do at least the reparenting thing to address that? Greetings, Andres Freund

Re: shared-memory based stats collector

2021-03-18 Thread Andres Freund
s in this > version. Since I am heavily editing the code, could you submit "functional" changes (as opposed to fixing rebase issues) as incremental patches? Greetings, Andres Freund

Replication slot stats misgivings

2021-03-19 Thread Andres Freund
Hi, I started to write this as a reply to https://postgr.es/m/20210318015105.dcfa4ceybdjubf2i%40alap3.anarazel.de but I think it doesn't really fit under that header anymore. On 2021-03-17 18:51:05 -0700, Andres Freund wrote: > It does make it easier for the shared memory stats patch, be

Re: Do we work with LLVM 12 on s390x?

2021-03-19 Thread Andres Freund
ences between the two, but there was an ABI change at some point, leading to that error. IIRC I hit that, but it vanished as soon as I used a matching libllvm and clang. Greetings, Andres Freund

Re: [HACKERS] Custom compression methods

2021-03-19 Thread Andres Freund
On 2021-03-19 17:35:58 -0300, Alvaro Herrera wrote: > I find this behavior confusing; I'd rather have configure error out if > it can't find the package support I requested, than continuing with a > set of configure options different from what I gave. +1

Re: shared-memory based stats collector

2021-03-19 Thread Andres Freund
Hi, On 2021-03-10 20:26:56 -0800, Andres Freund wrote: > > + shhashent = dshash_find_extended(pgStatSharedHash, &key, > > + > > create, nowait, create, &shfound); > > + if (shhashent) >

Re: [HACKERS] Custom compression methods

2021-03-19 Thread Andres Freund
On 2021-03-19 22:19:49 +0100, Tomas Vondra wrote: > Yeah. And why does it even require pkg-config, unlike any other library > that I'm aware of? IMO it's fine to require pkg-config to simplify the configure code. Especially for new optional features. Adding multiple alternative ways to discover li

shared memory stats: high level design decisions: consistency, dropping

2021-03-19 Thread Andres Freund
too hard, try to see how hard it'd be to "accurately" drop. But I'm also not sure - it might be smarter to go full in, to avoid introducing a system that we'll just rip out again. Comments? Greetings, Andres Freund [1] https://www.postgresql.org/message-id/2018062

Re: shared memory stats: high level design decisions: consistency, dropping

2021-03-19 Thread Andres Freund
Hi, On 2021-03-20 01:16:31 +0100, Hannu Krosing wrote: > > But now we could instead schedule stats to be removed at commit > time. That's not trivial of course, as we'd need to handle cases where > the commit fails after the commit record, but before processing the > dropped stats. > > We likely

Re: [HACKERS] Custom compression methods

2021-03-20 Thread Andres Freund
On 2021-03-19 15:44:34 -0400, Robert Haas wrote: > I committed the core patch (0003) with a bit more editing. Let's see > what the buildfarm thinks. Congrats Dilip, Robert, All. The slow toast compression has been a significant issue for a long time.

Re: Replication slot stats misgivings

2021-03-20 Thread Andres Freund
Hi, On 2021-03-20 09:25:40 +0530, Amit Kapila wrote: > On Sat, Mar 20, 2021 at 12:22 AM Andres Freund wrote: > > > > And then more generally about the feature: > > - If a slot was used to stream out a large amount of changes (say an > > initial data load), but then

Re: Replication slot stats misgivings

2021-03-20 Thread Andres Freund
user created a slot with the same name after > dropping it) exists with this as well. pgstat_report_replslot_drop() already prevents that, no? Greetings, Andres Freund

Re: Replication slot stats misgivings

2021-03-21 Thread Andres Freund
Hi, On 2021-03-21 16:08:00 +0530, Amit Kapila wrote: > On Sun, Mar 21, 2021 at 2:57 AM Andres Freund wrote: > > On 2021-03-20 10:28:06 +0530, Amit Kapila wrote: > > > On Sat, Mar 20, 2021 at 9:25 AM Amit Kapila > > > wrote: > > > > This idea is worth

Re: shared memory stats: high level design decisions: consistency, dropping

2021-03-21 Thread Andres Freund
, or at least put enough > effort into it to get a feeling of if it's going to require a *lot* more > work or not and then reconsider if necessary. I think my gut's argument is that it's already a huge patch, and that it's better to have the the very substantial memory and disk IO savings with the crappy vacuum approach, than neither. And given the timeframe there does seem to be a substantial danger of "neither" being the outcome... Anyway, I'm mentally sketching out what it'd take. Greetings, Andres Freund

Re: shared memory stats: high level design decisions: consistency, dropping

2021-03-21 Thread Andres Freund
Hi, On 2021-03-21 12:14:35 -0400, Tom Lane wrote: > Andres Freund writes: > > 1) What kind of consistency do we want from the pg_stats_* views? > > That's a hard choice to make. But let me set the record straight: > when we did the initial implementation, the stats snap

Re: replication cleanup code incorrect way to use of HTAB HASH_REMOVE ?

2021-03-21 Thread Andres Freund
l of dynahash. Having all those branches for partitioned hashes, actions are really not great. Greetings, Andres Freund

Re: Table AM and DDLs

2021-03-21 Thread Andres Freund
Hi, On 2021-03-03 22:15:18 +0100, Mats Kindahl wrote: > On Tue, Feb 23, 2021 at 2:11 AM Andres Freund wrote: > Thanks for the answer and sorry about the late reply. Mine is even later ;) > > I don't think that's quite right. It's not exactly obvious from the > &

Re: shared-memory based stats collector

2021-03-22 Thread Andres Freund
Hi, On 2021-03-19 14:27:38 -0700, Andres Freund wrote: > Yep, it's not even particularly hard to hit: > > S0: CREATE TABLE a_table(); > S0: INSERT INTO a_table(); > S0: disconnect > S1: set a breakpoint to just after the dshash_release_lock(), with an > if objid =

Re: shared-memory based stats collector

2021-03-22 Thread Andres Freund
Hi, On 2021-03-22 12:02:39 +0900, Kyotaro Horiguchi wrote: > At Mon, 22 Mar 2021 09:55:59 +0900 (JST), Kyotaro Horiguchi > wrote in > > At Thu, 18 Mar 2021 01:47:20 -0700, Andres Freund > > wrote in > > > Hi, > > > > > > On 2021-03-18 16:56:02 +

Re: shared-memory based stats collector

2021-03-22 Thread Andres Freund
Hi, On 2021-03-22 16:17:37 -0700, Andres Freund wrote: > and to reduce unnecessary diff noise This patch has just tons of stuff like: -/* - * Calculate function call usage and update stat counters. - * Called by the executor after invoking a funct

Re: Replication slot stats misgivings

2021-03-23 Thread Andres Freund
wrote: > > > > > > > > On Sat, Mar 20, 2021 at 3:52 AM Andres Freund > > > > wrote: > > > > > > > > > > - If max_replication_slots was lowered between a restart, > > > > > pgstat_read_statfile() wi

Re: make the stats collector shutdown without writing the statsfiles if the immediate shutdown is requested.

2021-03-23 Thread Andres Freund
ed before > PMSIGNAL_BEGIN_HOT_STANDBY is sent. FWIW, in the shared memory stats patch the stats subsystem is initialized early on by the startup process. Greetings, Andres Freund

Re: Add client connection check during the execution of the query

2021-03-23 Thread Andres Freund
ghed out of the room? > (I tried to make it work on Windows too by reading the manual, no idea > if that part compiles or works). If the patch had tests, cfbot might tell you :) Greetings, Andres Freund

wal stats questions

2021-03-24 Thread Andres Freund
uot; check at the top. It's probably rare to have pending wal stats without also passing one of the other conditions, but ... Generally the various patches seems to to have a lot of the boilerplate style comments (like "Prepare and send the message"), but very little in the way of explaining the design. Greetings, Andres Freund

Re: make the stats collector shutdown without writing the statsfiles if the immediate shutdown is requested.

2021-03-24 Thread Andres Freund
at next > + * startup after emergency termination. Normally there won't be any stats to remove, no? The permanent stats file has been removed when the stats collector started up. Greetings, Andres Freund

Re: wal stats questions

2021-03-24 Thread Andres Freund
Hi, On 2021-03-25 10:51:56 +0900, Masahiro Ikeda wrote: > On 2021/03/25 8:22, Andres Freund wrote: > > 1) What is the motivation to have both prevWalUsage and pgWalUsage, > >instead of just accumulating the stats since the last submission? > >There doesn

Re: shared memory stats: high level design decisions: consistency, dropping

2021-03-25 Thread Andres Freund
Hi, On 2021-03-24 14:42:11 +0100, Magnus Hagander wrote: > On Sun, Mar 21, 2021 at 11:34 PM Andres Freund wrote: > > > If I understand what you are proposing, all stats views would become > > > completely volatile, without even within-query consistency. That really > &g

Re: shared memory stats: high level design decisions: consistency, dropping

2021-03-25 Thread Andres Freund
Hi, On 2021-03-21 14:53:50 -0700, Andres Freund wrote: > On 2021-03-21 11:41:30 -0400, Stephen Frost wrote: > > > 2) How to remove stats of dropped objects? > > > > > > [...] > > > > The current approach sounds pretty terrible and propagating that forwa

Re: Replication slot stats misgivings

2021-03-25 Thread Andres Freund
out of space (that can probably happen only after > restart and when we lost some drop messages)? I suggest we wait doing anything about this until we know if the shared stats patch gets in or not (I'd give it 50% maybe). If it does get in things get a good bit easier, because we don't have to deal with the message loss issues anymore. Greetings, Andres Freund

Re: [PATCH] add concurrent_abort callback for output plugin

2021-03-25 Thread Andres Freund
that it is allowed to stream > remaining data. */ > ReorderBufferResetTXN(rb, txn, snapshot_now, > command_id, > prev_lsn, I don't think this would be ok, errors thrown in the callback wouldn't be handled as they would be in other callbacks. Greetings, Andres Freund

Re: wal stats questions

2021-03-26 Thread Andres Freund
hing, isn't that much compared to all the other stuff in there. The branch to check if wal stats timing etc is enabled is much more expensive. I think we should just set a boolean to true and leave it at that. Greetings, Andres Freund

Re: make the stats collector shutdown without writing the statsfiles if the immediate shutdown is requested.

2021-03-26 Thread Andres Freund
tats collector _exit(1). It's that that you added a function that sounded generic, but should basically not be used anywhere (we have very few non-shmem connected processes left - I don't think that number will increase). Greetings, Andres Freund

Re: make the stats collector shutdown without writing the statsfiles if the immediate shutdown is requested.

2021-03-26 Thread Andres Freund
o work with after the next start. After this, not so much? Greetings, Andres Freund

Re: Walsender may fail to send wal to the end.

2021-03-26 Thread Andres Freund
But it's > a great help to save the case of just 1 segment behind. -1. This seems like a bandaid to make a broken configuration work a tiny bit better, without actually being meaningfully better. Greetings, Andres Freund

Re: Getting better results from valgrind leak tracking

2021-03-29 Thread Andres Freund
itself should already be fixed, and was more my fault... commit 415ffdc2205e209b6a73fb42a3fdd6e57e16c7b2 Author: Tom Lane Date: 2021-03-18 20:50:56 -0400 Don't run RelationInitTableAccessMethod in a long-lived context. Greetings, Andres Freund

Re: Replication slot stats misgivings

2021-03-29 Thread Andres Freund
Hi, On 2021-03-26 07:58:58 +0530, Amit Kapila wrote: > On Fri, Mar 26, 2021 at 1:17 AM Andres Freund wrote: > > I suggest we wait doing anything about this until we know if the shared > > stats patch gets in or not (I'd give it 50% maybe). If it does get in > > th

Re: Replication slot stats misgivings

2021-03-29 Thread Andres Freund
Hi, On 2021-03-30 10:13:29 +0530, vignesh C wrote: > On Tue, Mar 30, 2021 at 6:28 AM Andres Freund wrote: > > Any chance you could write a tap test exercising a few of these cases? > > I can try to write a patch for this if nobody objects. Cool! > > E.g. things like: &

Re: row filtering for logical replication

2021-03-31 Thread Andres Freund
ExprState *exprstate; > + > + /* Prepare for expression execution */ > + exprstate = pgoutput_row_filter_prepare_expr(rfnode, estate); > + > + /* Evaluates row filter */ > + result = pgoutput_row_filter_exec_expr(exprstate, ecxt); Also, this still seems like an *extremely* expensive thing to do for each tuple. It'll often be *vastly* faster to just send the data than to the other side. This just cannot be done once per tuple. It has to be cached. I don't see how these issues can be addressed in the next 7 days, therefore I think this unfortunately needs to be marked as returned with feedback. Greetings, Andres Freund

Re: Crash in record_type_typmod_compare

2021-03-31 Thread Andres Freund
that cannot ever be found via HASH_FIND/HASH_REMOVE, because record_type_typmod_compare() returns false once there's a NULL in there. Greetings, Andres Freund

Re: Crash in record_type_typmod_compare

2021-03-31 Thread Andres Freund
LL); > recentry->tupdesc = entDesc; ISTM that the ensure_record_cache_typmod_slot_exists() should be moved to before find_or_make_matching_shared_tupledesc / CreateTupleDescCopy. Otherwise we can leak if the CreateTupleDescCopy() succeeds but ensure_record_cache_typmod_slot_exists() fails. Conversely

Re: What to call an executor node which lazily caches tuples in a hash table?

2021-03-31 Thread Andres Freund
arams are the key, rather than configuring the cache... I don't like "result cache" all that much, because it does sound like we'd be caching query results etc, or that it might be referring to Result nodes. Neither of which is the case... Greetings, Andres Freund

Process initialization labyrinth

2021-04-01 Thread Andres Freund
ust pointlessly confusing. Note that bootstrap itself does *not* go through AuxiliaryProcessMain(), and thus has yet another set of initialization needs. Greetings, Andres Freund

Re: shared-memory based stats collector

2021-04-01 Thread Andres Freund
stuff isn't quite right in my branch > > Ah, yeah. As I mentioned above I think it should be in the unified > stats and should have a special means of shotcut. And the global > stats also should be the same. The problem is that I use indexes for addressing, but that they can change between restarts. I think we can fix that fairly easily, by mapping names to indices once, pgstat_restore_stats(). At the point we call pgstat_restore_stats() StartupReplicationSlots() already was executed, so we can just inquire at that point... Greetings, Andres Freund

Re: Process initialization labyrinth

2021-04-02 Thread Andres Freund
elp here? > https://www.postgresql.org/message-id/flat/CAMN686FE0OdZKp9YPO=htC6LnA6aW4r-+jq=3q5raofqgw8...@mail.gmail.com I think parts of it could help, at least. It doesn't really do much about centralizing / de-mazing the actual initialization of sub-systems, but it'd make it a bit easier

Re: shared-memory based stats collector

2021-04-02 Thread Andres Freund
Hi, On 2021-04-02 15:34:54 +0900, Kyotaro Horiguchi wrote: > At Thu, 1 Apr 2021 19:44:25 -0700, Andres Freund wrote > in > > Hi, > > > > I spent quite a bit more time working on the patch. There's are large > > changes: > > > > - postmaster/pgs

Making wait events a bit more efficient

2021-04-02 Thread Andres Freund
rent usage, 1-byte is * reserved for future usage. * * NB: this *must* be able to survive being called before MyProc has been * initialized. * -- */ I.e. I'd like to remove the - framing, the repetition of the function name, and the varying indentation in the comment. Gre

Re: Making wait events a bit more efficient

2021-04-02 Thread Andres Freund
make the store volatile, nothing else. I think it's much better to annotate that we want individual stores to happen regardless of compiler optimizations, rather than all interactions with a variable. Greetings, Andres Freund

Re: Making wait events a bit more efficient

2021-04-02 Thread Andres Freund
to &MyProc->wait_event_info. Greetings, Andres Freund

Re: Making wait events a bit more efficient

2021-04-02 Thread Andres Freund
Hi, On 2021-04-02 12:44:58 -0700, Andres Freund wrote: > If we went for the my_wait_event_info approach there is one further > advantage, after my change to move the wait event code into a separate > file: wait_event.h does not need to include proc.h anymore, which seems > architec

Re: shared-memory based stats collector

2021-04-02 Thread Andres Freund
Hi, On 2021-03-16 12:54:40 -0700, Andres Freund wrote: > I did consider command_progress.c too - but that seems confusing because > there's src/include/commands/progress.h, which is imo a different layer > than what pgstat/backend_progress provide. So I thought splitting thing

Re: Making wait events a bit more efficient

2021-04-03 Thread Andres Freund
On 2021-04-02 19:55:16 -0700, Andres Freund wrote: > On 2021-04-02 12:44:58 -0700, Andres Freund wrote: > > If we went for the my_wait_event_info approach there is one further > > advantage, after my change to move the wait event code into a separate > > file: wait_event.h doe

Allowing dsm allocations in single user mode

2021-04-03 Thread Andres Freund
of Assert(IsUnderPostmaster || !IsPostmasterEnvironment); Greetings, Andres Freund

Re: shared memory stats: high level design decisions: consistency, dropping

2021-04-05 Thread Andres Freund
great > > to hook a reset into. > > > > So I'm inclined to instead have 'access', which caches individual stats > > object on first access. But lasts longer than the query. I went with stats_fetch_consistency = {snapshot, cache, none}, that seems a bit more obvious than cache. I haven't yet changed [auto]vacuum so it uses 'none' unconditionally - but that shouldn't be hard. Greetings, Andres Freund

Re: New IndexAM API controlling index vacuum strategies

2021-04-05 Thread Andres Freund
ith "appear to work"? Isn't, in 13, the only consequence of vac_strategy not being "propagated" that we'll not use a strategy in parallel workers? Presumably that was hard to notice because most people don't run manual VACUUM with cost limits turned on. And autovacuum doesn't use parallelism. Greetings, Andres Freund

Re: New IndexAM API controlling index vacuum strategies

2021-04-05 Thread Andres Freund
I first suspected. Feels like a v13 bug to me, one that should be fixed. Greetings, Andres Freund

subtransaction performance regression [kind of] due to snapshot caching

2021-04-05 Thread Andres Freund
I don't think that's why the issue exists, but I very much hate the XidCache* name. Makes it sound much less important than it is... Greetings, Andres Freund [1] https://postgr.es/m/20210317055718.v6qs3ltzrformqoa%40alap3.anarazel.de >From 1e1259df48373934be43049d8aa4dad328e96b96 Mon Sep 17

Re: subtransaction performance regression [kind of] due to snapshot caching

2021-04-05 Thread Andres Freund
Hi, On 2021-04-06 00:47:13 -0400, Tom Lane wrote: > Andres Freund writes: > > The time in 14 is spent mostly below: > > - 94.58% 0.01% postgres postgres[.] CreateFunction > >- 94.57% CreateFunction > > - 94.49% Procedure

Re: subtransaction performance regression [kind of] due to snapshot caching

2021-04-05 Thread Andres Freund
On 2021-04-06 01:34:02 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2021-04-06 00:47:13 -0400, Tom Lane wrote: > >> Not wanting to distract from your point about xactCompletionCount, > >> but ... I wonder if we could get away with defining "isObjectPin

Re: subtransaction performance regression [kind of] due to snapshot caching

2021-04-06 Thread Andres Freund
Hi, On 2021-04-05 21:35:21 -0700, Andres Freund wrote: > See the attached fix. I did include a test that verifies that the > kill_prior_tuples optimization actually prevents the index from growing, > when subtransactions are involved. I think it should be stable, even > with concurr

Re: shared-memory based stats collector

2021-04-06 Thread Andres Freund
Hi, On 2021-04-05 02:29:14 -0700, Andres Freund wrote: > I've spent most of the last 2 1/2 weeks on this now. Unfortunately I think > that, while it has gotten a lot closer, it's still about a week's worth of > work away from being committable. > > My main conc

Re: Minimal logical decoding on standbys

2021-04-06 Thread Andres Freund
ive from primary. Conflicting slots > would be handled in next commits. > > Andres Freund and Amit Khandekar. I think more people have worked on this by now... Does this strike you as an accurate description? Author: Andres Freund (in an older version), Amit Khandekar, Bertrand Drouvot

Re: Remove page-read callback from XLogReaderState.

2021-04-06 Thread Andres Freund
ate, void *priv) > { > - XLogDumpPrivate *private = state->private_data; > + XLogRecPtr targetPagePtr = state->readPagePtr; > + int reqLen= state->readLen; > + char *readBuff = state->readBuf; > + XLogDumpPrivate *private = (XLogDumpPrivate *) priv; It seems weird to pass a void *priv to a function that now doesn't at all need the type punning anymore. Greetings, Andres Freund

Re: Autovacuum on partitioned table (autoanalyze)

2021-04-06 Thread Andres Freund
goes in (not 14, unfortunately). The hardest part will be to avoid exploding the number of interface functions, but I think we can figure out a way to deal with that. Greetings, Andres Freund

Re: ModifyTable overheads in generic plans

2021-04-06 Thread Andres Freund
gt; 108672 > 100 2753 > 1000 314 > > and after the two patches I just pushed, it looks like: > > 0 12105 > 109928 > 100 5433 > 1000 938 > > So while there's certainly work left to do, that's not bad for > some low-hanging-fruit grabbing. Nice. 3x at the upper end is pretty good. Greetings, Andres Freund

Re: Minimal logical decoding on standbys

2021-04-07 Thread Andres Freund
# This issue doesn't exist in the patch. There's also no test for a recovery conflict due to row removal. Despite that being a substantial part of the patchset. I'm tempted to throw out 024 - all of its tests seem fragile and prove little. And then add a few more tests to 025 (and renaming it). Greetings, Andres Freund

Re: Minimal logical decoding on standbys

2021-04-07 Thread Andres Freund
Hi, On 2021-04-07 10:09:54 -0700, Andres Freund wrote: > There's also no test for a recovery conflict due to row removal. Despite > that being a substantial part of the patchset. Another aspect that wasn't tested *at all*: Whether logical decoding actually produces useful and

Race condition in InvalidateObsoleteReplicationSlots()

2021-04-07 Thread Andres Freund
needs to prepare a condition variable, signal the owner and then wait on the condition variable, to restart after. Greetings, Andres Freund

Re: Why is specifying oids = false multiple times in create table is silently ignored?

2021-04-07 Thread Andres Freund
is, per the same > argument of upthread that this could just break stuff for free, and > that's not a maintenance burden either. Agreed. Given that this case didn't error out before the OIDs removal, it seems like it'd be really strange to make it error out in the compat code... Greetings, Andres Freund

Re: Race condition in InvalidateObsoleteReplicationSlots()

2021-04-07 Thread Andres Freund
Hi, On 2021-04-07 17:10:37 -0700, Andres Freund wrote: > I think this can be solved in two different ways: > > 1) Hold ReplicationSlotAllocationLock with LW_SHARED across most of >InvalidateObsoleteReplicationSlots(). That way nobody could re-create a new >slot in the

Re: Minimal logical decoding on standbys

2021-04-07 Thread Andres Freund
Hi, On 2021-04-07 13:32:18 -0700, Andres Freund wrote: > While working on this I found a, somewhat substantial, issue: > > When the primary is idle, on the standby logical decoding via walsender > will typically not process the records until further WAL writes come in > from the pr

Re: SQL-standard function body

2021-04-07 Thread Andres Freund
or might not) have. Independent of this patch, it might be a good idea to have ExecInitParallelPlan() be robust against NULL querystrings. Places like executor_errposition() are certainly trying to be... Greetings, Andres Freund

Re: SQL-standard function body

2021-04-07 Thread Andres Freund
CT pg_stat_statements_reset() | 1 |1 Too tired to figure out why the BF doesn't see this. Perhaps the extra settings aren't used because it's scripted as an install check? Greetings, Andres Freund

Re: SQL-standard function body

2021-04-07 Thread Andres Freund
Hi, On 2021-04-08 01:41:40 -0400, Tom Lane wrote: > Andres Freund writes: > > Independent of this patch, it might be a good idea to have > > ExecInitParallelPlan() be robust against NULL querystrings. Places like > > executor_errposition() are certainly trying to be... &

test runner (was Re: SQL-standard function body)

2021-04-08 Thread Andres Freund
ual tests: ~/src/meson/meson.py test recovery/t/008_fsm_truncation.pl Obviously all very far from being ready, but this seemed like a good enough excuse to mention it ;) Greetings, Andres Freund

Re: test runner (was Re: SQL-standard function body)

2021-04-08 Thread Andres Freund
On 2021-04-08 10:50:39 -0700, Andres Freund wrote: > It's hard to convey just how much nicer it is to see a progress report > during the test, see the failing tests at the end, without needing to > wade through reams of log output. The output of the individual tests is > in testl

Re: test runner (was Re: SQL-standard function body)

2021-04-08 Thread Andres Freund
Hi, On 2021-04-09 08:39:46 +0900, Michael Paquier wrote: > On Thu, Apr 08, 2021 at 10:50:39AM -0700, Andres Freund wrote: > > Obviously all very far from being ready, but this seemed like a good > > enough excuse to mention it ;) > > This is nice. Are there any paralleli

Re: PANIC: wrong buffer passed to visibilitymap_clear

2021-04-09 Thread Andres Freund
cleared_new = true; PageClearAllVisible(BufferGetPage(newbuf)); visibilitymap_clear(relation, BufferGetBlockNumber(newbuf), vmbuffer_new, VISIBILITYMAP_VALID_BITS); } Greetings, Andres Freund

Re: PANIC: wrong buffer passed to visibilitymap_clear

2021-04-09 Thread Andres Freund
= {si_addr = 0x3e95e6c, _addr_lsb = 0, _addr_bnd = {_lower = 0x0, _upper = 0x0}}, _sigpoll = {si_band = 4299262287468, si_fd = 0}}} Greetings, Andres Freund

Re: PANIC: wrong buffer passed to visibilitymap_clear

2021-04-09 Thread Andres Freund
On 2021-04-09 16:27:39 -0700, Andres Freund wrote: > Just looking at the code in heap_update: I'm a bit confused about > RelationGetBufferForTuple()'s vmbuffer and vmbuffer_other > arguments. It looks like it's not at all clear which of the two > arguments will have th

Re: pgsql: autovacuum: handle analyze for partitioned tables

2021-04-09 Thread Andres Freund
thread (unlikely but ==3500389== possible), you can try to increase the size of the ==3500389== main thread stack using the --main-stacksize= flag. ==3500389== The main thread stack size used in this run was 8388608. Greetings, Andres Freund

Re: [PATCH] Identify LWLocks in tracepoints

2021-04-12 Thread Andres Freund
le can use so > they can understand why these probes are useful. -1. This is not scalable. Adding static probes all over has both a runtime (L1I, branches, code optimization) and maintenance overhead. > (Those can also be used with systemtap guru mode scripts to do things > like turn a particular elog(DEBUG) into a PANIC at runtime for > diagnostic purposes). Yikes. Greetings, Andres Freund

Re: PANIC: wrong buffer passed to visibilitymap_clear

2021-04-12 Thread Andres Freund
ot;Also, in a PITR log-shipping or 2PC environment ..."), we don't actually enforce that strictly anyway - so I am not sure why it's necessary to pay the price here. But maybe I'm all wet here, I certainly haven't had enough coffee. Greetings, Andres Freund

Re: Allowing to create LEAKPROOF functions to non-superuser

2021-04-12 Thread Andres Freund
ion can incur. There's certain things that are pretty intrinsically "can execute native code", like defining new 'C' functions, arbitrary ALTER SYSTEM, arbitrary file reads/writes, etc. Splitting them off from superuser is a fools errand. But it's not at all clear why adding LEAKPROOF to functions falls into that category? Greetings, Andres Freund

Re: Allowing to create LEAKPROOF functions to non-superuser

2021-04-12 Thread Andres Freund
Hi, On 2021-04-12 22:42:03 +0200, Tomas Vondra wrote: > It's unfortunate that we tie the this capability to being superuser, > so maybe the right solution would be to introduce a separate role with > this privilege? Perhaps DB owner + BYPASSRLS would be enough? Greetings, Andres Freund

Re: Allowing to create LEAKPROOF functions to non-superuser

2021-04-12 Thread Andres Freund
ting arbitrary native code", I don't think, if implemented sensibly, allowing to set LEAKPROOF on new functions would equate superuser permissions. But you soon after might hit further limitations where lifting them would have such a risk, e.g. defining new types with in/out functions. Greetings, Andres Freund

Re: Allowing to create LEAKPROOF functions to non-superuser

2021-04-12 Thread Andres Freund
s and that many use-cases for BYPASSRLS do not require the target to have the technical chops to determine if a function actually is leakproof. But that seems more an argument for providing a separate control over allowing to specify LEAKPROOF than against separating it from superuser. Greetings, Andres Freund

Re: Teaching users how they can get the most out of HOT in Postgres 14

2021-04-12 Thread Andres Freund
t;optimization", which scares me. Greetings, Andres Freund

Re: Teaching users how they can get the most out of HOT in Postgres 14

2021-04-12 Thread Andres Freund
is somewhat scary. That said, adding more and more reloptions has a significant cost, so I don't think it's clear cut that it's the right decision to add one. Perhaps vacuum_cleanup_index_scale_factor should just be reused for BYPASS_THRESHOLD_PAGES? Greetings, Andres Freund

Re: [PATCH] Identify LWLocks in tracepoints

2021-04-12 Thread Andres Freund
[Tracepoint event] > ... > sched:sched_wakeup [Tracepoint event] > > The kernel is packed with extremely useful trace events, and for very > good reasons. Some on very hot paths. IIRC those aren't really comparable - the kernel actually does modify the executable code to replace the tracepoints with nops. Greetings, Andres Freund

<    2   3   4   5   6   7   8   9   10   11   >