Re: Add statistics to pg_stat_wal view for wal related parameter tuning

2020-12-21 Thread Andres Freund
Hi, On 2020-12-21 13:16:50 -0800, Andres Freund wrote: > On 2020-12-02 13:52:43 +0900, Fujii Masao wrote: > > Pushed. Thanks! > > Why are wal_records/fpi long, instead of uint64? > longwal_records;/* # of WAL records produced */ > lo

Pre-allocating WAL files

2020-12-25 Thread Andres Freund
egment boundary. With a plain pgbench run I see a 2.5x reduction in throughput in the periods where we initialize WAL files. Greetings, Andres Freund

Re: Logical decoding without slots: decoding in lockstep with recovery

2020-12-25 Thread Andres Freund
rovide some of the infrastructure to do this properly. Should really commit it. /me adds an entry to the top of the todo list. Greetings, Andres Freund

Re: Better client reporting for "immediate stop" shutdowns

2020-12-25 Thread Andres Freund
s of repeated "terminating connection..." lines that add absolutely no additional information, just because I am shutting down the server. Similarly, trying to find the reason for a PANIC is often hard due to all the other messages. Greetings, Andres Freund

Re: Better client reporting for "immediate stop" shutdowns

2020-12-26 Thread Andres Freund
IGQUIT came > from the postmaster, and then it might be all right to suppress the > log entry altogether when it did. > > I'd be happy to write up a patch for either of these, but let's > decide what we want first. My vote would be #2, with the same reasoning as yours. Greetings, Andres Freund

Re: Wrong comment in tuptable.h

2020-12-26 Thread Andres Freund
); > > But acquire_sample_rows() calls ExecCopySlotHeapTuple(), and then > subsequently sorts the rows by TID. Is acquire_sample_rows() doing > something it shouldn't, or is the comment mistaken? I think the comment is too vague and thinking of system columns as xmin/xmax/cmin/cmax. Greetings, Andres Freund

Re: Better client reporting for "immediate stop" shutdowns

2020-12-28 Thread Andres Freund
ht now, but I again want to suggest it'd be nice if we split log levels into a bitmask. If we bits, separate from the log level, for do-not-log-to-client and do-not-log-to-server some of this code would imo look nicer. Looks good to me. Greetings, Andres Freund

Re: Lazy JIT IR code generation to increase JIT speed with partitions

2020-12-29 Thread Andres Freund
immediately emit function, instead do so the first time the > - * expression is actually evaluated. That allows to emit a lot of > - * functions together, avoiding a lot of repeated llvm and memory > - * remapping overhead. > - */ Greetings, Andres Freund

Re: [PATCH] LWLock self-deadlock detection

2020-12-29 Thread Andres Freund
at did that this week [1]. > You usually find that mistake very quickly when you start testing though, I > don't think I've seen it happen in production. I think something roughly along Craig's original patch, basically adding assert checks against holding a lock already, ma

lwlock optimization opportunities

2020-12-29 Thread Andres Freund
Hi, This started as a reply to https://postgr.es/m/CAH2-WzkbCvgKrmw%2Bf%2B1hwgXhmiv%2BUNRihotALXftUiNr%3D3VUKA%40mail.gmail.com but after typing for a while I decided that it's large and unrelated enough to be better handled as a new thread. On 2020-11-27 11:08:49 -0800, Peter Geoghegan wrote: >

Re: WIP: WAL prefetch (another approach)

2020-12-29 Thread Andres Freund
; + * Like pg_atomic_unlocked_write_u32, guarantees only that partial values > + * cannot be observed. Maybe add a note saying that that in particularly means that modifications could be lost when used concurrently? Greetings, Andres Freund

Re: Cache relation sizes?

2020-12-29 Thread Andres Freund
to scan (which > doesn't need to see blocks added after that) or an exclusive heavyweight > lock to extend, truncate or drop. XXX right? "free enough"? I am not sure this holds for non-mvcc scans? > @@ -699,7 +748,7 @@ smgrclearowner(SMgrRelation *owner, SMgrRelation reln) > bool > smgrexists(SMgrRelation reln, ForkNumber forknum) > { > - if (smgrnblocks_shared(reln, forknum) != InvalidBlockNumber) > + if (smgrnblocks_fast(reln, forknum) != InvalidBlockNumber) > return true; ISTM that all these places referencing _fast is leaking an implementation detail (but not very far) - seems like that should be an implementation detail for smgrnblocks() which can then call out to _fast/_shared or such. Greetings, Andres Freund

crash recovery vs partially written WAL

2020-12-30 Thread Andres Freund
a "correct" xlp_pageaddr, but that may reading a lot of WAL due to WAL file recycling. I hope I am missing some crosscheck making this a non-issue? Greetings, Andres Freund

Re: pg_waldump/heapdesc.c and struct field names

2021-01-04 Thread Andres Freund
nally mildly prefer remxid - anything that is understandable but shortens the line length is good for my uses of waldump. Greetings, Andres Freund

logical replication worker accesses catalogs in error context callback

2021-01-05 Thread Andres Freund
e: 2018-03-14 21:34:26 -0300 logical replication: fix OID type mapping mechanism Greetings, Andres Freund

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-05 Thread Andres Freund
l be quite larger. Also, the vacuum > runtime is still 15x longer. That's obviously an issue. > So maybe we should switch on wal_compression if we enable data checksums > by default. It unfortunately also hurts other workloads. If we moved towards a saner compression algorithm that'd perhaps not be an issue anymore... Greetings, Andres Freund

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-06 Thread Andres Freund
Hi, On 2021-01-06 12:02:40 -0500, Stephen Frost wrote: > * Andres Freund (and...@anarazel.de) wrote: > > On 2021-01-04 19:11:43 +0100, Michael Banck wrote: > > > Am Samstag, den 02.01.2021, 10:47 -0500 schrieb Stephen Frost: > > > > I agree with this, but I'd

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-06 Thread Andres Freund
x27;ll not be an option on most cloud providers... Greetings, Andres Freund

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-06 Thread Andres Freund
Hi, On 2021-01-06 13:01:59 -0500, Stephen Frost wrote: > * Andres Freund (and...@anarazel.de) wrote: > > > imv, enabling page checksums is akin to having fsync enabled by default. > > > Does it impact performance? Yes, surely quite a lot, but it's also the > >

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-08 Thread Andres Freund
Hi, On Fri, Jan 8, 2021, at 01:53, Laurenz Albe wrote: > On Thu, 2021-01-07 at 16:14 -0500, Stephen Frost wrote: > > I expected there'd be some disagreement on this, but I do continue to > > feel that it's sensible to enable checksums by default. > > +1 I don't disagree with this in principle, b

Re: WIP: WAL prefetch (another approach)

2021-02-11 Thread Andres Freund
Hi, On 2021-02-12 00:42:04 +0100, Tomas Vondra wrote: > Yeah, that's a good point. I think it'd make sense to keep track of recent > FPIs and skip prefetching such blocks. But how exactly should we implement > that, how many blocks do we need to track? If you get an FPI, how long > should we skip

Re: partial heap only tuples

2021-02-13 Thread Andres Freund
unds like it might be prohibitively painful. Adding effectively unremovable bloat to remove other bloat is not an uncomplicated premise. I think you'd really need a way to fully remove this as part of vacuum for this to be viable. Greetings, Andres Freund

Re: repeated decoding of prepared transactions

2021-02-13 Thread Andres Freund
at point, but a replication client will send the last location they've received and successfully processed, and that has to *guarantee* that they won't receive anything twice, or miss something. Otherwise you've broken the protocol. Greetings, Andres Freund

Re: repeated decoding of prepared transactions

2021-02-13 Thread Andres Freund
ceived the flush (network disconnect, primary crash, ...). Greetings, Andres Freund

Re: PG vs LLVM 12 on seawasp, next round

2021-02-15 Thread Andres Freund
later. You can also do something like LDFLAGS="$LDFLAGS -Wl,-rpath,$(llvm-config --libdir)" or such. Greetings, Andres Freund

Re: ERROR: invalid spinlock number: 0

2021-02-15 Thread Andres Freund
tartup adds a lot of complications, because e.g. somebody could already hold the spinlock because the previous walsender just disconnected, and they were looking at the stats. Greetings, Andres Freund

Re: Snapshot scalability patch issue

2021-02-15 Thread Andres Freund
m back at my computer... Greetings, Andres Freund

Re: Snapbuild woes followup

2021-02-15 Thread Andres Freund
Hi, On 2021-01-29 14:04:47 +0530, Amit Kapila wrote: > On Tue, Jan 26, 2021 at 2:18 AM Andres Freund wrote: > > > > Hi, > > > > On 2021-01-25 12:00:08 -0800, Andres Freund wrote: > > > > > /* > > > > >* For backward compatibilit

Re: Snapshot scalability patch issue

2021-02-15 Thread Andres Freund
Hi, On 2021-02-15 15:08:40 -0800, Andres Freund wrote: > On 2021-02-14 18:42:18 -0800, Peter Geoghegan wrote: > > The call to heap_page_prune() within lazy_scan_heap() passes a bool > > literal ('false') as its fourth argument. But the fourth argument is > > of typ

Re: Tid scan improvements

2021-02-16 Thread Andres Freund
> + TidOpExpr *tidopexpr; > + > + if (!IsA(opexpr, OpExpr)) > + elog(ERROR, "could not identify CTID expression"); > + > + tidopexpr = MakeTidOpExpr(opexpr, tidrangestate); > + tidexprs = lappend(tidexprs, tidopexpr); > + } > + > + tidrangestate->trss_tidexprs = tidexprs; > +} Architecturally it feels like this is something that really belongs more into plan time? > +/* > + * table_set_tidrange resets the minimum and maximum TID range to scan for a > + * TableScanDesc created by table_beginscan_tidrange. > + */ > +static inline void > +table_set_tidrange(TableScanDesc sscan, ItemPointer mintid, > +ItemPointer maxtid) > +{ > + /* Ensure table_beginscan_tidrange() was used. */ > + Assert((sscan->rs_flags & SO_TYPE_TIDRANGESCAN) != 0); > + > + sscan->rs_rd->rd_tableam->scan_set_tidrange(sscan, mintid, maxtid); > +} How does this interact with rescans? Greetings, Andres Freund

Re: [Patch] ALTER SYSTEM READ ONLY

2021-02-16 Thread Andres Freund
t in which it'd be useful to do disable ASRO, right? But 4) should make the node a normal standby. Greetings, Andres Freund

Re: A qsort template

2021-02-17 Thread Andres Freund
the crucial spot, and wouldn't be entirely wrong. OTOH, in my AIO branch I see checkpointer doing ~10GB/s, leading to the sort being a measurable portion of the overall time. Greetings, Andres Freund

Re: repeated decoding of prepared transactions

2021-02-20 Thread Andres Freund
Hi, On Fri, Feb 19, 2021, at 19:38, Amit Kapila wrote: > On Fri, Feb 19, 2021 at 8:23 PM Markus Wanner > wrote: > > > > With that line of thinking, the point in time (or in WAL) of the COMMIT > > PREPARED does not matter at all to reason about the decoding of the > > PREPARE operation. Instead,

Re: [PATCH] Present all committed transaction to the output plugin

2021-02-20 Thread Andres Freund
> not all empty transactions (on the decoder side)? I'm wondering the opposite: What's a potential use case for handing "trivially empty" transactions to the output plugin that's worth incurring some cost for everyone? Greetings, Andres Freund

Re: repeated decoding of prepared transactions

2021-02-20 Thread Andres Freund
but what you're proposing is a cost paid during ongoing replication. Greetings, Andres Freund

Re: [PATCH] Present all committed transaction to the output plugin

2021-02-20 Thread Andres Freund
Hi, On 2021-02-20 21:44:30 +0100, Markus Wanner wrote: > On 20.02.21 21:08, Andres Freund wrote: > > It's not free though > > Agreed. It's an additional call to a callback. If it were just a single indirection function call I'd not be bothered. But we need to do

Re: repeated decoding of prepared transactions

2021-02-21 Thread Andres Freund
restart and just send the commit > prepared. So, again that can lead to replica going out of sync. I don't think that an LSN based approach is susceptible to this. And it wouldn't require more memory etc than we'd now. Greetings, Andres Freund

Re: Catalog version access

2021-02-21 Thread Andres Freund
magic bump will also require a dump/restore or pg_upgrade, but won't be detected by just doing this. So perhaps we should instead add a pg_config option showing all the different versions that influence on-disk compatibility? Greetings, Andres Freund

Re: Is a connection max lifetime useful in a connection pool?

2021-02-21 Thread Andres Freund
nefit. Unfortunately we do not yet have effective "pressure" against that. Similarly, if you have an application using prepared statements you can end up with enough prepared statements for that to be a memory usage issue. Additionally, if there's ever a problem with memory leakage

Re: Catalog version access

2021-02-21 Thread Andres Freund
Hi, On 2021-02-21 19:54:01 -0500, Tom Lane wrote: > Vik Fearing writes: > > On 2/22/21 12:48 AM, Andres Freund wrote: > >> Seems roughly reasonable. Although I wonder if we rather should make it > >> something more generic than just catversion? E.g. a wal page magic bu

Re: Catalog version access

2021-02-21 Thread Andres Freund
Hi, On 2021-02-21 20:53:52 -0500, Tom Lane wrote: > Andres Freund writes: > >> If we're going to bother with providing a way > >> to get this info, we should make it possible to ask the running server. > > > In Vik's case there is no running server to

Re: Finding cause of test fails on the cfbot site

2021-02-21 Thread Andres Freund
n the number of these that just boil down to a variant of check: $(pg_regress_check) $(prove_check) installcheck: $(pg_regress_installcheck) it seems we should lift the REGRESS and TAP_TESTS specific logic in pgxs.mk up into src/Makefiles.global. Which then would make something list a global listchecks target easy. Greetings, Andres Freund

Re: repeated decoding of prepared transactions

2021-02-21 Thread Andres Freund
Hi, On 2021-02-22 08:22:35 +0530, Amit Kapila wrote: > On Mon, Feb 22, 2021 at 3:56 AM Andres Freund wrote: > > > > On 2021-02-21 11:32:29 +0530, Amit Kapila wrote: > > > Here, I am assuming you are asking to disable 2PC both via > > > apply-worker and tablesync

Re: repeated decoding of prepared transactions

2021-02-21 Thread Andres Freund
;t think that can be made work without disproportionate complexity. Especially not in cases where we start to be CONSISTENT based on pre-existing on-disk snapshots. Greetings, Andres Freund

Re: repeated decoding of prepared transactions

2021-02-22 Thread Andres Freund
Hi, On 2021-02-22 09:25:52 +0100, Markus Wanner wrote: > What issues do you see with the approach I proposed? Very significant increase in complexity for initializing a logical replica, because there's no easy way to just use the initial slot. - Andres

Re: repeated decoding of prepared transactions

2021-02-22 Thread Andres Freund
Hi, On 2021-02-22 14:29:09 +0530, Amit Kapila wrote: > On Mon, Feb 22, 2021 at 9:39 AM Andres Freund wrote: > > What I am proposing is to instead track the point at which the slot > > gained consistency - a simple LSN. That way you can change the above > > logic to inste

Re: libpq compression

2021-02-22 Thread Andres Freund
compress would be far more appealing than one that > causes a 30% overhead in such cases (which seems to be where we are > now). I personally don't think it's worth caring about pgbench. But there's obviously also other cases where "stateful" compression could be better... Greetings, Andres Freund

Re: Parser Hook

2021-02-22 Thread Andres Freund
don't think just adding the hook is a step worth doing on its own. Imo a functional approach would really need to do the work to allow to extend & reuse the parser in a piecemeal fashion and *then* add a hook. Greetings, Andres Freund

Re: libpq compression

2021-02-22 Thread Andres Freund
ttacker controls the data the client requests, but I don't think it's a comparable risk. Not a cryptographer by any means though. I think the majority of the cases where one would want to use libpq compression are pretty much unaffected by such attacks, even on a theoretical level. And for the remainder I suspect that we'd already be vulnerable, due to to wal_compression. Greetings, Andres Freund

locking [user] catalog tables vs 2pc vs logical rep

2021-02-22 Thread Andres Freund
ed to read pg_class to log in... And I can't remember this being reported to be a problem? Perhaps all that we need to do is to disallow 2PC prepare if [user] catalog tables have been locked exclusively? Similar to how we're disallowing preparing tables with temp table access. Greetings, Andres Freund

Re: Table AM and DDLs

2021-02-22 Thread Andres Freund
but in the file referenced by pg_class.relfilenode. I don't think anybody would object on principle to making the necessary infrastructure changes to support storing data elsewhere - but I think it'll also not quite as simple as the change you suggested :(. Greetings, Andres Freund

Re: Fallback table AM for relkinds without storage

2021-02-22 Thread Andres Freund
7;d vote for not naming it no_storage - to me that sounds like a name you'd give "blackhole_am". This concept kinda reminds me of pseudotypes - so maybe we should just name it pseudo_am.c or such? Greetings, Andres Freund

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2021-02-22 Thread Andres Freund
which do not inherit from Expr), we'd instead need to pass a new param to preprocess_expression() that stores the volatility somewhere in PlannerInfo? Seems a bit yucky to manage :(. Greetings, Andres Freund

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2021-02-22 Thread Andres Freund
Hi, On 2021-02-22 20:51:17 -0500, Tom Lane wrote: > Andres Freund writes: > > Except for the annoying issue that that we pervasively use Lists as > > expressions, I'd argue that we should actually cache "subtree > > volatility" in Expr nodes, similar to th

Re: locking [user] catalog tables vs 2pc vs logical rep

2021-02-22 Thread Andres Freund
Hi, On 2021-02-23 08:56:39 +0530, Amit Kapila wrote: > On Tue, Feb 23, 2021 at 3:58 AM Andres Freund wrote: > > Perhaps all that we need to do is to disallow 2PC prepare if [user] > > catalog tables have been locked exclusively? > Right, and we have discussed this during deve

Re: locking [user] catalog tables vs 2pc vs logical rep

2021-02-22 Thread Andres Freund
ed. There's plenty users of logical decoding that aren't going through the normal replication mechanism - so they can hit this. So I think it needs to be documented somewhere. Greetings, Andres Freund

Asynchronous and "direct" IO support for PostgreSQL.

2021-02-23 Thread Andres Freund
Hi, over the last ~year I spent a lot of time trying to figure out how we could add AIO (asynchronous IO) and DIO (direct IO) support to postgres. While there's still a *lot* of open questions, I think I now have a decent handle on most of the bigger architectural questions. Thus this long email.

Re: Some regular-expression performance hacking

2021-02-23 Thread Andres Freund
’ 2920 | bool hasmatch[DUPINF + 1]; |^~~~ Greetings, Andres Freund

Re: Some regular-expression performance hacking

2021-02-23 Thread Andres Freund
arning. I have been wondering about making Asserts do something along those lines - but it'd need to be opt-in, since we clearly have a lot of assertions that would cost too much. Greetings, Andres Freund

Re: Some regular-expression performance hacking

2021-02-23 Thread Andres Freund
On 2021-02-23 13:09:18 -0500, Tom Lane wrote: > Andres Freund writes: > > One of the recent commits have introduce a new warning with gcc 10, when > > building with optimizations: > > Oddly, I see no such warning with Fedora's current compiler, > gcc version 10.2

Re: Asynchronous and "direct" IO support for PostgreSQL.

2021-02-23 Thread Andres Freund
from data file writes. And - except that the current callbacks for that aren't great - it should even allow to split the IO by different relfilenodes etc. I assume pg_stat_aio_backends also can be helpful, e.g. by seeing which backends currently the deep IO queues that cause latency delays in other backends, and which backends do a lot of sequential IO (high executed_total_count / issued_total_count) and which only random... Greetings, Andres Freund

Re: Asynchronous and "direct" IO support for PostgreSQL.

2021-02-24 Thread Andres Freund
d of was planning to go somewhere roughly in the direction you suggest, but took a few shortcuts due to the size of the project. Having the views made it a lot easier to debug / develop, but supporting longer lived stuff wasn't yet crucial. But I agree, we really should have it... Greetings, Andres Freund

Re: Asynchronous and "direct" IO support for PostgreSQL.

2021-02-24 Thread Andres Freund
should have a generic infrastructure > for stats that automatically counts things associated with plan nodes > and automatically bubbles that data up to the per-transaction, > per-backend, per-relation, and pg_stat_statements stats. But that's a > whole other ball of wax :) Heh, yea, let's tackle that separately ;) Greetings, Andres Freund

Re: Asynchronous and "direct" IO support for PostgreSQL.

2021-02-24 Thread Andres Freund
ovacuum to have a low queue depth instead of using the current cost limiting. That way the impact on the overall system is limitted, but it's not slowed down unnecessarily as much. Greetings, Andres Freund

buildfarm windows checks / tap tests on windows

2021-03-01 Thread Andres Freund
make things to be at least somewhat less painful. This makes it even clearer to me that we really need a builtin testrunner that runs tests efficiently *and* debuggable on windows. Greetings, Andres Freund [1] https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=1e6e40447115ca7b4749d7d117b81b016ee5e2c2

Re: buildfarm windows checks / tap tests on windows

2021-03-02 Thread Andres Freund
guid => $^O eq "MSWin32" ? Win32::GuidGen() : 'FAKE', If I change them to accept msys perl then the build ends up working. The reason it'd be nice to accept msys perl is that git for windows bundles that - and that's already installed on most CI projects... Greetings, Andres Freund

Re: buildfarm windows checks / tap tests on windows

2021-03-02 Thread Andres Freund
run almost all the TAP > tests. Thanks! > The exception is the recovery tests, because > 021_row_visibility.pl crashes so badly it brings down the whole > buildfarm client. It still does, even after commit 1e6e40447115ca7b4749d7d117b81b016ee5e2c2 (upstream/master, master) Author: Andre

Re: buildfarm windows checks / tap tests on windows

2021-03-02 Thread Andres Freund
Hi, On 2021-03-02 12:57:57 -0800, Andres Freund wrote: > t/003_recovery_targets.pl 7/9 > # Failed test 'multiple conflicting settings' > # at t/003_recovery_targets.pl line 151. > > # Failed test 'recovery end before target reache

Re: buildfarm windows checks / tap tests on windows

2021-03-02 Thread Andres Freund
Hi, On 2021-03-02 21:20:52 -0500, Andrew Dunstan wrote: > On 3/2/21 8:27 PM, Michael Paquier wrote: > > On Tue, Mar 02, 2021 at 04:54:57PM -0800, Andres Freund wrote: > >> It still does, even after > >> > >> commit 1e6e40447115ca7b4749d7d117b81b016ee5e2c2 (

Re: buildfarm windows checks / tap tests on windows

2021-03-02 Thread Andres Freund
Hi, On 2021-03-02 21:20:11 -0800, Andres Freund wrote: > On 2021-03-02 12:57:57 -0800, Andres Freund wrote: > > t/003_recovery_targets.pl 7/9 > > # Failed test 'multiple conflicting settings' > > # at t/003_recovery_targets.pl line 151. > &g

Re: buildfarm windows checks / tap tests on windows

2021-03-03 Thread Andres Freund
the \q to psql, and then we need to wait for the process to finish? I'll try that, but given that I can't reproduce any problems... I suspect that at least the crash recovery test suffers from exactly the same problem and that we can re-enable it on windows after doign an equivalent change... Greetings, Andres Freund

Re: Fix DROP TABLESPACE on Windows with ProcSignalBarrier?

2021-03-03 Thread Andres Freund
E. I suspect it makes sense to tackle the problem as part of the same commit, but I'm not opposed to splitting it if that makes sense... Greetings, Andres Freund

Re: buildfarm windows checks / tap tests on windows

2021-03-04 Thread Andres Freund
kill() without a subsequent ->finish()? But anyway, that seems like a less critical test... Greetings, Andres Freund

CI/windows docker vs "am a service" autodetection on windows

2021-03-04 Thread Andres Freund
look around to see if there's a simple way to drop the problematic memberships that trigger pgwin32_is_service() - but there seem to be no commandline tools doing so (but there are C APIs). Does anybody have an alternative way of fixing this? Greetings, Andres Freund [1] commit ff30aec759b

Add .log file ending to tap test log files

2021-03-04 Thread Andres Freund
potentially would cause some short-term pain on the buildfarm, but I think it'd improve things longer term. Greetings, Andres Freund

Re: CI/windows docker vs "am a service" autodetection on windows

2021-03-04 Thread Andres Freund
ove the case of pg_ctl/postgres run interactively without breaking a sweat. And there is fstat() too, so if stderr in a service is something distinguishable... Greetings, Andres Freund

Re: CI/windows docker vs "am a service" autodetection on windows

2021-03-04 Thread Andres Freund
was thinking. Gah, I don't really want to know anything about windows, I just want to hack on aio with proper working CI. Greetings, Andres Freund

Re: CI/windows docker vs "am a service" autodetection on windows

2021-03-05 Thread Andres Freund
Hi, On 2021-03-04 12:48:59 -0800, Andres Freund wrote: > On 2021-03-04 21:36:23 +0100, Magnus Hagander wrote: > > > I think that's a good answer for pg_ctl - not so sure about postgres > > > itself, at least once it's up and running. I don't know what lead to a

Re: CI/windows docker vs "am a service" autodetection on windows

2021-03-05 Thread Andres Freund
Hi, On 2021-03-05 10:57:52 -0800, Andres Freund wrote: > On 2021-03-04 12:48:59 -0800, Andres Freund wrote: > > On 2021-03-04 21:36:23 +0100, Magnus Hagander wrote: > > > > I think that's a good answer for pg_ctl - not so sure about postgres > > > > itself

Re: Occasional tablespace.sql failures in check-world -jnn

2021-03-10 Thread Andres Freund
> So, I have spent more time checking the surroundings of this patch, > and finally applied it. Thanks for the review, Peter. Thanks! Greetings, Andres Freund

Re: shared-memory based stats collector

2021-03-10 Thread Andres Freund
*/ This comment in PostmasterStateMachine() is outdated now. Greetings, Andres Freund

Re: shared-memory based stats collector

2021-03-10 Thread Andres Freund
nd decide to free the memory. Then the code below will access already freed / reused memory, no? > @@ -2649,85 +3212,138 @@ pgstat_twophase_postabort(TransactionId xid, uint16 > info, > /* -- > * pgstat_fetch_stat_dbentry() - > * > - * Support function for the SQL-callable pgstat* functions. Returns > - * the collected statistics for one database or NULL. NULL doesn't mean > - * that the database doesn't exist, it is just not yet known by the > - * collector, so the caller is better off to report ZERO instead. > + * Find database stats entry on backends in a palloc'ed memory. > + * > + * The returned entry is stored in static memory so the content is valid > until > + * the next call of the same function for the different database. > * -- > */ > PgStat_StatDBEntry * > pgstat_fetch_stat_dbentry(Oid dbid) > { > - /* > - * If not done for this transaction, read the statistics collector stats > - * file into some hash tables. > - */ > - backend_read_statsfile(); > - > - /* > - * Lookup the requested database; return NULL if not found > - */ > - return (PgStat_StatDBEntry *) hash_search(pgStatDBHash, > - > (void *) &dbid, > - > HASH_FIND, NULL); > + PgStat_StatDBEntry *shent; > + > + /* should be called from backends */ > + Assert(IsUnderPostmaster); Can't these be called from single user mode? Greetings, Andres Freund

Re: Performance degradation of REFRESH MATERIALIZED VIEW

2021-03-11 Thread Andres Freund
sert() since it sets VM buffer after > filling the heap page with tuples. Therefore, there is no such > performance degradation between COPY and COPY FREEZE if they use > heap_multi_insert() (i.g., CIM_MULTI). Paul also reported it in that > thread. Probably worth adding as an open item for 14. Greetings, Andres Freund

Re: A qsort template

2021-03-11 Thread Andres Freund
ially large number of dirty buffers, the > checkpointer can benefit from a faster sort routine. One reported > improvement on a large buffer pool system was 1.4s -> 0.6s. > > Discussion: > https://postgr.es/m/CA%2BhUKGJ2-eaDqAum5bxhpMNhvuJmRDZxB_Tow0n-gse%2BHG0Yig%40mail.gmail.com Looks good to me. Greetings, Andres Freund

Re: shared-memory based stats collector

2021-03-11 Thread Andres Freund
Hi, On 2021-03-10 20:26:56 -0800, Andres Freund wrote: > > +static void > > +attach_shared_stats(void) > > +{ > > + MemoryContext oldcontext; > > > + /* > > +* The first attacher backend may still reading the stats file, or the > > +* la

Re: shared-memory based stats collector

2021-03-12 Thread Andres Freund
Hi, On 2021-03-11 19:22:57 -0800, Andres Freund wrote: > I started changing the patch to address my complaints. I'll try to do > it as an incremental patch ontop of your 0004, but it might become too > unwieldy. Not planning to touch other patches for now (and would be > happy

Re: pg_amcheck contrib application

2021-03-12 Thread Andres Freund
Hi, On 2021-03-13 01:22:54 -0500, Tom Lane wrote: > Mark Dilger writes: > > On Mar 12, 2021, at 10:16 PM, Noah Misch wrote: > >> hoverfly does configure with PERL=perl64. /usr/bin/prove is from the > >> 32-bit > >> Perl, so I suspect the TAP suites get 32-bit Perl that way. (There's no > >> "

Re: shared-memory based stats collector

2021-03-13 Thread Andres Freund
Hi, On 2021-03-13 12:53:30 +0100, Magnus Hagander wrote: > On Sat, Mar 13, 2021 at 7:20 AM Andres Freund wrote: > >I think before making things differently complicated with this patch, > >we need to clean this up, unfortunately. I think we should initially have >

Re: New IndexAM API controlling index vacuum strategies

2021-03-15 Thread Andres Freund
ly_frozen)) > + frozen[nfrozen++].offset = offnum; I'm not comfortable with this change without adding more safety checks. If there's ever a case in which the HEAPTUPLE_DEAD case is hit and the xid needs to be frozen, we'll either cause errors or corruption. Yes, that's already the case with params->index_cleanup == DISABLED, but that's not that widely used. See https://postgr.es/m/20200724165514.dnu5hr4vvgkssf5p%40alap3.anarazel.de for some discussion around the fragility. Greetings, Andres Freund

Re: [HACKERS] Custom compression methods

2021-03-15 Thread Andres Freund
it before returning it, since a compressed lz4 datum is > impossible. That seems fairly terrible. > I'm open to being convinced that we don't need to do either of these > things, and that the cost of iterating over all varlenas in the tuple > is not so bad as to preclude doing things as you have them here. But, > I'm afraid it's going to be too expensive. I mean, I would just define several of those places away by not caring about tuples in a different compressino formation ending up in a table... Greetings, Andres Freund

Re: New IndexAM API controlling index vacuum strategies

2021-03-15 Thread Andres Freund
Hi, On 2021-03-15 12:58:33 -0700, Peter Geoghegan wrote: > On Mon, Mar 15, 2021 at 12:21 PM Andres Freund wrote: > > It's evil sorcery. Fragile sorcery. I think Robert, Tom and me all run > > afoul of edge cases around it in the last few years. > > Right, which is why

Re: New IndexAM API controlling index vacuum strategies

2021-03-15 Thread Andres Freund
cessfully and correctly do the deletion? So there clearly is another option? See my email from a few minutes ago for a somewhat crude idea for how to tackle the issue differently... Greetings, Andres Freund

Re: shared-memory based stats collector

2021-03-15 Thread Andres Freund
Hi, On 2021-03-13 10:05:21 -0800, Andres Freund wrote: > Cool. I'll give it a try. I have a few questions about the patch: - Why was collect_oids() changed to a different hashtable as part of this change? Seems fairly independent? - What's the point of all those cached_* stuff

Re: [HACKERS] Custom compression methods

2021-03-16 Thread Andres Freund
I > insert data into a table, I as a user would expect that the > compression settings for that column are going to be respected. IDK. The user might also expect that INSERT .. SELECT is fast, instead of doing expensive decompression + compression (with pglz the former can be really slow). I think there's a good argument for having an explicit "recompress" operation, but I'm not convincd that doing things implicitly is good, especially if it causes complications in quite a few places. Greetings, Andres Freund

Re: shared-memory based stats collector

2021-03-16 Thread Andres Freund
Hi, On 2021-03-16 15:08:39 -0400, Robert Haas wrote: > On Mon, Mar 15, 2021 at 10:56 PM Andres Freund wrote: > > I did roughly the first steps of the split as I had outlined. I moved: > > > > 1) wait event related functions into utils/activity/wait_event.c / > >

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

2021-03-16 Thread Andres Freund
this > pushed early next weeks, barring opinions from others. It is *very* exciting to see this being merged. Thanks for all the work to all that contributed! Greetings, Andres Freund

Re: Getting better results from valgrind leak tracking

2021-03-16 Thread Andres Freund
52) ==2054558==by 0xA780015: plpgsql_yyparse (pl_gram.c:1945) ==2054558==by 0xA76A652: do_compile (pl_comp.c:788) ==2054558==by 0xA769130: plpgsql_compile (pl_comp.c:224) ==2054558==by 0xA78948F: plpgsql_validator (pl_handler.c:539) Based on the quick look I had (dinner is calling) I didn't yet understand how plpgsql_compile_tmp_cxt error handling works. Greetings, Andres Freund (*) or not so quick, I had to figure out why valgrind was so slow. It turned out that I had typed shared_buffers=32MB into shared_buffers=32GB...

Re: cutting down the TODO list thread

2020-10-27 Thread Andres Freund
e > problems" I think we ought to do something here. Mostly because the current situation makes it impossible to test many things on a single system. And we have a partial solution with the tablespace mapping files. Greetings, Andres Freund

More aggressive vacuuming of unlogged relations?

2020-10-27 Thread Andres Freund
slot based horizons. The obvious question is, is that worth doing? My intuition is that yes, it probably is: Unlogged tables are often used for hotly updated transient state, allowing that to be cleaned up more aggressively will reduce bloat. Comments? Greetings, Andres Freund [1] https://www.

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-10-27 Thread Andres Freund
Hi, On 2020-10-15 01:37:35 -0700, Andres Freund wrote: > Attached is a *prototype* implemention of this concept, which clearly is > lacking some comment work (and is intentionally lacking some > re-indentation). > > I described my thoughts about how to limit the horizons for

<    1   2   3   4   5   6   7   8   9   10   >