Re: Printing backtrace of postgres processes

2020-11-30 Thread Craig Ringer
On Tue, 1 Dec 2020 at 07:04, Tom Lane wrote: > I'd feel better about it if the mechanism had you specify exactly > one target process, and were restricted to a superuser requestor. Er, rather. I actually assumed the former was the case already, not having looked closely yet.

Re: Printing backtrace of postgres processes

2020-11-30 Thread Craig Ringer
On Tue, 1 Dec 2020 at 11:31, Andres Freund wrote: > > Hi, > > On 2020-11-30 13:35:46 +0800, Craig Ringer wrote: > > I find that when I most often want a backtrace of a running, live > > backend, it's because the backend is doing something that isn't > >

TAP test utility module 'PG_LSN.pm'

2020-11-30 Thread Craig Ringer
Hi all I'd like to share the attached PG_LSN.pm module that I use when writing TAP tests. I suggest that it be considered for inclusion in core. It defines a Perl datatype PG_LSN with operator support, so you can write things like cmp_ok($got_lsn, "<", $expected_lsn, "testname") in TAP test

Re: TAP test utility module 'PG_LSN.pm'

2020-11-30 Thread Craig Ringer
On Tue, 1 Dec 2020 at 13:58, Michael Paquier wrote: > > On Tue, Dec 01, 2020 at 12:03:41PM +0800, Craig Ringer wrote: > > I'd like to share the attached PG_LSN.pm module that I use when > > writing TAP tests. I suggest that it be considered for inclusion in > > co

Re: [PATCH] Runtime control of CLOBBER_CACHE_ALWAYS

2020-12-01 Thread Craig Ringer
On Mon, 30 Nov 2020, 20:38 Anastasia Lubennikova, < a.lubennik...@postgrespro.ru> wrote: > On 27.10.2020 11:34, Peter Eisentraut wrote: > > On 2020-09-25 09:40, Craig Ringer wrote: > >> While working on extensions I've often wanted to enable cache > >> c

Re: Two fsync related performance issues?

2020-12-01 Thread Craig Ringer
On Wed, 14 Oct 2020, 13:06 Michael Paquier, wrote: > On Wed, Oct 14, 2020 at 02:48:18PM +1300, Thomas Munro wrote: > > On Wed, Oct 14, 2020 at 12:53 AM Michael Banck > > wrote: > >> One question about this: Did you consider the case of a basebackup being > >> copied/restored somewhere and the re

Re: [PATCH] Runtime control of CLOBBER_CACHE_ALWAYS

2020-12-02 Thread Craig Ringer
On Tue, 27 Oct 2020 at 16:34, Peter Eisentraut < peter.eisentr...@2ndquadrant.com> wrote: > On 2020-09-25 09:40, Craig Ringer wrote: > > While working on extensions I've often wanted to enable cache clobbering > > for a targeted piece of code, without paying the price

Re: Two fsync related performance issues?

2020-12-02 Thread Craig Ringer
On Wed, 2 Dec 2020 at 15:41, Michael Paquier wrote: > On Tue, Dec 01, 2020 at 07:39:30PM +0800, Craig Ringer wrote: > > On Wed, 14 Oct 2020, 13:06 Michael Paquier, wrote: > >> Yeah, it is safer to assume that it is the responsability of the > >> backup tool to ensu

Re: pg_ctl.exe file deleted automatically

2020-12-03 Thread Craig Ringer
On Thu, 3 Dec 2020, 13:06 Joel Mariadasan (jomariad), wrote: > Hi, > > > > We are using Windows 2019 server. > > > > Sometimes we see the *pg_ctl.exe* getting automatically deleted. > > Due to this, while starting up Postgres windows service we are getting the > error. > > “Error 2: The system ca

Re: Single transaction in the tablesync worker?

2020-12-03 Thread Craig Ringer
On Thu, 3 Dec 2020 at 17:25, Amit Kapila wrote: > Is there any fundamental problem if > we commit the transaction after initial copy and slot creation in > LogicalRepSyncTableStart and then allow the apply of transactions as > it happens in apply worker? No fundamental problem. Both approaches a

Re: Single transaction in the tablesync worker?

2020-12-06 Thread Craig Ringer
On Sat, 5 Dec 2020, 10:03 Amit Kapila, wrote: > On Fri, Dec 4, 2020 at 7:12 PM Ashutosh Bapat > wrote: > > > > On Thu, Dec 3, 2020 at 7:24 PM Amit Kapila > wrote: > > > > > > On Thu, Dec 3, 2020 at 7:04 PM Ashutosh Bapat > > > wrote: > > > > > > > > On Thu, Dec 3, 2020 at 2:55 PM Amit Kapila

Re: Logical archiving

2020-12-06 Thread Craig Ringer
Reply follows inline. I addressed your last point first, so it's out of order. On Fri, 4 Dec 2020 at 15:33, Andrey Borodin wrote > If OLAP cannot consume data fast enough - we are out of space due to repl slot. There is a much simpler solution to this than logical PITR. What we should be doing

Re: Logical archiving

2020-12-06 Thread Craig Ringer
Actually CC'd Petr this time. On Mon, 7 Dec 2020 at 11:05, Craig Ringer wrote: > Reply follows inline. I addressed your last point first, so it's out of > order. > > On Fri, 4 Dec 2020 at 15:33, Andrey Borodin wrote > > > If OLAP cannot consume data fast enou

RFC: Giving bgworkers walsender-like grace during shutdown (for logical replication)

2020-12-06 Thread Craig Ringer
Hi folks TL;DR: Anyone object to a new bgworker flag that exempts background workers (such as logical apply workers) from the first round of fast shutdown signals, and instead lets them to finish their currently in-progress txn and exit? This is a change proposal raised for comment before patch s

RFC: Deadlock detector hooks for victim selection and edge injection

2020-12-06 Thread Craig Ringer
Hi folks Now that we're well on track for streaming logical decoding, it's becoming more practical to look at parallel logical apply. The support for this in pglogical3 benefits from a deadlock detector hook. It was added in the optional patched postgres pglogical can use to enable various extra

RFC: Deadlock detector hooks for edge injection

2020-12-06 Thread Craig Ringer
Hi all Related to my other post about deadlock detector hooks for victim selection, I'd like to gauge opinions here about whether it's feasible to inject edges into the deadlock detector's waits-for graph. Doing so would help with detecting deadlocks relating to shm_mq waits, help with implementi

Re: Single transaction in the tablesync worker?

2020-12-06 Thread Craig Ringer
On Mon, 7 Dec 2020 at 11:44, Peter Smith wrote: > > Basically, I was wondering why can't the "tablesync" worker just > gather messages in a similar way to how the current streaming feature > gathers messages into a "changes" file, so that they can be replayed > later. > > See the related thread "

Re: POC: Better infrastructure for automated testing of concurrency issues

2020-12-06 Thread Craig Ringer
On Wed, 25 Nov 2020 at 22:11, Alexander Korotkov wrote: > Hackers, > > PostgreSQL is a complex multi-process system, and we are periodically > faced with complicated concurrency issues. While the postgres community > does a great job on investigating and fixing the problems, our ability to > repr

Blocking I/O, async I/O and io_uring

2020-12-07 Thread Craig Ringer
Hi all A new kernel API called io_uring has recently come to my attention. I assume some of you (Andres?) have been following it for a while. io_uring appears to offer a way to make system calls including reads, writes, fsync()s, and more in a non-blocking, batched and pipelined manner, with or w

Re: Blocking I/O, async I/O and io_uring

2020-12-07 Thread Craig Ringer
References to get things started: * https://lwn.net/Articles/810414/ * https://unixism.net/loti/what_is_io_uring.html * https://blogs.oracle.com/linux/an-introduction-to-the-io_uring-asynchronous-io-framework * https://thenewstack.io/how-io_uring-and-ebpf-will-revolutionize-programming-in-linux/

Re: Blocking I/O, async I/O and io_uring

2020-12-07 Thread Craig Ringer
On Tue, 8 Dec 2020 at 12:02, Andres Freund wrote: > Hi, > > On 2020-12-08 10:55:37 +0800, Craig Ringer wrote: > > A new kernel API called io_uring has recently come to my attention. I > > assume some of you (Andres?) have been following it for a while. > > Yea, I'

Re: Blocking I/O, async I/O and io_uring

2020-12-08 Thread Craig Ringer
On Tue, 8 Dec 2020 at 15:04, Andres Freund wrote: > Hi, > > On 2020-12-08 04:24:44 +, tsunakawa.ta...@fujitsu.com wrote: > > I'm looking forward to this from the async+direct I/O, since the > > throughput of some write-heavy workload decreased by half or more > > during checkpointing (due to

Re: Logical archiving

2020-12-08 Thread Craig Ringer
On Tue, 8 Dec 2020 at 17:54, Andrey Borodin wrote: > > In pglogical3 we already support streaming decoded WAL data to > alternative writer downstreams including RabbitMQ and Kafka via writer > plugins. > Yes, Yandex.Cloud Transfer Manger supports it too. But it has to be > resynced after physica

Re: [PATCH] Runtime control of CLOBBER_CACHE_ALWAYS

2020-12-09 Thread Craig Ringer
On Thu, 3 Dec 2020 at 15:53, Craig Ringer wrote: > > On Tue, 27 Oct 2020 at 16:34, Peter Eisentraut < > peter.eisentr...@2ndquadrant.com> wrote: > >> On 2020-09-25 09:40, Craig Ringer wrote: >> > While working on extensions I've often wanted to enable c

[PATCH] Identify LWLocks in tracepoints

2020-12-18 Thread Craig Ringer
try to address some points I found confusing and hard to understand when investigating this topic. [1] https://www.postgresql.org/message-id/CAGRY4nz%3DSEs3qc1R6xD3max7sg3kS-L81eJk2aLUWSQAeAFJTA%40mail.gmail.com . From 583c818e3121c0f7c6506b434497c81ae94ee9cb Mon Sep 17 00:00:00 2001 From: Craig Ring

Re: Preventing hangups in bgworker start/stop during DB shutdown

2020-12-22 Thread Craig Ringer
On Wed, 23 Dec 2020 at 05:40, Tom Lane wrote: > Here's an attempt at closing the race condition discussed in [1] > (and in some earlier threads, though I'm too lazy to find them). > > The core problem is that the bgworker management APIs were designed > without any thought for exception condition

Logical decoding without slots: decoding in lockstep with recovery

2020-12-22 Thread Craig Ringer
Hi all I want to share an idea I've looked at a few times where I've run into situations where logical slots were inadvertently dropped, or where it became necessary to decode changes in the past on a slot. As most of you will know you can't just create a logical slot in the past. Even if it was

TAP PostgresNode function to gdb stacks and optional cores for all backends

2020-12-22 Thread Craig Ringer
Hi all I recently wrote a utility that adds a $node->gdb_backends() method to PostgresNode instances - figured I'd share it here in case anyone finds it useful, or wants to adopt it into the features of the TAP tools. This function provides a one-line way to dump stacks for all running backends t

Re: Improving LWLock wait events

2020-12-22 Thread Craig Ringer
On Mon, 21 Dec 2020 at 05:27, Andres Freund wrote: > Hi, > > The current wait events are already pretty useful. But I think we could > make them more informative without adding real runtime overhead. > > All 1-3 sound pretty sensible to me. I also think there's a 4, but I think the tradeoffs are

Re: Improving LWLock wait events

2020-12-22 Thread Craig Ringer
On Wed, 23 Dec 2020 at 15:51, Craig Ringer wrote: > > I've struggled with this quite a bit myself. > > By the way, I sent in a patch to enhance the static tracepoints available for LWLocks. See https://www.postgresql.org/message-id/cagry4nxjo+-hcc2i5h93ttsz4gzo-f

Re: Logical decoding without slots: decoding in lockstep with recovery

2020-12-25 Thread Craig Ringer
On Wed, 23 Dec 2020, 18:57 Amit Kapila, wrote: > On Wed, Dec 23, 2020 at 12:26 PM Craig Ringer > wrote: > > > > Hi all > > > > I want to share an idea I've looked at a few times where I've run into > situations where logical slots were inadvertently

Re: [PATCH] LWLock self-deadlock detection

2021-01-04 Thread Craig Ringer
On Wed, 30 Dec 2020 at 10:11, Andres Freund wrote: > Hi, > > On 2020-11-27 20:22:41 +0200, Heikki Linnakangas wrote: > > On 26/11/2020 04:50, Tom Lane wrote: > > > Craig Ringer writes: > > > > On Wed, Nov 25, 2020 at 9:23 PM Ashutosh Bapat < > a

Re: [PATCH] Runtime control of CLOBBER_CACHE_ALWAYS

2021-01-05 Thread Craig Ringer
On Tue, 5 Jan 2021, 22:41 Peter Eisentraut, < peter.eisentr...@2ndquadrant.com> wrote: > On 2020-12-03 07:01, Craig Ringer wrote: > > To try it out, apply the patch (git am), build with --enable-cassert, > > then compare: > > > > make -C s

Re: [PATCH] Runtime control of CLOBBER_CACHE_ALWAYS

2021-01-06 Thread Craig Ringer
On Wed, 6 Jan 2021 at 18:23, Peter Eisentraut < peter.eisentr...@2ndquadrant.com> wrote: > On 2021-01-06 00:30, Craig Ringer wrote: > > Perhaps debug_invalidate_system_caches_always ? It's a bit long but we > > use the debug prefix elsewhere too. > > Committed wit

Re: [PATCH] Identify LWLocks in tracepoints

2021-01-06 Thread Craig Ringer
On Mon, 28 Dec 2020 at 20:09, Masahiko Sawada wrote: > Hi Craig, > > On Sat, Dec 19, 2020 at 2:00 PM Craig Ringer > wrote: > > > > Hi all > > > > The attached patch set follows on from the discussion in [1] "Add LWLock > blocker(s) information&q

Re: [PATCH] Identify LWLocks in tracepoints

2021-01-07 Thread Craig Ringer
On Sat, 19 Dec 2020 at 13:00, Craig Ringer wrote: > Hi all > > The attached patch set follows on from the discussion in [1] "Add LWLock > blocker(s) information" by adding the actual LWLock* and the numeric > tranche ID to each LWLock related TRACE_POSTGRESQL_foo tracep

Re: PATCH: Batch/pipelining support for libpq

2021-02-15 Thread Craig Ringer
On Thu, 11 Feb 2021 at 07:51, Alvaro Herrera wrote: > On 2021-Jan-21, Alvaro Herrera wrote: > > > As you can see in an XXX comment in the libpq test program, the current > > implementation has the behavior that PQgetResult() returns NULL after a > > batch is finished and has reported PGRES_BATCH_

Re: PATCH: Batch/pipelining support for libpq

2021-02-15 Thread Craig Ringer
On Tue, 16 Feb 2021 at 09:19, Craig Ringer wrote: > > So long as there is a way to "send A", "send B", "send C", "read results > from A", "send D", and there's a way for the application to associate some > kind of state (an ap

libpq PQresultErrorMessage vs PQerrorMessage API issue

2021-02-15 Thread Craig Ringer
Hi all This morning for the the umpteenth time I saw: some error message: [blank here] output from a libpq program. That's because passing a NULL PGresult to PQgetResultErrorMessage() returns "". But a NULL PGresult is a normal result from PQexec when it fails to submit a query due to an inval

Re: PATCH: Batch/pipelining support for libpq

2021-02-19 Thread Craig Ringer
On Wed, 17 Feb 2021, 07:13 Alvaro Herrera, wrote: > Here's a new version, where I've renamed everything to "pipeline". Wow. Thanks. I > think the docs could use some additional tweaks now in order to make a > coherent story on pipeline mode, how it can be used in a batched > fashion, etc. > I

Re: libpq compression

2021-02-21 Thread Craig Ringer
On Thu, 11 Feb 2021, 21:09 Daniil Zakhlystov, wrote:: > > 3. Chunked compression allows to compress only well compressible messages > and save the CPU cycles by not compressing the others > 4. Chunked compression introduces some traffic overhead compared to the > permanent (1.2810G vs 1.2761G TX

Re: [PATCH] Identify LWLocks in tracepoints

2021-03-09 Thread Craig Ringer
till interested in whether Andres will be able to do anything about identifying LWLocks in a cross-backend manner. But this work doesn't really depend on that; it'd benefit from it, but would be easily adapted to it later if needed. From 36c7ddcbca2dbbcb2967f01cb92aa1f61620c838 Mon Sep 17 00

Re: PATCH: Report libpq version and configuration

2020-10-26 Thread Craig Ringer
On Tue, Oct 27, 2020 at 12:41 AM Alvaro Herrera wrote: > On 2020-Oct-26, Craig Ringer wrote: > > > Patch 0001 adds PQlibInfo(), which returns an array of key/value > > description items reporting on configuration like the full version > string, > > SSL support, gs

Re: PATCH: Report libpq version and configuration

2020-10-26 Thread Craig Ringer
On Tue, Oct 27, 2020 at 12:56 AM Tom Lane wrote: > > Alvaro Herrera writes: > > On 2020-Oct-26, Craig Ringer wrote: > >> also adds PQlibInfoPrint() which dumps PQlibInfo() keys/values to stdout. > > > Sounds useful. I'd have PQlibInfoPrint(FILE *) instead, so y

Re: Internal key management system

2020-10-27 Thread Craig Ringer
On Mon, Oct 26, 2020 at 11:02 PM Stephen Frost wrote: TL;DR: * Important to check that key rotation is possible on a replica, i.e. primary and standby can have different cluster passphrase and KEK encrypting the same WAL and heap keys; * with a HSM we can't read the key out, so a pluggable KEK

Re: Internal key management system

2020-10-27 Thread Craig Ringer
On Tue, 27 Oct 2020, 19:15 Bruce Momjian, wrote: > We could implement a 'command:' prefix now, and maybe > a 'pass:' one, and allow other methods like 'pkcs11' later. > We don't need to do anything except provide a way to tell OpenSSL where to get the KEK from, for situations where having Pg gen

Re: Internal key management system

2020-10-27 Thread Craig Ringer
On Wed, Oct 28, 2020 at 9:43 AM Bruce Momjian wrote: > > I don't know much about how to hook into that stuff so if you have an > idea, I am all ears. Yeah, I have a reasonable idea. The main thing will be to re-read the patch and put it into more concrete terms, which I'll try to find time for

Re: Internal key management system

2020-10-28 Thread Craig Ringer
On Wed, Oct 28, 2020 at 12:02 PM Craig Ringer wrote: > On Wed, Oct 28, 2020 at 9:43 AM Bruce Momjian wrote: > > > >> I don't know much about how to hook into that stuff so if you have an >> idea, I am all ears. > > > Yeah, I have a reasonable idea. The mai

Re: Internal key management system

2020-10-28 Thread Craig Ringer
On Thu, Oct 29, 2020 at 1:22 AM Stephen Frost wrote: > > Most importantly - I don't think the SQL key adds anything really > > crucial that we cannot do at the SQL level with an extension. An > > extension "pg_wrap" could provide pg_wrap() and pg_unwrap() already, > > using a single master key

[PATCH] TAP test showing that pg_replication_slot_advance() works on standby

2020-11-09 Thread Craig Ringer
h committing, but it's here so searchers can find it at least. From 218c607b64db493b975bb398528d6a952beeb32f Mon Sep 17 00:00:00 2001 From: Craig Ringer Date: Tue, 10 Nov 2020 13:06:41 +0800 Subject: [PATCH] Extend TAP test for pg_replication_slot_advance() to cover standby Show that pg_replica

Add docs stub for recovery.conf

2020-11-09 Thread Craig Ringer
hem instead of removing them. That'll help prevent us from breaking links on the wider web, in 3rd party documentation, etc. From 9d6db17c8bff093c08a90129d0ba181554b5d3e2 Mon Sep 17 00:00:00 2001 From: Craig Ringer Date: Tue, 10 Nov 2020 10:51:20 +0800 Subject: [PATCH] Link the old recovery.co

Re: PATCH: Report libpq version and configuration

2020-11-09 Thread Craig Ringer
On Tue, Nov 10, 2020 at 12:33 AM Tom Lane wrote: > Alvaro Herrera writes: > > Well, if we can make it run in more systems than just Linux, then it > > seems worth having. The submitted patch seems a little bit on the > > naughty side. > > I agree that the facility seems possibly useful, as long

Re: PATCH: Report libpq version and configuration

2020-11-10 Thread Craig Ringer
On Tue, Nov 10, 2020 at 2:22 PM Craig Ringer wrote: > > The main things I'd really like to get in place are a way to get the > version as an ELF data symbol, and a simple way to ID the binary. > > So the minimal change would be to declare: > > const char LIBPQ_VERS

Re: Add docs stub for recovery.conf

2020-11-11 Thread Craig Ringer
On Thu, Nov 12, 2020 at 3:44 AM Bruce Momjian wrote: > On Tue, Nov 10, 2020 at 01:38:14PM +0800, Craig Ringer wrote: > > Hi all > > > > I noticed that when recovery.conf was removed in 2dedf4d9a8 (yay!) the > docs for > > it were removed completely as well. T

Re: Add docs stub for recovery.conf

2020-11-11 Thread Craig Ringer
On Thu, Nov 12, 2020 at 4:01 AM Bruce Momjian wrote: > On Wed, Nov 11, 2020 at 08:59:40PM +0100, Daniel Gustafsson wrote: > > > On 11 Nov 2020, at 20:44, Bruce Momjian wrote: > > > On Tue, Nov 10, 2020 at 01:38:14PM +0800, Craig Ringer wrote: > > > > >>

Re: Add docs stub for recovery.conf

2020-11-12 Thread Craig Ringer
On Thu, Nov 12, 2020 at 11:25 PM Stephen Frost wrote: > > > Now, the pgweb feature that Jonathan wrote recently might actually be > > exactly what we need to fix that, and to address the issue with > > recovery config documentation that Craig raises. > > After chatting with Jonathan about this fo

Re: Add docs stub for recovery.conf

2020-11-12 Thread Craig Ringer
On Fri, Nov 13, 2020 at 11:31 AM Craig Ringer wrote: > > Can anyone tell me why the solution I proposed is not acceptable, and why > we have to invent a different one instead? The website redirect is good > and all, but doesn't really solve the problem, and I still don'

Re: Add docs stub for recovery.conf

2020-11-12 Thread Craig Ringer
On Fri, Nov 13, 2020 at 11:50 AM Bruce Momjian wrote: > > So you are saying you don't think you are getting sufficient thought > > into your proposal, and getting just a reflex? Just because we don't > > agree with you don't mean we didn't think about it. In fact, we have > > thought about it

Re: Add docs stub for recovery.conf

2020-11-15 Thread Craig Ringer
On Sun, Nov 15, 2020 at 1:49 PM David G. Johnston < david.g.johns...@gmail.com> wrote: > On Fri, Nov 13, 2020 at 10:42 AM Bruce Momjian wrote: > >> I think the big problem, and I have seen this repeatedly, is showing up >> with a patch without discussing whether people actually want the >> featur

Re: abstract Unix-domain sockets

2020-11-17 Thread Craig Ringer
On Fri, Oct 9, 2020 at 3:28 PM Peter Eisentraut < peter.eisentr...@2ndquadrant.com> wrote: > During the discussion on Unix-domain sockets on Windows, someone pointed > out[0] abstract Unix-domain sockets. > This reminds me on a somewhat random note that SSPI mode authentication should work out of

Re: Detecting File Damage & Inconsistencies

2020-11-17 Thread Craig Ringer
On Fri, Nov 13, 2020 at 7:24 PM Simon Riggs wrote: > > What I'm proposing is an option to add 16 bytes onto each COMMIT > record > Would it make sense to write this at the time we write a topxid assignment to WAL instead? Otherwise it won't be accessible to streaming-mode logical decoding.

[PATCH] LWLock self-deadlock detection

2020-11-19 Thread Craig Ringer
t of it even if nobody likes the idea of merging it. As written the check runs only for --enable-cassert builds or when LOCK_DEBUG is defined. From 0ec0beb294f4c5ed35dbb35260f53b069563638f Mon Sep 17 00:00:00 2001 From: Craig Ringer Date: Thu, 19 Nov 2020 14:24:55 +0800 Subject: [PATCH] LWLock self-dea

Re: Add LWLock blocker(s) information

2020-11-19 Thread Craig Ringer
thin a tranche except for individual locks and named tranches. By the way, the LWLock tracepoints currently fire T_NAME(lock) which calls GetLWTrancheName() for each tracepoint hit, so long as Pg is built with --enable-dtrace, even when nothing is actually tracing them. We might want to consider g

Re: ResourceOwner refactoring

2020-11-19 Thread Craig Ringer
and safer to author in general, like providing a simple way to do error trapping and recovery in an extension mainloop without copy/pasting a bunch of PostgresMain code, better default signal handlers, startup/shutdown that shares more with user backends, etc. Right now it's quite tricky to

Re: [PATCH] postgres_fdw connection caching - cause remote sessions linger till the local session exit

2020-11-24 Thread Craig Ringer
On Wed, Nov 25, 2020 at 2:43 AM Alexey Kondratov wrote: > On 2020-11-24 06:52, Bharath Rupireddy wrote: > > Thanks for the review comments. > > > > On Mon, Nov 23, 2020 at 9:57 PM Alexey Kondratov > > wrote: > >> > >> > v1-0001-postgres_fdw-function-to-discard-cached-connections.patch > >> > >>

Re: POC: postgres_fdw insert batching

2020-11-24 Thread Craig Ringer
On Thu, Oct 8, 2020 at 10:40 AM tsunakawa.ta...@fujitsu.com < tsunakawa.ta...@fujitsu.com> wrote: > > Thank you for picking up this. I'm interested in this topic, too. (As an > aside, we'd like to submit a bulk insert patch for ECPG in the near future.) > > As others referred, Andrey-san's fast

Re: [PATCH] LWLock self-deadlock detection

2020-11-24 Thread Craig Ringer
On Tue, Nov 24, 2020 at 10:11 PM Ashutosh Bapat < ashutosh.bapat@gmail.com> wrote: > This looks useful. LWLockCheckSelfDeadlock() could use LWLockHeldByMe > variant instead of copying that code with possibly a change in that > function to return the required information. > Yes, possibly so. I

Re: [PATCH] LWLock self-deadlock detection

2020-11-25 Thread Craig Ringer
On Wed, Nov 25, 2020 at 9:23 PM Ashutosh Bapat wrote: > On Wed, Nov 25, 2020 at 11:47 AM Craig Ringer > wrote: > >> I am also seeing a pattern > >> Assert(!LWLockHeldByMe()); > >> LWLockAcquire() > >> > >> at some places. Should we cha

Re: POC: postgres_fdw insert batching

2020-11-26 Thread Craig Ringer
On Fri, Nov 27, 2020 at 3:34 AM Tomas Vondra wrote: > Not sure how is this related to app developers? I think the idea was > that the libpq features might be useful between the two PostgreSQL > instances. I.e. the postgres_fdw would use the libpq batching to send > chunks of data to the other si

Re: POC: postgres_fdw insert batching

2020-11-26 Thread Craig Ringer
On Fri, Nov 27, 2020 at 10:47 AM tsunakawa.ta...@fujitsu.com < tsunakawa.ta...@fujitsu.com> wrote: Covering this one first: I expect postgresExecForeignBatchInsert() would be able to use the libpq > batching API, because it receives an array of tuples and can generate and > issue INSERT statement

Re: POC: postgres_fdw insert batching

2020-11-27 Thread Craig Ringer
On Sat, 28 Nov 2020, 10:10 Tomas Vondra, wrote: > > > On 11/27/20 7:05 AM, tsunakawa.ta...@fujitsu.com wrote: > > However, the FDW interface as it's implemented today is not designed to > allow that, I believe (we pretty much just invoke the FWD callbacks as > if it was a local AM). It assumes th

Re: Add docs stub for recovery.conf

2020-11-29 Thread Craig Ringer
On Sat, Nov 14, 2020 at 1:42 AM Bruce Momjian wrote: > > Clearly we have need for documenting these renamings somewhere. We were > going to go with a simple URL redirect and a "tip" for > default/pre-installed roles, but I like the idea of doing something more > wholistic that covers all of our r

Re: Printing LSN made easy

2020-11-29 Thread Craig Ringer
On Mon, Nov 30, 2020 at 1:10 AM Tom Lane wrote: > Michael Paquier writes: > > On Fri, Nov 27, 2020 at 04:10:27PM +0530, Ashutosh Bapat wrote: > >> Off list Craig Ringer suggested introducing a new format specifier > >> similar to %m for LSN but I did not get

Re: POC: postgres_fdw insert batching

2020-11-29 Thread Craig Ringer
On Fri, 27 Nov 2020, 14:06 tsunakawa.ta...@fujitsu.com, wrote: > > Also, I'm afraid it requires major surgery or reform of executor. I don't want it to delay the release of reasonably good (10x) improvement with the synchronous interface.) Totally sensible. If it isn't feasible without signific

Re: Asynchronous Append on postgres_fdw nodes.

2020-11-29 Thread Craig Ringer
"On Thu, Nov 26, 2020 at 9:28 AM movead...@highgo.ca wrote: > > > I test the patch and occur several issues as blow: > > Issue one: > Get a Assert error at 'Assert(bms_is_member(i, node->as_needrequest));' in > ExecAppendAsyncRequest() function when I use more than two foreign table > on different

RFC: extension_search_path to supplement dynamic_library_path

2020-11-29 Thread Craig Ringer
Hi all We have a dynamic_library_path setting to allow postgres to search additional locations for loadable libraries (since 8.1, commit 761a0bb69b). This permits libraries to be located in alternate locations for loading by shared_preload_libraries, LOAD commands, implicit loading via function ca

Notes on physical replica failover with logical publisher or subscriber

2020-11-29 Thread Craig Ringer
Hi all I recently wrote some notes on interaction between physical replication failover/promotion and logical replication publisher and/or standby. As you probably all know, right now we don't support physical failover for logical replication publishers at all, either for in-core logical replicat

Re: Printing backtrace of postgres processes

2020-11-29 Thread Craig Ringer
> Surely this is *utterly* unsafe. You can't do that sort of stuff in > a signal handler. Not safely, anyway. The signal handler could be called in the middle of a malloc(), a pfree(), or all sorts of other exciting circumstances. It'd have to be extremely careful to use only local resources on t

Re: proposal: function pg_setting_value_split() to parse shared_preload_libraries etc.

2020-11-29 Thread Craig Ringer
On Mon, 30 Nov 2020 at 03:24, Anastasia Lubennikova wrote: > > On 23.10.2020 05:06, Ian Lawrence Barwick wrote: > > Having just submitted this, I realised I'm focussing on the GUCs which call > > "SplitDirectoriesString()" (as my specific uses case is for > > "shared_preload_libraries") > > but th

Re: Logical decoding without slots: decoding in lockstep with recovery

2021-01-12 Thread Craig Ringer
On Sat, 26 Dec 2020 at 06:51, Andres Freund wrote: > Hi, > > On 2020-12-23 14:56:07 +0800, Craig Ringer wrote: > > I want to share an idea I've looked at a few times where I've run into > > situations where logical slots were inadvertently dropped, or where it

Re: [PATCH] Identify LWLocks in tracepoints

2021-01-14 Thread Craig Ringer
On Wed, 13 Jan 2021 at 19:19, Dmitry Dolgov <9erthali...@gmail.com> wrote: > > On Sat, Dec 19, 2020 at 01:00:01PM +0800, Craig Ringer wrote: > > > > The attached patch set follows on from the discussion in [1] "Add LWLock > > blocker(s) information" by a

Re: [PATCH] Identify LWLocks in tracepoints

2021-01-14 Thread Craig Ringer
On Thu, 14 Jan 2021 at 15:56, Peter Eisentraut < peter.eisentr...@enterprisedb.com> wrote: > On 2020-12-19 06:00, Craig Ringer wrote: > > Patch 1 fixes a bogus tracepoint where an lwlock__acquire event would be > > fired from LWLockWaitForVar, despite that function never a

Re: Add docs stub for recovery.conf

2021-01-17 Thread Craig Ringer
On Thu, 14 Jan 2021 at 03:44, Stephen Frost wrote: > > Alright, how does this look? The new entries are all under the > 'obsolete' section to keep it out of the main line, but should work to > 'fix' the links that currently 404 and provide a bit of a 'softer' > landing for the other cases that c

Re: Printing backtrace of postgres processes

2021-01-17 Thread Craig Ringer
On Mon, 18 Jan 2021 at 00:56, vignesh C wrote: > > Thanks for your comments Andres, I will ignore it for the processes > which do not have access to ProcSignal. I will make the changes and > post a patch for this soon. > I think that's sensible. I've had variable results with glibc's backtrace(

[PATCH] Cross-reference comments on signal handling logic

2021-01-17 Thread Craig Ringer
various scattered but inter-related code. I hope this helps some new developers in future. From a16d0a0f8f502ba3631d20d51c7bb50cedea6d57 Mon Sep 17 00:00:00 2001 From: Craig Ringer Date: Mon, 18 Jan 2021 12:21:18 +0800 Subject: [PATCH v1 1/2] Comments and cross-references for signal handling To help

[PATCH] More docs on what to do and not do in extension code

2021-01-17 Thread Craig Ringer
art providing people with clear references to the right place in the code, READMEs, etc to look when they need to understand specific topics. From 96ce89cb7e1a97d9d247fbacba73ade85a01ea38 Mon Sep 17 00:00:00 2001 From: Craig Ringer Date: Mon, 18 Jan 2021 14:05:15 +0800 Subject: [PATCH v1 2/2] Expan

[PATCH] ProcessInterrupts_hook

2021-01-18 Thread Craig Ringer
don't want to use the memory for a ProcSignal slot for something that won't be relevant for most backends for most of the postmaster lifetime. Ideas welcomed. From 1c8c477a5814420011fa034323e82d8eabc6bc5f Mon Sep 17 00:00:00 2001 From: Craig Ringer Date: Mon, 18 Jan 2021 14:14:46 +0800

Re: [PATCH] ProcessInterrupts_hook

2021-01-18 Thread Craig Ringer
On Tue, 19 Jan 2021, 02:01 Robert Haas, wrote: > On Mon, Jan 18, 2021 at 11:56 AM Tom Lane wrote: > > > I've wanted this in the past, too, so +1 from me. > > > > I dunno, this seems pretty scary and easily abusable. There's not all > > that much that can be done safely in ProcessInterrupts(), a

How to expose session vs txn lock info in pg_locks view?

2021-01-18 Thread Craig Ringer
Presently there doesn't seem to be a way to tell whether a lock is session-level or transaction-level in the pg_locks view. I was expecting this to be a quick patch, but the comment on the definition of PROCLOCKTAG in lock.h notes that shmem state for heavyweight locks does not track whether the l

Re: [PATCH] ProcessInterrupts_hook

2021-01-18 Thread Craig Ringer
On Tue, 19 Jan 2021 at 12:44, Craig Ringer wrote: > > > We're about halfway there already, see 7e784d1dc. I didn't do the >> > other half because it wasn't necessary to the problem, but exposing >> > the shutdown state more fully seems reasonable.

Re: Printing backtrace of postgres processes

2021-01-20 Thread Craig Ringer
On Wed, 20 Jan 2021 at 01:31, Robert Haas wrote: > On Sat, Jan 16, 2021 at 3:21 PM Tom Lane wrote: > > I'd argue that backtraces for those processes aren't really essential, > > and indeed that trying to make the syslogger report its own backtrace > > is damn dangerous. > > I agree. Ideally I'd

Re: Printing backtrace of postgres processes

2021-01-20 Thread Craig Ringer
On Wed, 20 Jan 2021 at 05:23, Tom Lane wrote: > > Recursion is scary, but it should (I think) not be possible if this > is driven off CHECK_FOR_INTERRUPTS. There will certainly be none > of those in libbacktrace. > We can also hold interrupts for the call, and it might be wise to do so. One po

Re: Add docs stub for recovery.conf

2021-01-20 Thread Craig Ringer
On Wed, 20 Jan 2021 at 02:45, Stephen Frost wrote: > Greetings, > > * Stephen Frost (sfr...@snowman.net) wrote: > > * Craig Ringer (craig.rin...@enterprisedb.com) wrote: > > > On Thu, 14 Jan 2021 at 03:44, Stephen Frost > wrote: > > > > Alright, how d

Re: Printing backtrace of postgres processes

2021-01-20 Thread Craig Ringer
On Thu, 21 Jan 2021 at 09:56, Tom Lane wrote: > Craig Ringer writes: > > On Wed, 20 Jan 2021 at 05:23, Tom Lane wrote: > >> BTW, it also looks like the patch is doing nothing to prevent the > >> backtrace from being sent to the connected client. > > > I don

Re: [PATCH] More docs on what to do and not do in extension code

2021-01-21 Thread Craig Ringer
Hi Thanks so much for reading over this! Would you mind attaching a revised version of the patch with your edits? Otherwise I'll go and merge them in once you've had your say on my comments inline below. Bruce, Robert, can I have an opinion from you on how best to locate and structure these docs

Re: Preventing hangups in bgworker start/stop during DB shutdown

2021-01-22 Thread Craig Ringer
On Fri, 25 Dec 2020 at 06:07, Tom Lane wrote: > I wrote: > > Bharath Rupireddy writes: > >> 4) IIUC, in the patch we mark slot->terminate = true only for > >> BGW_NEVER_RESTART kind bg workers, what happens if a bg worker has > >> bgw_restart_time seconds and don't we hit the hanging issue(that

Re: How to expose session vs txn lock info in pg_locks view?

2021-02-01 Thread Craig Ringer
On Sun, 24 Jan 2021 at 09:12, Andres Freund wrote: > Hi, > > On 2021-01-19 14:16:07 +0800, Craig Ringer wrote: > > AFAICS it'd be necessary to expand PROCLOG to expose this in shmem. > > Probably by adding a small bitfield where bit 0 is set if there's a txn >

Re: Detecting File Damage & Inconsistencies

2021-03-17 Thread Craig Ringer
On Mon, 15 Mar 2021 at 21:01, David Steele wrote: > On 11/18/20 5:23 AM, Simon Riggs wrote: > > On Wed, 18 Nov 2020 at 06:42, Craig Ringer > > wrote: > >> > >> On Fri, Nov 13, 2020 at 7:24 PM Simon Riggs > wrote: > >>> > >>> >

Re: [PATCH] Identify LWLocks in tracepoints

2021-03-17 Thread Craig Ringer
On Thu, 11 Mar 2021 at 15:57, Peter Eisentraut < peter.eisentr...@enterprisedb.com> wrote: > On 10.03.21 06:38, Craig Ringer wrote: > > On Wed, 3 Mar 2021 at 20:50, David Steele > <mailto:da...@pgmasters.net>> wrote: > > > > On 1/22/21 6:02 AM, Peter E

Re: [PATCH] Identify LWLocks in tracepoints

2021-03-19 Thread Craig Ringer
On Sat, 20 Mar 2021, 04:21 Peter Eisentraut, < peter.eisentr...@enterprisedb.com> wrote: > > On 18.03.21 07:34, Craig Ringer wrote: > > The main reason I didn't want to add more tracepoints than were strictly > > necessary is that Pg doesn't enable the systemta

  1   2   3   4   5   6   >