Re: [HACKERS] logical decoding of two-phase transactions

2017-11-29 Thread Craig Ringer
On 30 November 2017 at 07:40, Petr Jelinek wrote: > Hi, > > On 24/11/17 07:41, Craig Ringer wrote: > > On 24 November 2017 at 13:44, Nikhil Sontakke > > > > > > How practical is adding a lock class? > > > > Am open to suggestions. This look

Re: Would a BGW need shmem_access or database_connection to enumerate databases?

2017-12-04 Thread Craig Ringer
tart_manager_workers in pglogical.c https://github.com/2ndQuadrant/pglogical/blob/REL2_x_STABLE/pglogical.c#L594 and the caller pglogical_supervisor_main . -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Would a BGW need shmem_access or database_connection to enumerate databases?

2017-12-04 Thread Craig Ringer
n the postmaster, > but also not in a "regular backend", but rather another BGW? > Yes. BDR does it a lot. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Signals in a BGW

2017-12-04 Thread Craig Ringer
and a bad example. Personally I'd rather change the default bgw handler to 'die', make the sample bgworkers use CHECK_FOR_INTERRUPTS() properly, and be done with it. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] logical decoding of two-phase transactions

2017-12-04 Thread Craig Ringer
es than ROLLBACKed. It's a pretty tiny corner case for decoding of 2pc but a bigger one when we're addressing streaming decoding. Otherwise I'm really, really happy with how this is progressing and want to find time to play with it. -- Craig Ringer http://ww

Re: [HACKERS] Walsender timeouts and large transactions

2017-12-04 Thread Craig Ringer
things fixed. This version > fixes the original issue as well. > > I'm happy with what I see here. Commit message needs tweaking, but any committer would do that anyway. To me it looks like it's time to get this committed, marking as such. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

BUGFIX: standby disconnect can corrupt serialized reorder buffers

2017-12-04 Thread Craig Ringer
le this bug looks similar to https://www.postgresql.org/message-id/54e4e488-186b-a056-6628-50628e4e4...@lab.ntt.co.jp "Failed to delete old ReorderBuffer spilled files" it's really quite a separate issue. Both this bugfix and the above need backpatching to 9.4. -- Craig Ringer

Re: [HACKERS] logical decoding of two-phase transactions

2017-12-05 Thread Craig Ringer
nd to be added > to the core for this to work. We can add this functionality as a > follow-on separate patch after discussing how we want to implement > that in core. Yeah, definitely a different patch, but assuredly valuable. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Walsender timeouts and large transactions

2017-12-05 Thread Craig Ringer
On 6 December 2017 at 04:07, Robert Haas wrote: > On Mon, Dec 4, 2017 at 10:59 PM, Craig Ringer > wrote: > > To me it looks like it's time to get this committed, marking as such. > > This version has noticeably more code rearrangement than before, and > I'm not

Re: Postgres with pthread

2017-12-06 Thread Craig Ringer
y convert to a threaded backend with minimal changes to callers, and probably only limited changes to shm_mq its self. So maybe these abstractions will prove to have been a win in some ways. Except DSA, and even then it could serve as a transitional API... -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Postgres with pthread

2017-12-06 Thread Craig Ringer
ore practical if we didn't have to serialize and deserialize the entire session state to do it. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Postgres with pthread

2017-12-06 Thread Craig Ringer
On 7 December 2017 at 11:44, Tsunakawa, Takayuki < tsunakawa.ta...@jp.fujitsu.com> wrote: > From: Craig Ringer [mailto:cr...@2ndquadrant.com] > > I'd personally expect that an immediate conversion would result > > in very > > little speedup, a b

Re: Add PGDLLIMPORT lines to some variables

2017-12-06 Thread Craig Ringer
missed the boat on your commit, but ... please? Patches attached. MyReplicationSlot was only made PGDLLIMPORT in 9.6, so there's one for 9.4/9.5 and one for 9.6, 10, and master. Personally I don't care about 9.4/9.5 in the slightest for this, but that's where c572599c is backpatched

Re: Logical replication without a Primary Key

2017-12-07 Thread Craig Ringer
tid (since if it were a table with oids, the rows wouldn't be identical anymore). It'd make no sense to send the ctid to the downstream since it'd bear no relationship to the downstream table. I expect it'd probably update both rows. If it doesn't, it probably should.

Re: Postgres with pthread

2017-12-07 Thread Craig Ringer
ntally clobbering postmaster memory/state would be my main worry there. Right now we gain a lot of protection from our copy-on-write shared-nothing-by-default model, and we rely on it in quite a lot of places where backends merrily stomp on inherited postmaster state. The more I think about it, th

Re: Postgres with pthread

2017-12-07 Thread Craig Ringer
On 8 December 2017 at 03:58, Andres Freund wrote: > On 2017-12-07 11:26:07 +0800, Craig Ringer wrote: > > PostgreSQL's architecture conflates "connection", "session" and > "executor" > > into one somewhat muddled mess. > > How is t

Re: [HACKERS] Logical decoding on standby

2017-12-08 Thread Craig Ringer
On 27 June 2017 at 13:24, Craig Ringer wrote: > On 21 June 2017 at 17:30, sanyam jain wrote: > > Hi, > > After changing > > sendTimeLineIsHistoric = state->currTLI == ThisTimeLineID; > > to > > sendTimeLineIsHistoric = state->currTLI != ThisTimeLineID; >

Re: Added PostgreSQL internals learning materials in Developer FAQ

2017-12-11 Thread Craig Ringer
we're talking wiki changes, some folks may be interested in the recent stuff I added to the profiling with perf page https://wiki.postgresql.org/wiki/Profiling_with_perf on using postgres's predefined tracepoints, injecting userspace tracepoints, capturing variable values at tracepoints, etc. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Testing Extension Upgrade Paths

2017-12-11 Thread Craig Ringer
th, but you still have the 1.0.0 -> 1.1.0 path. You can handle that with TAP if you're willing to write something to do the various combinations of steps and exercise them. It's not practical with pg_regress. More thorough testing benefits from an external harness. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: proposal: alternative psql commands quit and exit

2017-12-11 Thread Craig Ringer
help them either. Experienced users on unixes will try control-D, but unless you've used unix a while that's not going to occur to you. I doubt I've ever written just "exit" or "quit" without indentation. I think if it requires them to be a bareword with no indentation, strictly ^(exit|quit)\n when isatty, then that's probably a safe and helpful choice. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: proposal: alternative psql commands quit and exit

2017-12-11 Thread Craig Ringer
stant psql user I wouldn't have learned to be blind to that yet. But I think many users are probably almost as blind to welcome messages are they are to banner ads. And lets face it, the users we're trying to reach aren't necessarily the best readers/problem solvers already. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Using ProcSignal to get memory context stats from a running backend

2017-12-11 Thread Craig Ringer
place, attaching, etc. You still have to go fishing for stderr to find the output, but that's usually the same place as the rest of the logs. Barring objections I'll send a patch in a while. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Using ProcSignal to get memory context stats from a running backend

2017-12-11 Thread Craig Ringer
On 12 December 2017 at 12:25, Tsunakawa, Takayuki < tsunakawa.ta...@jp.fujitsu.com> wrote: > From: Craig Ringer [mailto:cr...@2ndquadrant.com] > > TL;DR: Lets add a ProcSignalReason that makes a backend call > > MemoryContextStats when it sees it and a C func that users can u

Re: Using ProcSignal to get memory context stats from a running backend

2017-12-11 Thread Craig Ringer
On 12 December 2017 at 12:43, Andres Freund wrote: > Hi, > > On 2017-12-12 11:57:41 +0800, Craig Ringer wrote: > > TL;DR: Lets add a ProcSignalReason that makes a backend > > call MemoryContextStats when it sees it and a C func that users can use > to > > set it

Re: Using ProcSignal to get memory context stats from a running backend

2017-12-11 Thread Craig Ringer
farm critters. Letting it go to stderr isn't too bad, but it'd certainly make it that bit nicer if it didn't so I'm not opposed to adding the needed indirection. I'll give it a try in a bit. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: proposal: alternative psql commands quit and exit

2017-12-11 Thread Craig Ringer
On 12 December 2017 at 13:58, Tom Lane wrote: > Craig Ringer writes: > > I doubt I've ever written just "exit" or "quit" without indentation. I > > think if it requires them to be a bareword with no indentation, strictly > > ^(exit|quit)\n when i

Re: proposal: alternative psql commands quit and exit

2017-12-12 Thread Craig Ringer
the fact that table metadata formatting and display is done wholly client-side and cannot be used in applications is absolutely horrid, and a real wart in the system. If we had the server-side functionality then some special case HINTs in error messages for common systems would be reasonable enough. Frankly that's probably a good idea anyway, adding a specific errhint. But it's really a separate topic. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: proposal: alternative psql commands quit and exit

2017-12-12 Thread Craig Ringer
and bail aren't necessarily the ones you want to lose; sometimes they're just busy or are evaluating a few things at once. Early UX matters. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Using ProcSignal to get memory context stats from a running backend

2017-12-12 Thread Craig Ringer
On 13 December 2017 at 12:10, Michael Paquier wrote: > On Mon, Dec 11, 2017 at 10:07 PM, Craig Ringer > wrote: > > On 12 December 2017 at 12:43, Andres Freund wrote: > >> On 2017-12-12 11:57:41 +0800, Craig Ringer wrote: > >> But that'd have > >> the

Re: [HACKERS] Walsender timeouts and large transactions

2017-12-13 Thread Craig Ringer
of if (big_condition && big_condition) one_linerdo_something; as awfully unreadable, but I guess code convention means you live with things you don't like. Anyway, I've just hit this bug in the wild for the umpteenth time this year, and I'd like to know wha

Re: [HACKERS] Walsender timeouts and large transactions

2017-12-14 Thread Craig Ringer
On 15 December 2017 at 00:36, Andrew Dunstan wrote: > > > On 12/14/2017 01:46 AM, Craig Ringer wrote: > > On 7 December 2017 at 01:22, Petr Jelinek > > mailto:petr.jeli...@2ndquadrant.com>> > > wrote: > > > > On 05/12/17 21:07, Robert Haas wrote:

Re: Using ProcSignal to get memory context stats from a running backend

2017-12-14 Thread Craig Ringer
On 15 December 2017 at 09:24, Greg Stark wrote: > Another simpler option would be to open up a new file in the log > directory ... if we have one. We might be logging to syslog, or whatever else. I'd rather keep it simple(ish). -- Craig Ringer http://www.2ndQ

Re: genomic locus

2017-12-18 Thread Craig Ringer
uite so narrowly focused as genetics alone, then I could see adding it as a secondary type in the same extension. But it's more likely that the best course would be to extract the seg extension from core, rename it, hack it as desired, and build it as an extension maintained out-of-tree. -- C

Re: Using ProcSignal to get memory context stats from a running backend

2017-12-19 Thread Craig Ringer
On 18 December 2017 at 10:05, Robert Haas wrote: > On Thu, Dec 14, 2017 at 9:34 PM, Craig Ringer > wrote: > > On 15 December 2017 at 09:24, Greg Stark wrote: > >> Another simpler option would be to open up a new file in the log > >> directory > > > &g

Re: Using ProcSignal to get memory context stats from a running backend

2017-12-19 Thread Craig Ringer
ethod, which I thought would be more likely to get complaints than the global hook. I'm actually happier to do it with a passed callback. Will revise when I get a chance in the next couple of days. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: access/parallel.h lacks PGDLLIMPORT

2017-12-19 Thread Craig Ringer
er and InitializingParallelWorker. > Especially since all non-static *functions* are exported unconditionally, so it's not like we set a high bar for public API. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: AS OF queries

2017-12-20 Thread Craig Ringer
hing") without some rather > > enormous pushups. > > The SQL standard syntax appears to be something like > > "tablename" [ AS OF SYSTEM TIME 'something' ] [ [ AS ] "alias" ] > > That's not going to be fun to parse. Well, the SQL committe

Re: [HACKERS] REINDEX CONCURRENTLY 2.0

2017-12-20 Thread Craig Ringer
; hand. > > Yeah, that would be likely possible. I am not volunteering for that in > the short term though.. > It sounds like that'd make some of ALTER TABLE a bit less ... upsetting ... too. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Using ProcSignal to get memory context stats from a running backend

2017-12-20 Thread Craig Ringer
On 20 December 2017 at 08:46, Craig Ringer wrote: > On 20 December 2017 at 02:35, Andres Freund wrote: > > >> > Yeah. But please don't mess with MemoryContextStats per se --- >> > I dunno about you guys but "call MemoryContextStats(TopMemoryContext)"

Re: Using ProcSignal to get memory context stats from a running backend

2017-12-20 Thread Craig Ringer
On 21 December 2017 at 14:58, Andres Freund wrote: > Hi, > > On 2017-12-21 14:49:28 +0800, Craig Ringer wrote: > > +/* > > + * Accumulate writes into the buffer in diag_request_buf, > > + * for use with functions that expect a printf-like callback

Re: != should give error?

2017-12-20 Thread Craig Ringer
rs Schema | Name | Left arg type | Right arg type | Result type | Function | Description +--+---++-+-+- pg_catalog | !| bigint | | numeric | numeric_fac | factorial (1 row) -- Cr

Re: Using ProcSignal to get memory context stats from a running backend

2017-12-20 Thread Craig Ringer
On 21 December 2017 at 15:24, Andres Freund wrote: > Hi, > > On 2017-12-21 15:13:13 +0800, Craig Ringer wrote: > > There tons of callers to enlargeStringInfo, so a 'noerror' parameter > would > > be viable. > > Not sure what you mean with that

Re: The pg_indent on on ftp is outdated

2017-12-21 Thread Craig Ringer
On 21 Dec. 2017 22:42, "Tom Lane" wrote: Magnus Hagander writes: > On Thu, Dec 21, 2017 at 12:42 PM, Craig Ringer > wrote: >>> The pg_indent on www is outdated, and doesn't understand --version . >>> https://www.postgresql.org/ftp/dev/ > Is there a re

Re: Finalizing logical replication limitations as well as potential features

2017-12-21 Thread Craig Ringer
stent view. It could possibly be worked around with some tricky key-range-based filtering of the applied change-stream if you were willing to require that no PK updates may occur, but it'd probably be bug city. It's hard enough to get sync correct at all. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Using ProcSignal to get memory context stats from a running backend

2017-12-21 Thread Craig Ringer
On 21 December 2017 at 15:55, Craig Ringer wrote: > On 21 December 2017 at 15:24, Andres Freund wrote: > >> Hi, >> >> On 2017-12-21 15:13:13 +0800, Craig Ringer wrote: >> > There tons of callers to enlargeStringInfo, so a 'noerror' parameter >&

Re: Using ProcSignal to get memory context stats from a running backend

2017-12-21 Thread Craig Ringer
On 22 December 2017 at 13:05, Craig Ringer wrote: > On 21 December 2017 at 15:55, Craig Ringer wrote: > >> On 21 December 2017 at 15:24, Andres Freund wrote: >> >>> Hi, >>> >>> On 2017-12-21 15:13:13 +0800, Craig Ringer wrote: >>> > T

Re: Using ProcSignal to get memory context stats from a running backend

2017-12-22 Thread Craig Ringer
On 22 December 2017 at 20:50, Maksim Milyutin wrote: > On 19.12.2017 16:54, Pavel Stehule wrote: > > Hi > > 2017-12-19 14:44 GMT+01:00 Craig Ringer : > >> On 18 December 2017 at 10:05, Robert Haas wrote: >> >>> On Thu, Dec 14, 2017 at 9:34 PM, Craig Ringe

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2017-12-23 Thread Craig Ringer
7 slots, one for each DB. So there's lots of room for difficulty with unpredictable memory use. So the patch does two things. Firstly, it introduces logical_work_mem, a > GUC restricting memory consumed by all transactions currently kept in > the reorder buffer > Does this consider the

Re: AS OF queries

2017-12-24 Thread Craig Ringer
. > Unfortunately it is not so easy to implement. > I think you can learn a lot from studying logical decoding here. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: AS OF queries

2017-12-26 Thread Craig Ringer
On 25 December 2017 at 15:59, Konstantin Knizhnik wrote: > > > On 25.12.2017 06:26, Craig Ringer wrote: > > On 24 December 2017 at 04:53, konstantin knizhnik < > k.knizh...@postgrespro.ru> wrote: > >> >> >> But what if I just forbid to chan

Re: Using ProcSignal to get memory context stats from a running backend

2017-12-26 Thread Craig Ringer
On 22 December 2017 at 23:19, Maksim Milyutin wrote: > On 22.12.2017 16:56, Craig Ringer wrote: > > On 22 December 2017 at 20:50, Maksim Milyutin > wrote: > >> On 19.12.2017 16:54, Pavel Stehule wrote: >> >> sorry for small offtopic. Can be used this mechani

Re: Contributing some code

2017-12-27 Thread Craig Ringer
in turn, since Pg doesn't support cross-DB queries. Logical decoding handles this with the RelidByRelfilenode function, but there are issues there around making sure you have the right snapshot etc. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: The pg_indent on on ftp is outdated

2017-12-28 Thread Craig Ringer
On 28 December 2017 at 23:59, Magnus Hagander wrote: > > > On Fri, Dec 22, 2017 at 1:33 AM, Craig Ringer > wrote: > >> >> >> On 21 Dec. 2017 22:42, "Tom Lane" wrote: >> >> Magnus Hagander writes: >> > On Thu, Dec 21, 2017 at 12

Re: Contributing with code

2017-12-28 Thread Craig Ringer
explicitly to the various lookup functions. Relmapper invalidations would be a problem, I don't see how it'd be practical to handle them, you'd have to be able to register as interested in another DB's invals somehow. And you'd have no locking. So you could probably only use it for things like this - mapping relfilenodes to oids/relnames for other DBs. Worth the effort? Probably not. But potentially fun. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: TAP test module - PostgresClient

2017-12-28 Thread Craig Ringer
ally random people running the test suite to install and maintain them. But if we teach the test suite how to build and run them scoped to the current Pg build using PERL5LIB, users wouldn't *have* to. Internet access could be a sticking point for some, but the same would be true of using

Re: [PATCH] session_replication_role = replica with TRUNCATE

2017-12-29 Thread Craig Ringer
o stop you creating the FK. I'd still like to know if it was a cascade when applying it, so I can possibly make some client-determined behaviour choice, but that's for the other patch really. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [PATCH] session_replication_role = replica with TRUNCATE

2017-12-29 Thread Craig Ringer
it's > implemented using triggers, but that's not true for TRUNCATE. In any > case it does not hurt to mention the FKs explicitly rather than > implicitly here. Yeah. I'd argue that's an oversight in the current docs that "can cause FK violations" is

Re: Postgres, fsync, and OSs (specifically linux)

2018-08-29 Thread Craig Ringer
On 15 August 2018 at 07:32, Thomas Munro wrote: > On Wed, Aug 15, 2018 at 11:08 AM, Asim R P wrote: > > I was looking at the commitfest entry for feature > > (https://commitfest.postgresql.org/19/1639/) for the most recent list > > of patches to try out. The list doesn't look correct/complete.

Re: Code of Conduct plan

2018-09-19 Thread Craig Ringer
came their way, it's a shame it wan't handled by a complaint to a conference CoC group instead. I'd like the CoC to emphasise that while we don't want to restrain people from "calling out" egregious behaviour, going via the CoC team is often more likely to lead to constructive communication and positive change. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

<    1   2   3   4   5   6