Re: [HACKERS] Some thoughts about SCRAM implementation

2017-04-12 Thread Craig Ringer
On 12 Apr. 2017 17:27, "Magnus Hagander" wrote: On Wed, Apr 12, 2017 at 11:13 AM, Heikki Linnakangas wrote: > On 04/12/2017 11:22 AM, Magnus Hagander wrote: > >> On Wed, Apr 12, 2017 at 3:25 AM, Bruce Momjian wrote: >> >> And which enterprises are using SSL without certificates? And I thoug

Re: [HACKERS] GSOC'17 project introduction: Parallel COPY execution with errors handling

2017-04-12 Thread Craig Ringer
ossible before allocating a new (sub)xid and writing to the heap. We'd still abort but we'd only be aborting a vtxid. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsq

Re: [HACKERS] [PATCH v1] Add and report the new "in_hot_standby" GUC pseudo-variable.

2017-04-13 Thread Craig Ringer
On 13 April 2017 at 14:59, Tsunakawa, Takayuki wrote: > 2. Make transaction_read_only GUC_REPORT > This is to avoid the added round-trip by SHOW command. It also benefits > client apps that want to know when the server gets promoted? And this may > simplify the libpq code. > I don't understan

Re: [HACKERS] Letting the client choose the protocol to use during a SASL exchange

2017-04-14 Thread Craig Ringer
On 14 Apr. 2017 10:44, "Michael Paquier" wrote: On Fri, Apr 14, 2017 at 1:37 AM, Heikki Linnakangas wrote: > On 04/13/2017 05:53 AM, Michael Paquier wrote: >> +* Parse the list of SASL authentication mechanisms in the >> +* AuthenticationSASL message, and select the best mechanism that w

Re: [HACKERS] On How To Shorten the Steep Learning Curve Towards PG Hacking...

2017-04-18 Thread Craig Ringer
On 18 April 2017 at 01:29, Alvaro Herrera wrote: > Craig Ringer wrote: > >> Personally I have to agree that the learning curve is very steep. Some >> of the docs and presentations help, but there's a LOT to understand. > > There is a wiki page "Developer_FAQ

Re: [HACKERS] On How To Shorten the Steep Learning Curve Towards PG Hacking...

2017-04-18 Thread Craig Ringer
s, so you can read the code and have more idea what it does and why. > But almost nothing about The Internals of PostgreSQL: Not surprising. They'd go out of date fast, be a huge effort to write and maintain, and sell poorly given the small audience. Print books probably aren't th

Re: [HACKERS] Logical replication and synchronous replication

2017-04-18 Thread Craig Ringer
ought? Definitely should be documented. I think it's covered under logical decoding, but needs at least an xref. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hac

[HACKERS] PATCH: Making constant StringInfo

2017-06-18 Thread Craig Ringer
f call for a StringInfo that can start with an external buffer and append to it until it runs out of room, then copy it only if needed. Patch for constrant StringInfo attached. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Ser

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

2017-06-19 Thread Craig Ringer
I'm glad folks are interested, it's not something I can dedicate much time to. Especially with a 6-week-old baby now > FWIW, I still think this needs a pgbench or similar example integration, > so we can actually properly measure the benefits. I agree. I originally wanted to patch

Re: [HACKERS] REPLICA IDENTITY FULL

2017-06-19 Thread Craig Ringer
the whole old tuple not just keys, so they can be used in conflict processing etc. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] REPLICA IDENTITY FULL

2017-06-19 Thread Craig Ringer
On 20 June 2017 at 09:47, Andres Freund wrote: > On 2017-06-20 09:45:27 +0800, Craig Ringer wrote: >> I frequently want to be able to use REPLICA IDENTITY DEFAULT, but also >> record the whole old tuple not just keys, so they can be used in >> conflict processing etc. &g

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

2017-06-21 Thread Craig Ringer
On 22 Jun. 2017 07:40, "Andres Freund" wrote: On 2017-06-20 17:51:23 +0200, Daniel Verite wrote: > Andres Freund wrote: > > > FWIW, I still think this needs a pgbench or similar example integration, > > so we can actually properly measure the benefits. > > Here's an updated version of the p

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

2017-06-21 Thread Craig Ringer
we don't and shouldn't), and the syscall overhead is IMO not worth worrying about just yet. > and then completely unnecessarily call recv() over and over again > without polling. To me it looks very much like we're just doing either > exactly once per command... Yeah, t

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

2017-06-21 Thread Craig Ringer
On 22 June 2017 at 09:07, Andres Freund wrote: > On 2017-06-22 09:03:05 +0800, Craig Ringer wrote: >> On 22 June 2017 at 08:29, Andres Freund wrote: >> >> > I.e. we're doing tiny write send() syscalls (they should be coalesced) >> >> That's l

Re: [HACKERS] SQL MERGE patches for PostgreSQL Versions

2017-06-22 Thread Craig Ringer
patch. Otherwise you have to rely on what's in the email thread. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to you

Re: [HACKERS] SQL MERGE patches for PostgreSQL Versions

2017-06-22 Thread Craig Ringer
st entry in the 'index' is the git commit hash of the base commit, IIRC. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make chang

Re: [HACKERS] Timing-sensitive case in src/test/recovery TAP tests

2017-06-25 Thread Craig Ringer
ng poll_query_until? This should do the trick: $node_standby_1->poll_query_until('postgres', q[SELECT xmin IS NULL from pg_replication_slots WHERE slot_name = '] . $slotname_2 . q[']); -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development,

Re: [HACKERS] Timing-sensitive case in src/test/recovery TAP tests

2017-06-25 Thread Craig Ringer
On 26 June 2017 at 10:09, Tom Lane wrote: > Michael Paquier writes: >> On Mon, Jun 26, 2017 at 10:48 AM, Craig Ringer wrote: >>> $node_standby_1->poll_query_until('postgres', q[SELECT xmin IS NULL >>> from pg_replication_slots WHERE slot_name = ']

Re: [HACKERS] Timing-sensitive case in src/test/recovery TAP tests

2017-06-25 Thread Craig Ringer
27;s > pg_replication_slot on slotname_2? It would really seem better to make > the nullness check conditional in get_slot_xmins instead. Sorry for > changing opinion here. I'm not sure I understand this. -- Craig Ringer http://www.2ndQuadrant.com/ Post

Re: [HACKERS] Logical decoding on standby

2017-06-26 Thread Craig Ringer
istoric is true when state->currTLI is less than > ThisTimeLineID. Correct, that was a bug. I thought it got fixed upthread though. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers m

Re: [HACKERS] Logical decoding on standby

2017-06-26 Thread Craig Ringer
is correct or not.In any case please someone clarify. That's a reasonable thing to do, and again, I thought I did it in a later revision, but apparently not (?). I've been working on other things and have lost track of progress here a bit. I'll check more closely. -- Craig Ringer

Re: protocol version negotiation (Re: [HACKERS] Libpq PGRES_COPY_BOTH - version compatibility)

2017-06-28 Thread Craig Ringer
an just not ask for them. Capabilities will make startup messages bigger. Personally I don't care much about that, as on modern networks it's all about latency not message size. We'd use abbreviated capability names I expect. If the list gets too big we could always roll up capabi

Re: protocol version negotiation (Re: [HACKERS] Libpq PGRES_COPY_BOTH - version compatibility)

2017-06-28 Thread Craig Ringer
On 29 June 2017 at 09:44, Craig Ringer wrote: > I > can't personally think of much right away that wouldn't work pretty > well in a follow-on message. Actually, I take that back, there's one thing that's bugged me for a while that wouldn't work well this

Re: protocol version negotiation (Re: [HACKERS] Libpq PGRES_COPY_BOTH - version compatibility)

2017-06-28 Thread Craig Ringer
On 29 June 2017 at 10:27, Tom Lane wrote: > Craig Ringer writes: >> On 29 June 2017 at 03:01, Robert Haas wrote: >>> It wouldn't be >>> so bad if unrecognized parameters were just ignored; the client would >>> know from the ServerProtocolVersion (or Par

Re: protocol version negotiation (Re: [HACKERS] Libpq PGRES_COPY_BOTH - version compatibility)

2017-06-28 Thread Craig Ringer
On 29 June 2017 at 12:23, Craig Ringer wrote: > It does. But I don't see anywhere that extra round trips have been discussed. Ah, right, they're implied by having the server respond with some downversion message and ignore input until the client sends a new startup message. That&#

Re: [HACKERS] Start logical decoding from any lsn present in the wal segment

2017-06-29 Thread Craig Ringer
running xacts are precisely known; see xl_running_xacts and the snapshot builder. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: protocol version negotiation (Re: [HACKERS] Libpq PGRES_COPY_BOTH - version compatibility)

2017-06-29 Thread Craig Ringer
initial response to the startup message is > anything other than a ServerProtocolVersion message, the client should > assume it's talking to a 3.0 server. (To make this work, we would > back-patch a change into existing releases to allow any 3.x protocol > version and ignore any pg

Re: [HACKERS] Postgres process invoking exit resulting in sh-QUIT core

2017-06-30 Thread Craig Ringer
> startup process waiting for 000102EB Looks like an archive_command or restore_command . If 'sh' is dumping core, you probably have issues at a low level in the kernel, file system, etc. Check dmesg. -- Craig Ringer http://www.2ndQuadrant.com/

Re: [HACKERS] Race-like failure in recovery/t/009_twophase.pl

2017-07-02 Thread Craig Ringer
B7E8" with normal timing, but with > enough delay in there, you get "|physical|||t|11542|||" which > triggers split's default behavior of ignoring the trailing empty > fields. It looks like the way to get split to not do that is > to pass it a "limit"

Re: [BUGS] [HACKERS] Segmentation fault in libpq

2017-07-02 Thread Craig Ringer
tead of our code. > > Usually using a tool like valgrind is quite helpful to find issues like > that, because it'll show you the call-stack accessing the memory and > *also* the call-stack that lead to the memory being freed. Yep, huge help. BTW, on Windows, the free tool

Re: [HACKERS] Postgres process invoking exit resulting in sh-QUIT core

2017-07-05 Thread Craig Ringer
On 3 Jul. 2017 23:01, "K S, Sandhya (Nokia - IN/Bangalore)" < sandhya@nokia.com> wrote: Hi Craig, Thanks for the response. Scenario tried here is restart of the system multiple times. sh-QUIT core is generated when Postgres is invoking the shell to exit and may not be due to kernel or file s

Re: [HACKERS] Postgres process invoking exit resulting in sh-QUIT core

2017-07-07 Thread Craig Ringer
would get signalled too. Can't immediately explain the exit code, and SIGQUIT should _not_ generate a core file. Can you show the result of attaching 'gdb' to the core file and running 'bt full' ? -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Postgres process invoking exit resulting in sh-QUIT core

2017-07-07 Thread Craig Ringer
ations. > > I have attached output for 2 such instance. > > You seem to be missing debug symbols. Install appropriate debuginfo packages. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] New partitioning - some feedback

2017-07-09 Thread Craig Ringer
gt; > (7 rows) > > Would showing relispartition=tru tables only in \d+ fix this? > <http://www.postgresql.org/mailpref/pgsql-hackers> > I think so. I'd like to add a flag of some kind to \d column output that marks a table as having partitions, but I can't think of anything narrow enough and still useful. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] building libpq.a static library

2017-07-12 Thread Craig Ringer
inux, use $ORIGIN in your rpath. Beware of quoting issues with $ORIGIN though. I'm not trying to block support for a static libpq, I just don't see the point. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] building libpq.a static library

2017-07-12 Thread Craig Ringer
On 13 July 2017 at 10:58, Craig Ringer wrote: > On 12 July 2017 at 23:46, Jeroen Ooms wrote: > >> On Wed, Jul 12, 2017 at 5:11 PM, Tom Lane wrote: >> > Jeroen Ooms writes: >> >> I maintain static libraries for libpq for the R programming language >> >

Re: [HACKERS] [GENERAL] huge RAM use in multi-command ALTER of table heirarchy

2017-07-18 Thread Craig Ringer
he v11 cycle opens, unless someone can show an example > of non-broken coding that requires it. (And if so, there ought to > be a regression test incorporating that.) Just FYI, I believe Simon's currently on holiday, so may not notice this discussion as promptly as usual. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

[HACKERS] [PATCH] pageinspect function to decode infomasks

2017-07-19 Thread Craig Ringer
D_ONLY or HEAP_LOCKED_UPGRADED, and filtering them out could be just as confusing as leaving them in. The infomask2 natts mask is ignored. You can bitwise-and it out in SQL pretty easily if needed. I could output it here as a constructed text datum, but it seems mostly pointless. -- Craig R

Re: [HACKERS] [PATCH] pageinspect function to decode infomasks

2017-07-19 Thread Craig Ringer
On 20 July 2017 at 11:33, Craig Ringer wrote: > Hi > > Whenever I'm debugging some kind of corruption incident, possible > visibility bug, etc, I always land up staring at integer infomasks or using > a SQL helper function to decode them. > > That's silly, so here

Re: [HACKERS] Dealing with logical replication

2017-07-19 Thread Craig Ringer
corner case. If we had a hook in the logical apply worker's insert or wal-message routines it'd be possible to write an extension to do this for pg10, but AFAICS we don't. So schema changes in logical replication currently require more care than in physical replication. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] [TRAP: FailedAssertion] causing server to crash

2017-07-19 Thread Craig Ringer
01000 )) - try to start pg, note the missing clog segment it complains about - dd if=/dev/zero bs=1 count=262144 of=datadir/pg_clog/$MISSINGSEGNAME - start Pg That should put you about 1000 txn's from the 1 million xid limit, assuming I got my maths right (don't assume that), and assuming

Re: [HACKERS] [TRAP: FailedAssertion] causing server to crash

2017-07-19 Thread Craig Ringer
ur script assume a newly initdb'd instance with no custom configuration? If not, what setup steps/configuration precede your script run? > well short of the 2-million mark. > Er, billion. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] [TRAP: FailedAssertion] causing server to crash

2017-07-20 Thread Craig Ringer
to be, you might get faster results by using a custom pgbench script for one or more workers that just runs txid_current() a whole lot. Or jump the server's xid space forward. I've got a few other things on right now but I'll keep an eye out and hope for a core dump. -- Craig Ringer

Re: [HACKERS] [PATCH] pageinspect function to decode infomasks

2017-07-20 Thread Craig Ringer
On 20 Jul. 2017 19:09, "Ashutosh Sharma" wrote: I had a quick look into this patch and also tested it and following are my observations. Thanks very much. I'll expand the tests to cover various normal and nonsensical masks and combinations and fix the identified issues. This was a quick morni

Re: [HACKERS] [PATCH] A hook for session start

2017-07-20 Thread Craig Ringer
l replication code in pg11 for similar reasons, and so features like DDL replication can be prototyped as extensions more practically). That said, isn't ExecutorStart_hook + ProcessUtility_hook able to serve the same job as a session-start hook, albeit at slightly higher overhead? You can just te

Re: [HACKERS] Better error message for trying to drop a DB with open subscriptions?

2017-07-20 Thread Craig Ringer
prepared xacts separately already, and errdetail_busy_db uses that to report the two separately. Since we have slot attachment data I expect reporting attached replication slots would not be hard either; you might be able to prep a patch for that in a few hours. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] More optimization effort?

2017-07-20 Thread Craig Ringer
7;ve originally matched? I'm not sure it's a blocker, but it bears consideration, and Pg might have to do more work on partial index matching too. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Definitional questions for pg_sequences view

2017-07-20 Thread Craig Ringer
but if it's a view to help users out exposing that would seem sensible. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] xlogfilename

2017-07-20 Thread Craig Ringer
multiple physical files of 16MB which are called > WAL segments. The second 8 characters indicate the id of the logical > xlog file, and the last 8 characters indicate the sequencial number of > the segment in this xlog. > <http://www.postgresql.org/mailpref/pgsql-hackers> > You missed the timeline ID, which is the first 8 digits. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] [PATCH] pageinspect function to decode infomasks

2017-07-20 Thread Craig Ringer
inition, the argument name used is > 'include_combined' whereas in documentation you have mentioned > 'show_combined'. > Fixed, thanks. I want to find time to expand the tests on this some more and look more closely, but here it is for now. -- Craig Ringer

Re: [HACKERS] [PATCH] A hook for session start

2017-07-21 Thread Craig Ringer
On 21 Jul. 2017 21:58, "Yugo Nagata" wrote: On Fri, 21 Jul 2017 10:31:57 -0300 Fabrízio de Royes Mello wrote: > On Fri, Jul 21, 2017 at 9:35 AM, Yugo Nagata wrote: > > > > On Fri, 21 Jul 2017 09:53:19 +0800 > > Craig Ringer wrote: > > > > >

Re: [HACKERS] Syncing sql extension versions with shared library versions

2017-07-22 Thread Craig Ringer
On 22 Jul. 2017 04:19, "Mat Arye" wrote: Hi All, I am developing the TimescaleDB extension for postgres ( https://github.com/timescale/timescaledb) and have some questions about versioning. First of all, I have to say that the versioning system on the sql side is wonderful. It's really simple to

[HACKERS] Re: [SQL] Postgresql “alter column type” creates an event which contains “temp_table_xxx”

2017-07-25 Thread Craig Ringer
in-core logical rep doesn't natively handle truncation yet, and this is one of the things it'd be good to do for pg11, especially if more people get interested in contributing. In the mean time, logical decoding clients can special case "pg_temp_" relation names in thei

Re: [HACKERS] WIP: Failover Slots

2017-07-25 Thread Craig Ringer
On 26 July 2017 at 00:16, Thom Brown wrote: > On 8 April 2016 at 07:13, Craig Ringer wrote: > > On 6 April 2016 at 22:17, Andres Freund wrote: > > > >> > >> Quickly skimming 0001 in [4] there appear to be a number of issues: > >> * LWLockHeldBy

Re: [HACKERS] TAP: allow overriding PostgresNode in get_new_node

2017-07-25 Thread Craig Ringer
er > (if at all). > > Pushed to 9.6 and HEAD. > Thanks. An upvote from our resident Perl wizard certainly does help :) -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Syncing sql extension versions with shared library versions

2017-07-25 Thread Craig Ringer
tension function definitions can >> fail at runtime if funcs are removed or change signature, but won't fail at >> startup or load. >> >> So we let the C extension detect when it's newer than the loaded SQL ext >> during its startup and run an ALTER EXTENSION

Re: [HACKERS] On Complex Source Code Reading Strategy

2017-07-27 Thread Craig Ringer
//wiki.postgresql.org/wiki/Developer_FAQ (some of which need to be added to the "developer information" wiki page I think) -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

[HACKERS] Persistent wait event sets and socket changes

2017-07-30 Thread Craig Ringer
event set when the fd-set changes. So I'm posting mostly to confirm that it's not supposed to work, and ask if anyone thinks I should submit a comment patch to latch.c documenting it. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Effect of dropping a partitioned table's column over time

2017-08-06 Thread Craig Ringer
ntal > materialized views (at least in one proposal) which is why I asked > about it on this list recently[2]. > Can we instead create the new partitions with the same dropped columns? Ensure that every partition, parent and child, has the same column-set? -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] [TRAP: FailedAssertion] causing server to crash

2017-08-06 Thread Craig Ringer
isedb.com> > I think so - specifically, that it's a leftover from a revision where the xid limit was advanced before clog truncation. I'll be finding time in the next couple of days to look more closely and ensure that's all it is. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] WIP: Failover Slots

2017-08-08 Thread Craig Ringer
On 3 August 2017 at 04:35, Robert Haas wrote: > On Tue, Jul 25, 2017 at 8:44 PM, Craig Ringer > wrote: > > No. The whole approach seems to have been bounced from core. I don't > agree > > and continue to think this functionality is desirable but I don't get to >

Re: [HACKERS] Walsender timeouts and large transactions

2017-08-09 Thread Craig Ringer
; > + } > > + if (now <= TimestampTzPlusMilliseconds(last_reply_timestamp, > wal_sender_timeout / 2)) > > + return; > > + } > > > > If not, what problem prevents? > > We should do CHECK_FOR_INTERRUPTS() independently of

Re: [HACKERS] WIP: Failover Slots

2017-08-09 Thread Craig Ringer
On 9 August 2017 at 23:42, Robert Haas wrote: > On Tue, Aug 8, 2017 at 4:00 AM, Craig Ringer > wrote: > >> - When a standby connects to a master, it can optionally supply a list > >> of slot names that it cares about. > > > > Wouldn't that immediately e

Re: [HACKERS] [TRAP: FailedAssertion] causing server to crash

2017-08-10 Thread Craig Ringer
On 10 August 2017 at 23:25, Robert Haas wrote: > On Mon, Aug 7, 2017 at 2:06 AM, Craig Ringer > wrote: > > I think so - specifically, that it's a leftover from a revision where the > > xid limit was advanced before clog truncation. > > > > I'll be finding

Re: [HACKERS] Thoughts on unit testing?

2017-08-10 Thread Craig Ringer
#x27; and move toward higher level visibility (https://gcc.gnu.org/wiki/Visibility: -fvisibility=hidden and __attribute__((dllexport)) ). It'd make it easier not to forget needed PGDLLEXPORTs, let us hide stuff we consider really internal but still share across a few files, etc.) -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] WIP: Failover Slots

2017-08-10 Thread Craig Ringer
vail as https://gist.github.com/ringerc/d4a8fe97f5fd332d8b883d596d61e257 ) To actually use the slot once decoding on standby is supported: a decoding client on "C" can consume xacts and cause slot "X" to advance catalog_xmin, confirmed_flush_lsn, etc. walreceiver on "C" will tell walsender on "B" about the new slot state, and it'll get synced up-tree, then B will tell A. Since slot is already marked permanent, state won't get copied back down-tree, that only happens once when slot is first fully created on master. Some node "D" can exist as a phys rep of "C". If C fails and is replace with D, admin can promote the down-mirror slot on "D" to an owned slot. Make sense? -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Thoughts on unit testing?

2017-08-13 Thread Craig Ringer
t strict unit testing as the rest of Pg's APIs aren't mocked away, but it's very practical small-unit integration testing that helps catch issues. I wouldn't mind having an easier and nicer way to do that built in to Pg, but don't have many ideas about practical, low-maint

Re: [HACKERS] WIP: Failover Slots

2017-08-13 Thread Craig Ringer
too, just as proposed for the physical case, though no replica->master reporting would be needed for logical failover. So despite my initial expectations they can be moderately similar in broad structure. But I don't think there's going to be much actual code overlap beyond minor things like both wanting a way to query slot state on the upstream. Both *could* use decoding on standby to advance slot positions, but for the physical case that's just a slower (and unfinished) way to do what we already have, wheras it's necessary for logical failover. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] [PATCH] pageinspect function to decode infomasks

2017-08-14 Thread Craig Ringer
On 15 August 2017 at 09:11, Moon Insung wrote: > Dear Craig Ringer > > > > Frist, thank you for implementing the necessary function. > > > > but, i have some question. > > > > question 1) vacuum freeze hint bits > > If run a vacuum freeze, bits in

Re: [HACKERS] make async slave to wait for lsn to be replayed

2017-08-14 Thread Craig Ringer
all over pg_stat_replication and pg_replication_slots and so on. They're already routinely used for monitoring replication lag in bytes, waiting for a peer to catch up, etc. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] What users can do with custom ICU collations in Postgres 10

2017-08-14 Thread Craig Ringer
s come first, which we see here: > > Ooh, this finally gives us a path toward case-insensitive default database collation via CLDR caseLevel. http://userguide.icu-project.org/collation http://www.unicode.org/reports/tr35/tr35-collation.html#Algorithm_Case That *definitely* should be

Re: [HACKERS] What users can do with custom ICU collations in Postgres 10

2017-08-14 Thread Craig Ringer
On 15 August 2017 at 10:16, Michael Paquier wrote: > On Tue, Aug 15, 2017 at 11:10 AM, Craig Ringer > wrote: > > Ooh, this finally gives us a path toward case-insensitive default > database > > collation via CLDR caseLevel. > > > > http://userguide.i

Re: [HACKERS] [PATCH] pageinspect function to decode infomasks

2017-08-15 Thread Craig Ringer
erpret the bitmasks (omitting some of > the information) assuming all the bits were set correctly. I agree, and the patch already does half of this: it can output just the raw bit flags, or it can interpret them to show HEAP_XMIN_FROZEN etc. So the required change, which seems to have broad agreement, is to have the "interpret the bits" mode show only HEAP_XMIN_FROZEN when it sees HEAP_XMIN_COMMITTED|HEAP_XMIN_INVALID, etc. We can retain raw-flags output as-is for when seriously bogus state is suspected. Any takers? -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Function to move the position of a replication slot

2017-08-16 Thread Craig Ringer
all xacts, or by lower level use of the decoding code. Reasonable? -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] [PATCH] pageinspect function to decode infomasks

2017-08-16 Thread Craig Ringer
s we have macros, and I think it'd make sense to use them here too. Eschew direct use of HEAP_XMIN_COMMITTED, HEAP_XMIN_INVALID and HEAP_XMIN_FROZEN in tests. Instead, consistently use HeapXminIsFrozen(), HeapXminIsCommitted(), and HeapXminIsInvalid() or something like that. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Function to move the position of a replication slot

2017-08-16 Thread Craig Ringer
hink it's quite a useful > function to be used by an automated system. E.g. to ensure enough, but > not too much, WAL is available for a tertiary standby both on the actual > primary and a failover node. > I strongly agree. If you really need to move a physical slot back (why?)

Re: [HACKERS] [RFC] What would be difficult to make data models pluggable for making PostgreSQL a multi-model database?

2017-08-20 Thread Craig Ringer
tes, like the path model. With some kind of interaction so the sub-planner for the other model could know to generate a different sub-plan based on the context of the outer plan. I have no idea how that could work. But I think you have about zero chance of achieving what you want by going straight

Re: [HACKERS] Updating line length guidelines

2017-08-20 Thread Craig Ringer
fers side by side, or when I'm working in poor conditions where I've set my terminal to "giant old people text" sizes, I remember the advantages of a width limit. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

[HACKERS] shm_mq_wait_internal gets stuck forever on fast shutdown

2017-08-20 Thread Craig Ringer
function pointer called on each iteration to test whether looping should continue, to be passed to shm_mq_attach. So if you can't supply a bgw handle, you supply that instead. Provide a shm_mq_set_handle equivalent for it too. Any objections to the last approach? -- Craig Ringer

Re: [HACKERS] shm_mq_wait_internal gets stuck forever on fast shutdown

2017-08-20 Thread Craig Ringer
On 21 August 2017 at 10:57, Craig Ringer wrote: > Hi all > > I've noticed a possible bug / design limitation where shm_mq_wait_internal > sleep in a latch wait forever, and the postmaster gets stuck waiting for > the bgworker the wait is running in to exit. > > This ha

Re: [HACKERS] walsender & parallelism

2017-04-20 Thread Craig Ringer
y recovery conflicts for logical decoding on standby patch does, FWIW. I haven't found any issues yet.. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] multithreading in Batch/pipelining mode for libpq

2017-04-21 Thread Craig Ringer
On 22 Apr. 2017 6:04 am, "Ilya Roublev" wrote: 1) is it possible technically (possibly by changing some part of libpq code) to ignore results (especially for this sort of queries like insert), processing somehow separately the situation when some error occurs? There is a patch out there to all

Re: [HACKERS] Old versions of Test::More

2017-04-21 Thread Craig Ringer
On 22 Apr. 2017 4:23 am, "Tom Lane" wrote: Peter Eisentraut writes: > On 4/21/17 14:49, Andrew Dunstan wrote: >> I'll add a comment, but doing it in PostgresNode.pm would mean jacana >> (for instance) couldn't run any of the TAP tests. I'mm looking at >> installing a sufficiently modern Test::Si

Re: [HACKERS] A note about debugging TAP failures

2017-04-23 Thread Craig Ringer
On 23 Apr. 2017 10:32, "Michael Paquier" wrote: On Sun, Apr 23, 2017 at 7:48 AM, Daniel Gustafsson wrote: > Skipping the tempdir and instead using ${testname}_data_${name} without a > random suffix, we can achieve this with something along the lines of the > attached PoC. It works as now (retai

Re: [HACKERS] OK, so culicidae is *still* broken

2017-04-24 Thread Craig Ringer
us/library/bb384887.aspx It's rather better than the old registry hack, but it's a compat option we're likely to lose at some point. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hac

Re: [HACKERS] OK, so culicidae is *still* broken

2017-04-24 Thread Craig Ringer
ost.org/doc/libs/1_64_0/doc/html/interprocess/sharedmemorybetweenprocesses.html#interprocess.sharedmemorybetweenprocesses.mapped_region.mapped_region_address_mapping [3] http://stackoverflow.com/a/36145019/398670 [4] https://github.com/golang/go/issues/2323 -- Craig Ringer http://www.2

Re: [HACKERS] A note about debugging TAP failures

2017-04-24 Thread Craig Ringer
;make' target nuke the relevant tmp_check dir before rerunning the tests. Right now it just deletes the logs. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (p

Re: [HACKERS] OK, so culicidae is *still* broken

2017-04-24 Thread Craig Ringer
On 25 Apr. 2017 02:51, "Andres Freund" wrote: On 2017-04-24 11:08:48 -0700, Andres Freund wrote: > On 2017-04-24 23:14:40 +0800, Craig Ringer wrote: > > In the long run we'll probably be forced toward threading or far pointers. > > I'll vote for removing the

Re: [HACKERS] OK, so culicidae is *still* broken

2017-04-25 Thread Craig Ringer
On 25 Apr. 2017 13:37, "Heikki Linnakangas" wrote: For some data shared memory structures, that store no pointers, we wouldn't need to insist that they are mapped to the same address in every backend, though. In particular, shared_buffers. It wouldn't eliminate the problem, though, only make it

Re: [HACKERS] OK, so culicidae is *still* broken

2017-04-25 Thread Craig Ringer
On 25 April 2017 at 22:07, Tom Lane wrote: > Craig Ringer writes: >> On 25 Apr. 2017 13:37, "Heikki Linnakangas" wrote: >>> For some data shared memory structures, that store no pointers, we wouldn't >>> need to insist that they are mapped to the sam

Re: [HACKERS] [PostgreSQL 10] default of hot_standby should be "on"?

2017-04-25 Thread Craig Ringer
start standby with -W option. Yeah. That's a good reason to change it. I think at this point warm standby is clearly the less-used secondary option and hot_standby should be default. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training

Re: [HACKERS] Separation walsender & normal backends

2017-04-25 Thread Craig Ringer
nd logical decoding on normal backends, allow normal backends to be signaled when there's new WAL, etc. I think there's a fair bit to do in order to do this well though. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training &

Re: [HACKERS] On How To Shorten the Steep Learning Curve Towards PG Hacking...

2017-04-28 Thread Craig Ringer
On 28 Apr. 2017 17:04, "Kang Yuzhe" wrote: Hello Simon, The journey that caused and is causing me a lot of pain is finding my way in PG development. Complex Code Reading like PG. Fully understanding the science of DBMS Engines: Query Processing, Storage stuff, Transaction Management and so on...

Re: [HACKERS] CTE inlining

2017-04-29 Thread Craig Ringer
On 30 Apr. 2017 07:56, "Ilya Shkuratov" wrote: Hello, dear hackers! There is task in todo list about optional CTE optimization fence disabling. I am not interested at this point in disabling mechanism implementation, but I would like to discuss the optimization mechanism, that should work when

Re: [HACKERS] CTE inlining

2017-04-29 Thread Craig Ringer
On 30 Apr. 2017 13:28, "Andres Freund" wrote: On 2017-04-30 00:28:46 -0400, Tom Lane wrote: > There's already a pretty large hill to climb here in the way of > breaking peoples' expectations about CTEs being optimization > fences. Breaking the documented semantics about CTEs being > single-evalu

[HACKERS] Newbie project: Detect attempts to run pg_dump etc on psql, \copy on server

2017-04-30 Thread Craig Ringer
s. But I see users getting confused by psql metacommands vs postgres server commands a lot. Similarly by pg_dump and pg_restore being shell programs. If it can be done at minimal cost to everyone else, it might be worth it. Just an idea. -- Craig Ringer http://www.2ndQuadrant

Re: [HACKERS] snapbuild woes

2017-04-30 Thread Craig Ringer
appears before the xl_running_xacts, but the xl_running_xacts still has the commited xact listed as running, right? Because we update PGXACT only after we write the commit to WAL, so there's a window where an xact is committed in WAL but not shown as committed in shmem. -- Craig Ringer

Re: [HACKERS] CTE inlining

2017-05-01 Thread Craig Ringer
query hint. And yes, that's what it is, because we'd only inline when we could produce semantically equivalent results anyway. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing l

Re: [HACKERS] CTE inlining

2017-05-01 Thread Craig Ringer
t's what we do right now so we can pretend we don't have query hints while still having query hints. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgr

Re: [HACKERS] CTE inlining

2017-05-01 Thread Craig Ringer
uot; and wonder why we haven't fixed this limitation yet, viewing it as a bug not a feature. The same logic being applied here should've prevented us from ever introducing: * inlining of SQL functions * inlining of views * inlining of subqueries ... but somehow, this one is different

<    2   3   4   5   6   7   8   9   10   11   >