Re: Binary support for pgoutput plugin

Petr Jelinek writes: > On 20/07/2020 17:51, Tom Lane wrote: >> I'm fixing that, but even after that, there's a semantic problem: >> LOGICALREP_COLUMN_UNCHANGED is just a weak optimization, cf the code >> that sends it, in proto.c around line 480. colstatus will often *not* >> be that for columns

Re: Binary support for pgoutput plugin

Hi, On 20/07/2020 17:51, Tom Lane wrote: Peter Geoghegan writes: Skink's latest run reports a failure that I surmise was caused by this patch: Yeah, I've just been digging through that. The patch didn't create the bug, but it allowed valgrind to detect it, because the column status array is

Re: Binary support for pgoutput plugin

Peter Geoghegan writes: > Skink's latest run reports a failure that I surmise was caused by this patch: Yeah, I've just been digging through that. The patch didn't create the bug, but it allowed valgrind to detect it, because the column status array is now "just big enough" rather than being alw

Re: Binary support for pgoutput plugin

On Sat, Jul 18, 2020 at 9:53 AM Tom Lane wrote: > I've pushed this patch, with a number of adjustments, some cosmetic > and some not so much (no pg_dump support!?). We're not quite > done though ... Skink's latest run reports a failure that I surmise was caused by this patch: ==722318== VALGRIN

Re: Binary support for pgoutput plugin

I've pushed this patch, with a number of adjustments, some cosmetic and some not so much (no pg_dump support!?). We're not quite done though ... Dave Cramer writes: > So looking at how to confirm that the subscriber has receive functions for > all of the types. > AFAICT we don't have that inform

Re: Binary support for pgoutput plugin

Working through this ... what is the rationale for having changed the API of logicalrep_read_update? It seems kind of random, especially since no comparable change was made to logicalrep_read_insert. If there's actually a good reason, it seems like it'd apply to both. If there's not, I'd be incl

Re: Binary support for pgoutput plugin

On Tue, 14 Jul 2020 at 22:47, Andres Freund wrote: > Hi, > > On 2020-07-14 22:28:48 -0400, Tom Lane wrote: > > Andres Freund writes: > > > What is the gain in having these checks? recv functions need to be safe > > > against arbitrary input, so a type crosscheck doesn't buy additional > > > safe

Re: Binary support for pgoutput plugin

Hi, On 2020-07-14 22:28:48 -0400, Tom Lane wrote: > Andres Freund writes: > > What is the gain in having these checks? recv functions need to be safe > > against arbitrary input, so a type crosscheck doesn't buy additional > > safety in that regard. Not that a potential attacker couldn't just > >

Re: Binary support for pgoutput plugin

Andres Freund writes: > What is the gain in having these checks? recv functions need to be safe > against arbitrary input, so a type crosscheck doesn't buy additional > safety in that regard. Not that a potential attacker couldn't just > change the content anyways? You're confusing security issue

Re: Binary support for pgoutput plugin

Hi, On 2020-07-14 19:46:52 -0400, Tom Lane wrote: > Andres Freund writes: > > There's also send/receive functions that do not work across systems, > > unfortunately :(. In particular record and array send functions embed > > type oids and their receive functions verify that they match the local >

Re: Binary support for pgoutput plugin

Andres Freund writes: > There's also send/receive functions that do not work across systems, > unfortunately :(. In particular record and array send functions embed > type oids and their receive functions verify that they match the local > system. Which basically means that if there's any differen

Re: Binary support for pgoutput plugin

Hi, On 2020-07-14 14:08:53 -0400, Dave Cramer wrote: > On Tue, 14 Jul 2020 at 12:59, Tom Lane wrote: > > > So I started looking through this seriously, and my first question > > is why do the docs and code keep saying that "base types" are sent > > in binary? Why not just "data"? Are there any

Re: Binary support for pgoutput plugin

Dave Cramer writes: > On Tue, 14 Jul 2020 at 12:59, Tom Lane wrote: >> So I started looking through this seriously, and my first question >> is why do the docs and code keep saying that "base types" are sent >> in binary? Why not just "data"? Are there any cases where we >> don't use binary for

Re: Binary support for pgoutput plugin

On Tue, 14 Jul 2020 at 12:59, Tom Lane wrote: > So I started looking through this seriously, and my first question > is why do the docs and code keep saying that "base types" are sent > in binary? Why not just "data"? Are there any cases where we > don't use binary format, if the subscription r

Re: Binary support for pgoutput plugin

So I started looking through this seriously, and my first question is why do the docs and code keep saying that "base types" are sent in binary? Why not just "data"? Are there any cases where we don't use binary format, if the subscription requests it? If there's not a concrete reason to use tha

Re: Binary support for pgoutput plugin

On Tue, 14 Jul 2020 at 09:26, Daniel Gustafsson wrote: > > On 13 Jul 2020, at 15:11, Dave Cramer wrote: > > I took another look at the updated version today. Since there now were > some > unused variables and (I believe) unnecessary checks (int size and > endianness > etc) left, I took the libe

Re: Binary support for pgoutput plugin

> On 13 Jul 2020, at 15:11, Dave Cramer wrote: I took another look at the updated version today. Since there now were some unused variables and (I believe) unnecessary checks (int size and endianness etc) left, I took the liberty to fix those. I also fixed some markup in the catalog docs, did s

Re: Binary support for pgoutput plugin

On Sat, 11 Jul 2020 at 10:20, Tom Lane wrote: > Petr Jelinek writes: > > On 11/07/2020 14:14, Dave Cramer wrote: > >> So is there any point in having them as options then ? > > > I am guessing this is copied from pglogical, right? We have them there > > because it can optionally send data in the

Re: Binary support for pgoutput plugin

Petr Jelinek writes: > On 11/07/2020 14:14, Dave Cramer wrote: >> So is there any point in having them as options then ? > I am guessing this is copied from pglogical, right? We have them there > because it can optionally send data in the on-disk format (not the > network binary format) and the

Re: Binary support for pgoutput plugin

Hi, On 11/07/2020 14:14, Dave Cramer wrote: On Fri, 10 Jul 2020 at 14:21, Tom Lane > wrote: > Reading through the new patch, and running the tests, I'm marking this as Ready > for Committer.  It does need some cosmetic TLC, quite possibly just fro

Re: Binary support for pgoutput plugin

On Fri, 10 Jul 2020 at 14:21, Tom Lane wrote: > Daniel Gustafsson writes: > > Thanks for the update! Do note that my patch included a new file which > is > > missing from this patchset: > > src/test/subscription/t/014_binary.pl > > This is, IMO, the most interesting test of this feature s

Re: Binary support for pgoutput plugin

Daniel Gustafsson writes: > Thanks for the update! Do note that my patch included a new file which is > missing from this patchset: > src/test/subscription/t/014_binary.pl > This is, IMO, the most interesting test of this feature so it would be good to > be included. It's a basic test and

Re: Binary support for pgoutput plugin

> On 7 Jul 2020, at 22:53, Dave Cramer wrote: > I have put all your requests other than the indentation as that can be dealt > with by pg_indent into another patch which I reordered ahead of yours > This should make it easier to see that all of your issues have been addressed. Thanks for the up

Re: Binary support for pgoutput plugin

On Tue, 7 Jul 2020 at 10:01, Daniel Gustafsson wrote: > > On 7 Jul 2020, at 02:16, Dave Cramer wrote: > > > OK, rebased it down to 2 patches, attached. > > I took a look at this patchset today. The feature clearly seems like > something > which we'd benefit from having, especially if it allows

Re: Binary support for pgoutput plugin

> On 7 Jul 2020, at 02:16, Dave Cramer wrote: > OK, rebased it down to 2 patches, attached. I took a look at this patchset today. The feature clearly seems like something which we'd benefit from having, especially if it allows for the kind of extensions that were discussed at the beginning of t

Re: Binary support for pgoutput plugin

On Mon, 6 Jul 2020 at 09:35, Dave Cramer wrote: > > > On Mon, 6 Jul 2020 at 09:03, Daniel Gustafsson wrote: > >> > On 6 Jul 2020, at 14:58, Dave Cramer wrote: >> >> > as far as rebase -i do what is advised here for squashing them. Just >> one patch now ? >> >> One patch per logical change, if t

Re: Binary support for pgoutput plugin

On Mon, 6 Jul 2020 at 09:03, Daniel Gustafsson wrote: > > On 6 Jul 2020, at 14:58, Dave Cramer wrote: > > > as far as rebase -i do what is advised here for squashing them. Just one > patch now ? > > One patch per logical change, if there are two disjoint changes in the > patchset > where one bui

Re: Binary support for pgoutput plugin

> On 6 Jul 2020, at 14:58, Dave Cramer wrote: > as far as rebase -i do what is advised here for squashing them. Just one > patch now ? One patch per logical change, if there are two disjoint changes in the patchset where one builds on top of the other then multiple patches are of course fine. M

Re: Binary support for pgoutput plugin

On Sun, 5 Jul 2020 at 17:28, Daniel Gustafsson wrote: > > On 5 Jul 2020, at 23:11, Alvaro Herrera > wrote: > > > > On 2020-Jul-05, Daniel Gustafsson wrote: > > > >>> On 2 Jul 2020, at 18:41, Dave Cramer wrote: > >>> > >>> rebased > >> > >> Thanks! The new version of 0001 patch has a compiler w

Re: Binary support for pgoutput plugin

> On 5 Jul 2020, at 23:11, Alvaro Herrera wrote: > > On 2020-Jul-05, Daniel Gustafsson wrote: > >>> On 2 Jul 2020, at 18:41, Dave Cramer wrote: >>> >>> rebased >> >> Thanks! The new version of 0001 patch has a compiler warning due to mixed >> declarations and code: >> >> worker.c: In functi

Re: Binary support for pgoutput plugin

On 2020-Jul-05, Daniel Gustafsson wrote: > > On 2 Jul 2020, at 18:41, Dave Cramer wrote: > > > > rebased > > Thanks! The new version of 0001 patch has a compiler warning due to mixed > declarations and code: > > worker.c: In function ‘slot_store_data’: > worker.c:366:5: error: ISO C90 forbids

Re: Binary support for pgoutput plugin

> On 2 Jul 2020, at 18:41, Dave Cramer wrote: > > rebased Thanks! The new version of 0001 patch has a compiler warning due to mixed declarations and code: worker.c: In function ‘slot_store_data’: worker.c:366:5: error: ISO C90 forbids mixed declarations and code [-Werror=declaration-after-sta

Re: Binary support for pgoutput plugin

rebased Thanks, Dave Cramer On Wed, 1 Jul 2020 at 06:43, Dave Cramer wrote: > Honestly I'm getting a little weary of fixing this up only to have the > patch not get reviewed. > > Apparently it has no value so unless someone is willing to review it and > get it committed I'm just going to let

Re: Binary support for pgoutput plugin

Honestly I'm getting a little weary of fixing this up only to have the patch not get reviewed. Apparently it has no value so unless someone is willing to review it and get it committed I'm just going to let it go. Thanks, Dave Cramer On Wed, 1 Jul 2020 at 04:53, Daniel Gustafsson wrote: > >

Re: Binary support for pgoutput plugin

> On 7 Apr 2020, at 21:45, Dave Cramer wrote: > New patch that fixes a number of errors in the check for validity as well as > reduces the memory usage by > dynamically allocating the data changed as well as collapsing the changed and > binary arrays into a format array. The 0001 patch fails t

Re: Binary support for pgoutput plugin

On Fri, 3 Apr 2020 at 16:44, Dave Cramer wrote: > > > On Fri, 3 Apr 2020 at 03:43, Petr Jelinek wrote: > >> Hi, >> >> On 08/03/2020 00:18, Dave Cramer wrote: >> > On Fri, 6 Mar 2020 at 08:54, Petr Jelinek > > > wrote: >> > >> > Hi Dave, >> > >> > On 29/02/202

Re: Binary support for pgoutput plugin

On Fri, 3 Apr 2020 at 03:43, Petr Jelinek wrote: > Hi, > > On 08/03/2020 00:18, Dave Cramer wrote: > > On Fri, 6 Mar 2020 at 08:54, Petr Jelinek > > wrote: > > > > Hi Dave, > > > > On 29/02/2020 18:44, Dave Cramer wrote: > > > > > > > > > rebas

Re: Binary support for pgoutput plugin

Hi, On 08/03/2020 00:18, Dave Cramer wrote: On Fri, 6 Mar 2020 at 08:54, Petr Jelinek > wrote: Hi Dave, On 29/02/2020 18:44, Dave Cramer wrote: > > > rebased and removed the catversion bump. Looked into this and it generally seems okay,

Re: Binary support for pgoutput plugin

On Fri, 6 Mar 2020 at 08:54, Petr Jelinek wrote: > Hi Dave, > > On 29/02/2020 18:44, Dave Cramer wrote: > > > > > > rebased and removed the catversion bump. > > Looked into this and it generally seems okay, but I do have one gripe here: > > > + tuple->values[i]

Re: Binary support for pgoutput plugin

Hi Dave, On 29/02/2020 18:44, Dave Cramer wrote: rebased and removed the catversion bump. Looked into this and it generally seems okay, but I do have one gripe here: + tuple->values[i].data = palloc(len + 1); + /*

Re: Binary support for pgoutput plugin

On Fri, 28 Feb 2020 at 18:34, Tom Lane wrote: > Dave Cramer writes: > > Rebased against head > > The cfbot's failing to apply this [1]. It looks like the reason is only > that you included a catversion bump in what you submitted. Protocol is to > *not* do that in submitted patches, but rely on

Re: Binary support for pgoutput plugin

Dave Cramer writes: > Rebased against head The cfbot's failing to apply this [1]. It looks like the reason is only that you included a catversion bump in what you submitted. Protocol is to *not* do that in submitted patches, but rely on the committer to add it at the last minute --- otherwise y

Re: Binary support for pgoutput plugin

On Mon, 2 Dec 2019 at 14:35, Dave Cramer wrote: > Rebased against head > > Dave Cramer > > > On Sat, 30 Nov 2019 at 20:48, Michael Paquier wrote: > >> Hi, >> >> On Mon, Nov 11, 2019 at 03:24:59PM -0500, Dave Cramer wrote: >> > Attached, >> >> The latest patch set does not apply correctly. Could

Re: Binary support for pgoutput plugin

Rebased against head Dave Cramer On Sat, 30 Nov 2019 at 20:48, Michael Paquier wrote: > Hi, > > On Mon, Nov 11, 2019 at 03:24:59PM -0500, Dave Cramer wrote: > > Attached, > > The latest patch set does not apply correctly. Could you send a > rebase please? I am moving the patch to next CF, wa

Re: Binary support for pgoutput plugin

Hi, On Mon, Nov 11, 2019 at 03:24:59PM -0500, Dave Cramer wrote: > Attached, The latest patch set does not apply correctly. Could you send a rebase please? I am moving the patch to next CF, waiting on author. -- Michael signature.asc Description: PGP signature

Re: Binary support for pgoutput plugin

On Mon, 11 Nov 2019 at 15:17, Alvaro Herrera wrote: > On 2019-Nov-11, Dave Cramer wrote: > > > Following 2 patches address Dmitry's concern and check for compatibility. > > Please resend the whole patchset, so that the patch tester can verify > the series. (Doing it helps humans, too). > > Attac

Re: Binary support for pgoutput plugin

On 2019-Nov-11, Dave Cramer wrote: > Following 2 patches address Dmitry's concern and check for compatibility. Please resend the whole patchset, so that the patch tester can verify the series. (Doing it helps humans, too). -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQ

Re: Binary support for pgoutput plugin

> On Mon, Nov 11, 2019 at 11:15:45AM -0500, Dave Cramer wrote: > On Fri, 8 Nov 2019 at 11:20, Dmitry Dolgov <9erthali...@gmail.com> wrote: > > > > On Tue, Nov 05, 2019 at 07:16:10AM -0500, Dave Cramer wrote: > > > > > > See attached > > > > --- a/src/backend/replication/logical/worker.c > >

Re: Binary support for pgoutput plugin

On Mon, 11 Nov 2019 at 12:07, Dave Cramer wrote: > > > On Mon, 11 Nov 2019 at 12:04, Alvaro Herrera > wrote: > >> On 2019-Nov-11, Dave Cramer wrote: >> >> > Previously someone mentioned that we need to confirm whether the two >> > servers are compatible for binary or not. >> > >> > Checking to m

Re: Binary support for pgoutput plugin

On Mon, 11 Nov 2019 at 12:04, Alvaro Herrera wrote: > On 2019-Nov-11, Dave Cramer wrote: > > > Previously someone mentioned that we need to confirm whether the two > > servers are compatible for binary or not. > > > > Checking to make sure the two servers have the same endianness is > obvious. >

Re: Binary support for pgoutput plugin

On 2019-Nov-11, Dave Cramer wrote: > Previously someone mentioned that we need to confirm whether the two > servers are compatible for binary or not. > > Checking to make sure the two servers have the same endianness is obvious. > Sizeof int, long, float, double, timestamp (float/int) at a minimu

Re: Binary support for pgoutput plugin

On Fri, 8 Nov 2019 at 11:20, Dmitry Dolgov <9erthali...@gmail.com> wrote: > > On Tue, Nov 05, 2019 at 07:16:10AM -0500, Dave Cramer wrote: > > > > See attached > > --- a/src/backend/replication/logical/worker.c > +++ b/src/backend/replication/logical/worker.c > @@ -1779,6 +1779,7 @@ Ap

Re: Binary support for pgoutput plugin

> On Tue, Nov 05, 2019 at 07:16:10AM -0500, Dave Cramer wrote: > > See attached --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -1779,6 +1779,7 @@ ApplyWorkerMain(Datum main_arg) options.slotname = myslotname; options.pro

Re: Binary support for pgoutput plugin

On Sun, 3 Nov 2019 at 21:47, Thomas Munro wrote: > On Thu, Oct 31, 2019 at 3:03 AM Dave Cramer wrote: > > Ok, I've rebased and reverted logicalrep_read_insert > > Hi Dave, > > From the code style police (actually just from cfbot, which is set up > to complain about declarations after statements,

Re: Binary support for pgoutput plugin

On Thu, Oct 31, 2019 at 3:03 AM Dave Cramer wrote: > Ok, I've rebased and reverted logicalrep_read_insert Hi Dave, >From the code style police (actually just from cfbot, which is set up to complain about declarations after statements, a bit of C99 we aren't ready for): proto.c:557:6: error: ISO

Re: Binary support for pgoutput plugin

On Sun, 27 Oct 2019 at 11:00, Dmitry Dolgov <9erthali...@gmail.com> wrote: > > On Mon, Jun 17, 2019 at 10:29:26AM -0400, Dave Cramer wrote: > > > Which is what I have done. Thanks > > > > > > I've attached both patches for comments. > > > I still have to add documentation. > > > > Additional patch

Re: Binary support for pgoutput plugin

> On Mon, Jun 17, 2019 at 10:29:26AM -0400, Dave Cramer wrote: > > Which is what I have done. Thanks > > > > I've attached both patches for comments. > > I still have to add documentation. > > Additional patch for documentation. Thank you for the patch! Unfortunately 0002 has some conflicts, could

Re: Binary support for pgoutput plugin

On Wed, 5 Jun 2019 at 18:50, Andres Freund wrote: > Hi, > > On 2019-06-05 18:47:57 -0400, Dave Cramer wrote: > > So one of the things they would like added is to get not null information > > in the schema record. This is so they can mark the field Optional in > Java. > > I presume this would also

Re: Binary support for pgoutput plugin

On Fri, 14 Jun 2019 at 15:42, Dave Cramer wrote: > > Dave Cramer > > > On Fri, 14 Jun 2019 at 14:36, Tomas Vondra > wrote: > >> On Wed, Jun 12, 2019 at 10:35:48AM -0400, Dave Cramer wrote: >> >On Mon, 10 Jun 2019 at 07:49, Petr Jelinek > > >> >wrote: >> > >> >> Hi, >> >> >> >> On 10/06/2019 13:2

Re: Binary support for pgoutput plugin

Dave Cramer On Fri, 14 Jun 2019 at 14:36, Tomas Vondra wrote: > On Wed, Jun 12, 2019 at 10:35:48AM -0400, Dave Cramer wrote: > >On Mon, 10 Jun 2019 at 07:49, Petr Jelinek > >wrote: > > > >> Hi, > >> > >> On 10/06/2019 13:27, Dave Cramer wrote: > >> > So back to binary output. > >> > > >> > Fro

Re: Binary support for pgoutput plugin

On Wed, Jun 12, 2019 at 10:35:48AM -0400, Dave Cramer wrote: On Mon, 10 Jun 2019 at 07:49, Petr Jelinek wrote: Hi, On 10/06/2019 13:27, Dave Cramer wrote: > So back to binary output. > > From what I can tell the place to specify binary options would be in the > create publication and or in re

Re: Binary support for pgoutput plugin

On Mon, 10 Jun 2019 at 07:49, Petr Jelinek wrote: > Hi, > > On 10/06/2019 13:27, Dave Cramer wrote: > > So back to binary output. > > > > From what I can tell the place to specify binary options would be in the > > create publication and or in replication slots? > > > > The challenge as I see it

Re: Binary support for pgoutput plugin

OK, before I go too much further down this rabbit hole I'd like feedback on the current code. See attached patch There is one obvious hack where in binary mode I reset the input cursor to allow the binary input to be re-read >From what I can tell the alternative is to convert the data in logicalre

Re: Binary support for pgoutput plugin

Hi, On 10/06/2019 13:27, Dave Cramer wrote: > So back to binary output. > > From what I can tell the place to specify binary options would be in the > create publication and or in replication slots? > > The challenge as I see it is that the subscriber would have to be able > to decode binary out

Re: Binary support for pgoutput plugin

So back to binary output. >From what I can tell the place to specify binary options would be in the create publication and or in replication slots? The challenge as I see it is that the subscriber would have to be able to decode binary output. Any thoughts on how to handle this? At the moment I'

Re: Binary support for pgoutput plugin

On Sat, Jun 08, 2019 at 08:40:43PM -0400, Dave Cramer wrote: On Sat, 8 Jun 2019 at 20:09, Andres Freund wrote: Hi, On 2019-06-08 19:41:34 -0400, Dave Cramer wrote: > So the reason we are discussing using pgoutput plugin is because it is part > of core and guaranteed to be in cloud providers s

Re: Binary support for pgoutput plugin

On Sat, 8 Jun 2019 at 20:09, Andres Freund wrote: > Hi, > > On 2019-06-08 19:41:34 -0400, Dave Cramer wrote: > > So the reason we are discussing using pgoutput plugin is because it is > part > > of core and guaranteed to be in cloud providers solutions. > > IMO people needing this should then ban

Re: Binary support for pgoutput plugin

Hi, On 2019-06-08 19:41:34 -0400, Dave Cramer wrote: > So the reason we are discussing using pgoutput plugin is because it is part > of core and guaranteed to be in cloud providers solutions. IMO people needing this should then band together and write one that's suitable, rather than trying to co

Binary support for pgoutput plugin

This should have gone to hackers as well -- Forwarded message - From: Dave Cramer Date: Sat, Jun 8, 2019, 6:41 PM Subject: Re: Binary support for pgoutput plugin To: Tomas Vondra On Sat, Jun 8, 2019, 6:27 PM Tomas Vondra, wrote: > On Fri, Jun 07, 2019 at 06:01:12PM -0

Re: Binary support for pgoutput plugin

On Fri, Jun 07, 2019 at 06:01:12PM -0700, Andres Freund wrote: Hi, On 2019-06-07 20:52:38 -0400, Chapman Flack wrote: It seems they had ended up designing a whole 'nother "protocol level" involving queries wrapping their results as JSON and an app layer that unwraps again, after trying a simple

Re: Binary support for pgoutput plugin

Hi, On 2019-06-07 21:16:12 -0400, Chapman Flack wrote: > On 06/07/19 21:01, Andres Freund wrote: > > On 2019-06-07 20:52:38 -0400, Chapman Flack wrote: > > That'd be a *lot* of additional complexity, and pretty much prohibitive > > from a performance POV. We'd have to not continue decoding on the

Re: Binary support for pgoutput plugin

On 06/07/19 21:01, Andres Freund wrote: > On 2019-06-07 20:52:38 -0400, Chapman Flack wrote: >> It seems they had ended up designing a whole 'nother "protocol level" >> involving queries wrapping their results as JSON and an app layer that >> unwraps again, after trying a simpler first approach tha

Re: Binary support for pgoutput plugin

Hi, On 2019-06-07 20:52:38 -0400, Chapman Flack wrote: > It seems they had ended up designing a whole 'nother "protocol level" > involving queries wrapping their results as JSON and an app layer that > unwraps again, after trying a simpler first approach that was foiled by the > inability to see i

Re: Binary support for pgoutput plugin

On 06/07/19 19:27, Andres Freund wrote: > The problem is that I don't recognize a limiting principle: > > If we want NOT NULL information for clients, why don't we include the > underlying types for arrays, and the fields in composite types? What > about foreign keys? And unique keys? This remind

Re: Binary support for pgoutput plugin

Hi, On 2019-06-05 19:05:05 -0400, Dave Cramer wrote: > I am curious why you are "strongly" opposed however. We already have the > information. Adding doesn't seem onerous. (thought I'd already replied with this) The problem is that I don't recognize a limiting principle: If we want NOT NULL inf

Re: Binary support for pgoutput plugin

Hi, On Wed, 5 Jun 2019 at 18:50, Andres Freund wrote: > Hi, > > On 2019-06-05 18:47:57 -0400, Dave Cramer wrote: > > So one of the things they would like added is to get not null information > > in the schema record. This is so they can mark the field Optional in > Java. > > I presume this woul

Re: Binary support for pgoutput plugin

Hi, On 2019-06-05 18:47:57 -0400, Dave Cramer wrote: > So one of the things they would like added is to get not null information > in the schema record. This is so they can mark the field Optional in Java. > I presume this would also have some uses in other languages. As I > understand it this wou

Re: Binary support for pgoutput plugin

Hi, On Wed, 5 Jun 2019 at 12:01, Andres Freund wrote: > Hi > > On June 5, 2019 8:51:10 AM PDT, Dave Cramer wrote: > >On Wed, 5 Jun 2019 at 07:21, Dave Cramer wrote: > > > >> Hi, > >> > >> > >> On Wed, 5 Jun 2019 at 07:18, Petr Jelinek > > > >> wrote: > >> > >>> Hi, > >>> > >>> On 05/06/2019 00

Re: Binary support for pgoutput plugin

Hi On June 5, 2019 8:51:10 AM PDT, Dave Cramer wrote: >On Wed, 5 Jun 2019 at 07:21, Dave Cramer wrote: > >> Hi, >> >> >> On Wed, 5 Jun 2019 at 07:18, Petr Jelinek > >> wrote: >> >>> Hi, >>> >>> On 05/06/2019 00:08, Andres Freund wrote: >>> > Hi, >>> > >>> > On 2019-06-05 00:05:02 +0200, David Fe

Re: Binary support for pgoutput plugin

On Wed, 5 Jun 2019 at 07:21, Dave Cramer wrote: > Hi, > > > On Wed, 5 Jun 2019 at 07:18, Petr Jelinek > wrote: > >> Hi, >> >> On 05/06/2019 00:08, Andres Freund wrote: >> > Hi, >> > >> > On 2019-06-05 00:05:02 +0200, David Fetter wrote: >> >> Would it make sense to work toward a binary format th

Re: Binary support for pgoutput plugin

Hi, On Wed, 5 Jun 2019 at 07:18, Petr Jelinek wrote: > Hi, > > On 05/06/2019 00:08, Andres Freund wrote: > > Hi, > > > > On 2019-06-05 00:05:02 +0200, David Fetter wrote: > >> Would it make sense to work toward a binary format that's not > >> architecture-specific? I recall from COPY that our b

Re: Binary support for pgoutput plugin

Hi, On 05/06/2019 00:08, Andres Freund wrote: > Hi, > > On 2019-06-05 00:05:02 +0200, David Fetter wrote: >> Would it make sense to work toward a binary format that's not >> architecture-specific? I recall from COPY that our binary format is >> not standardized across, for example, big- and littl

Re: Binary support for pgoutput plugin

On Tue, 4 Jun 2019 at 18:08, Andres Freund wrote: > Hi, > > On 2019-06-05 00:05:02 +0200, David Fetter wrote: > > Would it make sense to work toward a binary format that's not > > architecture-specific? I recall from COPY that our binary format is > > not standardized across, for example, big- an

Re: Binary support for pgoutput plugin

Hi, On 2019-06-05 00:05:02 +0200, David Fetter wrote: > Would it make sense to work toward a binary format that's not > architecture-specific? I recall from COPY that our binary format is > not standardized across, for example, big- and little-endian machines. I think you recall wrongly. It's obv

Re: Binary support for pgoutput plugin

On Tue, Jun 04, 2019 at 04:55:33PM -0400, Dave Cramer wrote: > On Tue, 4 Jun 2019 at 16:46, Andres Freund wrote: > > > Hi, > > > > On 2019-06-04 16:39:32 -0400, Dave Cramer wrote: > > > On Tue, 4 Jun 2019 at 16:30, Andres Freund < > > andres.fre...@enterprisedb.com> > > > wrote: > > > > > There's

Re: Binary support for pgoutput plugin

On 6/4/19 4:39 PM, Dave Cramer wrote: > I haven't really thought this through completely but one place JDBC has > problems with binary is with > timestamps with timezone as we don't know which timezone to use. Is it safe > to assume everything is in UTC > since the server stores in UTC ? PL/Java,

Re: Binary support for pgoutput plugin

On Tue, 4 Jun 2019 at 16:46, Andres Freund wrote: > Hi, > > On 2019-06-04 16:39:32 -0400, Dave Cramer wrote: > > On Tue, 4 Jun 2019 at 16:30, Andres Freund < > andres.fre...@enterprisedb.com> > > wrote: > > > > There's also no reason that I am aware that binary outputs can't be > > > > supported.

Re: Binary support for pgoutput plugin

Hi, On 2019-06-04 16:39:32 -0400, Dave Cramer wrote: > On Tue, 4 Jun 2019 at 16:30, Andres Freund > wrote: > > > There's also no reason that I am aware that binary outputs can't be > > > supported. > > > > Well, it *does* increase version dependencies, and does make replication > > more complicat

Re: Binary support for pgoutput plugin

Dave Cramer On Tue, 4 Jun 2019 at 16:30, Andres Freund wrote: > Hi, > > On 2019-06-04 15:47:04 -0400, Dave Cramer wrote: > > On Mon, 3 Jun 2019 at 20:54, David Fetter wrote: > > > > > On Mon, Jun 03, 2019 at 10:49:54AM -0400, Dave Cramer wrote: > > > > Is there a reason why pgoutput sends data

Re: Binary support for pgoutput plugin

Hi, On 2019-06-04 15:47:04 -0400, Dave Cramer wrote: > On Mon, 3 Jun 2019 at 20:54, David Fetter wrote: > > > On Mon, Jun 03, 2019 at 10:49:54AM -0400, Dave Cramer wrote: > > > Is there a reason why pgoutput sends data in text format? Seems to > > > me that sending data in binary would provide a

Re: Binary support for pgoutput plugin

Dave Cramer On Mon, 3 Jun 2019 at 20:54, David Fetter wrote: > On Mon, Jun 03, 2019 at 10:49:54AM -0400, Dave Cramer wrote: > > Is there a reason why pgoutput sends data in text format? Seems to > > me that sending data in binary would provide a considerable > > performance improvement. > > Are

Re: Binary support for pgoutput plugin

On Mon, Jun 03, 2019 at 10:49:54AM -0400, Dave Cramer wrote: > Is there a reason why pgoutput sends data in text format? Seems to > me that sending data in binary would provide a considerable > performance improvement. Are you seeing something that suggests that the text output is taking a lot of

Binary support for pgoutput plugin

Is there a reason why pgoutput sends data in text format? Seems to me that sending data in binary would provide a considerable performance improvement. Dave Cramer