Re: Extensibility of the PostgreSQL wire protocol

2021-03-04 Thread Jan Wieck
On 3/4/21 7:38 PM, Hannu Krosing wrote: On Thu, Mar 4, 2021 at 9:55 PM Jan Wieck wrote: but the whole thing was developed that way from the beginning and it is working. I don't have a definitive date when that code will be presented. Kuntal or Prateek may be able to fill in more details. Are

Re: Extensibility of the PostgreSQL wire protocol

2021-03-04 Thread Hannu Krosing
On Thu, Mar 4, 2021 at 9:55 PM Jan Wieck wrote: > > Another possibility, and this is what is being used by the AWS team > implementing the TDS protocol for Babelfish, is to completely replace > the entire TCOP mainloop function PostgresMain(). I suspect this is the only reasonable way to do it fo

Re: Extensibility of the PostgreSQL wire protocol

2021-03-04 Thread Jan Wieck
On 3/3/21 2:43 PM, Peter Eisentraut wrote: I think, the way the abstractions are chosen in this patch, it is still very much tied to how the libpq protocol works. For example, there is a cancel key and a ready-for-query message. Some of the details of the simple and the extended query are expo

Re: Extensibility of the PostgreSQL wire protocol

2021-03-03 Thread Hannu Krosing
I have not looked at the actual patch, but does it allow you to set up its own channels to listen to ? For example if I'd want to set up a server to listen to incoming connections over QUIC [1] - a protocol which create a connection over UDP and allows clients to move to new IP addresses (among ot

Re: Extensibility of the PostgreSQL wire protocol

2021-03-03 Thread Peter Eisentraut
I think, the way the abstractions are chosen in this patch, it is still very much tied to how the libpq protocol works. For example, there is a cancel key and a ready-for-query message. Some of the details of the simple and the extended query are exposed. So you could create a protocol th

Re: Extensibility of the PostgreSQL wire protocol

2021-02-24 Thread Jan Wieck
On 2/19/21 10:13 AM, Jan Wieck wrote: Give the function, that postmaster is calling to accept a connection when a server_fd is ready, a return code that it can use to tell postmaster "forget about it, don't fork or do anything else with it". This function is normally calling StreamConnection() b

Re: Extensibility of the PostgreSQL wire protocol

2021-02-22 Thread Jan Wieck
On 2/22/21 1:01 PM, Tom Lane wrote: Yeah, and as I pointed out somewhere upthread, trying to replace the whole parser will just end in a maintenance nightmare. The constructs that the parser has to emit are complex, Postgres-specific, and constantly evolving. We are NOT going to promise any sor

Re: Extensibility of the PostgreSQL wire protocol

2021-02-22 Thread Jonah H. Harris
On Mon, Feb 22, 2021 at 1:01 PM Tom Lane wrote: > Jan Wieck writes: > > As Jim Mlodgenski just posted in [0], having the ability to also extend > > and/or replace the parser will give them the ability to do just that. > > Yeah, and as I pointed out somewhere upthread, trying to replace the > who

Re: Extensibility of the PostgreSQL wire protocol

2021-02-22 Thread Tom Lane
Jan Wieck writes: > On 2/10/21 1:10 PM, Tom Lane wrote: >> What I'm actually more concerned about, in this whole line of development, >> is the follow-on requests that will surely occur to kluge up Postgres >> to make its behavior more like $whatever. As in "well, now that we >> can serve MySQL c

Re: Extensibility of the PostgreSQL wire protocol

2021-02-22 Thread Jan Wieck
On 2/10/21 1:10 PM, Tom Lane wrote: What I'm actually more concerned about, in this whole line of development, is the follow-on requests that will surely occur to kluge up Postgres to make its behavior more like $whatever. As in "well, now that we can serve MySQL clients protocol-wise, can't we

Re: Extensibility of the PostgreSQL wire protocol

2021-02-22 Thread David Fetter
On Mon, Feb 22, 2021 at 07:34:51AM -0500, Dave Cramer wrote: > On Fri, 19 Feb 2021 at 15:39, Álvaro Hernández wrote: > > > On 19/2/21 19:30, Jan Wieck wrote: > > > [...] > > > > > > I also am not sure if building a connection pool into a > > > background worker or postmaster is a good idea to beg

Re: Extensibility of the PostgreSQL wire protocol

2021-02-22 Thread Jan Wieck
On 2/19/21 4:36 AM, Kuntal Ghosh wrote: On Thu, Feb 18, 2021 at 9:32 PM Jan Wieck wrote: Few comments in the extension code (although experimental): 1. In telnet_srv.c, + static intpe_port; .. + DefineCustomIntVariable("telnet_srv.port", +

Re: Extensibility of the PostgreSQL wire protocol

2021-02-22 Thread Jan Wieck
On 2/22/21 7:34 AM, Dave Cramer wrote: Yes, when did it become a good idea to put a connection pooler in the backend??? Dave Cramer www.postgres.rocks As Alvaro said, different pool purposes lead to different pool architecture and placement. However, the changes proposed here, aiming at t

Re: Extensibility of the PostgreSQL wire protocol

2021-02-22 Thread Dave Cramer
On Fri, 19 Feb 2021 at 15:39, Álvaro Hernández wrote: > > > On 19/2/21 19:30, Jan Wieck wrote: > > [...] > > > > I also am not sure if building a connection pool into a background > > worker or postmaster is a good idea to begin with. One of the > > important features of a pool is to be able to s

Re: Extensibility of the PostgreSQL wire protocol

2021-02-19 Thread Álvaro Hernández
On 19/2/21 19:30, Jan Wieck wrote: > [...] > > I also am not sure if building a connection pool into a background > worker or postmaster is a good idea to begin with. One of the > important features of a pool is to be able to suspend traffic and make > the server completely idle to for example b

Re: Extensibility of the PostgreSQL wire protocol

2021-02-19 Thread Álvaro Hernández
On 19/2/21 14:48, Heikki Linnakangas wrote: > [...] > > I can see value in supporting different protocols. I don't like the > approach discussed in this thread, however. > > For example, there has been discussion elsewhere about integrating > connection pooling into the server itself. For that,

Re: Extensibility of the PostgreSQL wire protocol

2021-02-19 Thread Damir Simunic
> On 19 Feb 2021, at 19:30, Jan Wieck wrote: > > An "extended" libpq protocol could allow the pool to give clients a unique > ID. The protocol handler would then maintain maps with the SQL of prepared > statements and what the client thinks their prepared statement name is. Or, the connecti

Re: Extensibility of the PostgreSQL wire protocol

2021-02-19 Thread Jan Wieck
On 2/19/21 12:18 PM, Damir Simunic wrote: On 19 Feb 2021, at 14:48, Heikki Linnakangas wrote: For example, there has been discussion elsewhere about integrating connection pooling into the server itself. For that, you want to have a custom process that listens for incoming connections, and

Re: Extensibility of the PostgreSQL wire protocol

2021-02-19 Thread Damir Simunic
> On 19 Feb 2021, at 14:48, Heikki Linnakangas wrote: > > For example, there has been discussion elsewhere about integrating connection > pooling into the server itself. For that, you want to have a custom process > that listens for incoming connections, and launches backends independently of

Re: Extensibility of the PostgreSQL wire protocol

2021-02-19 Thread Jan Wieck
On 2/19/21 8:48 AM, Heikki Linnakangas wrote: I can see value in supporting different protocols. I don't like the approach discussed in this thread, however. For example, there has been discussion elsewhere about integrating connection pooling into the server itself. For that, you want to have a

Re: Extensibility of the PostgreSQL wire protocol

2021-02-19 Thread Jonah H. Harris
On Fri, Feb 19, 2021 at 8:48 AM Heikki Linnakangas wrote: > With the hooks that exist today, would it possible to write a background > worker that listens on a port, instead of postmaster? Can you launch > backends from a background worker? And communicate the backend processes > using a shared m

Re: Extensibility of the PostgreSQL wire protocol

2021-02-19 Thread Heikki Linnakangas
On 19/02/2021 14:29, Damir Simunic wrote: On 11 Feb 2021, at 16:06, Tom Lane wrote: Maybe there is some useful thing that can be accomplished here, but we need to consider the bigger picture rather than believing (without proof) that a few hook variables will be enough to do anything. Plugg

Re: Extensibility of the PostgreSQL wire protocol

2021-02-19 Thread Jan Wieck
Thank you Kuntal, On Fri, Feb 19, 2021 at 4:36 AM Kuntal Ghosh wrote: > On Thu, Feb 18, 2021 at 9:32 PM Jan Wieck wrote: > > > Few comments in the extension code (although experimental): > > 1. In telnet_srv.c, > + static intpe_port; > .. > + DefineCustomIntVariable("telnet_srv.po

Re: Extensibility of the PostgreSQL wire protocol

2021-02-19 Thread Damir Simunic
> On 11 Feb 2021, at 16:06, Tom Lane wrote: > > Maybe there is some useful thing that can be accomplished here, but we > need to consider the bigger picture rather than believing (without proof) > that a few hook variables will be enough to do anything. > > regards, tom l

Re: Extensibility of the PostgreSQL wire protocol

2021-02-19 Thread Kuntal Ghosh
On Thu, Feb 18, 2021 at 9:32 PM Jan Wieck wrote: > And, here is how it looks with the following configuration: telnet_srv.port = 1433 telnet_srv.listen_addresses = '*' telnet localhost 1433   master Trying 127.0.0.1... Connected to localhost. Escape character is '

Re: Extensibility of the PostgreSQL wire protocol

2021-02-18 Thread Jan Wieck
Attached are a first patch and a functioning extension that implements a telnet protocol server. The extension needs to be loaded via shared_preload_libraries and configured for a port number and listen_addresses as follows: shared_preload_libraries = 'telnet_srv' telnet_srv.listen_addresses = '

Re: Extensibility of the PostgreSQL wire protocol

2021-02-14 Thread Dave Cramer
On Thu, 11 Feb 2021 at 09:28, Robert Haas wrote: > On Wed, Feb 10, 2021 at 2:04 PM Tom Lane wrote: > > That is a spot-on definition of where I do NOT want to end up. Hooks > > everywhere and enormous extensions that break anytime we change anything > > in the core. It's not really clear that a

Re: Extensibility of the PostgreSQL wire protocol

2021-02-12 Thread Fabrízio de Royes Mello
On Thu, Feb 11, 2021 at 12:07 PM Tom Lane wrote: > > Robert Haas writes: > > On Thu, Feb 11, 2021 at 9:42 AM Jonah H. Harris wrote: > >> As Jan said in his last email, they're not proposing all the different > >> aspects needed. In fact, nothing has actually been proposed yet. This > >> is an en

Re: Extensibility of the PostgreSQL wire protocol

2021-02-11 Thread Joshua Drake
On Wed, Feb 10, 2021 at 11:04 AM Tom Lane wrote: > "Jonah H. Harris" writes: > > On Wed, Feb 10, 2021 at 1:10 PM Tom Lane wrote: > >> ... If we start having > >> modes for MySQL identifier quoting, Oracle outer join syntax, yadda > >> yadda, it's going to be way more of a maintenance nightmare

Re: Extensibility of the PostgreSQL wire protocol

2021-02-11 Thread Jim Mlodgenski
On Thu, Feb 11, 2021 at 10:29 AM Andrew Dunstan wrote: > > On 2/11/21 10:06 AM, Tom Lane wrote: > > Robert Haas writes: > >> On Thu, Feb 11, 2021 at 9:42 AM Jonah H. Harris > wrote: > >>> As Jan said in his last email, they're not proposing all the different > >>> aspects needed. In fact, nothi

Re: Extensibility of the PostgreSQL wire protocol

2021-02-11 Thread Andrew Dunstan
On 2/11/21 10:06 AM, Tom Lane wrote: > Robert Haas writes: >> On Thu, Feb 11, 2021 at 9:42 AM Jonah H. Harris >> wrote: >>> As Jan said in his last email, they're not proposing all the different >>> aspects needed. In fact, nothing has actually been proposed yet. This >>> is an entirely philos

Re: Extensibility of the PostgreSQL wire protocol

2021-02-11 Thread Tom Lane
Robert Haas writes: > On Thu, Feb 11, 2021 at 9:42 AM Jonah H. Harris > wrote: >> As Jan said in his last email, they're not proposing all the different >> aspects needed. In fact, nothing has actually been proposed yet. This >> is an entirely philosophical debate. I don't even know what's being

Re: Extensibility of the PostgreSQL wire protocol

2021-02-11 Thread Robert Haas
On Thu, Feb 11, 2021 at 9:42 AM Jonah H. Harris wrote: > I'm quite sure I said I'd open source my MySQL implementation, which allows > Postgres to appear to MySQL clients as a MySQL/MariaDB server. This is > neither proprietary nor Amazon-related and makes Postgres substantially more > useful f

Re: Extensibility of the PostgreSQL wire protocol

2021-02-11 Thread Jonah H. Harris
On Thu, Feb 11, 2021 at 9:28 AM Robert Haas wrote: > That being said, I'm not in favor of transferring maintenance work to > the community for this set of hooks any more than I am for something > on the parsing side. In general, I'm in favor of as much extensibility > as we can reasonably create,

Re: Extensibility of the PostgreSQL wire protocol

2021-02-11 Thread Robert Haas
On Wed, Feb 10, 2021 at 2:04 PM Tom Lane wrote: > That is a spot-on definition of where I do NOT want to end up. Hooks > everywhere and enormous extensions that break anytime we change anything > in the core. It's not really clear that anybody is going to find that > more maintainable than a str

Re: Extensibility of the PostgreSQL wire protocol

2021-02-10 Thread Jan Wieck
On Wed, Feb 10, 2021 at 11:43 AM Robert Haas wrote: > On Mon, Jan 25, 2021 at 10:07 AM Jan Wieck wrote: > > Our current plan is to create a new set of API calls and hooks that > allow to register additional wire protocols. The existing backend libpq > implementation will be modified to register

Re: Extensibility of the PostgreSQL wire protocol

2021-02-10 Thread Jonah H. Harris
On Wed, Feb 10, 2021 at 2:04 PM Tom Lane wrote: > That is a spot-on definition of where I do NOT want to end up. Hooks > everywhere and enormous extensions that break anytime we change anything > in the core. It's not really clear that anybody is going to find that > more maintainable than a st

Re: Extensibility of the PostgreSQL wire protocol

2021-02-10 Thread Tom Lane
"Jonah H. Harris" writes: > On Wed, Feb 10, 2021 at 1:10 PM Tom Lane wrote: >> ... If we start having >> modes for MySQL identifier quoting, Oracle outer join syntax, yadda >> yadda, it's going to be way more of a maintenance nightmare than some >> hook functions. So if we accept any patch alon

Re: Extensibility of the PostgreSQL wire protocol

2021-02-10 Thread Jonah H. Harris
On Wed, Feb 10, 2021 at 1:10 PM Tom Lane wrote: > What I'm actually more concerned about, in this whole line of development, > is the follow-on requests that will surely occur to kluge up Postgres > to make its behavior more like $whatever. As in "well, now that we > can serve MySQL clients prot

Re: Extensibility of the PostgreSQL wire protocol

2021-02-10 Thread Tom Lane
Robert Haas writes: > If we're going to end up with an open source implementation of > something useful in contrib or whatever, then I think this is fine. > But, if not, then we're just making it easier for Amazon to do > proprietary stuff without getting any benefit for the open-source > project.

Re: Extensibility of the PostgreSQL wire protocol

2021-02-10 Thread Jonah H. Harris
On Wed, Feb 10, 2021 at 11:43 AM Robert Haas wrote: > On Mon, Jan 25, 2021 at 10:07 AM Jan Wieck wrote: > > Our current plan is to create a new set of API calls and hooks that > allow to register additional wire protocols. The existing backend libpq > implementation will be modified to register

Re: Extensibility of the PostgreSQL wire protocol

2021-02-10 Thread Fabrízio de Royes Mello
On Wed, Feb 10, 2021 at 1:43 PM Robert Haas wrote: > > On Mon, Jan 25, 2021 at 10:07 AM Jan Wieck wrote: > > Our current plan is to create a new set of API calls and hooks that allow to register additional wire protocols. The existing backend libpq implementation will be modified to register itse

Re: Extensibility of the PostgreSQL wire protocol

2021-02-10 Thread Robert Haas
On Mon, Jan 25, 2021 at 10:07 AM Jan Wieck wrote: > Our current plan is to create a new set of API calls and hooks that allow to > register additional wire protocols. The existing backend libpq implementation > will be modified to register itself using the new API. This will serve as a > proof

Re: Extensibility of the PostgreSQL wire protocol

2021-01-25 Thread Jan Wieck
Hi Jonah, On Mon, Jan 25, 2021 at 10:18 AM Jonah H. Harris wrote: > On Mon, Jan 25, 2021 at 10:07 AM Jan Wieck wrote: > >> The following is a request for discussion and comments, not a refined >> proposal accompanied by a working patch. >> > > After implementing this three different ways inside

Re: Extensibility of the PostgreSQL wire protocol

2021-01-25 Thread Jonah H. Harris
On Mon, Jan 25, 2021 at 10:07 AM Jan Wieck wrote: > The following is a request for discussion and comments, not a refined > proposal accompanied by a working patch. > After implementing this three different ways inside the backend over the years, I landed on almost this identical approach for ha