Re: Add Pipelining support in psql

On Wed, Apr 23, 2025 at 04:13:14PM +0900, Michael Paquier wrote: > The tweak for psql_fails_like() was kind of independent of the rest of > the fix, so I have applied that as a commit of its own. I am not > convinced about the addition of a 4th test where we use the queries > with semicolons witho

Re: Add Pipelining support in psql

On Tue, Apr 22, 2025 at 02:37:19PM +0200, Anthonin Bonnefoy wrote: > Also, we still have a triggered assertion failure with the following: > CREATE TABLE psql_pipeline(a text); > \startpipeline > COPY psql_pipeline FROM STDIN; > SELECT 'val1'; > \syncpipeline > \endpipeline > ... > Assertion failed

Re: Add Pipelining support in psql

On Tue, Apr 22, 2025 at 2:06 AM Michael Paquier wrote: > I am wondering if we could not be smarter with the handling of > the counters, but I really doubt that there is much more we can do > under a PGRES_FATAL_ERROR. One thing that bothers me is that the reported error is silently discarded with

Re: Add Pipelining support in psql

On Mon, Apr 21, 2025 at 03:22:28PM +0900, Michael Paquier wrote: > Anyway, I don't think that there is much we can do under a > PGRES_FATAL_ERROR in this code path when discarding the pipe results. > As far as I can tell, the server has failed the query suddenly and the > whole pipeline flow is bor

Re: Add Pipelining support in psql

On Wed, Apr 16, 2025 at 09:18:01AM -0700, Michael Paquier wrote: > On Wed, Apr 16, 2025 at 09:31:59PM +0700, a.kozhemyakin wrote: >> After commit 2cce0fe on master >> >> ERROR:  unexpected message type 0x50 during COPY from stdin >> CONTEXT:  COPY psql_pipeline, line 1 >> Pipeline aborted, command

Re: Add Pipelining support in psql

On Wed, Apr 16, 2025 at 09:31:59PM +0700, a.kozhemyakin wrote: > After commit 2cce0fe on master > > ERROR:  unexpected message type 0x50 during COPY from stdin > CONTEXT:  COPY psql_pipeline, line 1 > Pipeline aborted, command did not run > psql: common.c:1510: discardAbortedPipelineResults: Asser

Re: Add Pipelining support in psql

Hello, After commit 2cce0fe on master When executing query: psql postgres

Re: Add Pipelining support in psql

Anthonin Bonnefoy wrote: > 0002: Allows ';' to send a query using extended protocol when within a > pipeline by using PQsendQueryParams It's a nice improvement! > with 0 parameters. It is not > possible to send parameters with extended protocol this way and > everything will be propagate

Re: Add Pipelining support in psql

Michael Paquier wrote: > Perhaps an \extended command that behaves outside a pipeline makes > sense to force the use of queries without parameters to use the > extended mode, but I cannot get much excited about the concept knowing > all the meta-commands we have now (not talking about the

Re: Add Pipelining support in psql

On Tue, Mar 18, 2025 at 09:55:21AM +0100, Anthonin Bonnefoy wrote: > I've added additional tests when piping queries with ';': > - I've reused the same scenario with \sendpipeline: single query, > multiple queries, flushes, syncs, using COPY... > - Using ';' will replace the unnamed prepared statem

Re: Add Pipelining support in psql

On Tue, Mar 18, 2025 at 10:27:38AM +0100, Jelte Fennema-Nio wrote: > One thing that comes to mind that I think would be quite useful and > pretty easy to implement if we have this functionality within a > pipeline: An \extended command. That puts psql in "extended protocol > mode" (without enabling

Re: Add Pipelining support in psql

On Tue, Mar 18, 2025 at 10:36:28AM +0100, Daniel Verite wrote: > It's actually possible to use parameters > > \startpipeline > \bind 'foo' > select $1; > \endpipeline > > ?column? > -- > foo > (1 row) > > I suspect there's a misunderstanding that \bind can only be placed > after the

Re: Add Pipelining support in psql

On Tue, Mar 18, 2025 at 1:50 AM Michael Paquier wrote: > - count > > - 4 > -(1 row) > > This removal done in the regression tests was not intentional. Yes, thanks for fixing that. > How about adding a check on PIPELINE_COMMAND_COUNT when sending a > query through this path? Should

Re: Add Pipelining support in psql

On Tue, 18 Mar 2025 at 09:55, Anthonin Bonnefoy wrote: > I've added additional tests when piping queries with ';': > - I've reused the same scenario with \sendpipeline: single query, > multiple queries, flushes, syncs, using COPY... > - Using ';' will replace the unnamed prepared statement. It's a

Re: Add Pipelining support in psql

On Mon, Mar 17, 2025 at 10:50:50AM +0100, Anthonin Bonnefoy wrote: > 0001: This introduces the \sendpipeline meta-command and forbid \g in > a pipeline. This is to fix the formatting options of \g that are not > supported in a pipeline. - count - 4 -(1 row) This removal done in the

Re: Add Pipelining support in psql

On Mon, Mar 17, 2025 at 10:50:50AM +0100, Anthonin Bonnefoy wrote: > 0001: This introduces the \sendpipeline meta-command and forbid \g in > a pipeline. This is to fix the formatting options of \g that are not > supported in a pipeline. > > 0002: Allows ';' to send a query using extended protocol

Re: Add Pipelining support in psql

Here is a new patch set: 0001: This introduces the \sendpipeline meta-command and forbid \g in a pipeline. This is to fix the formatting options of \g that are not supported in a pipeline. 0002: Allows ';' to send a query using extended protocol when within a pipeline by using PQsendQueryParams w

Re: Add Pipelining support in psql

Jelte Fennema-Nio wrote: > As an example you can copy paste this tiny script: > > \startpipeline > select pg_sleep(5) \bind \g > \endpipeline > > And then it will show these "extra argument ... ignored" warnings > > \startpipeline: extra argument "select" ignored > \startpipeline: extra

Re: Add Pipelining support in psql

On Thu, Mar 6, 2025 at 5:20 AM Michael Paquier wrote: > That was not a test case we had in mind originally here, but if it is > possible to keep the implementation simple while supporting your > demand, well, let's do it. If it's not that straight-forward, let's > use the new meta-command, forbid

Re: Add Pipelining support in psql

On Tue, 25 Feb 2025 at 02:11, Michael Paquier wrote: > Initial digestion has gone well. One thing I've noticed is that \startpipeline throws warnings when copy pasting multiple lines. It seems to still execute everything as expected though. As an example you can copy paste this tiny script: \sta

Re: Add Pipelining support in psql

On Wed, Mar 05, 2025 at 03:25:12PM +0100, Daniel Verite wrote: > Anthonin Bonnefoy wrote: >> I do see the idea to make it easier to convert existing scripts into >> using pipelining. The main focus of the initial implementation was >> more on protocol regression tests with psql, so that's not

Re: Add Pipelining support in psql

Anthonin Bonnefoy wrote: > So if I understand correctly, you want to automatically convert a > simple query into an extended query when we're within a pipeline. That > would be doable with: > > --- a/src/bin/psql/common.c > +++ b/src/bin/psql/common.c > @@ -1668,7 +1668,16 @@ ExecQueryAnd

Re: Add Pipelining support in psql

On Tue, Mar 04, 2025 at 06:37:09PM +0100, Anthonin Bonnefoy wrote: > I do see the idea to make it easier to convert existing scripts into > using pipelining. The main focus of the initial implementation was > more on protocol regression tests with psql, so that's not necessarily > something I had i

Re: Add Pipelining support in psql

On Tue, Mar 04, 2025 at 10:31:46AM +0100, Anthonin Bonnefoy wrote: > Saving the printQueryOpt when a command is pushed was an option I had > in mind if that was straightforward to implement. However, even with > savePsetInfo, you will need to save values like gfname and gset_prefix > since it impac

Re: Add Pipelining support in psql

On Tue, Mar 4, 2025 at 1:32 PM Daniel Verite wrote: > But if the code triggered the use of the extended query protocol > if \bind is in effect *or* a pipeline is active, then the first sequence > would just push "select 1" into the pipeline. > > This would have the advantage that, to submit into

Re: Add Pipelining support in psql

Anthonin Bonnefoy wrote: > Another possible option would be to directly send the command without > requiring an additional meta-command, like "SELECT 1 \bind". However, > this would make it more painful to introduce new parameters, plus it > makes the \bind and \bind_named inconsistent as

Re: Add Pipelining support in psql

On Tue, Mar 4, 2025 at 1:57 AM Michael Paquier wrote: > Anyway, can we really say that the set of printQueryOpt saved at > execution is the correct set to use? It is possible to have the > opposite argument and say that we should just apply the printQueryOpt > at the moment where \getresult is ru

Re: Add Pipelining support in psql

(My previous message did not reach the lists, so re-sending with some edits.) On Fri, Feb 28, 2025 at 05:31:00PM +0100, Daniel Verite wrote: > The following invocations don't respect the desired output destination > and formats (ignoring them), when a pipeline is active: > > select ... \bind \g f

Re: Add Pipelining support in psql

Anthonin Bonnefoy wrote: > > What is the reasoning here behind this restriction? \gx is a wrapper > > of \g with expanded mode on, but it is also possible to call \g with > > expanded=on, bypassing this restriction. > > The issue is that \gx enables expanded mode for the duration of the

Re: Add Pipelining support in psql

On Tue, Feb 25, 2025 at 2:11 AM Michael Paquier wrote: > For the prompt part, I have added a > couple of tests with \echo and the variables. The patch felt > incomplete without these. Perhaps we could extend them more, at least > we have a start point. Good catch. I guess that's also another be

Re: Add Pipelining support in psql

On Fri, Feb 21, 2025 at 11:33:41AM +0900, Michael Paquier wrote: > Attached are the remaining pieces, split here because they are > different bullet points: > - Tests for implicit transactions with various commands, with some > edits. > - Prompt support, with more edits. > > I'm putting these on

Re: Add Pipelining support in psql

On Thu, Feb 20, 2025 at 10:29:33AM +0100, Anthonin Bonnefoy wrote: > Ha yeah, I forgot about the inactive branches. I've added the new > commands and fixed the behaviour. And I did not notice that it was as simple as forcing the status in the routines for the new meta-commands, as we do for the ex

Re: Add Pipelining support in psql

On Thu, Feb 20, 2025 at 9:02 AM Michael Paquier wrote: > You have forgotten the expected output. Not a big issue as the input > was sent. I was writing the mail with the missing file when you sent this mail. This is fixed. > While testing, I have been able to run into an assertion failure by >

Re: Add Pipelining support in psql

On Tue, Feb 18, 2025 at 06:34:20PM +0100, Anthonin Bonnefoy wrote: > On Tue, Feb 18, 2025 at 8:23 AM Michael Paquier wrote: >> The tests in psql.sql are becoming really long. Perhaps it would be >> better to split that into its own file, say psql_pipeline.sql? The >> input file is already 2k lin

Re: Add Pipelining support in psql

On Tue, Feb 18, 2025 at 8:23 AM Michael Paquier wrote: > This is a pretty cool patch. I like the structure you have used for > the integration with the tracking of the number of commands, the > number of syncs (like pgbench) put in a pipeline, the number of > results requested and the number of r

Re: Add Pipelining support in psql

On Tue, Jan 14, 2025 at 09:49:02AM +0100, Anthonin Bonnefoy wrote: > During my tests, I've noticed I didn't handle query cancellation, this > is now fixed. I've also added additional comments related to > available_results to make it clearer that it depends on what the > server has flushed to the c

Re: Add Pipelining support in psql

During my tests, I've noticed I didn't handle query cancellation, this is now fixed. I've also added additional comments related to available_results to make it clearer that it depends on what the server has flushed to the client. v07-0001-Add-pipelining-support-in-psql.patch Description: Binary

Re: Add Pipelining support in psql

> I feel there's a large overlap between \flush and \flushrequest. On > the other hand, if people want to test the behaviour of pushing data > with and without a flush request message, then \flush can be useful. I've been looking into some issues related to a backend process stuck in ClientWrite s

Re: Add Pipelining support in psql

Thanks for the review! On Thu, Dec 12, 2024 at 12:53 AM Jelte Fennema-Nio wrote: > I think that new prompt is super useful, so useful in fact that I'd > suggest linking to it from the \startpipeline docs. Good point, I've added a paragraph with the link to the %P prompt. > I do think that > the

Re: Add Pipelining support in psql

On Tue, 10 Dec 2024 at 11:44, Anthonin Bonnefoy wrote: > num_queries (2nd element in the pipeline status prompt) is now used to > track queued queries that were not flushed (with a flush request or > sync) to the server. It used to count both unflushed queries and > flushed queries. I skimmed the

Re: Add Pipelining support in psql

An improved version with simplifications and refinements. num_queries (2nd element in the pipeline status prompt) is now used to track queued queries that were not flushed (with a flush request or sync) to the server. It used to count both unflushed queries and flushed queries. Code in ExecQueryA

Re: Add Pipelining support in psql

An updated version of the patch, still a bit rough around the edges. I've added \flushrequest, \flush and \getrequests meta-commands. \flushrequest and \getresults look interesting as they add an additional protocol message to test, but it also introduces the possibility to be in the middle of an

Re: Add Pipelining support in psql

On Wed, Nov 27, 2024 at 11:46 AM Kirill Reshke wrote: > I'm very doubtful about the \syncpipeline . Maybe we should instead > support \sync meta-command in psql? This will be a useful contribution > itself. \syncpipeline is useful to cover regression tests involving implicit transaction blocks wi

Re: Add Pipelining support in psql

On Thu, 28 Nov 2024 at 07:43, Michael Paquier wrote: > Hmm. The start, end and sync meta-commands are useful for testing. I > find the flush control a bit less interesting, TBH. > > What would you use these for? I guess mostly for interactively playing around with pipelining from psql. But I t

Re: Add Pipelining support in psql

On Wed, Nov 27, 2024 at 01:45:32PM +0100, Jelte Fennema-Nio wrote: > I played around quickly with this patch and it works quite well. A few > things that would be nice improvements I think. Feel free to change > the command names: > 1. Add a \flush command that calls PQflush > 2. Add a \flushreques

Re: Add Pipelining support in psql

On Wed, 27 Nov 2024 at 13:05, Kirill Reshke wrote: > ``` > db1=# \startpipeline > db1=# begin \parse p1 > db1=*# > ``` > Notice the asterisks that appeared after parse the message. This > typically indicates we are in the tx block. this is however untrue > before the bind+exec message for p1 will

Re: Add Pipelining support in psql

On Wed, 27 Nov 2024 at 10:50, Anthonin Bonnefoy wrote: > With \bind, \parse, \bind_named and \close, it is possible to issue > queries from psql using the extended protocol. However, it wasn't > possible to send those queries using pipelining and the only way to > test pipelined queries was throug

Re: Add Pipelining support in psql

On Wed, 27 Nov 2024 at 14:50, Anthonin Bonnefoy wrote: > > Hi, > > With \bind, \parse, \bind_named and \close, it is possible to issue > queries from psql using the extended protocol. However, it wasn't > possible to send those queries using pipelining and the only way to > test pipelined queries

Re: Add Pipelining support in psql

On Wed, 27 Nov 2024 at 14:50, Anthonin Bonnefoy wrote: > > Hi, > > With \bind, \parse, \bind_named and \close, it is possible to issue > queries from psql using the extended protocol. However, it wasn't > possible to send those queries using pipelining and the only way to > test pipelined queries

Add Pipelining support in psql

Hi, With \bind, \parse, \bind_named and \close, it is possible to issue queries from psql using the extended protocol. However, it wasn't possible to send those queries using pipelining and the only way to test pipelined queries was through pgbench's tap tests. The attached patch adds pipelining