Interested, yes. But there may be reasons not to do that. Last time I
looked the binary format wasn't documented.
Anyway, I tried re-implementing pqGetCopyData3() using the callback.
Wasn't hard, but I did have to add a way for the callback to return an
error. Kept it pretty dumb for now, hopin
Jeroen Vermeulen writes:
> The printf() is just the simplest example that sprang to mind though.
> There may be other use-cases out there involving libraries that require
> zero-terminated strings, and I figured an ability to set a sentinel could
> help those.
Well, since it won't help for binar
On Fri, 3 Mar 2023 at 18:14, Tom Lane wrote:
> Jeroen Vermeulen writes:
> > On Fri, 3 Mar 2023 at 17:33, Tom Lane wrote:
> >> Let's not do that. Declare it const char *, or maybe better const void
> *.
>
> > Personally I would much prefer "char" over "void" here:
> > * It really is a char buff
Jelte Fennema writes:
> On Fri, 3 Mar 2023 at 17:33, Tom Lane wrote:
>> If you mean exposing PQExpBuffer to users of libpq-fe.h, I'd be very
>> seriously against that. I realize that libpq exposes it at an ABI
>> level, but that doesn't mean we want non-Postgres code to use it.
> The code comme
On Fri, 3 Mar 2023 at 17:33, Tom Lane wrote:
> If you mean exposing PQExpBuffer to users of libpq-fe.h, I'd be very
> seriously against that. I realize that libpq exposes it at an ABI
> level, but that doesn't mean we want non-Postgres code to use it.
The code comment in the pqexpbuffer.h header
Jeroen Vermeulen writes:
> On Fri, 3 Mar 2023 at 17:33, Tom Lane wrote:
>> Let's not do that. Declare it const char *, or maybe better const void *.
> Personally I would much prefer "char" over "void" here:
> * It really is a char buffer, containing text.
Not in binary-mode COPY.
> As for con
On Fri, 3 Mar 2023 at 17:33, Tom Lane wrote:
>
> If you mean exposing PQExpBuffer to users of libpq-fe.h, I'd be very
> seriously against that. I realize that libpq exposes it at an ABI
> level, but that doesn't mean we want non-Postgres code to use it.
> I also don't see what it'd add for this
Jelte Fennema writes:
> Did you try with PQExpBuffer? I still think that probably fits better
> in the API design of libpq.
If you mean exposing PQExpBuffer to users of libpq-fe.h, I'd be very
seriously against that. I realize that libpq exposes it at an ABI
level, but that doesn't mean we want
Thank you.
I meant to try PQExpBuffer — as you say, it fits much better with existing
libpq style. But then I hit on the callback idea which was even more
efficient, by a fair margin. It was also considerably simpler both inside
libpq and in the client code, eliminating all sorts of awkward ques
On Thu, 2 Mar 2023 at 20:45, Jeroen Vermeulen wrote:
> I'm attaching a diff now. This is not a patch, it's just a discussion piece.
Did you try with PQExpBuffer? I still think that probably fits better
in the API design of libpq. Obviously if it's significantly slower
than the callback approach
My apologies. The wiki said to discuss early, even before writing the code
if possible, but I added a link to the PR for those who really wanted to
see the details.
I'm attaching a diff now. This is not a patch, it's just a discussion
piece.
The problem was that PQgetCopyData loops use a lot of
> On 1 Mar 2023, at 15:23, Jeroen Vermeulen wrote:
> PR for easy discussion: https://github.com/jtv/postgres/pull/1
The process for discussing work on pgsql-hackers is to attach the patch to the
email and discuss it inline in the thread. That way all versions of the patch
as well as the discuss
Update: in the latest iteration, I have an alternative that reduces CPU
time by more than half, compared to PQgetCopyData().
And the code is simpler, too, both in the client and in libpq itself. The
one downside is that it breaks with libpq's existing API style.
PR for easy discussion: https://g
Done. Thanks for looking!
Jelte Fennema pointed out that I should probably be using PQExpBuffer for
this. I'll look into that later; this is a proof of concept, not a
production-ready API proposal.
Jeroen
On Mon, 27 Feb 2023 at 14:48, Bharath Rupireddy <
bharath.rupireddyforpostg...@gmail.com
Instead of implementing new growable buffer logic in this patch. It
seems much nicer to reuse the already existing PQExpBuffer type for
this patch.
On Mon, 27 Feb 2023 at 14:48, Bharath Rupireddy
wrote:
>
> On Fri, Feb 10, 2023 at 5:49 PM Jeroen Vermeulen wrote:
> >
> > OK, I've updated the PR
On Fri, Feb 10, 2023 at 5:49 PM Jeroen Vermeulen wrote:
>
> OK, I've updated the PR with a benchmark (in the main directory).
>
> On this benchmark I'm seeing about a 24% reduction in "user" CPU time, and a
> 8% reduction in "system" CPU time. (Almost no reduction in wall-clock time.)
I can hel
OK, I've updated the PR with a benchmark (in the main directory).
On this benchmark I'm seeing about a 24% reduction in "user" CPU time, and
a 8% reduction in "system" CPU time. (Almost no reduction in wall-clock
time.)
Jeroen
On Fri, 10 Feb 2023 at 11:32, Jeroen Vermeulen wrote:
> Here's th
Here's the patch (as a PR just to make it easy to read):
https://github.com/jtv/postgres/pull/1
I don't have an easily readable benchmark yet, since I've been timing the
potential impact on libpqxx. But can do that next.
Jeroen
On Fri, Feb 10, 2023, 11:26 Bharath Rupireddy <
bharath.rupireddyf
On Fri, Feb 10, 2023 at 3:43 PM Jeroen Vermeulen wrote:
>
> Would there be interest in a variant of PQgetCopyData() that re-uses the same
> buffer for a new row, rather than allocating a new buffer on each iteration?
>
> I tried it on a toy benchmark, and it reduced client-side CPU time by about
19 matches
Mail list logo
