Re: Make COPY format extendable: Extract COPY TO format implementations

2025-06-17 Thread Sutou Kouhei
Hi, In "Re: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 17 Jun 2025 09:38:54 +0900, Michael Paquier wrote: > On Tue, Jun 17, 2025 at 08:50:37AM +0900, Sutou Kouhei wrote: >> OK. I'll implement the initial version with this >>

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-06-16 Thread Michael Paquier
On Tue, Jun 17, 2025 at 08:50:37AM +0900, Sutou Kouhei wrote: > OK. I'll implement the initial version with this > design. (Allocating IDs local not shared.) Sounds good to me. Thanks Sutou-san! -- Michael signature.asc Description: PGP signature

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-06-16 Thread Sutou Kouhei
Hi, In "Re: Make COPY format extendable: Extract COPY TO format implementations" on Thu, 12 Jun 2025 10:00:12 -0700, Masahiko Sawada wrote: >> So, my opinion is to rely on _PG_init(), with a shared ID if you want >> to expose the method used somewhere for monito

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-06-12 Thread Masahiko Sawada
On Wed, Jun 11, 2025 at 7:34 PM Michael Paquier wrote: > > On Mon, May 26, 2025 at 10:04:05AM +0900, Sutou Kouhei wrote: > > As I already said, I don't have a strong opinion on which > > approach is better. My opinion for the (important) second > > point is no. I feel that the pros of a. isn't rea

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-06-11 Thread Michael Paquier
On Mon, May 26, 2025 at 10:04:05AM +0900, Sutou Kouhei wrote: > As I already said, I don't have a strong opinion on which > approach is better. My opinion for the (important) second > point is no. I feel that the pros of a. isn't realistic. If > users want to improve text/csv/binary performance (or

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-05-25 Thread Sutou Kouhei
Hi, In "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 9 May 2025 21:29:23 -0700, Masahiko Sawada wrote: >> > So the idea is that the backend process sets the format ID somewhere >> > in st_progress_param, and then the progre

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-05-25 Thread Sutou Kouhei
Hi, In "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 9 May 2025 17:57:35 -0700, Masahiko Sawada wrote: >> Proposed approaches to register custom COPY formats: >> a. Create a function that has the same name of custom COPY >

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-05-09 Thread Masahiko Sawada
On Fri, May 9, 2025 at 1:51 AM Sutou Kouhei wrote: > > Hi, > > In > "Re: Make COPY format extendable: Extract COPY TO format implementations" > on Fri, 2 May 2025 23:37:46 -0700, > Masahiko Sawada wrote: > > > The progress information

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-05-09 Thread Masahiko Sawada
On Fri, May 9, 2025 at 2:41 AM Sutou Kouhei wrote: > > Hi, > > In > "Re: Make COPY format extendable: Extract COPY TO format implementations" > on Sat, 3 May 2025 22:27:36 -0700, > "David G. Johnston" wrote: > > > In any case, I’m

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-05-09 Thread Sutou Kouhei
Hi, In "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 2 May 2025 23:37:46 -0700, Masahiko Sawada wrote: > The progress information is stored in PgBackendStatus defined in > backend_status.h: > > /* > * Command progres

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-05-09 Thread Sutou Kouhei
Hi, In "Re: Make COPY format extendable: Extract COPY TO format implementations" on Sat, 3 May 2025 22:27:36 -0700, "David G. Johnston" wrote: > In any case, I’m doubtful either of us can make a convincing enough > argument to sway the other fully. Both

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-05-03 Thread David G. Johnston
On Saturday, May 3, 2025, Masahiko Sawada wrote: > On Sat, May 3, 2025 at 7:42 AM David G. Johnston > wrote: > > > > On Saturday, May 3, 2025, Masahiko Sawada wrote: > > > >> > >> I think that we need to ensure that if users specify text/csv/binary > >> the built-in formats are always used, to

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-05-03 Thread Masahiko Sawada
On Sat, May 3, 2025 at 7:42 AM David G. Johnston wrote: > > On Saturday, May 3, 2025, Masahiko Sawada wrote: > >> >> I think that we need to ensure that if users specify text/csv/binary >> the built-in formats are always used, to keep backward compatibility. > > > That was my original thinking, b

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-05-03 Thread David G. Johnston
On Saturday, May 3, 2025, Masahiko Sawada wrote: > I think that we need to ensure that if users specify text/csv/binary > the built-in formats are always used, to keep backward compatibility. That was my original thinking, but it’s inconsistent with how functions behave today. We don’t promis

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-05-03 Thread Masahiko Sawada
On Fri, May 2, 2025 at 11:37 PM David G. Johnston wrote: > > On Friday, May 2, 2025, Masahiko Sawada wrote: >> >> >> I'm concerned about allowing multiple 'text' format implementations >> with identical names within the database, as this could lead to >> considerable confusion. When users specify

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-05-02 Thread Masahiko Sawada
On Fri, May 2, 2025 at 11:20 PM Sutou Kouhei wrote: > > Hi, > > In > "Re: Make COPY format extendable: Extract COPY TO format implementations" > on Fri, 2 May 2025 23:02:25 -0700, > Masahiko Sawada wrote: > > > The progress view can know only num

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-05-02 Thread David G. Johnston
On Friday, May 2, 2025, Masahiko Sawada wrote: > > I'm concerned about allowing multiple 'text' format implementations > with identical names within the database, as this could lead to > considerable confusion. When users specify 'text', it would be more > logical to guarantee that the built-in '

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-05-02 Thread Sutou Kouhei
Hi, In "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 2 May 2025 23:02:25 -0700, Masahiko Sawada wrote: > The progress view can know only numbers. We need to extend the > progress view infrastructure so that we can pass other data type

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-05-02 Thread Masahiko Sawada
On Fri, May 2, 2025 at 9:56 PM Sutou Kouhei wrote: > > Hi, > > In > "Re: Make COPY format extendable: Extract COPY TO format implementations" > on Fri, 2 May 2025 21:38:32 -0700, > Masahiko Sawada wrote: > > >> How about requiring sch

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-05-02 Thread Masahiko Sawada
On Fri, May 2, 2025 at 10:36 PM David G. Johnston wrote: > > On Thursday, May 1, 2025, Masahiko Sawada wrote: >> >> >> In light of these concerns, I've been contemplating alternative >> interface designs. One promising approach would involve registering >> custom copy formats via a C function dur

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-05-02 Thread David G. Johnston
On Thursday, May 1, 2025, Masahiko Sawada wrote: > > In light of these concerns, I've been contemplating alternative > interface designs. One promising approach would involve registering > custom copy formats via a C function during module loading > (specifically, in _PG_init()). This method woul

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-05-02 Thread Sutou Kouhei
Hi, In "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 2 May 2025 21:38:32 -0700, Masahiko Sawada wrote: >> How about requiring schema for all custom formats? >> >> Valid: >> >> COPY ... TO ... (FORM

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-05-02 Thread Masahiko Sawada
On Fri, May 2, 2025 at 7:20 PM Sutou Kouhei wrote: > > Hi, > > In > "Re: Make COPY format extendable: Extract COPY TO format implementations" > on Thu, 1 May 2025 12:15:30 -0700, > Masahiko Sawada wrote: > > > One of the primary considerations we nee

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-05-02 Thread Sutou Kouhei
Hi, In "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 2 May 2025 15:52:49 -0700, Masahiko Sawada wrote: >> Hmm. How much should we care about the observability of the COPY >> format used by a given backend? Storing this informa

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-05-02 Thread Sutou Kouhei
Hi, In "Re: Make COPY format extendable: Extract COPY TO format implementations" on Thu, 1 May 2025 12:15:30 -0700, Masahiko Sawada wrote: > One of the primary considerations we need to address is the treatment > of the specified format name. The current patch set u

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-05-02 Thread Masahiko Sawada
On Thu, May 1, 2025 at 4:04 PM Michael Paquier wrote: > > On Thu, May 01, 2025 at 12:15:30PM -0700, Masahiko Sawada wrote: > > In light of these concerns, I've been contemplating alternative > > interface designs. One promising approach would involve registering > > custom copy formats via a C fun

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-05-01 Thread Michael Paquier
On Thu, May 01, 2025 at 12:15:30PM -0700, Masahiko Sawada wrote: > In light of these concerns, I've been contemplating alternative > interface designs. One promising approach would involve registering > custom copy formats via a C function during module loading > (specifically, in _PG_init()). This

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-05-01 Thread Masahiko Sawada
On Fri, Apr 25, 2025 at 5:45 AM Sutou Kouhei wrote: > > Hi, > > I've updated the patch set. See the attached v40 patch set. > > In > "Re: Make COPY format extendable: Extract COPY TO format implementations" > on Wed, 23 Apr 2025 23:44:55 -0700, >

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-04-25 Thread Sutou Kouhei
Hi, I've updated the patch set. See the attached v40 patch set. In "Re: Make COPY format extendable: Extract COPY TO format implementations" on Wed, 23 Apr 2025 23:44:55 -0700, Masahiko Sawada wrote: >> Are the followings correct? >> >> 1. Move invalid

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-04-23 Thread Masahiko Sawada
On Fri, Apr 4, 2025 at 1:38 AM Sutou Kouhei wrote: > > Hi, > > In > "Re: Make COPY format extendable: Extract COPY TO format implementations" > on Mon, 31 Mar 2025 12:35:23 -0700, > Masahiko Sawada wrote: > > > Most of the queries under test_copy_fo

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-04-06 Thread Sutou Kouhei
Hi, In "Re: Make COPY format extendable: Extract COPY TO format implementations" on Sun, 6 Apr 2025 19:29:46 +0800, jian he wrote: > I did a brief review of v39-0001 and v39-0002. > > text: > COPY_FILE > COPY_FRONTEND > still appear on comments in copy

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-04-06 Thread David G. Johnston
On Sun, Apr 6, 2025 at 4:30 AM jian he wrote: > > CREATE FUNCTION test_copy_format(internal) > RETURNS copy_handler > AS 'MODULE_PATHNAME', 'test_copy_format' > LANGUAGE C; > src/backend/commands/copy.c: ProcessCopyOptions > if (strcmp(fmt, "text") == 0) >

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-04-06 Thread jian he
On Thu, Mar 27, 2025 at 11:29 AM Sutou Kouhei wrote: > We can merge 0001 quickly, right? I did a brief review of v39-0001 and v39-0002. text: COPY_FILE COPY_FRONTEND still appear on comments in copyfrom_internal.h and copyto.c, Should it be removed? +#include "commands/copyto_internal.h" #incl

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-04-05 Thread Masahiko Sawada
On Wed, Mar 26, 2025 at 8:28 PM Sutou Kouhei wrote: > > > We need more regression tests for handling the given format name. For > > example, > > > > - more various input patterns. > > - a function with the specified format name exists but it returns an > > unexpected Node. > > - looking for a han

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-04-05 Thread Masahiko Sawada
On Sat, Mar 29, 2025 at 1:57 AM Sutou Kouhei wrote: > > Hi, > > In > "Re: Make COPY format extendable: Extract COPY TO format implementations" > on Fri, 28 Mar 2025 22:37:03 -0700, > Masahiko Sawada wrote: > > >> I've added the following tes

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-04-04 Thread Masahiko Sawada
On Sat, Mar 29, 2025 at 9:49 AM David G. Johnston wrote: > > On Wed, Mar 26, 2025 at 8:28 PM Sutou Kouhei wrote: >> >> >> The attached v39 patch set uses the followings: >> >> 0001: Create copyto_internal.h and change COPY_XXX to >> COPY_SOURCE_XXX and COPY_DEST_XXX accordingly. >> (S

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-04-04 Thread David G. Johnston
On Friday, March 28, 2025, Masahiko Sawada wrote: > > > One problem in the following chunk I can see is: > > + qualified_format = stringToQualifiedNameList(format, NULL); > + DeconstructQualifiedName(qualified_format, &schema, &fmt); > + if (!schema || strcmp(schema,

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-04-04 Thread Sutou Kouhei
Hi, In "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 31 Mar 2025 12:35:23 -0700, Masahiko Sawada wrote: > Most of the queries under test_copy_format/sql verifies the input > patterns of the FORMAT option. I find that the regression tes

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-04-03 Thread Sutou Kouhei
Hi, In "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 31 Mar 2025 10:05:34 -0700, "David G. Johnston" wrote: > The CopyFromInFunc API allows for each attribute to somehow > have its I/O format individualized. B

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-03-31 Thread David G. Johnston
On Mon, Mar 31, 2025 at 11:52 AM Masahiko Sawada wrote: > On Sat, Mar 29, 2025 at 9:49 AM David G. Johnston > wrote: > > > > On Wed, Mar 26, 2025 at 8:28 PM Sutou Kouhei wrote: > >> > >> > >> The attached v39 patch set uses the followings: > >> > >> 0001: Create copyto_internal.h and change COP

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-03-31 Thread David G. Johnston
On Wed, Mar 26, 2025 at 8:28 PM Sutou Kouhei wrote: > > > --- > > +static const CopyFromRoutine CopyFromRoutineTestCopyFormat = { > > +.type = T_CopyFromRoutine, > > +.CopyFromInFunc = CopyFromInFunc, > > +.CopyFromStart = CopyFromStart, > > +.CopyFromOneRow = Copy

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-03-29 Thread Sutou Kouhei
Hi, In "Re: Make COPY format extendable: Extract COPY TO format implementations" on Sat, 29 Mar 2025 09:48:22 -0700, "David G. Johnston" wrote: > I don't think this module should be responsible for testing the validity of > "qualified names in a strin

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-03-29 Thread David G. Johnston
On Wed, Mar 26, 2025 at 8:28 PM Sutou Kouhei wrote: > > The attached v39 patch set uses the followings: > > 0001: Create copyto_internal.h and change COPY_XXX to > COPY_SOURCE_XXX and COPY_DEST_XXX accordingly. > (Same as 1. in your suggestion) > 0002: Support custom format for both C

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-03-29 Thread David G. Johnston
On Sat, Mar 29, 2025 at 1:57 AM Sutou Kouhei wrote: > * I still think that someone may don't like defining COPY > handlers for built-in formats. If we don't define COPY > handlers for built-in formats finally, we can just drop > 0004. > We should (and usually do) dog-food APIs when reason

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-03-29 Thread Sutou Kouhei
Hi, In "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 28 Mar 2025 22:37:03 -0700, Masahiko Sawada wrote: >> I've added the following tests: >> >> * Wrong input type handler without namespace >> * Wrong input type h

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-03-26 Thread Sutou Kouhei
Hi, In "Re: Make COPY format extendable: Extract COPY TO format implementations" on Sun, 23 Mar 2025 02:01:59 -0700, Masahiko Sawada wrote: > --- > +/* > + * Process the "format" option. > + * > + * This function checks whether the option value is a

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-03-24 Thread Masahiko Sawada
On Wed, Mar 19, 2025 at 6:25 PM Sutou Kouhei wrote: > > Hi, > > In > "Re: Make COPY format extendable: Extract COPY TO format implementations" > on Wed, 19 Mar 2025 17:49:49 -0700, > "David G. Johnston" wrote: > > >> And could someone

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-03-23 Thread Masahiko Sawada
On Wed, Mar 19, 2025 at 6:25 PM Sutou Kouhei wrote: > > Hi, > > In > "Re: Make COPY format extendable: Extract COPY TO format implementations" > on Wed, 19 Mar 2025 17:49:49 -0700, > "David G. Johnston" wrote: > > >> And could someone

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-03-22 Thread David G. Johnston
On Fri, Mar 21, 2025 at 10:23 PM David G. Johnston < david.g.johns...@gmail.com> wrote: > Then this would benefit from the new function I suggest creating since it > apparently has the same, IMO, bug. > > Concretely like I posted here: https://www.postgresql.org/message-id/cakfquwybtck+uw-byfchhp8

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-03-21 Thread David G. Johnston
On Friday, March 21, 2025, Masahiko Sawada wrote: > On Fri, Mar 21, 2025 at 5:32 PM David G. Johnston > wrote: > > > > On Tue, Mar 18, 2025 at 7:56 PM Sutou Kouhei wrote: > >> > >> Hi, > >> > >> In > >> "Re: Make COPY for

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-03-21 Thread Masahiko Sawada
On Fri, Mar 21, 2025 at 5:32 PM David G. Johnston wrote: > > On Tue, Mar 18, 2025 at 7:56 PM Sutou Kouhei wrote: >> >> Hi, >> >> In >> "Re: Make COPY format extendable: Extract COPY TO format implementations" >> on Mon, 17 Mar 2025 13:50:03

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-03-21 Thread David G. Johnston
On Tue, Mar 18, 2025 at 7:56 PM Sutou Kouhei wrote: > Hi, > > In > "Re: Make COPY format extendable: Extract COPY TO format > implementations" on Mon, 17 Mar 2025 13:50:03 -0700, > Masahiko Sawada wrote: > > > I think that built-in formats also nee

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-03-19 Thread Sutou Kouhei
Hi, In "Re: Make COPY format extendable: Extract COPY TO format implementations" on Wed, 19 Mar 2025 17:49:49 -0700, "David G. Johnston" wrote: >> And could someone help (take over if possible) writing a >> document for this feature? I'm not goo

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-03-19 Thread David G. Johnston
On Tue, Mar 18, 2025 at 7:56 PM Sutou Kouhei wrote: > And could someone help (take over if possible) writing a > document for this feature? I'm not good at writing a > document in English... 0009 in the attached v37 patch set > has a draft of it. It's based on existing documents in > doc/src/sgml

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-03-18 Thread Sutou Kouhei
Hi, In "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 17 Mar 2025 13:50:03 -0700, Masahiko Sawada wrote: > I think that built-in formats also need to have their handler > functions. This seems to be a conventional way for customizable &g

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-03-17 Thread Masahiko Sawada
On Tue, Mar 4, 2025 at 4:06 PM Sutou Kouhei wrote: > > Hi, > > In > "Re: Make COPY format extendable: Extract COPY TO format implementations" > on Mon, 3 Mar 2025 11:06:39 -0800, > Masahiko Sawada wrote: > > > I agree with the fix and the patch looks

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-03-04 Thread Sutou Kouhei
Hi, In "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 3 Mar 2025 11:06:39 -0800, Masahiko Sawada wrote: > I agree with the fix and the patch looks good to me. I've updated the > commit message and am going to push, barring any objec

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-03-03 Thread Masahiko Sawada
On Sun, Mar 2, 2025 at 4:19 PM Sutou Kouhei wrote: > > Hi, > > In <3191030.1740932...@sss.pgh.pa.us> > "Re: Make COPY format extendable: Extract COPY TO format implementations" > on Sun, 02 Mar 2025 11:27:20 -0500, > Tom Lane wrote: > > >> W

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-03-02 Thread Junwang Zhao
On Mon, Mar 3, 2025 at 8:19 AM Sutou Kouhei wrote: > > Hi, > > In <3191030.1740932...@sss.pgh.pa.us> > "Re: Make COPY format extendable: Extract COPY TO format implementations" > on Sun, 02 Mar 2025 11:27:20 -0500, > Tom Lane wrote: > > >> W

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-03-02 Thread Sutou Kouhei
Hi, In <3191030.1740932...@sss.pgh.pa.us> "Re: Make COPY format extendable: Extract COPY TO format implementations" on Sun, 02 Mar 2025 11:27:20 -0500, Tom Lane wrote: >> While review another thread (Emitting JSON to file using COPY TO), >> I found the recent

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-03-02 Thread Tom Lane
Junwang Zhao writes: > While review another thread (Emitting JSON to file using COPY TO), > I found the recently committed patches on this thread pass the > CopyFormatOptions struct directly rather a pointer of the struct > as a function parameter of CopyToGetRoutine and CopyFromGetRoutine. Cover

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-03-01 Thread Junwang Zhao
On Sat, Mar 1, 2025 at 10:50 AM Sutou Kouhei wrote: > > Hi, > > Our 0001/0002 patches were merged into master. I've rebased > on master. Can we discuss how to proceed rest patches? > > The contents of them aren't changed but I'll show a summary > of them again: > > 0001-0003 are for COPY TO and 00

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-02-28 Thread Sutou Kouhei
Hi, Our 0001/0002 patches were merged into master. I've rebased on master. Can we discuss how to proceed rest patches? The contents of them aren't changed but I'll show a summary of them again: 0001-0003 are for COPY TO and 0004-0007 are for COPY FROM. For COPY TO: 0001: Add support for adding

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-02-28 Thread Sutou Kouhei
Hi, In "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 28 Feb 2025 14:00:18 -0800, Masahiko Sawada wrote: > Yes, I've submitted the patch to re-export that function[1]. Could you > review it? > > [1] > https:/

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-02-28 Thread Masahiko Sawada
On Fri, Feb 28, 2025 at 1:58 PM Sutou Kouhei wrote: > > Hi, > > In > "Re: Make COPY format extendable: Extract COPY TO format implementations" > on Fri, 28 Feb 2025 11:50:39 -0800, > Masahiko Sawada wrote: > > > I initially thought it would be acce

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-02-28 Thread Sutou Kouhei
Hi, In "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 28 Feb 2025 11:50:39 -0800, Masahiko Sawada wrote: > I initially thought it would be acceptable to stop > NextCopyFromRawFields exposed since NextCopyFrom() could serve as an &g

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-02-28 Thread Masahiko Sawada
On Thu, Feb 27, 2025 at 7:57 PM Sutou Kouhei wrote: > > Hi, > > In > "Re: Make COPY format extendable: Extract COPY TO format implementations" > on Thu, 27 Feb 2025 15:24:26 -0800, > Masahiko Sawada wrote: > > > Pushed the 0001 patch. > > Than

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-02-27 Thread Sutou Kouhei
Hi, In "Re: Make COPY format extendable: Extract COPY TO format implementations" on Thu, 27 Feb 2025 15:24:26 -0800, Masahiko Sawada wrote: > Pushed the 0001 patch. Thanks! > Regarding the 0002 patch, I realized we stopped exposing > NextCopyFromRawFields() functi

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-02-27 Thread Masahiko Sawada
On Tue, Feb 25, 2025 at 6:08 PM Sutou Kouhei wrote: > > Hi, > > In > "Re: Make COPY format extendable: Extract COPY TO format implementations" > on Tue, 25 Feb 2025 17:14:43 -0800, > Masahiko Sawada wrote: > > > I've attached updated patches.

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-02-25 Thread Sutou Kouhei
Hi, In "Re: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 25 Feb 2025 17:14:43 -0800, Masahiko Sawada wrote: > I've attached updated patches. Thanks. I found one more missing last ".": 0002: > --- a/src/backend/commands

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-02-25 Thread Masahiko Sawada
On Tue, Feb 25, 2025 at 3:52 PM Sutou Kouhei wrote: > > Thank you for reviewing the patches. I've addressed comments except for the following comment: > > --- a/src/backend/commands/copyfromparse.c > > +++ b/src/backend/commands/copyfromparse.c > > > @@ -1087,7 +1132,7 @@ NextCopyFrom(CopyFromSt

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-02-25 Thread Masahiko Sawada
On Thu, Feb 20, 2025 at 6:48 PM Sutou Kouhei wrote: > > Hi, > > In > "Re: Make COPY format extendable: Extract COPY TO format implementations" > on Thu, 20 Feb 2025 15:28:26 -0800, > Masahiko Sawada wrote: > > > Looking at the 0001

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-02-25 Thread Sutou Kouhei
Hi, In "Re: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 25 Feb 2025 14:05:28 -0800, Masahiko Sawada wrote: > The first two patches are refactoring patches (+ small performance > improvements). I've reviewed these patches again and a

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-02-20 Thread Sutou Kouhei
Hi, In "Re: Make COPY format extendable: Extract COPY TO format implementations" on Thu, 20 Feb 2025 15:28:26 -0800, Masahiko Sawada wrote: > Looking at the 0001 patch again, I have a question: we have > CopyToTextLikeOneRow() for both CSV and text format: > > +/*

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-02-20 Thread Masahiko Sawada
On Fri, Feb 7, 2025 at 5:01 AM Sutou Kouhei wrote: > > Hi, > > In > "Re: Make COPY format extendable: Extract COPY TO format implementations" > on Tue, 4 Feb 2025 22:20:51 -0800, > Masahiko Sawada wrote: > > >> I was just looking at bit at this

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-02-07 Thread Sutou Kouhei
Hi, In "Re: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 4 Feb 2025 22:20:51 -0800, Masahiko Sawada wrote: >> I was just looking at bit at this series of patch labelled with v31, >> to see what is happening here. >&

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-02-06 Thread Sutou Kouhei
Hi, In "Re: Make COPY format extendable: Extract COPY TO format implementations" on Wed, 5 Feb 2025 12:29:44 -0800, Masahiko Sawada wrote: >> It seems that >> FmgrInfo doesn't have return value type information. Should >> we read pg_catalog.pg_proc or some

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-02-05 Thread Masahiko Sawada
On Tue, Feb 4, 2025 at 11:37 PM Sutou Kouhei wrote: > > Hi, > > In > "Re: Make COPY format extendable: Extract COPY TO format implementations" > on Tue, 4 Feb 2025 17:32:07 -0800, > Masahiko Sawada wrote: > > > BTW we need to check if the return v

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-02-05 Thread Álvaro Herrera
On 2025-Feb-05, Masahiko Sawada wrote: > I think that the patch needs to check if the function's result type is > COPY_HANDLEROID by using get_func_rettype(), before calling it. But > with this check, we can prevent arbitrary functions from being called > via COPY. Why do we need to extend CREATE

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-02-05 Thread Masahiko Sawada
On Wed, Feb 5, 2025 at 3:49 AM Álvaro Herrera wrote: > > On 2025-Feb-03, Vladlen Popolitov wrote: > > > You use FORMAT option to add new formats, filling it with routine name > > in shared library. As result any caller can call any routine in PostgreSQL > > kernel. > > I think, it will start compe

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-02-05 Thread Vladlen Popolitov
Álvaro Herrera писал(а) 2025-02-05 18:49: On 2025-Feb-03, Vladlen Popolitov wrote: You use FORMAT option to add new formats, filling it with routine name in shared library. As result any caller can call any routine in PostgreSQL kernel. I think, it will start competition, who can find most da

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-02-05 Thread Álvaro Herrera
On 2025-Feb-03, Vladlen Popolitov wrote: > You use FORMAT option to add new formats, filling it with routine name > in shared library. As result any caller can call any routine in PostgreSQL > kernel. > I think, it will start competition, who can find most dangerous routine > to call just from COP

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-02-04 Thread Sutou Kouhei
Hi, In "Re: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 4 Feb 2025 17:32:07 -0800, Masahiko Sawada wrote: > BTW we need to check if the return value type of the handler function > is copy_handler. Oh, can we do it without calling a functio

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-02-04 Thread Sutou Kouhei
Hi, In "Re: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 04 Feb 2025 17:46:10 +0700, Vladlen Popolitov wrote: > I think, in case of USING PostgreSQL kernel will call corresponding > handler, > and it looks secure - the same as for tabl

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-02-04 Thread Masahiko Sawada
On Tue, Feb 4, 2025 at 9:10 PM Michael Paquier wrote: > > On Sat, Feb 01, 2025 at 07:12:01PM +0900, Sutou Kouhei wrote: > > For the propose, copyapi.h should not include > > copy{to,from}_internal.h. If we do it, copyto.c includes > > CopyFromState and copyfrom*.c include CopyToState. > > > > What

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-02-04 Thread Masahiko Sawada
On Tue, Feb 4, 2025 at 6:19 PM Vladlen Popolitov wrote: > > Masahiko Sawada писал(а) 2025-02-05 08:32: > > On Tue, Feb 4, 2025 at 2:46 AM Vladlen Popolitov > > >> >> Standard PostgreSQL realisation for new methods to use USING > >> >> keyword. Every > >> >> new method could have own options (FOR

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-02-04 Thread Michael Paquier
On Sat, Feb 01, 2025 at 07:12:01PM +0900, Sutou Kouhei wrote: > For the propose, copyapi.h should not include > copy{to,from}_internal.h. If we do it, copyto.c includes > CopyFromState and copyfrom*.c include CopyToState. > > What do you think about the following change? Note that > extensions mus

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-02-04 Thread Vladlen Popolitov
Masahiko Sawada писал(а) 2025-02-05 08:32: On Tue, Feb 4, 2025 at 2:46 AM Vladlen Popolitov >> Standard PostgreSQL realisation for new methods to use USING >> keyword. Every >> new method could have own options (FORMAT is option of internal 'copy >> from/to' >> methods), > > Ah, I didn't thi

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-02-04 Thread Masahiko Sawada
On Tue, Feb 4, 2025 at 2:46 AM Vladlen Popolitov wrote: > > Sutou Kouhei писал(а) 2025-02-04 13:29: > Hi > > Hi, > > > > In > > "Re: Make COPY format extendable: Extract COPY TO format > > implementations" on Mon, 03 Feb 2025 13:38:04 +0700,

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-02-04 Thread Vladlen Popolitov
Sutou Kouhei писал(а) 2025-02-04 13:29: Hi Hi, In "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 03 Feb 2025 13:38:04 +0700, Vladlen Popolitov wrote: I would like to inform about the security breach in your design of COPY TO/FROM. Thanks

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-02-03 Thread Sutou Kouhei
Hi, In "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 03 Feb 2025 13:38:04 +0700, Vladlen Popolitov wrote: > I would like to inform about the security breach in your design of > COPY TO/FROM. Thanks! I didn't notice it. > You

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-02-02 Thread Vladlen Popolitov
Sutou Kouhei писал(а) 2025-02-01 17:12: Hi, Hi I would like to inform about the security breach in your design of COPY TO/FROM. You use FORMAT option to add new formats, filling it with routine name in shared library. As result any caller can call any routine in PostgreSQL kernel. I think

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-02-01 Thread Sutou Kouhei
Hi, In "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 31 Jan 2025 16:34:52 -0800, Masahiko Sawada wrote: > Again, what I'd like to avoid is that we end up adding everything > (including new fields we add in the future) related t

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-01-31 Thread Masahiko Sawada
On Fri, Jan 31, 2025 at 3:10 PM Sutou Kouhei wrote: > > Hi, > > In > "Re: Make COPY format extendable: Extract COPY TO format implementations" > on Fri, 31 Jan 2025 14:25:34 -0800, > Masahiko Sawada wrote: > > > I think that CopyToState and CopyFrom

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-01-31 Thread Sutou Kouhei
Hi, In "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 31 Jan 2025 14:25:34 -0800, Masahiko Sawada wrote: > I think that CopyToState and CopyFromState are not APIs but the > execution states. I'm not against exposing CopyToState and

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-01-31 Thread Masahiko Sawada
On Thu, Jan 30, 2025 at 7:42 AM Sutou Kouhei wrote: > > Hi, > > In > "Re: Make COPY format extendable: Extract COPY TO format implementations" > on Tue, 28 Jan 2025 15:00:03 -0800, > Masahiko Sawada wrote: > > > While 0001 and 0002 look goo

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-01-30 Thread Sutou Kouhei
Hi, In "Re: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 28 Jan 2025 15:00:03 -0800, Masahiko Sawada wrote: > While 0001 and 0002 look good to me overall, we still need to polish > subsequent patches. Here are review comments: I attached t

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-01-28 Thread Masahiko Sawada
On Thu, Jan 23, 2025 at 1:12 AM Sutou Kouhei wrote: > > Hi, > > I noticed that the last patch set (v27) can't be applied to > the current master. I've rebased on the current master and > created v28 patch set. No code change. Thank you for updating the patch! While 0001 and 0002 look good to me

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-01-23 Thread Sutou Kouhei
Hi, I noticed that the last patch set (v27) can't be applied to the current master. I've rebased on the current master and created v28 patch set. No code change. Thanks, -- kou >From 016ccfc63d2faa441a6996e3dcfd3cdbff7c185f Mon Sep 17 00:00:00 2001 From: Sutou Kouhei Date: Sat, 28 Sep 2024 23:

Re: Make COPY format extendable: Extract COPY TO format implementations

2025-01-23 Thread Sutou Kouhei
Hi, In "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 29 Nov 2024 10:15:04 +0800, Junwang Zhao wrote: > This thread and [1] thread are kind of interleaved, I chose this thread > to share the numbers because I think this feature sho

  1   2   3   >