Re: New "single" COPY format

2025-02-18 Thread Andrew Dunstan
On 2025-02-17 Mo 7:05 PM, David G. Johnston wrote: On Sat, Dec 21, 2024 at 1:57 AM Joel Jacobson wrote: How about: COPY table_name [ ( column_name ) ] { FROM | TO } 'filename' (FORMAT jsonb); - If column list is omitted, table_name must have exactly one column. - If colum

Re: New "single" COPY format

2025-02-17 Thread David G. Johnston
On Sat, Dec 21, 2024 at 1:57 AM Joel Jacobson wrote: > How about: > > COPY table_name [ ( column_name ) ] { FROM | TO } 'filename' (FORMAT > jsonb); > > - If column list is omitted, table_name must have exactly one column. > - If column list is specified, it must be of length one. > - The column

Re: New "single" COPY format

2024-12-21 Thread Joel Jacobson
On Thu, Dec 19, 2024, at 14:40, Andrew Dunstan wrote: > We seem to have got seriously into the weeds, here. I'd be sorry to see > this dropped. After all, it's not something new, and while we have a > sort of workaround for "one json doc per line" it's far from obvious, > and except in a few blo

Re: New "single" COPY format

2024-12-19 Thread Andrew Dunstan
On 2024-12-16 Mo 10:09 AM, Joel Jacobson wrote: Hi hackers, After further consideration, I'm withdrawing the patch. Some fundamental questions remain unresolved: - Should round-trip fidelity be a strict goal? By "round-trip fidelity", I mean that data exported and then re-imported should y

Re: New "single" COPY format

2024-12-19 Thread Joel Jacobson
On Thu, Dec 19, 2024, at 07:48, jian he wrote: > I have reviewed v21-0001 again. > > v21-0001-Introduce-CopyFormat-and-replace-csv_mode-and-binary.patch > is a good refactor. > > > overall looks good to me. OK, I could submit it as a separate patch. Would we also want the reorganization of existi

Re: New "single" COPY format

2024-12-18 Thread jian he
I have reviewed v21-0001 again. v21-0001-Introduce-CopyFormat-and-replace-csv_mode-and-binary.patch is a good refactor. overall looks good to me.

Re: New "single" COPY format

2024-12-16 Thread Joel Jacobson
Hi hackers, After further consideration, I'm withdrawing the patch. Some fundamental questions remain unresolved: - Should round-trip fidelity be a strict goal? By "round-trip fidelity", I mean that data exported and then re-imported should yield exactly the original values, including the dis

Re: New "single" COPY format

2024-11-10 Thread Joel Jacobson
On Sun, Nov 10, 2024, at 09:00, Joel Jacobson wrote: >> It's a bit of a hack, but I kinda like it., since it seems like the >> only option without an error situation. I forgot about the error situation when a textual contain newline characters, that remains the same for option A, B and C. /Joel

Re: New "single" COPY format

2024-11-10 Thread Joel Jacobson
On Sun, Nov 10, 2024, at 08:48, Joel Jacobson wrote: > On Sun, Nov 10, 2024, at 08:32, Joel Jacobson wrote: >> Option A: >> COPY TO: Empty string field gets exported as an empty line. NULL field >> is an error. >> COPY FROM: Empty line is imported as an empty string. >> >> Option B: >> COPY TO: NU

Re: New "single" COPY format

2024-11-09 Thread Joel Jacobson
On Sun, Nov 10, 2024, at 08:32, Joel Jacobson wrote: > Option A: > COPY TO: Empty string field gets exported as an empty line. NULL field > is an error. > COPY FROM: Empty line is imported as an empty string. > > Option B: > COPY TO: NULL field gets exported as an empty line. Empty string field >

Re: New "single" COPY format

2024-11-09 Thread Joel Jacobson
On Sun, Nov 10, 2024, at 05:55, David G. Johnston wrote: > On Saturday, November 9, 2024, jian he wrote: >> >> The list format does not distinguish a >> NULL >> value from an empty string. Empty lines are imported as empty strings, >> not >> as NULL values. >> >> we only ment

Re: New "single" COPY format

2024-11-09 Thread Joel Jacobson
On Sun, Nov 10, 2024, at 05:26, jian he wrote: > On Sun, Nov 10, 2024 at 3:29 AM Joel Jacobson wrote: >> >> Cool. I've drafted a new patch on this approach. >> The list of newline-free built-in types is not exhaustive, yet. > > > do we care that COPY back and forth always work? Yes, I think that'

Re: New "single" COPY format

2024-11-09 Thread David G. Johnston
On Saturday, November 9, 2024, jian he wrote: > > > > The list format does not distinguish a > NULL > value from an empty string. Empty lines are imported as empty strings, > not > as NULL values. > > we only mentioned import, not export (COPY TO) dealing with > NULL value. > >

Re: New "single" COPY format

2024-11-09 Thread jian he
On Sun, Nov 10, 2024 at 3:29 AM Joel Jacobson wrote: > > Cool. I've drafted a new patch on this approach. > The list of newline-free built-in types is not exhaustive, yet. do we care that COPY back and forth always work? doc not mentioned, but seems it's an implicit idea. copy the_table to '/t

Re: New "single" COPY format

2024-11-09 Thread David G. Johnston
On Sat, Nov 9, 2024 at 1:48 PM Joel Jacobson wrote: > On Sat, Nov 9, 2024, at 15:28, David G. Johnston wrote: > > PostgreSQL cannot store the NUL byte. Would that be an option for the > > record separator. Default to new line but accept NUL if one needs to > > input/output lists containing newl

Re: New "single" COPY format

2024-11-09 Thread Joel Jacobson
On Sat, Nov 9, 2024, at 15:28, David G. Johnston wrote: > PostgreSQL cannot store the NUL byte. Would that be an option for the > record separator. Default to new line but accept NUL if one needs to > input/output lists containing newlines. Or whatever character the user > believes is not par

Re: New "single" COPY format

2024-11-09 Thread Joel Jacobson
On Sat, Nov 9, 2024, at 15:13, David G. Johnston wrote: > On Saturday, November 9, 2024, Joel Jacobson wrote: >> On Sat, Nov 9, 2024, at 08:07, Joel Jacobson wrote: >> > Here's a draft of an idea I'm considering (not yet implemented): >> >> I realize the last part about optional quoting is unnece

Re: New "single" COPY format

2024-11-09 Thread David G. Johnston
On Saturday, November 9, 2024, Joel Jacobson wrote: > On Fri, Nov 8, 2024, at 22:47, David G. Johnston wrote: > > On Fri, Nov 8, 2024 at 2:20 PM Joel Jacobson wrote: > >> > >> 1. Text files containing \. in the middle of the file > >> % cat /tmp/test.txt > >> foo > >> \. > >> bar > >> > >> Or an

Re: New "single" COPY format

2024-11-09 Thread David G. Johnston
On Saturday, November 9, 2024, Joel Jacobson wrote: > On Sat, Nov 9, 2024, at 08:07, Joel Jacobson wrote: > > Here's a draft of an idea I'm considering (not yet implemented): > > I realize the last part about optional quoting is unnecessary, > since if quoting is desired, users could just use the

Re: New "single" COPY format

2024-11-09 Thread Joel Jacobson
On Sat, Nov 9, 2024, at 08:07, Joel Jacobson wrote: > Here's a draft of an idea I'm considering (not yet implemented): I realize the last part about optional quoting is unnecessary, since if quoting is desired, users could just use the 'csv' format. Revised draft of the idea (not yet implemented)

Re: New "single" COPY format

2024-11-08 Thread Joel Jacobson
On Fri, Nov 8, 2024, at 22:47, David G. Johnston wrote: > On Fri, Nov 8, 2024 at 2:20 PM Joel Jacobson wrote: >> >> 1. Text files containing \. in the middle of the file >> % cat /tmp/test.txt >> foo >> \. >> bar >> >> Or another option to turn off the special meaning of \.? > > This does seem l

Re: New "single" COPY format

2024-11-08 Thread David G. Johnston
On Fri, Nov 8, 2024 at 2:20 PM Joel Jacobson wrote: > > 1. Text files containing \. in the middle of the file > % cat /tmp/test.txt > foo > \. > bar > > Or another option to turn off the special meaning of \.? > This does seem like an orthogonal option worth considering. > > Besides, "single"

Re: New "single" COPY format

2024-11-08 Thread Joel Jacobson
On Fri, Nov 8, 2024, at 20:44, Daniel Verite wrote: > Aleksander Alekseev wrote: > >> IMO it should be 'text' we already have with special options e.g. >> DELIMITER AS NULL ESCAPE AS NULL. If there are no escape characters >> and column delimiters (and no NULLs designations, and what else I >> forg

Re: New "single" COPY format

2024-11-08 Thread Daniel Verite
Aleksander Alekseev wrote: > IMO it should be 'text' we already have with special options e.g. > DELIMITER AS NULL ESCAPE AS NULL. If there are no escape characters > and column delimiters (and no NULLs designations, and what else I > forgot) then your text file just contains one tuple per

Re: New "single" COPY format

2024-11-08 Thread Joel Jacobson
On Thu, Nov 7, 2024, at 17:15, Joel Jacobson wrote: > Attachments: > * v18-0001-Introduce-CopyFormat-and-replace-csv_mode-and-binary.patch > * v18-0002-Add-COPY-format-single.patch > * v18-0003-Reorganize-option-validations.patch I want to bring up a potential problem with v18, which has been disc

Re: New "single" COPY format

2024-11-08 Thread Joel Jacobson
On Fri, Nov 8, 2024, at 12:25, Aleksander Alekseev wrote: > Sorry for being late for the discussion. No worries, better late than never, thanks for chiming in. > I disagree with the idea of adding a new format name for this. Mostly > because it is *not* a new format and pretending that it is will

Re: New "single" COPY format

2024-11-08 Thread Aleksander Alekseev
Hi Joel, > Recap: This is about adding support to import/export text-based formats such > as > JSONL, or any unstructured text file, where wanting to import each line "as > is" > into a single column, or wanting to export a single column to a text file. Sorry for being late for the discussion.

Re: New "single" COPY format

2024-11-08 Thread Aleksander Alekseev
Hi, > > Recap: This is about adding support to import/export text-based formats > > such as > > JSONL, or any unstructured text file, where wanting to import each line "as > > is" > > into a single column, or wanting to export a single column to a text file. > > Sorry for being late for the disc

Re: New "single" COPY format

2024-11-08 Thread Joel Jacobson
On Fri, Nov 8, 2024, at 08:42, Joel Jacobson wrote: >> I’d be concerned choosing “single” given this future possibility. I do >> agree that such an enhancement would be best done in its own patch. > > OK, sounds good to do it in its own patch. > > If the name "single" doesn't work for this reason

Re: New "single" COPY format

2024-11-07 Thread Joel Jacobson
On Fri, Nov 8, 2024, at 07:14, David G. Johnston wrote: > On Thursday, November 7, 2024, Joel Jacobson wrote: >> On Fri, Nov 8, 2024, at 00:13, Masahiko Sawada wrote: >> > In earlier versions, we supported loading the whole file into a single >> > tuple. Is there any reason that it doesn't support

Re: New "single" COPY format

2024-11-07 Thread David G. Johnston
On Thursday, November 7, 2024, Joel Jacobson wrote: > On Fri, Nov 8, 2024, at 00:13, Masahiko Sawada wrote: > > In earlier versions, we supported loading the whole file into a single > > tuple. Is there any reason that it doesn't support it in v18? I think > > if it's useful we can improve it in

Re: New "single" COPY format

2024-11-07 Thread Joel Jacobson
On Fri, Nov 8, 2024, at 00:13, Masahiko Sawada wrote: > In earlier versions, we supported loading the whole file into a single > tuple. Is there any reason that it doesn't support it in v18? I think > if it's useful we can improve it in a separate patch. Not sure how useful it is, since we already

Re: New "single" COPY format

2024-11-07 Thread Masahiko Sawada
Hi, On Thu, Nov 7, 2024 at 8:16 AM Joel Jacobson wrote: > > Hi hackers, > > Thread [1] renamed, since the format name has now been changed from 'raw' to > 'single', as suggested by Andrew Dunstan and Jacob Champion. > > [1] https://postgr.es/m/c12516b1-77dc-4ad3-94a7-88527360a...@app.fastmail.com

New "single" COPY format

2024-11-07 Thread Joel Jacobson
Hi hackers, Thread [1] renamed, since the format name has now been changed from 'raw' to 'single', as suggested by Andrew Dunstan and Jacob Champion. [1] https://postgr.es/m/c12516b1-77dc-4ad3-94a7-88527360a...@app.fastmail.com Recap: This is about adding support to import/export text-based form