Re: confusing / inefficient "need_transcoding" handling in copy

2024-12-15 Thread Sutou Kouhei
Hi, In "Re: confusing / inefficient "need_transcoding" handling in copy" on Mon, 16 Dec 2024 11:25:16 +0900, Michael Paquier wrote: > On Sat, Dec 14, 2024 at 04:46:57PM +0900, Michael Paquier wrote: >> Note that using "test" as table name for the tes

Re: confusing / inefficient "need_transcoding" handling in copy

2024-12-15 Thread Michael Paquier
On Sat, Dec 14, 2024 at 04:46:57PM +0900, Michael Paquier wrote: > Note that using "test" as table name for the tests is not a good idea, > as this could very likely conflict with some concurrent activity. I > would also add two RESET queries to remove the dependency to > client_encoding once the

Re: confusing / inefficient "need_transcoding" handling in copy

2024-12-13 Thread Michael Paquier
On Fri, Dec 13, 2024 at 12:27:37PM +0900, Sutou Kouhei wrote: > Oh, sorry... I attached wrong patch... > I attach the v4 patch that includes this case. Sounds fair to me as a beginning for the code paths setting need_transcoding. Note that using "test" as table name for the tests is not a good id

Re: confusing / inefficient "need_transcoding" handling in copy

2024-12-12 Thread Sutou Kouhei
Hi, In "Re: confusing / inefficient "need_transcoding" handling in copy" on Fri, 13 Dec 2024 12:03:45 +0900, Michael Paquier wrote: >> OK. I've added valid cases too by using LATIN1 as you >> suggested. > > I may have missed something but v3

Re: confusing / inefficient "need_transcoding" handling in copy

2024-12-12 Thread Michael Paquier
On Thu, Dec 12, 2024 at 03:25:41PM +0900, Sutou Kouhei wrote: > In > "Re: confusing / inefficient "need_transcoding" handling in copy" on Tue, > 10 Dec 2024 13:59:25 +0900, > Michael Paquier wrote: >> Another one would be valid conversions back and

Re: confusing / inefficient "need_transcoding" handling in copy

2024-12-11 Thread Sutou Kouhei
Hi, In "Re: confusing / inefficient "need_transcoding" handling in copy" on Tue, 10 Dec 2024 13:59:25 +0900, Michael Paquier wrote: > client_encoding would be used by COPY when not specifying ENCODING > option. Perhaps more tests should be added with this

Re: confusing / inefficient "need_transcoding" handling in copy

2024-12-09 Thread Michael Paquier
On Fri, Dec 06, 2024 at 04:20:42PM +0900, Sutou Kouhei wrote: > (Do you think that this patch is still needed?) This thread has fallen off my radar, my apologies about that. Yes, I think that this is a good thing to expand these tests. Let's take one step at a time. I have a couple of comments.

Re: confusing / inefficient "need_transcoding" handling in copy

2024-12-05 Thread Sutou Kouhei
Hi Michael, ping. (Do you think that this patch is still needed?) Thanks, -- kou In <20240214.114608.2091541942684063981@clear-code.com> "Re: confusing / inefficient "need_transcoding" handling in copy" on Wed, 14 Feb 2024 11:46:08 +0900 (JST), Sutou Kou

Re: confusing / inefficient "need_transcoding" handling in copy

2024-02-13 Thread Sutou Kouhei
Hi, In "Re: confusing / inefficient "need_transcoding" handling in copy" on Wed, 14 Feb 2024 06:56:16 +0900, Michael Paquier wrote: > We have a couple of non-ASCII characters in the tests, but I suspect > that this one will not be digested correctly everywhere,

Re: confusing / inefficient "need_transcoding" handling in copy

2024-02-13 Thread Michael Paquier
On Thu, Feb 08, 2024 at 05:25:01PM +0900, Sutou Kouhei wrote: > In <20240206222445.hzq22pb2nye7r...@awork3.anarazel.de> > "Re: confusing / inefficient "need_transcoding" handling in copy" on Tue, 6 > Feb 2024 14:24:45 -0800, > Andres Freund wrote: >

Re: confusing / inefficient "need_transcoding" handling in copy

2024-02-08 Thread Andres Freund
Hi, On 2024-02-09 09:36:28 +0900, Michael Paquier wrote: > On Thu, Feb 08, 2024 at 10:25:07AM +0200, Heikki Linnakangas wrote: > > There's no validation, just conversion. I'd suggest: > > > > "Set up encoding conversion info if the file and server encodings differ > > (see also pg_server_to_any).

Re: confusing / inefficient "need_transcoding" handling in copy

2024-02-08 Thread Michael Paquier
On Thu, Feb 08, 2024 at 10:25:07AM +0200, Heikki Linnakangas wrote: > There's no validation, just conversion. I'd suggest: > > "Set up encoding conversion info if the file and server encodings differ > (see also pg_server_to_any)." > > Other than that, +1 Cool. I've used your wording and applie

Re: confusing / inefficient "need_transcoding" handling in copy

2024-02-08 Thread Michael Paquier
On Thu, Feb 08, 2024 at 05:29:46PM +0900, Sutou Kouhei wrote: > Oh, sorry. I missed the Michael's patch: > https://www.postgresql.org/message-id/flat/ZcR9Q9hJ8GedFSCd%40paquier.xyz#e73272b042a22befac7a95f7bcb4fb9a > > I withdraw my change. No problem. Thanks for caring about that. -- Michael s

Re: confusing / inefficient "need_transcoding" handling in copy

2024-02-08 Thread Sutou Kouhei
Hi, In <20240208.172501.2177371292839763981@clear-code.com> "Re: confusing / inefficient "need_transcoding" handling in copy" on Thu, 08 Feb 2024 17:25:01 +0900 (JST), Sutou Kouhei wrote: > How about the following to avoid needless transcoding? Oh, sorry.

Re: confusing / inefficient "need_transcoding" handling in copy

2024-02-08 Thread Heikki Linnakangas
On 08/02/2024 09:05, Michael Paquier wrote: On Tue, Feb 06, 2024 at 02:24:45PM -0800, Andres Freund wrote: I think the code is just very confusing - there actually *is* verification of the encoding, it just happens at a different, earlier, layer, namely in copyfromparse.c: CopyConvertBuf() which

Re: confusing / inefficient "need_transcoding" handling in copy

2024-02-08 Thread Sutou Kouhei
Hi, In <20240206222445.hzq22pb2nye7r...@awork3.anarazel.de> "Re: confusing / inefficient "need_transcoding" handling in copy" on Tue, 6 Feb 2024 14:24:45 -0800, Andres Freund wrote: > One unfortunate issue: We don't have any tests verifying that COPY FR

Re: confusing / inefficient "need_transcoding" handling in copy

2024-02-07 Thread Michael Paquier
On Tue, Feb 06, 2024 at 02:24:45PM -0800, Andres Freund wrote: > I think the code is just very confusing - there actually *is* verification of > the encoding, it just happens at a different, earlier, layer, namely in > copyfromparse.c: CopyConvertBuf() which says: > /* >* If the file

Re: confusing / inefficient "need_transcoding" handling in copy

2024-02-06 Thread Andres Freund
Hi, On 2024-02-06 12:51:48 -0500, Tom Lane wrote: > Michael Paquier writes: > > On Mon, Feb 05, 2024 at 06:05:04PM -0800, Andres Freund wrote: > >> I haven't yet dug into the code history. One guess is that this should only > >> have been set this way for COPY FROM. > > > Looking the git history,

Re: confusing / inefficient "need_transcoding" handling in copy

2024-02-06 Thread Tom Lane
Michael Paquier writes: > On Mon, Feb 05, 2024 at 06:05:04PM -0800, Andres Freund wrote: >> I haven't yet dug into the code history. One guess is that this should only >> have been set this way for COPY FROM. > Looking the git history, this looks like an oversight of c61a2f58418e > that has added

Re: confusing / inefficient "need_transcoding" handling in copy

2024-02-05 Thread Michael Paquier
On Mon, Feb 05, 2024 at 06:05:04PM -0800, Andres Freund wrote: > I don't really understand why we need to validate anything during COPY TO? > Which is good, because it turns out that we don't actually validate anything, > as pg_server_to_any() returns without doing anything if the encoding matches: