Re: [HACKERS] Logical Replication and Character encoding

2017-04-06 Thread Kyotaro HORIGUCHI
At Thu, 6 Apr 2017 14:43:56 -0400, Peter Eisentraut wrote in <330a093a-d155-c866-cbf2-8f36fdf51...@2ndquadrant.com> > On 4/6/17 11:47, Peter Eisentraut wrote: > > On 4/5/17 21:32, Kyotaro HORIGUCHI wrote: > >> At Wed, 5 Apr 2017 11:33:51 -0400, Peter Eisentraut > >> wrote in > >> <5401fef6-c0

Re: [HACKERS] Logical Replication and Character encoding

2017-04-06 Thread Peter Eisentraut
On 4/6/17 11:47, Peter Eisentraut wrote: > On 4/5/17 21:32, Kyotaro HORIGUCHI wrote: >> At Wed, 5 Apr 2017 11:33:51 -0400, Peter Eisentraut >> wrote in >> <5401fef6-c0c0-7e8a-d8b1-169e30cbd...@2ndquadrant.com> >>> After further thinking, I prefer the alternative approach of using >>> pq_sendcoun

Re: [HACKERS] Logical Replication and Character encoding

2017-04-06 Thread Peter Eisentraut
On 4/5/17 21:32, Kyotaro HORIGUCHI wrote: > At Wed, 5 Apr 2017 11:33:51 -0400, Peter Eisentraut > wrote in > <5401fef6-c0c0-7e8a-d8b1-169e30cbd...@2ndquadrant.com> >> After further thinking, I prefer the alternative approach of using >> pq_sendcountedtext() as is and sticking the trailing zero b

Re: [HACKERS] Logical Replication and Character encoding

2017-04-05 Thread Kyotaro HORIGUCHI
At Wed, 5 Apr 2017 11:33:51 -0400, Peter Eisentraut wrote in <5401fef6-c0c0-7e8a-d8b1-169e30cbd...@2ndquadrant.com> > After further thinking, I prefer the alternative approach of using > pq_sendcountedtext() as is and sticking the trailing zero byte on on the > receiving side. This is a more lo

Re: [HACKERS] Logical Replication and Character encoding

2017-04-05 Thread Peter Eisentraut
After further thinking, I prefer the alternative approach of using pq_sendcountedtext() as is and sticking the trailing zero byte on on the receiving side. This is a more localized change, and keeps the logical replication protocol consistent with the main FE/BE protocol. (Also, we don't need to

Re: [HACKERS] Logical Replication and Character encoding

2017-03-21 Thread Kyotaro HORIGUCHI
Mmm. I shot the previous mail halfway. At Mon, 6 Mar 2017 11:13:48 +0100, Petr Jelinek wrote in > > By the way, I noticed that postmaster launches logical > > replication launcher even if wal_level < logical. Is it right? > > Yes, that came up couple of times in various threads. The logical >

Re: [HACKERS] Logical Replication and Character encoding

2017-03-21 Thread Kyotaro HORIGUCHI
At Mon, 6 Mar 2017 11:13:48 +0100, Petr Jelinek wrote in > >>> Well the length is necessary to be able to add binary format support in > >>> future so it's definitely not an omission. > >> > >> Right. So it appears the right function to use here is > >> pq_sendcountedtext(). However, this send

Re: [HACKERS] Logical Replication and Character encoding

2017-03-06 Thread Petr Jelinek
On 06/03/17 11:06, Kyotaro HORIGUCHI wrote: > At Fri, 3 Mar 2017 21:31:24 -0500, Peter Eisentraut > wrote in > <88397afa-a8ec-8d8a-1c94-94a4795a3...@2ndquadrant.com> >> On 3/3/17 14:51, Petr Jelinek wrote: >>> On 03/03/17 20:37, Peter Eisentraut wrote: On 2/27/17 00:23, Kyotaro HORIGUCHI wr

Re: [HACKERS] Logical Replication and Character encoding

2017-03-06 Thread Kyotaro HORIGUCHI
At Fri, 3 Mar 2017 21:31:24 -0500, Peter Eisentraut wrote in <88397afa-a8ec-8d8a-1c94-94a4795a3...@2ndquadrant.com> > On 3/3/17 14:51, Petr Jelinek wrote: > > On 03/03/17 20:37, Peter Eisentraut wrote: > >> On 2/27/17 00:23, Kyotaro HORIGUCHI wrote: > >>> Yeah, the patch sends converted string w

Re: [HACKERS] Logical Replication and Character encoding

2017-03-03 Thread Peter Eisentraut
On 3/3/17 14:51, Petr Jelinek wrote: > On 03/03/17 20:37, Peter Eisentraut wrote: >> On 2/27/17 00:23, Kyotaro HORIGUCHI wrote: >>> Yeah, the patch sends converted string with the length of the >>> orignal length. Usually encoding conversion changes the length of >>> a string. I doubt that the reve

Re: [HACKERS] Logical Replication and Character encoding

2017-03-03 Thread Petr Jelinek
On 03/03/17 20:37, Peter Eisentraut wrote: > On 2/27/17 00:23, Kyotaro HORIGUCHI wrote: >> Yeah, the patch sends converted string with the length of the >> orignal length. Usually encoding conversion changes the length of >> a string. I doubt that the reverse case was working correctly. > > I thin

Re: [HACKERS] Logical Replication and Character encoding

2017-03-03 Thread Peter Eisentraut
On 2/27/17 00:23, Kyotaro HORIGUCHI wrote: > Yeah, the patch sends converted string with the length of the > orignal length. Usually encoding conversion changes the length of > a string. I doubt that the reverse case was working correctly. I think we shouldn't send the length value at all. This m

Re: [HACKERS] Logical Replication and Character encoding

2017-02-26 Thread Kyotaro HORIGUCHI
; >Cc: cr...@2ndquadrant.com; Shinoda, Noriyoshi ; > >pgsql-hackers@postgresql.org > >Subject: Re: [HACKERS] Logical Replication and Character encoding > > > >On 2/17/17 10:14, Peter Eisentraut wrote: > >> Well, it is sort of a libpq connection, and a proper libpq client

Re: [HACKERS] Logical Replication and Character encoding

2017-02-23 Thread Shinoda, Noriyoshi
>From: Peter Eisentraut [mailto:peter.eisentr...@2ndquadrant.com] >Sent: Friday, February 24, 2017 1:32 AM >To: Petr Jelinek ; Kyotaro HORIGUCHI > >Cc: cr...@2ndquadrant.com; Shinoda, Noriyoshi ; >pgsql-hackers@postgresql.org >Subject: Re: [HACKERS] Logical Replication a

Re: [HACKERS] Logical Replication and Character encoding

2017-02-23 Thread Peter Eisentraut
On 2/17/17 10:14, Peter Eisentraut wrote: > Well, it is sort of a libpq connection, and a proper libpq client should > set the client encoding, and a proper libpq server should do encoding > conversion accordingly. If we just play along with this, it all works > correctly. > > Other output plugin

Re: [HACKERS] Logical Replication and Character encoding

2017-02-17 Thread Peter Eisentraut
On 2/15/17 17:55, Petr Jelinek wrote: > I am not quite convinced that this should be handled by logical decoding > itself. It's quite possible to have output plugins that will handle this > correctly for their use-cases (by doing similar conversion you did in > the original patch) so they should no

Re: [HACKERS] Logical Replication and Character encoding

2017-02-15 Thread Petr Jelinek
On 14/02/17 03:23, Kyotaro HORIGUCHI wrote: > At Sat, 4 Feb 2017 21:27:32 +0100, Petr Jelinek > wrote in > >> Hmm I wonder if we should just make the subscriber send the >> client_encoding always (based on server encoding of the subscriber). >> That should solve the issue in combination with yo

Re: [HACKERS] Logical Replication and Character encoding

2017-02-13 Thread Kyotaro HORIGUCHI
At Sat, 4 Feb 2017 21:27:32 +0100, Petr Jelinek wrote in > Hmm I wonder if we should just make the subscriber send the > client_encoding always (based on server encoding of the subscriber). > That should solve the issue in combination with your patch no? Yeah, right. I considered that a subscri

Re: [HACKERS] Logical Replication and Character encoding

2017-02-04 Thread Petr Jelinek
On 01/02/17 04:05, Kyotaro HORIGUCHI wrote: > Hello, > > At Tue, 31 Jan 2017 12:46:18 +, "Shinoda, Noriyoshi" > wrote in > >> I tried a committed Logical Replication environment. I found >> that replication between databases of different encodings did >> not convert encodings in charact

Re: [HACKERS] Logical Replication and Character encoding

2017-02-03 Thread Kyotaro HORIGUCHI
Hello, At Fri, 3 Feb 2017 13:47:54 +0800, Craig Ringer wrote in > On 3 Feb. 2017 15:47, "Kyotaro HORIGUCHI" > wrote: > > Hello, > > At Fri, 3 Feb 2017 09:16:47 +0800, Craig Ringer > wrote in > > > On 2 February 2017 at 11:45, Euler Taveira wrote: > > > > > I don't think storage without co

Re: [HACKERS] Logical Replication and Character encoding

2017-02-02 Thread Craig Ringer
On 3 Feb. 2017 15:47, "Kyotaro HORIGUCHI" wrote: Hello, At Fri, 3 Feb 2017 09:16:47 +0800, Craig Ringer wrote in > On 2 February 2017 at 11:45, Euler Taveira wrote: > > > I don't think storage without conversion is an acceptable approach. We > > should provide options to users such as ignore

Re: [HACKERS] Logical Replication and Character encoding

2017-02-02 Thread Kyotaro HORIGUCHI
Hello, At Fri, 3 Feb 2017 09:16:47 +0800, Craig Ringer wrote in > On 2 February 2017 at 11:45, Euler Taveira wrote: > > > I don't think storage without conversion is an acceptable approach. We > > should provide options to users such as ignore tuple or NULL for > > column(s) with conversion p

Re: [HACKERS] Logical Replication and Character encoding

2017-02-02 Thread Craig Ringer
On 2 February 2017 at 11:45, Euler Taveira wrote: > I don't think storage without conversion is an acceptable approach. We > should provide options to users such as ignore tuple or NULL for > column(s) with conversion problem. I wouldn't consider storage data > without conversion because it silen

Re: [HACKERS] Logical Replication and Character encoding

2017-02-01 Thread Euler Taveira
On 01-02-2017 00:05, Kyotaro HORIGUCHI wrote: > - Subscriber connects with client_encoding specification and the > output plugin pgoutput decide whether it accepts the encoding > or not. If the subscriber doesn't, pgoutput send data without > conversion. > I don't think storage without conve

Re: [HACKERS] Logical Replication and Character encoding

2017-02-01 Thread Kyotaro HORIGUCHI
> From: Kyotaro HORIGUCHI [mailto:horiguchi.kyot...@lab.ntt.co.jp] > Sent: Wednesday, February 01, 2017 3:31 PM > To: Shinoda, Noriyoshi > Cc: pgsql-hackers@postgresql.org > Subject: Re: [HACKERS] Logical Replication and Character encoding > > At Wed, 01 Feb 2017 12:13:04 +0900 (Tokyo

Re: [HACKERS] Logical Replication and Character encoding

2017-02-01 Thread Shinoda, Noriyoshi
...@lab.ntt.co.jp] Sent: Wednesday, February 01, 2017 3:31 PM To: Shinoda, Noriyoshi Cc: pgsql-hackers@postgresql.org Subject: Re: [HACKERS] Logical Replication and Character encoding At Wed, 01 Feb 2017 12:13:04 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20170201.121304.267734

Re: [HACKERS] Logical Replication and Character encoding

2017-01-31 Thread Kyotaro HORIGUCHI
At Wed, 01 Feb 2017 12:13:04 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20170201.121304.267734380.horiguchi.kyot...@lab.ntt.co.jp> > > > I tried a committed Logical Replication environment. I found > > > that replication between databases of different encodings did > > > not con

Re: [HACKERS] Logical Replication and Character encoding

2017-01-31 Thread Kyotaro HORIGUCHI
At Wed, 01 Feb 2017 12:05:40 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20170201.120540.183393194.horiguchi.kyot...@lab.ntt.co.jp> > Hello, > > At Tue, 31 Jan 2017 12:46:18 +, "Shinoda, Noriyoshi" > wrote in > > > I tried a committed Logical Replication environment. I fou

Re: [HACKERS] Logical Replication and Character encoding

2017-01-31 Thread Kyotaro HORIGUCHI
Hello, At Tue, 31 Jan 2017 12:46:18 +, "Shinoda, Noriyoshi" wrote in > I tried a committed Logical Replication environment. I found > that replication between databases of different encodings did > not convert encodings in character type columns. Is this > behavior correct? The output

[HACKERS] Logical Replication and Character encoding

2017-01-31 Thread Shinoda, Noriyoshi
Hi hackers, I tried a committed Logical Replication environment. I found that replication between databases of different encodings did not convert encodings in character type columns. Is this behavior correct? I expected that the character 0xe6bca2 (UTF-8) would be converted to the same chara