Re: [HACKERS] Making TEXT NUL-transparent

2011-12-23 Thread Florian Weimer
* Florian Pflug: > On Nov24, 2011, at 10:54 , Florian Weimer wrote: >>> Or is it not only about being able to *store* NULs in a text field? >> >> No, the entire core should be NUL-transparent. > > That's unlikely to happen. Yes, with the type input/output functions tied to NUL-terminated strings

Re: [HACKERS] Making TEXT NUL-transparent

2011-11-24 Thread Tom Lane
Andrew Dunstan writes: > On Thu, Nov 24, 2011 at 4:54 AM, Florian Weimer wrote: >> By the way, I refuse the notion that UTF-8 strings with embedded NULs >> are "broken". I can't recall any other system which enforces UTF-8 >> well-formedness, but does not permit embedded NULs. > Refuse away, bu

Re: [HACKERS] Making TEXT NUL-transparent

2011-11-24 Thread Andrew Dunstan
On Thu, Nov 24, 2011 at 4:54 AM, Florian Weimer wrote: > > By the way, I refuse the notion that UTF-8 strings with embedded NULs > are "broken". I can't recall any other system which enforces UTF-8 > well-formedness, but does not permit embedded NULs. > > Refuse away, but I don't think you know

Re: [HACKERS] Making TEXT NUL-transparent

2011-11-24 Thread Florian Pflug
On Nov24, 2011, at 10:54 , Florian Weimer wrote: >> Or is it not only about being able to *store* NULs in a text field? > > No, the entire core should be NUL-transparent. That's unlikely to happen. A more realistic approach would be to solve this only for UTF-8 encoded strings by encoding the NUL

Re: [HACKERS] Making TEXT NUL-transparent

2011-11-24 Thread Robert Haas
On Thu, Nov 24, 2011 at 4:54 AM, Florian Weimer wrote: > By the way, I refuse the notion that UTF-8 strings with embedded NULs > are "broken".  I can't recall any other system which enforces UTF-8 > well-formedness, but does not permit embedded NULs. This seems like a key point. If UTF-8 allows

Re: [HACKERS] Making TEXT NUL-transparent

2011-11-24 Thread Pavel Stehule
2011/11/24 Alexander Shulgin : > > Excerpts from Florian Weimer's message of Thu Nov 24 12:59:09 +0200 2011: >> >> > I have a different question. What is reason for embedded NULs inside >> > strings? >> >> The source system does not enforce that constraint, so from time to >> time, such data slips

Re: [HACKERS] Making TEXT NUL-transparent

2011-11-24 Thread Alexander Shulgin
Excerpts from Florian Weimer's message of Thu Nov 24 12:59:09 +0200 2011: > > > I have a different question. What is reason for embedded NULs inside > > strings? > > The source system does not enforce that constraint, so from time to > time, such data slips through. I don't know why it's there

Re: [HACKERS] Making TEXT NUL-transparent

2011-11-24 Thread Florian Weimer
* Pavel Stehule: >> By the way, I refuse the notion that UTF-8 strings with embedded NULs >> are "broken".  I can't recall any other system which enforces UTF-8 >> well-formedness, but does not permit embedded NULs. > I have a different question. What is reason for embedded NULs inside > strings?

Re: [HACKERS] Making TEXT NUL-transparent

2011-11-24 Thread Pavel Stehule
2011/11/24 Florian Weimer : > * Alexander Shulgin: > >>> It's actually UTF-8 text, and some PostgreSQL functions are only >>> available for TEXT, but not BYTEA, e.g.: >>> >>> bfk_int=> SELECT '\x006500'::bytea ~ 'A'; >>> ERROR:  operator does not exist: bytea ~ unknown >> >> And how will those TEXT

Re: [HACKERS] Making TEXT NUL-transparent

2011-11-24 Thread Florian Weimer
* Alexander Shulgin: >> It's actually UTF-8 text, and some PostgreSQL functions are only >> available for TEXT, but not BYTEA, e.g.: >> >> bfk_int=> SELECT '\x006500'::bytea ~ 'A'; >> ERROR: operator does not exist: bytea ~ unknown > > And how will those TEXT functions behave on a value with an

Re: [HACKERS] Making TEXT NUL-transparent

2011-11-24 Thread Pavel Stehule
2011/11/24 Florian Weimer : > * Pavel Stehule: > >> Hello >> >> 2011/11/24 Florian Weimer : >>> Occasionally, we get bitten by embedded NUL bytes in TEXT values.  We >>> take care of generating proper UTF-8, but this additional restriction >>> sometimes slips by.  It would be really helpful if Post

Re: [HACKERS] Making TEXT NUL-transparent

2011-11-24 Thread Alexander Shulgin
Excerpts from Florian Weimer's message of Thu Nov 24 11:27:51 +0200 2011: > > > and why you don't use bytea ? Text should be correct literal. > > It's actually UTF-8 text, and some PostgreSQL functions are only > available for TEXT, but not BYTEA, e.g.: > > bfk_int=> SELECT '\x006500'::bytea ~

Re: [HACKERS] Making TEXT NUL-transparent

2011-11-24 Thread Florian Weimer
* Pavel Stehule: > Hello > > 2011/11/24 Florian Weimer : >> Occasionally, we get bitten by embedded NUL bytes in TEXT values.  We >> take care of generating proper UTF-8, but this additional restriction >> sometimes slips by.  It would be really helpful if PostgreSQL could >> store such TEXT field

Re: [HACKERS] Making TEXT NUL-transparent

2011-11-24 Thread Pavel Stehule
Hello 2011/11/24 Florian Weimer : > Occasionally, we get bitten by embedded NUL bytes in TEXT values.  We > take care of generating proper UTF-8, but this additional restriction > sometimes slips by.  It would be really helpful if PostgreSQL could > store such TEXT fields as-is (at least if they a

[HACKERS] Making TEXT NUL-transparent

2011-11-24 Thread Florian Weimer
Occasionally, we get bitten by embedded NUL bytes in TEXT values. We take care of generating proper UTF-8, but this additional restriction sometimes slips by. It would be really helpful if PostgreSQL could store such TEXT fields as-is (at least if they are computed internally, or come from query