On Tue, 03 Apr 2018, Michael Lange wrote: > I believe (please anyone correct me if I am wrong) that "text" files > won't contain any null byte; many text editors even refuse to open such a
Depends on the encoding. For ASCII, ISO-8859-* and UTF-8 (and any other modern encoding AFAIK, other than modified UTF-8), any zero bytes map one-to-one to the NUL character/code point. I don't recall how it is on other common encodings of the 80's and 90's, though. Some even-more-modern encodings (modified UTF-8 :p) simply do NOT use bytes with the value of zero when encoding characters, so NUL is encoded by a different sequence, and you can safely use a byte with the value of zero for some out-of-band control (like zero-terminated strings that can contain NULs, etc) -- note that NUL is a character, and it might be represented by a sequence of bytes that has nothing to do with zeroes on a particular encoding... (in fact, C strings are *zero-terminated*, not NUL-terminated, but most of the time this is irrelevant :p). Also, a text file MAY contain NULs (the character), it is just considered bad practice (nowadays?). Don't assume you won't see any. For example, received e-mail is *more* likely to have NULs in it than normal text due to the quality of some mail agents out there. I recall postfix would reject a *lot* of crap when we configured it to refuse to accept NULs outside of 8-bit bodies, because Cyrus-IMAPd *refuses* any such crap, and we wanted it bounced as early as possible. (note that NULs are forbidden in MIME-compliant email text and ESMTP, unless encoded or guarded by a 8-bit transfer area of known size, so there you have it: NULs in one text format that actually forbids them!). > Probably it is the same with some other control characters like 04 (End > of Transmission). When I look at https://en.wikipedia.org/wiki/ASCII > it seems like 1C (File Separator) or 1E (Record Separator) might be > appropriate choices for you. I'm no expert on this, though. Well, ASCII control characters were inherited by ISO-8859-* and Unicode, so yes, you can use them. But so could the data file. It would be perfectly ok for a text data file to use the record separator control characters to delimit records in a table, for example... Here's a good definition of them (follow the hyperlinks for the definition of each control character): https://en.wikipedia.org/wiki/Basic_Latin_(Unicode_block) Here is also a proper solution: use modified UTF-8 (which encodes NUL so that zero bytes are *never* present in the stream): encode every input format to modified UTF-8, then add the zero-byte separators you want. You'll have to normalize the input data set into known charset/encodings and then recode them to modified UTF-8, of course. You can't blindly call any random data "UTF-8" (let alone modified UTF-8) and expect things not to break horribly. -- Henrique Holschuh
