On 2019-09-06 15:22, Tomas Hajny wrote:
On 2019-09-06 07:24, LacaK wrote:
Hi *,
As promised, I discussed the idea of adding support for UTF-16 encoded
text files (and preferably UTF-32 as well while at it) to the RTL with
other core team members. Overall, I didn't come across anybody oposing
this idea, the only (logical) requirement is taking care of the
performance implications of this change, i.e. avoiding considerable
performance decrease in processing of 8-bit encoded files (actually,
this is one of reasons of my suggestion to add codepoint size
information to the text file record and use that instead of checking
individual values of the codepage variable to find out the codepoint
size implications every time working with the file - see below).
Yes, I believe that extending SetTextCodePage with supporting UTF-16
makes sense (with certain caveats like that calling it should be
performed before Rewrite in case of new files creation, or otherwise
the BOM mark will not be added to the beginning of the file). The
other question is what needs to happen within the text file record -
as mentioned in my other post, I'd prefer adding a new field
specifying the codepoint size rather than having to check for specific
codepage values in all code branches which would need to be created
for handling the difference.
Moreover, the case of opening a file is somewhat trickier, because the
file may have the encoding specified within the file itself. Would we
add code for reading the first bytes every time Reset is called for a
text file not associated with another device (console) and set the
fields in the text file record (possibly overriding an explicit
setting from SetTextCodePage)? Personally, I'd do so, but others may
have a different opinion.
.
.
After the discussion with some people from the core team, I suggest the
following:
1) New attribute for the codepoint size will be added to the text file
record and all the text file I/O needs to be checked and possibly
extended to with using this attribute instead of current implicit
expectation that the codepoint size is always 1 byte.
2) Support for UTF-16BE/LE and UTF-32BE/LE will be added to
SetTextCodePage, the new codepoint size attribute will be updated as
appropriate.
3) New function 'DetectUtfBom (var T: text): boolean' will be added.
This function may be called after the call to 'Reset (T: text)' to check
for existence of BOM at the beginning of the text file. If it is found
(Result=true), SetTextCodePage is invoked automatically from
DetectUtfBom with the codepage value corresponding to the found BOM and
encoding variant. If BOM is not found (Result=false), nothing changes.
4) A new procedure 'SetUtfBom (var T: text; CodePage: word; BOM:
boolean)' will be added. This procedure may be called after the call to
Rewrite and allows writing BOM to the respective text file.
SetTextCodePage with the respective value will be called from SetUtfBom.
Comments, anybody?
Tomas
_______________________________________________
fpc-pascal maillist - fpc-pascal@lists.freepascal.org
https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
