Thanks for the question Alex, I’m wrestling with the same issues - but so far got no responses from encoding gurus here :)
This is my understanding: 1) Yes its recommended to textEncode text that comes from outside into Livecode’s internal native format (which is utf16). Livecode handles everything internally “transparently” from then on - which I guess means all usual language and control operations expect this utf16 internal format. My guess is this is why a few things have got slower as compared with early versions of Livecode. 2) Without doing textEncode the engine tries to guess the encoding (duck-typing?) and does this in a platform specific way? Again exactly what is going on there is a bit opaque to me, but the take-home message is that this is slower and less robust. So yes -losing nothing (assuming the original file is utf8, and yes its the best alternative. I thing the hard thing to find out is exactly what type of encoding some files are - would be great if there was a duck-typing service where we could paste text or upload files and it would say - hey this looks like utf8 - but that’s asking too much 📆 Schedule a call with me On 2 Sep 2021, 12:12 +0100, Alex Tweedly via use-livecode <use-livecode@lists.runrev.com>, wrote: > Sorry to drag us off the interesting topic of licensing :-) into some > Livecode question. > > I know little or nothing about Unicode, text encodings, etc. - so my > question is indeed naive. > > I have a text file (War & Peace from Project Gutenberg), about 3.4Mb. > The Mac describes it simply as "Plain text". > > When I read that into a variable, and then do > replace tChar by SPACE in tWholeText > it takes between 1000 and 4000 millisecs - versus the 8-10 msecs I had > expected from other samples. > > If I put in > put textEncode(tWHoleText, "UTF8") into tWholeText > before the replace then it does indeed tae 8-10 msecs. > > Q1. What (if anything) am I losing by doing that ? > > Q2. Is this the best alternative ? > > Additional info - I just discovered that according to 'more' command > line, the file start with : > > <U+FEFF>The Project .... > > if that is useful. > > Many thanks, > > Alex. > > > _______________________________________________ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your subscription > preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode _______________________________________________ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
