On 2020-03-21 14:09, Paul Dupuis via use-livecode wrote:
So far the only person who has read my post and replied with what I
was looking for was Peter - and although the routine was written in
Rebol rather than LiveCode, he kindly provided a link to information
about it.
It might have got lost in amongst other replies but I did suggest:
<https://pypi.org/project/chardet/>
It even comes with a command-line script (chardetect) which would allow
to compare your detector with that one.
However, on further digging it appears that this does not (as it stands)
detect MacRoman which is obviously a key requirement here.
There is a stale PR for that though
<https://github.com/chardet/chardet/pull/5> so the method used here is
obviously possible to extend to that.
From what I have read the Python one is a python reimplementation of
Mozilla's 'Universal Charset Detector' which, from what I have read,
is/was pretty much state of the art - reading through the chardet docs
(https://chardet.readthedocs.io/en/latest/how-it-works.html#single-byte-encodings
is perhaps the most pertinent) it sounds like its single-byte detectors
use 2-byte sequences to try and distinguish.
There is a special case for Latin-1 (1252) which is needed because
English text looks the same in a large number of encodings - this works
by looking for curly quotes and other special symbols by the look of it.
(The MacRoman addition in the stale PR above, is also a Latin-1 like
special-case - which makes sense as Latin-1 and MacRoman are almost just
permutations of each other).
My general feeling is that if you already have a process which works to
detect the differences between MacRoman and Latin-1, then it is likely
largely equivalent to any other means which exists (the accepted answer
here
<https://stackoverflow.com/questions/4198804/how-to-reliably-guess-the-encoding-between-macroman-cp1252-latin1-utf-8-and>
sounds like it pretty much sums up the situation!) so beyond fixing the
bug(s) you found recently, you might find that there is nothing more you
can do.
Warmest Regards,
Mark.
--
Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/
LiveCode: Everyone can create apps
_______________________________________________
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
