On 2018-11-13 12:43, Ben Rubinstein via use-livecode wrote:
I'm grateful for all the information, but _outraged_ that the thread
that I carefully created separate from the offset thread was so
quickly hijacked for the continuing (useful!) detailed discussion on
that topic.
The phrase 'attempting to herd cats' springs to mind ;)
From recent contributions on both threads I'm getting some more
insights, but I'd really like to understand clearly what's going on. I
do think that I should have asked this question more broadly: how does
the engine represent values internally?
The engine uses a number of distinct types 'behind the scenes'. The ones
pertinent to LCS (there are many many more which LCS never sees) are:
- nothing: a type with a single value nothing/null)
- boolean: a type with two values true/false
- number: a type which can either store a 32-bit integer *or* a double
- string: a type which can either store a sequence of native (single byte)
codes, or a sequence of unicode (two byte - UTF-16) codes
- name: a type which stores a string, but uniques the string so that
caseless and exact equality checking is constant time
- data: a type which stores a sequence of bytes
- array: a type which stores (using a hashtable) a mapping from 'names' to
any other storage value type
The LCS part of the engine then sits on top of these core types, providing
various conversions depending on context.
All LCS syntax is actually typed - meaning that when you pass a value to any
piece of LCS syntax, each argument is converted to the type required.
e.g. charToNativeNum() has signature 'integer charToNativeNum(string)' meaning
that it
expects a string as input and will return a number as output.
Some syntax is overloaded - meaning that it can act in slightly different (but
always consistent) ways depending on the type of the arguments.
e.g. & has signatures 'string &(string, string)' and 'data &(data, data)'.
In simple cases where there is no overload, type conversion occurs exactly as
required:
e.g. In the case of charToNativeNum() - it has no overload, so always expects
a string
which means that the input argument will always undergo a 'convert to string'
operation.
The convert to string operation operates as follows:
- nothing -> ""
- boolean -> "true" or "false"
- number -> decimal representation of the number, using numberFormat
- string -> stays the same
- name -> uses the string the name contains
- data -> converts to a string using the native encoding
- array -> converts to empty (a very old semantic which probably does more
harm than good!)
In cases where syntax is overloaded, type conversion generally happens in
syntax-specific sequence in order to preserve consistency:
e.g. In the case of &, it can either take two data arguments, or two string
arguments. In this case,
if both arguments are data, then the result will be data. Otherwise both
arguments will be converted
to strings, and a string returned.
From Monte I get that the internal encoding for 'string' may be
MacRoman, ISO 8859 (I thought it would be CP1252), or UTF16 -
presumably with some attribute to tell the engine which one in each
case.
Monte wasn't quite correct - on Mac it is MacRoman or UTF-16, on Windows it
is CP1252 or UTF-16, on Linux it is IOS8859-1 or UTF-16. There is an internal
flag in a string value which says whether its character sequence is
single-byte (native)
or double-byte (UTF_16).
So then my question is whether a 'binary string' is a pure blob, with
no clues as to interpretation; or whether in fact it does have some
attributes to suggest that it might be interpreted as UTF8, UTF132
etc?
Data (binary string) values are pure blobs - they are sequences of bytes - it
has
no knowledge of where it came from. Indeed, that would generally be a bad idea
as you
wouldn't get repeatable semantics (i.e. a value from one codepath which is
data, might
have a different effect in context from one which is fetched from somewhere
else).
That being said, the engine does store some flags on values - but purely for
optimization.
i.e. To save later work. For example, a string value can store its (double)
numeric value in
it - which saves multiple 'convert to number' operations performed on the same
(pointer wise) string (due to the copy-on-write nature of values, and the fact
that all literals are unique names, pointer-wise equality of values occurs a
great deal).
If there are no such attributes, how does codepointOffset operate when
passed a binary string?
CodepointOffset is has signature 'integer codepointOffset(string)', so when you
pass a binary string (data) value to it, the data value gets converted to a
string
by interpreting it as a sequence of bytes in the native encoding.
If there are such attributes, how do they get set? Evidently if
textEncode is used, the engine knows that the resulting value is the
requested encoding. But what happens if the program reads a file as
'binary' - presumable the result is a binary string, how does the
engine treat it?
There are no attributes of that ilk. When you read a file as binary you get
data (binary
string) values - which means when you pass them to string taking
functions/commands that
data gets interpreted as a sequence of bytes in the native encoding. This is
why you must
always explicitly textEncode/textDecode data values when you know they are not
representing
native encoded text.
Is there any way at LiveCode script level to detect what a value is,
in the above terms?
Yes - the 'is strictly' operators:
is strictly nothing
is strictly a boolean
is strictly an integer - a number which has internal rep 32-bit int
is strictly a real - a number which has internal rep double
is strictly a string
is strictly a binary string
is strictly an array
It should be noted that 'is strictly' reports only how that value is stored
and not anything based on the value itself. This only really applies to 'an
integer' and 'a real' - you can store an integer in a double and all LCS
arithmetic operators act on doubles.
e.g. (1+2) is strictly an integer -> false
(1+2) is strictly a real -> true
In contrast, though, *some* syntax will return numbers which are stored
internally as integers:
e.g. nativeCharToNum("a") is strictly an integer -> true
I should point out that what 'is strictly' operators return for any given
context is not stable in the sense that future engine versions might return
different things. e.g. We might optimize arithmetic in the future (if we can
figure out a way to do it without performance penalty!) so that things which
are definitely integers, are stored as integers (e.g. 1 + 2 in the above).
And one more question: if a string, or binary string, is saved in a
'binary' file, are the bytes stored on disk a faithful rendition of
the bytes that composed the value in memory, or an interpretation of
some kind?
What happens when you read or write data or string values to a file depends on
how you opened the file.
If you opened the file for binary (whether reading or writing), when you read
you will get data, when you write string values will be converted to data via
the native encoding (default rule).
If you opened the file for text, then the engine will try and determine (using
a BOM) the existing text encoding of the file. If it can't determine it (if
for example, you are opening a file for write which doesn't exist), it will
assume it is encoded as native.
Otherwise the file will have an explicit encoding associated with it specified
by you - reading from it will interpret the bytes in that explicit encoding;
while writing to it will expect string values which will be encoded
appropriately. In the latter case if you write data values, they will first be
converted to a string (assuming native encoding) and then written as strings
in the file's encoding (i.e. default type conversion applies).
Essentially you can view file's a typed-stream - if you opened for binary
read/write give/take data; if you opened for text then read/write give/take
strings and default type conversion rules apply.
Warmest Regards,
Mark.
