Graham Samuel wrote:
> On Mon, 2 Apr 2012 Mark Schonewille wrote:
...
>> You can use only one index, i.e. the setProp/getProp handlers
>> are one-dimensional.
...
> Mark, thanks for that. There is absolutely nothing of this in the
> LC Dictionary AFAICS, although after a search I realised that
> there is a very hard to read (IMHO) description on section 7.10
> of the User Guide. I think at the very least a note should be
> added to the Dictionary entry, so I have submitted one, but I
> would be very happy if someone such as yourself corrects it.
Or you could do that yourself, with the Comments feature. Probably a
useful note to add until that Dict entry gets enhanced.
In fact, it would be great to submit an RQCC request for this Dict
issue, so it will be on RunRev's radar.
At the heart of this is a core difference between how arrays and custom
props are handled. They seem similar but have diverged in recent years:
In the olden days, both LiveCode's associative arrays and the custom
properties it allows in objects were parallel: they were one
dimensional, and you could move data between them easily:
put the customProperties of tObj into tMyArray
set the customProperties of tObj to tMyArray
When arrays became multi-dimensional, that made it difficult to use the
same syntax to accomplish similar goals, so they created a workaround to
support that which also gave rise to two new functions, arrayEncode and
arrayDecode.
ArrayEncode translates an array from the memory-specific hash table that
drives it into a flat form that can be saved to disk. ArrayDecode
translates that flattened form back into an array in memory.
But since memory-mapped hash tables can't be used from disk, the format
of data returned from arrayEncode is not easily traversed, requiring
considerable clock cycles to translate back into an array when run
though arrayDecode.
In essence, what those functions do is somewhat similar to what you'd
have to do if you wanted to flatten array data in a script, walking
through each element and writing it out into a linear format that
retains key-value association, nesting, etc.
When you save an array to a custom prop, from what I'm seeing in terms
of performance it seems that the array data is passed through something
very similar to the internal routine used for arrayEncode, allowing that
binary data to be storable, but unfortunately not in a form that's
easily traversed.
And since those can't be traversed using the same rapid methods that
memory-based arrays use, we have no ability to use multi-dimensional
array syntax to address them as properties like we do as arrays.
For the long term I've submitted a request to see if we can come up with
a b-tree storage method which might allow us to use the same syntax for
both arrays and nested props:
Make arrays and customProperties behave the same
<http://quality.runrev.com/show_bug.cgi?id=6912>
In the meantime, to access array data stored in a custom prop requires
two steps, first to extract the array and then to access its nested
elements:
put the MyPropArray of tObj into tSomeArray
put tSomeArray[1][2] into tSomeValue
Note that performance of nested arrays as props is roughly on par with
running data that had been saved with arrayEncode through the
arrayEncode function, not nearly as fast as using a true array in a
variable. So for performance-critical uses it may be helpful to extract
the array early on and use it from a global, storing it back only when
needed.
This is a good opportunity to share a bit of geeky fun news with you folks:
Kevin Miller and Ben Beaumont have generously asked Mark Waddingham to
take a moment away from his very busy schedule to provide a description
of the format of the data we get from arrayEncode.
I had requested this because I have a few cases where I need to be able
to extract a specific data element from an array stored on disk, without
the overhead of reading the entire file and running it through
arrayDecode. This is for a CGI where every millisecond counts, and
some of the arrays I'm working with have hundreds of thousands of nested
keys.
To justify the time required for Mark to do this, we made a deal: Mark
would provide a sparse description only sufficient to get me started so
he could get back to more critical tasks, provided I take that info and
flesh it out into an article for LiveCode Journal which explains the
format in detail and provides a working example of how to use it.
Mark came through on his end of the bargain wonderfully, providing a
sample function that worked on the first level of array keys, which was
enough for me to expand it to be able extract the data from any key
regardless of depth.
The article explaining it all is in progress now and close to done. I
expect to be able to post it to LiveCode Journal by Saturday.
Here's a tantalizing tidbit to whet your appetite:
The sample stack I wrote for this creates a three-dimensional array with
100 items in each dimension, resulting in an arrayEncoded file with 1
million elements that's about 20 MBs on disk.
The function I made from Mark's example code can extract the data from
any element in that file in about 3 to 5 milliseconds.
Hint: the function makes heavy use of the binaryDecode function and the
seek command, both of which are explained in depth in the article and
which you may find very valuable for a wide variety of uses. The seek
command is a god for rapidly traversing large files easily.
I look forward to sharing this with you ASAP -
--
Richard Gaskin
Fourth World
LiveCode training and consulting: http://www.fourthworld.com
Webzine for LiveCode developers: http://www.LiveCodeJournal.com
LiveCode Journal blog: http://LiveCodejournal.com/blog.irv
_______________________________________________
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
