Super thorough work there, Neville. Thanks.
Could I trouble you to post code listings for the various algos?
I'd like to try them on my MBOX archives, and they may also be useful
for others looking for parsing routines in the archives.
--
Richard Gaskin
Fourth World Systems
Neville wrote:> Just for interest, and to see just how slow lineOffset
is, I added a couple of more tests to the search for occurrences of
“Valjean” in the Gutenberg English translation of Les Miserables. I also
wanted find how filter performs.
The searches were first applied to the raw binary text as read from the utf-8
encoded file, without using textDecode; then on the text utf-8 decoded
Parse 0 : using itemdelimiter ‘Valjean’ (case insensitive)
Parse 1: using offset with skips
Parse 2: using offset, truncating the text and 0 skip
Parse 3: use lineOffset with skips
Parse 4: use lineOffset, truncating the text and 0 skip
filter: use filter to find lines containing '*Valjean*'
Parse 1 and 2 produced 1120 hits. Parse 0 gave 1121 hits, the extra one being a false positive at the end of the file, which needs to be accounted for in an implementation. I was slightly surprised that the character offsets produced were the same for raw and for utf-8 text, I guess I was expecting the latter to give the unicode character offset. Parse 3 and 4 and filter all output 1099 lines.
Results:
searches on raw text:
parse0 11 ms
parse1 9 ms
parse2 751 ms
parse3 2551 ms
parse4 753 ms
filter 16 ms
searches on utf-8 text:
parse0 4386 ms
parse1 224367 ms
parse2 3461 ms
parse3 636554 ms —— !!!!
parse4 7242 ms
filter 2258 ms
So for long texts it is best to use raw binary text and search with character
offset(pSt,pSrc,skip) [Parse 1]. If you have to search on utf-8 encode text
then use Parse 2, deleting initial sections of the text as you go. Never use
lineOffset (except for small text) even if that means extra code to find line
endings on either side of the character offset when you really want the found
line. If you don’t actually need the offset of the hits in the original file -
for example for editing the original - then filter is the fastest on long text
and just as fast on short text, but depending on your needs you probably have
to do another search on the filtered text; but this would be a viable approach
if the number of lines produced is itself small.
Neville
_______________________________________________
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
