Just for interest, and to see just how slow lineOffset is, I added a couple of more tests to the search for occurrences of “Valjean” in the Gutenberg English translation of Les Miserables. I also wanted find how filter performs.
The searches were first applied to the raw binary text as read from the utf-8 encoded file, without using textDecode; then on the text utf-8 decoded Parse 0 : using itemdelimiter ‘Valjean’ (case insensitive) Parse 1: using offset with skips Parse 2: using offset, truncating the text and 0 skip Parse 3: use lineOffset with skips Parse 4: use lineOffset, truncating the text and 0 skip filter: use filter to find lines containing '*Valjean*' Parse 1 and 2 produced 1120 hits. Parse 0 gave 1121 hits, the extra one being a false positive at the end of the file, which needs to be accounted for in an implementation. I was slightly surprised that the character offsets produced were the same for raw and for utf-8 text, I guess I was expecting the latter to give the unicode character offset. Parse 3 and 4 and filter all output 1099 lines. Results: searches on raw text: parse0 11 ms parse1 9 ms parse2 751 ms parse3 2551 ms parse4 753 ms filter 16 ms searches on utf-8 text: parse0 4386 ms parse1 224367 ms parse2 3461 ms parse3 636554 ms —— !!!! parse4 7242 ms filter 2258 ms So for long texts it is best to use raw binary text and search with character offset(pSt,pSrc,skip) [Parse 1]. If you have to search on utf-8 encode text then use Parse 2, deleting initial sections of the text as you go. Never use lineOffset (except for small text) even if that means extra code to find line endings on either side of the character offset when you really want the found line. If you don’t actually need the offset of the hits in the original file - for example for editing the original - then filter is the fastest on long text and just as fast on short text, but depending on your needs you probably have to do another search on the filtered text; but this would be a viable approach if the number of lines produced is itself small. Neville _______________________________________________ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
