> One thing to remember is as much as we would like it to be the engine is not prescient. > It does not > know how much data it will need to read before it will find your sentinel string. > What this means in > practice is when reading until a string the engine needs to: > - constantly resize the memory buffer when it nears overflow for data > that is read in because it > doesn’t know how big it needs to be > read one byte at a time then check if the last N bytes match your sentinel string
Monte and Mark. Exactly this is the point! I completely agree. What I tried to say - maybe I am not expressing it so well - is that by experience so far it is WORKING WELL READING VERY LARGE FILES just using "READ ... FOR" in a large file and then processing later in a separate step. The problem I had was when testing the "READ ... UNTIL" - and you commented this in depth. I already have been successful before building an index and then reading just pieces of data using the index. No odd behaviour. No "non responsiveness". YES Mark, I already was splitting the process! And even before writing to this list, I had done that intuitively in a similar way. It was working and it is working. And I can only suggest to anybody interested to follow your advise and study the scripts you and other provided. From all you said I get a picture which I did not have before in such detail. And I hope others get it too. It makes us understand more and more. So, all my thanks to your kind contributions. I will follow up here... Greetings, Roland On 15 April 2016 at 10:05, Monte Goulding <mo...@appisle.net> wrote: > > > On 15 Apr 2016, at 4:59 PM, Roland Huettmann <roland.huettm...@gmail.com> > wrote: > > > > Though it does NOT work for me as I am reading much larger chunks of data > > using "read until..." > > > > (It is still running for 60 minutes now... I think I have to force-stop > it. > > No... now it finished with 3,060,000. -:) > > > > I think it is important to know how to handle "big" data with files, in > > memory and placing in fields. And of course in a future nearby 8.1 > version > > (or sooner?) those in need would love to see this taken care of in the > > engine. > > Roland are you ensuring to encode your sentinel string as the same > encoding as the file? I’m interested to know how much difference that makes > as I think this could be the main source of any difference you are seeing > between LC 6 and 7/8 on this. > > One thing to remember is as much as we would like it to be the engine is > not prescient. It does not know how much data it will need to read before > it will find your sentinel string. What this means in practice is when > reading until a string the engine needs to: > - constantly resize the memory buffer when it nears overflow for data > that is read in because it doesn’t know how big it needs to be > - read one byte at a time then check if the last N bytes match your > sentinel string > > Obviously this is significantly more expensive than read for because the > buffer is allocated once and the read can read the number of bytes you want > and give it to you straight away because it doesn’t need to check it > anything matches. > > I second Mark’s recommendation to move the file parsing to a helper > process if possible because no matter what this is going to be a bottleneck > in your app. > > Cheers > > Monte > _______________________________________________ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your > subscription preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode > _______________________________________________ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
