Re: LC7 and 8 - Non responsive processing large text files

ddingham and the team. It will be fixed in LC 8 RC2 Kind regards Bernd -- View this message in context: http://runtime-revolution.278305.n4.nabble.com/LC7-and-8-Non-responsive-processing-large-text-files-tp4703419p4703566.html Sent from the Revolution - User mailing list archive at Nabble.com.

Re: LC7 and 8 - Non responsive processing large text files

Roland Huettmann wrote: > Also to be noted by Bernd, the speed also dramatically increases > using an SSD. I am still using old fashioned hard drive (. SSDs are indeed much faster. After I put in even an old one in my laptop, boot times are now under 9 seconds. But there's no shame in using H

Re: LC7 and 8 - Non responsive processing large text files

Absolutely Richard It changed the execution speed dramatically reducing the buffer size. I really still need the time to set up various tests and check it carefully trying to find the right size on my machine. Maybe 128k - we will see. >From Bernd - in an internal mail - he worked on my script an

Re: LC7 and 8 - Non responsive processing large text files

Roland Huettmann wrote: < Richard: "You may find the sample stack attached to this report useful: " Richard, thanks a lot for the sample stack. I downloaded and tested "LargeFileRead.livecode". Unfortunately it also creates the same symptoms on

Re: LC7 and 8 - Non responsive processing large text files

Mark Waddingham wrote: P.S. One could argue that 'multi-threading' might help here - but it is actually absolutely no different in effect from splitting into two processes. Indeed, for this kind of thing separate processes is a much much better idea as it completely insulates the master from the

Re: LC7 and 8 - Non responsive processing large text files

> One thing to remember is as much as we would like it to be the engine is not prescient. > It does not > know how much data it will need to read before it will find your sentinel string. > What this means in > practice is when reading until a string the engine needs to: > - constantly resize the

Re: LC7 and 8 - Non responsive processing large text files

> On 15 Apr 2016, at 4:59 PM, Roland Huettmann > wrote: > > Though it does NOT work for me as I am reading much larger chunks of data > using "read until..." > > (It is still running for 60 minutes now... I think I have to force-stop it. > No... now it finished with 3,060,000. -:) > > I think

Re: LC7 and 8 - Non responsive processing large text files

On 2016-04-15 08:59, Roland Huettmann wrote: I think it is important to know how to handle "big" data with files, in memory and placing in fields. And of course in a future nearby 8.1 version (or sooner?) those in need would love to see this taken care of in the engine. I don't think the prob

Re: LC7 and 8 - Non responsive processing large text files

< Richard: "You may find the sample stack attached to this report useful: " Richard, thanks a lot for the sample stack. I downloaded and tested "LargeFileRead.livecode". Unfortunately it also creates the same symptoms on Windows: Stack becomes un

Re: LC7 and 8 - Non responsive processing large text files

On 4/14/2016 12:27 PM, Roland Huettmann wrote: There is a work-around for me: Not using "read until " but reading "at" "for" As Mark suggested, that implies that the search string is uncommon in the text and large amounts of data need to be parsed before a match is found. By reading for a s

Re: LC7 and 8 - Non responsive processing large text files

Roland Huettmann wrote: > There is a work-around for me: > > Not using "read until " but reading "at" "for" > > I can read even 100 MB into memory (it does not create a big problem) > and then process using offset() and then reading the next pack of > data. You may find the sample stack attach

Re: LC7 and 8 - Non responsive processing large text files

Hi all, great response. Enjoyable to read, and motivating a lot ))). > Mark: ... "When you say 'non-responsiveness' I take it you mean that Windows thinks that the application has 'hung'? (i.e. the windows go slightly opaque)." That is correct, Mark. As I understand it now, it is Windows ma

Re: LC7 and 8 - Non responsive processing large text files

Roland Huettmann wrote: > And a more generalized question also discussed before: What exactly > happens when LC tries to read from a very very large file? Maybe it > is Gigabyte or even Terabyte file? It could just be too big to read > and it should then still not return empty or become unrespons

Re: LC7 and 8 - Non responsive processing large text files

Mark Talluto wrote: > Version 7/8 are slower than version 6 in almost every way. Here are a > couple of bug reports on the issue: > > > > You may want to add this enhancement request to that l

Re: LC7 and 8 - Non responsive processing large text files

Mark Waddingham wrote: > I wrote the above whilst waiting for my Windows VM to spin up... One more reason to consider VirtualBox. Being free is just icing: I switched because I found it much faster in restoring VM sessions than any other VM software I've used. -- Richard Gaskin Fourth Worl

Re: LC7 and 8 - Non responsive processing large text files

On 4/14/2016 5:43 AM, Mark Waddingham wrote: > When you say 'non-responsiveness' I take it you mean that Windows > thinks that the application has 'hung'? (i.e. the windows go slightly > opaque). This may or may not be related, but I have some code that does lots of data manipulation in a big loop

Re: LC7 and 8 - Non responsive processing large text files

Hi Roland, On 2016-04-13 12:03, Roland Huettmann wrote: NON-RESPONSIVENESS USING "UNTIL" READ I found opening a very large text file (my file is 26 GB), simply reading from it up to 90 MB of data in each iteration, using an offset () function processing the read junk of data, and iterating th

Re: LC7 and 8 - Non responsive processing large text files

Thanks a lot for replies Mark and Monte and for everyone ) I like to stress that this is planned to be a version for a real product to everyone out there... working with large files. The anomaly starts immediately after calling the handler whether or not an individual block of data is parsed. HE

Re: LC7 and 8 - Non responsive processing large text files

> On 14 Apr 2016, at 3:25 PM, Mark Talluto wrote: > >> But reading something into memory using "read from file UNTIL >> " and doing this many times over in such large text file creates >> non-responsiveness of LC (tested on 7 and 8). It would be interesting to know if the engine were doing lot

Re: LC7 and 8 - Non responsive processing large text files

> On Apr 13, 2016, at 3:03 AM, Roland Huettmann > wrote: …snip > But reading something into memory using "read from file UNTIL > " and doing this many times over in such large text file creates > non-responsiveness of LC (tested on 7 and 8). > > So, what do I do? > > One more note: Additiona

LC7 and 8 - Non responsive processing large text files

This issue was addressed before within another context, but I am running into the same problems with all versions of 7 and 8 including the latest rc1 (using Windows 8.1 to Windows 10). The question is important as it is related to a planned project for very serious development and a product to be