On 2016-04-15 08:59, Roland Huettmann wrote:
I think it is important to know how to handle "big" data with files, in
memory and placing in fields. And of course in a future nearby 8.1
version
(or sooner?) those in need would love to see this taken care of in the
engine.
I don't think the problem here is whether the engine has the facilities
to process large files - it clearly does given the number of examples of
how to do it that have been provided ;)
The issue here is how to keep a UI responsive on the Desktop whilst
running a long processing operation - this isn't a problem unique to
LiveCode it is one which pertinent to all languages on all OSes.
From what I can understand of your use-case you are wanting to index an
*exceptionally large* file so that you can look up things from it at a
later date. Indeed, it seems to me that the problem you are trying to
solve falls into two pieces:
1) Process the very large file and produce an index of 'points of
interest' in the file.
2) Subsequently use the index to perform the main function of your
application.
With this in mind, I'd suggest considering separating these two parts
and run them in separate processes.
The foreground (UI) application can launch the second process to
generate the index and provide a simple progress update periodically
without having any impact on the performance of its job, and then when
that operation is completed the foreground application can load the
index to use.
The LiveCode Installer (albeit no longer used on Mac) works like this.
It is a single application which can run in two modes. In the foreground
mode you see the UI which you interact with, but when you finally get to
the 'install' button, when you click that it launches a background
process which runs a installation script - it feeds back progress
information back to the foreground process at suitable intervals so that
the UI can update a progress bar. (Admittedly the installer *has* to
work this way as the background 'slave' process has to run with
administrator rights, however it also happily solves the UI progress bar
update / cancellation problem though too).
This is basically what you could call a 'master/slave' pattern. You have
a master process which provides the UI and the application's main
function, and a slave process which is run and controlled by the master
during indexing operations. This model has all kinds of advantages - for
example, you can rerun the slave 'indexing' process whilst the user is
still using an existing index; or indeed launch several slave processes
to index multiple files at once.
In LiveCode such a pattern can be implemented in the master using an
'open process read/write process close process' loop; whilst all the
slave has to do is read from stdin and write to stdout.
The above might seem a little opaque - however, it might help to have a
look at:
https://github.com/livecode/livecode/blob/develop/builder/installer_utilities.livecodescript
Which is the main installer implementation. When run as slave, the only
handler which runs is 'runInstallerActions'; and when run as master
there is a UI which runs a 'send in time' loop 'installerMonitor' to
talk to the slave. The main entry point to the installer application is
a startup handler which decides (by looking at the command line
arguments) what it should do:
on startup
local tAction
if $1 is "install" then
put "install" into tAction
else if $1 is "doinstall" then
put "doinstall" into tAction
else if $1 is "uninstall" then
put "uninstall" into tAction
else if $1 is "douninstall" then
put "douninstall" into tAction
else
set the itemDelimiter to slash
if the last item of $0 contains "setup" then
put "uninstall" into tAction
else
put "install" into tAction
end if
end if
switch tAction
case "install"
if $2 is "noui" then
runFacelessInstall
else
hide me
send "runInstallerUI" to me in 0 millisecs
end if
break
case "uninstall"
if $2 is "noui" then
runFacelessUninstall
else
hide me
send "runUninstallerUI" to me in 0 millisecs
end if
break
case "doinstall"
runInstallerActions
break
case "douninstall"
runUninstallerActions
break
default
quit 1
end switch
end startup
Here 'install' and 'uninstall' run the master UI (the default is
'install' if no arguments are specified); whilst doinstall/douninstall
run the slave.
Anyway, I'll try to find some time to distill out a skeleton application
which does this - it seems like it might be useful thing as an example
:)
Warmest Regards,
Mark.
P.S. One could argue that 'multi-threading' might help here - but it is
actually absolutely no different in effect from splitting into two
processes. Indeed, for this kind of thing separate processes is a much
much better idea as it completely insulates the master from the slave
and is much more failsafe as the two processes run in their own memory
spaces with their own resources and only communicate over a very thin
connection. i.e. Problems in one cannot propagate to the other. It also
provides more options in terms of running the indexing operation - it
doesn't necessarily need to be done in response to the master
application, indexing becomes an operation in its own right which can be
managed in any way which might be required.
--
Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/
LiveCode: Everyone can create apps
_______________________________________________
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
