Phil wrote:
> ... man's ability to bring the machine into his own service, instead of the
> other way around.
Ah, the reason I continue to code.
It will not beat me!
Never surrender!
It will not beat me!
;)
JIm Lambert
___
> Indeed. One of my co-workers was munging/aggregating using
> SQL queries -- not a good idea, and was still not done after
> an hour. Out of curiosity I got him to send me his data
> before lunch, and my first pass yesterday worked by
> processing all the (test) data into an array, and finish
On Thu, Sep 12, 2013 at 6:46 AM, Geoff Canyon wrote:
> Indeed. One of my co-workers was munging/aggregating using SQL queries --
> not a good idea, and was still not done after an hour.
>
Hi Geoff,
Was the time eaten up by inserting the data into the SQL database or
retrieving it?
Pete
lcSQL So
Geoff Canyon wrote:
> I opened my mouth at work when I shouldn't, and now I'm writing a
> function to process server log files: multi-gigabytes of data, and
> tens of million rows of data. Speed optimization will be key...
What sort of processing are you doing on those logs? What are you
lookin
Big data = big fun! :-)
I enjoy those challenges because it (when successful) showcases man's
ability to bring the machine into his own service, instead of the other
way around. At least that's what I tell myself.
Phil Davis
On 9/12/13 6:46 AM, Geoff Canyon wrote:
On Thu, Sep 12, 2013 at
I, for one, welcome our new machine overlords... ;-)
On Sep 12, 2013, at 12:34 PM, Jim Lambert wrote:
>
> Phil wrote:
>
>
>> ... man's ability to bring the machine into his own service, instead of the
>> other way around.
>
> Ah, the reason I continue to code.
> It will not beat me!
On Thu, Sep 12, 2013 at 2:35 AM, FlexibleLearning.com <
ad...@flexiblelearning.com> wrote:
> With multi-gigs of data, you will certainly have to chunk the data. And
> access your progress bar judiciously!
>
Indeed. One of my co-workers was munging/aggregating using SQL queries --
not a good idea
Hi Geoff,
Me too :( and yes, 'split/combine' with 'for each' are useful!
Ony processing 10-20Mb here, but parsing into arrays I run out of memory. I
am maxed out for XP (4Gb). Will have to upgrade to Win7 to access more, I
fear, and/or re-write to chunk the data. I didn't realise they were going
Geoff-
Wednesday, September 11, 2013, 3:23:05 PM, you wrote:
> I opened my mouth at work when I shouldn't, and now I'm writing a function
> to process server log files: multi-gigabytes of data, and tens of million
> rows of data. Speed optimization will be key...
...some people have all the fun.
I opened my mouth at work when I shouldn't, and now I'm writing a function
to process server log files: multi-gigabytes of data, and tens of million
rows of data. Speed optimization will be key...
___
use-livecode mailing list
use-livecode@lists.runrev.co
10 matches
Mail list logo
