Re: Array Split vs Combine

2021-03-10 Thread Monte Goulding via use-livecode
Ideally we wouldn’t introduce an additional iteration of the array to pre-calculate but yes there’s probably a few ways to go. It could also be possible to flag a string with a buffer allocation strategy on creation so the cases where the engine creates a string then repeatedly appends to it can

Re: Array Split vs Combine

2021-03-10 Thread Sean Cole (Pi) via use-livecode
Monte, Would it be possible to precalculate (ish) how big a string buffer is required from the size (memalloc) of the array? Then, if later down the process, it works out it doesn't have enough, it can add a bunch more memory to the buffer, thus reducing the frequency of buffer resizing. Or maybe d

Re: Array Split vs Combine

2021-03-10 Thread Monte Goulding via use-livecode
It’s probably most likely to do with the string buffer needing to be constantly resized as the array is iterated for combine. Some googling suggests Windows may have issues with this. Our strategy for growing string buffers at the moment is to allocate just enough for the string. Mark would need

Re: Array Split vs Combine

2021-03-10 Thread Sean Cole (Pi) via use-livecode
The code for 'Split': void MCArraysExecSplit(MCExecContext& ctxt, MCStringRef p_string, MCStringRef p_element_delimiter, MCStringRef p_key_delimiter, MCArrayRef& r_array) { if (MCStringSplit(p_string, p_element_delimiter, p_key_delimiter, ctxt . GetStringComparisonType(), r_array)) return; ctxt .

Re: Array Split vs Combine

2021-03-10 Thread Tom Glod via use-livecode
following this . On Tue, Mar 9, 2021 at 4:06 PM Sean Cole (Pi) via use-livecode < use-livecode@lists.runrev.com> wrote: > It's looking to be a Windows only issue. I need to see how far this goes > back and then I'll post a bug report. It's making a process that should > only take 30s max on a

Re: Array Split vs Combine

2021-03-10 Thread Bob Sneidar via use-livecode
Now THAT is fascinating, considering the Windows performance issues with file access reported in the past. Could it be that combine is somehow caching data to virtual memory? Bob S On Mar 9, 2021, at 1:05 PM, Sean Cole (Pi) via use-livecode mailto:use-livecode@lists.runrev.com>> wrote: It's

Re: Array Split vs Combine

2021-03-09 Thread Sean Cole (Pi) via use-livecode
It's looking to be a Windows only issue. I need to see how far this goes back and then I'll post a bug report. It's making a process that should only take 30s max on a single thread 2GHz remote Win server take 16mins to process 2 of these files, so it will be good to find a solution for this. Than

RE: Array Split vs Combine

2021-03-09 Thread Ralph DiMola via use-livecode
LC 9.6.1 Win 10 VM on SSD. 524 ms,512 ms, 3.28 minutes Ralph DiMola IT Director Evergreen Information Services rdim...@evergreeninfo.net -Original Message- From: use-livecode [mailto:use-livecode-boun...@lists.runrev.com] On Behalf Of Sean Cole (Pi) via use-livecode Sent: Tuesday, March

Re: Array Split vs Combine

2021-03-09 Thread Tore Nilsen via use-livecode
I got 248 393 257 also on a Mac Tore Nilsen > 9. mar. 2021 kl. 18:40 skrev Sean Cole (Pi) via use-livecode > : > > Extraordinary. Thanks Alex. This is helpful. This is on windows and a > windows server. I will try on a Mac and Linux and see if it has the same > issue. > > Sean > > On Tu

Re: Array Split vs Combine

2021-03-09 Thread Sean Cole (Pi) via use-livecode
Extraordinary. Thanks Alex. This is helpful. This is on windows and a windows server. I will try on a Mac and Linux and see if it has the same issue. Sean On Tue, 9 Mar 2021 at 17:25, Alex Tweedly via use-livecode < use-livecode@lists.runrev.com> wrote: > > On 09/03/2021 16:56, Sean Cole (Pi) vi

Re: Array Split vs Combine

2021-03-09 Thread Bob Sneidar via use-livecode
I get 483, 4325, 4280. It actually takes less time for me. Bob S > On Mar 9, 2021, at 9:25 AM, Alex Tweedly via use-livecode > wrote: > > > On 09/03/2021 16:56, Sean Cole (Pi) via use-livecode wrote: >> Hi all, >> >> I have a tsv I split out as an array, process, then combine back to a tsv

Re: Array Split vs Combine

2021-03-09 Thread Alex Tweedly via use-livecode
On 09/03/2021 16:56, Sean Cole (Pi) via use-livecode wrote: Hi all, I have a tsv I split out as an array, process, then combine back to a tsv. The tsv is 53MB (55,635,198 bytes) and has 228320 lines with 21 columns (55406878 chars). To load the tsv file : 434ms To split the tsv to array : 506