Ideally we wouldn’t introduce an additional iteration of the array to
pre-calculate but yes there’s probably a few ways to go. It could also be
possible to flag a string with a buffer allocation strategy on creation so the
cases where the engine creates a string then repeatedly appends to it can use a
different strategy. Anyway, I’m actually only guessing at the problem so best
to create an issue from which we can create a benchmark test.
Cheers
Monte
> On 11 Mar 2021, at 1:28 pm, Sean Cole (Pi) via use-livecode
> <use-livecode@lists.runrev.com> wrote:
>
> Monte,
> Would it be possible to precalculate (ish) how big a string buffer is
> required from the size (memalloc) of the array? Then, if later down the
> process, it works out it doesn't have enough, it can add a bunch more
> memory to the buffer, thus reducing the frequency of buffer resizing. Or
> maybe do it in kind-of blocks of n bytes. I'm just spit-balling but I guess
> you get my thinking.
>
> Sean
>
> On Thu, 11 Mar 2021 at 00:52, Monte Goulding via use-livecode <
> use-livecode@lists.runrev.com> wrote:
>
>> It’s probably most likely to do with the string buffer needing to be
>> constantly resized as the array is iterated for combine. Some googling
>> suggests Windows may have issues with this. Our strategy for growing string
>> buffers at the moment is to allocate just enough for the string. Mark would
>> need to chime in on whether growing the buffer exponentially would be
>> suitable. It would result in more memory being allocated than necessary but
>> much less frequent allocations so it depends on what’s most costly as
>> memory gets cheaper.
>>
>> Cheers
>>
>> Monte
>>
>>> On 11 Mar 2021, at 11:34 am, Sean Cole (Pi) via use-livecode <
>> use-livecode@lists.runrev.com> wrote:
>>>
>>> The code for 'Split':
>>>
>>> void MCArraysExecSplit(MCExecContext& ctxt, MCStringRef p_string,
>>> MCStringRef p_element_delimiter, MCStringRef p_key_delimiter, MCArrayRef&
>>> r_array)
>>> {
>>> if (MCStringSplit(p_string, p_element_delimiter, p_key_delimiter, ctxt .
>>> GetStringComparisonType(), r_array))
>>> return;
>>>
>>> ctxt . Throw();
>>> }
>>>
>>>
>>> vs
>>>
>>> The code for 'Combine' (// comments added by me):
>>>
>>> void MCArraysExecCombine(MCExecContext& ctxt, MCArrayRef p_array,
>>> MCStringRef p_element_delimiter, MCStringRef p_key_delimiter,
>> MCStringRef&
>>> r_string)
>>> {
>>> bool t_success;
>>> t_success = true; // Create a register to check progress success
>>>
>>> uindex_t t_count; // Create a new (t)emp counter for indices
>>> t_count = MCArrayGetCount(p_array); // Find out how many arrays there
>> are
>>>
>>> MCAutoStringRef t_string; // Create (t)emp string to store the result
>>> if (t_success)
>>>
>>> t_success = MCStringCreateMutable(0, &t_string); // t_success is always
>>> true here initially and is reset as true if t_string is now mutable,
>> false
>>> if not
>>>
>>>
>>> combine_array_t t_lisctxt; // create a new array object
>>> t_lisctxt . elements = nil; // initialise t_lisctxt array
>>> if (t_success)
>>>
>>> t_success = MCMemoryNewArray(t_count, t_lisctxt . elements); // make sure
>>> the array was created
>>>
>>>
>>> if (t_success)
>>> {
>>>
>>> t_lisctxt . index = 0;
>>>
>>> MCArrayApply(p_array, list_array_elements, &t_lisctxt);
>>>
>>> qsort(t_lisctxt . elements, t_count, sizeof(array_element_t),
>>> compare_array_element); // sort the elements
>>>
>>> for(uindex_t i = 0; i < t_count; i++)
>>>
>>> { // Loop through all indices
>>>
>>> MCAutoStringRef t_value_as_string; // create a (t)emp string for element
>>> value
>>>
>>>
>>>
>>> t_success = ctxt . ConvertToString(t_lisctxt . elements[i] . value,
>>> &t_value_as_string); // convert array value to string
>>>
>>> if (!t_success)
>>>
>>> break; skip if unable to convert to string
>>>
>>>
>>> t_success =
>>>
>>> (p_key_delimiter == nil ||
>>>
>>> (MCStringAppend(*t_string, MCNameGetString(t_lisctxt . elements[i] .
>> key))
>>> &&
>>>
>>> MCStringAppend(*t_string, p_key_delimiter)))&&
>>>
>>> MCStringAppend(*t_string, *t_value_as_string) &&
>>>
>>> (i == t_count - 1 ||
>>>
>>> MCStringAppend(*t_string, p_element_delimiter)); // t_success is true if
>>> the array element and values are added correctly
>>>
>>>
>>> if (!t_success)
>>>
>>> break; // skip if unable to add value
>>>
>>> }
>>>
>>> }
>>>
>>> if (t_success)
>>>
>>> t_success = MCStringCopy(*t_string, r_string); // Copies the (t)emp
>> string
>>> into the (r)eturn string
>>>
>>>
>>> MCMemoryDeleteArray(t_lisctxt . elements);
>>>
>>> if (t_success)
>>>
>>> return;
>>>
>>>
>>> // Throw the current error code (since last library call returned false).
>>> ctxt . Throw();
>>> }
>>>
>>>
>>> Following on from Bob's VM comment, there is reference to
>>> 'MCMemoryNewArray(t_count,
>>> t_lisctxt . elements)' which does highlight that some memory management
>> for
>>> the arrays is necessary in the combine command. This only creates a
>>> temporary copy of the array for working through. How this plays out
>>> differently for Windows vs Mac/Linux and why this would be increasing the
>>> time necessary by a factor of about 4:1 I can't see.
>>>
>>> I've tested as far back as LC7
>>> (Times - Read into memory, Split to array, Combine from array)
>>> LC9.5.0 Win64 - 0.437s, 0.516s, 3m 1.378s
>>> LC9.0.5 Win32 - 0.446s, 0.547s, 3m 27.9s
>>> LC8.2.0 DP2 - 0.543s, 0.577s, 3m 30.208s
>>> LC8.0.0 - 0.542s, 0.545s, 3m 30.815s
>>> LC7.0.0 - 0.827s, 0.460s , 3m 37.896s
>>>
>>> On mac all times are less than 1sec, 3 sec total.
>>>
>>> Sean
>>>
>>>
>>> On Wed, 10 Mar 2021 at 17:08, Bob Sneidar via use-livecode <
>>> use-livecode@lists.runrev.com> wrote:
>>>
>>>> Now THAT is fascinating, considering the Windows performance issues with
>>>> file access reported in the past. Could it be that combine is somehow
>>>> caching data to virtual memory?
>>>>
>>>> Bob S
>>>>
>>>>
>>>> On Mar 9, 2021, at 1:05 PM, Sean Cole (Pi) via use-livecode <
>>>> use-livecode@lists.runrev.com<mailto:use-livecode@lists.runrev.com>>
>>>> wrote:
>>>>
>>>> It's looking to be a Windows only issue. I need to see how far this goes
>>>> back and then I'll post a bug report. It's making a process that should
>>>> only take 30s max on a single thread 2GHz remote Win server take 16mins
>> to
>>>> process 2 of these files, so it will be good to find a solution for
>> this.
>>>>
>>>> Thanks everyone for confirming and providing your input.
>>>>
>>>> Regards
>>>> Sean
>>>>
>>>> _______________________________________________
>>>> use-livecode mailing list
>>>> use-livecode@lists.runrev.com
>>>> Please visit this url to subscribe, unsubscribe and manage your
>>>> subscription preferences:
>>>> http://lists.runrev.com/mailman/listinfo/use-livecode
>>>>
>>> _______________________________________________
>>> use-livecode mailing list
>>> use-livecode@lists.runrev.com
>>> Please visit this url to subscribe, unsubscribe and manage your
>> subscription preferences:
>>> http://lists.runrev.com/mailman/listinfo/use-livecode
>>
>>
>> _______________________________________________
>> use-livecode mailing list
>> use-livecode@lists.runrev.com
>> Please visit this url to subscribe, unsubscribe and manage your
>> subscription preferences:
>> http://lists.runrev.com/mailman/listinfo/use-livecode
>>
> _______________________________________________
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
_______________________________________________
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
