On 4/17/15 5:20 PM, Gary Gregory wrote:
> I thought I'd share this read with you guys:
> http://coopsoft.com/ar/CalamityArticle.html
>
> I'm not sure how closely these problems relate with what [math] is trying
> to do, but it's a interesting read.
Thanks! Kind of supports the idea that somehow allowing the
execution framework to be pluggable would be good.
Phil
>
> Gary
>
> On Fri, Apr 17, 2015 at 9:01 AM, Gilles <gil...@harfang.homelinux.org>
> wrote:
>
>> On Fri, 17 Apr 2015 08:35:42 -0700, Phil Steitz wrote:
>>
>>> On 4/17/15 3:14 AM, Gilles wrote:
>>>
>>>> Hello.
>>>>
>>>> On Thu, 16 Apr 2015 17:06:21 -0500, James Carman wrote:
>>>>
>>>>> Consider me poked!
>>>>>
>>>>> So, the Java answer to "how do I run things in multiple threads"
>>>>> is to
>>>>> use an Executor (java.util). This doesn't necessarily mean that you
>>>>> *have* to use a separate thread (the implementation could execute
>>>>> inline). However, in order to accommodate the separate thread case,
>>>>> you would need to code to a Future-like API. Now, I'm not saying to
>>>>> use Executors directly, but I'd provide some abstraction layer above
>>>>> them or in lieu of them, something like:
>>>>>
>>>>> public interface ExecutorThingy {
>>>>> Future<T> execute(Function<T> fn);
>>>>> }
>>>>>
>>>>> One could imagine implementing different ExecutorThingy
>>>>> implementations which allow you to parallelize things in different
>>>>> ways (simple threads, JMS, Akka, etc, etc.)
>>>>>
>>>> I did not understand what is being suggested: parallelization of a
>>>> single algorithm or concurrent calls to multiple instances of an
>>>> algorithm?
>>>>
>>> Really both. It's probably best to look at some concrete examples.
>>>
>> Certainly...
>>
>> The two I mentioned in my apachecon talk are:
>>> 1. Threads managed by some external process / application gathering
>>> statistics to be aggregated.
>>>
>>> 2. Allowing multiple threads to concurrently execute GA
>>> transformations within the GeneticAlgorithm "evolve" method.
>>>
>> I could not view the presentation from the link previously mentioned
>> (it did not work with my browser...).
>> Can I download the PDF file from somewhere?
>>
>> It would be instructive to think about how to handle both of these
>>> use cases using something like what James is suggesting. What is
>>> nice about his idea is that it could give us a way to let users /
>>> systems decide whether they want to have [math] algorithms spawn
>>> threads to execute concurrently or to allow an external execution
>>> framework to handle task distribution across threads.
>>>
>> Some (all?) cases of "external" parallelism are trivial for the CM
>> developers: the user must chop his data, pass the chunks as arguments
>> to the CM methods, then collect and reassemble the results, all by
>> himself.
>> IIUC the scenario, this cannot be deemed a "feature".
>>
>> Since 2. above is a good example of "internal" parallelism and it
>>> also has data sharing / transfer challenges, maybe its best to start
>>> with that one.
>>>
>> That's the scenario where usage is simple and performance can match
>> the user's machine capability when running CM algorithms that are
>> inherently parallel.
>>
>> There is an example in CM: see
>> testTravellerSalesmanSquareTourParallelSolver()
>> in
>> org.apache.commons.math4.ml.neuralnet.sofm.KohonenTrainingTaskTest
>>
>> I have just started thinking about this and would
>>> love to get better ideas than my own hacking about how to do it
>>>
>>> a) Using Spark with RDD's to maintain population state data
>>> b) Hadoop with HDFS (or something else?)
>>>
>> I have zero experience with this but I'm interested to know more. :-)
>>
>> Regards,
>> Gilles
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
>> For additional commands, e-mail: dev-h...@commons.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org
