Oh... That was embarrassingly easy!
Thank you that was exactly the understanding of partitions that I needed.
P
On Thu, Jul 30, 2015 at 6:35 AM, Simon Elliston Ball <
si...@simonellistonball.com> wrote:
> You might also want to consider broadcasting the models to ensure you get
> one instance
You might also want to consider broadcasting the models to ensure you get one
instance shared across cores in each machine, otherwise the model will be
serialised to each task and you'll get a copy per executor (roughly core in
this instance)
Simon
Sent from my iPhone
> On 30 Jul 2015, at 10
Like this?
val data = sc.textFile("/sigmoid/audio/data/", 24).foreachPartition(urls =>
speachRecognizer(urls))
Let 24 be the total number of cores that you have on all the workers.
Thanks
Best Regards
On Wed, Jul 29, 2015 at 6:50 AM, Peter Wolf wrote:
> Hello, I am writing a Spark application
