There is a large overhead to distributing this type of workload. I imagine
that for a small problem, the overhead dominates. You do not nearly need to
distribute a problem of this size, so more workers is probalby just worse.
On Sun, Apr 30, 2023 at 1:46 AM second_co...@yahoo.com <
second_co...@ya
I re-test with cifar10 example and below is the result . can advice why
lesser num_slot is faster compared with more slots?
num_slots=20 231 seconds
num_slots=5 52 seconds
num_slot=134 seconds
the code is at below
https://gist.github.com/cometta/240bbc549155e22f80f6ba670c9a2e32
Do you have
You don't want to use CPUs with Tensorflow.
If it's not scaling, you may have a problem that is far too small to
distribute.
On Sat, Apr 29, 2023 at 7:30 AM second_co...@yahoo.com.INVALID
wrote:
> Anyone successfully run native tensorflow on Spark ? i tested example at
> https://github.com/tenso
Dear Sean,
I do agree with you to a certain extent, makes sense. Perhaps I am wrong in
asking for native integrations and not depending on over engineered
external solutions which have their own performance issues, and bottlenecks
in live production environment. But asking and stating ones opinion
ibraries. I faced
>>>> the issue of version dependencies on those and had a hard time fixing the
>>>> library compatibilities. Hence a couple of below doubts:-
>>>>
>>>>
>>>>
>>>>- Does *Horovod* have any de
On the contrary, distributed deep learning is not data parallel. It's
dominated by the need to share parameters across workers.
Gourav, I don't understand what you're looking for. Have you looked at
Petastorm and Horovod? they _use Spark_, not another platform like Ray. Why
recreate this which has
t;>> - Does *Horovod* have any dependencies?
>>>- Any other library which is suitable for my use case.?
>>>- Any example code would really be of great help to understand.
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Vijayant
&
e any dependencies?
>>- Any other library which is suitable for my use case.?
>>- Any example code would really be of great help to understand.
>>
>>
>>
>> Thanks,
>>
>> Vijayant
>>
>>
>>
>> *From:* Sean Owen
>&
.?
>> Any example code would really be of great help to understand.
>>
>>
>> Thanks,
>>
>> Vijayant
>>
>>
>>
>> From: Sean Owen
>> Sent: Wednesday, February 23, 2022 8:40 AM
>> To: Vijayant Kumar
>> Cc: user
es data
>>>>> where I am using LSTM and transformers to forecast.
>>>>>
>>>>>
>>>>>
>>>>> I evaluated *SparkFlow* and *spark_tensorflow_distributor *libraries, and
>>>>> there has been no major development recent
t; I evaluated *SparkFlow* and *spark_tensorflow_distributor *libraries, and
>>>> there has been no major development recently on those libraries. I faced
>>>> the issue of version dependencies on those and had a hard time fixing the
>>>> library comp
gt;>
>>>- Does *Horovod* have any dependencies?
>>> - Any other library which is suitable for my use case.?
>>>- Any example code would really be of great help to understand.
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Vijaya
Horovod* have any dependencies?
>>- Any other library which is suitable for my use case.?
>>- Any example code would really be of great help to understand.
>>
>>
>>
>> Thanks,
>>
>> Vijayant
>>
>>
>>
>> *From:
is suitable for my use case.?
>- Any example code would really be of great help to understand.
>
>
>
> Thanks,
>
> Vijayant
>
>
>
> *From:* Sean Owen
> *Sent:* Wednesday, February 23, 2022 8:40 AM
> *To:* Vijayant Kumar
> *Cc:* user @spark
> *S
ve any dependencies?
>- Any other library which is suitable for my use case.?
>- Any example code would really be of great help to understand.
>
>
>
> Thanks,
>
> Vijayant
>
>
>
> *From:* Sean Owen
> *Sent:* Wednesday, February 23, 2022 8:40 AM
> *T
is suitable for my use case.?
* Any example code would really be of great help to understand.
Thanks,
Vijayant
From: Sean Owen
Sent: Wednesday, February 23, 2022 8:40 AM
To: Vijayant Kumar
Cc: user @spark
Subject: [E] COMMERCIAL BULK: Re: TensorFlow on Spark
Email is from a Free Mail
Sure, Horovod is commonly used on Spark for this:
https://horovod.readthedocs.io/en/stable/spark_include.html
On Tue, Feb 22, 2022 at 8:51 PM Vijayant Kumar
wrote:
> Hi All,
>
>
>
> Anyone using Apache spark with TensorFlow for building models. My
> requirement is to use TensorFlow distributed m
17 matches
Mail list logo
