Spark does do distributed ML, but not Tensorflow. Barrier execution mode is an element that things like Horovod uses. Not sure what you are getting at? Ray is not Spark. As I say -- Horovod does this already. The upside over TF distributed is that Spark sets up and manages the daemon processes rather than doing it by hand.
On Wed, Feb 23, 2022 at 2:43 AM Gourav Sengupta <gourav.sengu...@gmail.com> wrote: > Hi, > > the SPARK community should have been able to build distributed ML > capabilities, and as far as I remember that was the idea initially behind > SPARK 3.x roadmap (barrier execution mode, > https://issues.apache.org/jira/browse/SPARK-24579). > > Ray, another Berkeley Labs output like SPARK, is trying to capture that > market space. > > I am not sure whether there is any steer by the SPARK community leaders to > seriously prioritise building those capabilities at all. But I am sure if > the brilliant and fantastic minds behind SPARK did actually want to allow > building those capabilities, they can easily do so, and achieve that :) > > I would sincerely request the open source SPARK community to prioritise > building the SPARK capabilities to scale ML applications. > > > > Thanks and Regards, > Gourav Sengupta > > On Wed, Feb 23, 2022 at 3:53 AM Bitfox <bit...@bitfox.top> wrote: > >> tensorflow itself can implement the distributed computing via a >> parameter server. Why did you want spark here? >> >> regards. >> >> On Wed, Feb 23, 2022 at 11:27 AM Vijayant Kumar >> <vijayant.ku...@mavenir.com.invalid> wrote: >> >>> Thanks Sean for your response. !! >>> >>> >>> >>> Want to add some more background here. >>> >>> >>> >>> I am using Spark3.0+ version with Tensorflow 2.0+. >>> >>> My use case is not for the image data but for the Time-series data where >>> I am using LSTM and transformers to forecast. >>> >>> >>> >>> I evaluated *SparkFlow* and *spark_tensorflow_distributor *libraries, and >>> there has been no major development recently on those libraries. I faced >>> the issue of version dependencies on those and had a hard time fixing the >>> library compatibilities. Hence a couple of below doubts:- >>> >>> >>> >>> - Does *Horovod* have any dependencies? >>> - Any other library which is suitable for my use case.? >>> - Any example code would really be of great help to understand. >>> >>> >>> >>> Thanks, >>> >>> Vijayant >>> >>> >>> >>> *From:* Sean Owen <sro...@gmail.com> >>> *Sent:* Wednesday, February 23, 2022 8:40 AM >>> *To:* Vijayant Kumar <vijayant.ku...@mavenir.com.invalid> >>> *Cc:* user @spark <user@spark.apache.org> >>> *Subject:* [E] COMMERCIAL BULK: Re: TensorFlow on Spark >>> >>> >>> >>> *Email is from a Free Mail Service (Gmail/Yahoo/Hotmail….) *: Beware of >>> Phishing Scams, Report questionable emails to s...@mavenir.com >>> >>> Sure, Horovod is commonly used on Spark for this: >>> >>> https://horovod.readthedocs.io/en/stable/spark_include.html >>> >>> >>> >>> On Tue, Feb 22, 2022 at 8:51 PM Vijayant Kumar < >>> vijayant.ku...@mavenir.com.invalid> wrote: >>> >>> Hi All, >>> >>> >>> >>> Anyone using Apache spark with TensorFlow for building models. My >>> requirement is to use TensorFlow distributed model training across the >>> Spark executors. >>> >>> Please help me with some resources or some sample code. >>> >>> >>> >>> Thanks, >>> >>> Vijayant >>> ------------------------------ >>> >>> This e-mail message may contain confidential or proprietary information >>> of Mavenir Systems, Inc. or its affiliates and is intended solely for the >>> use of the intended recipient(s). If you are not the intended recipient of >>> this message, you are hereby notified that any review, use or distribution >>> of this information is absolutely prohibited and we request that you delete >>> all copies in your control and contact us by e-mailing to >>> secur...@mavenir.com. This message contains the views of its author and >>> may not necessarily reflect the views of Mavenir Systems, Inc. or its >>> affiliates, who employ systems to monitor email messages, but make no >>> representation that such messages are authorized, secure, uncompromised, or >>> free from computer viruses, malware, or other defects. Thank You >>> >>> ------------------------------ >>> >>> This e-mail message may contain confidential or proprietary information >>> of Mavenir Systems, Inc. or its affiliates and is intended solely for the >>> use of the intended recipient(s). If you are not the intended recipient of >>> this message, you are hereby notified that any review, use or distribution >>> of this information is absolutely prohibited and we request that you delete >>> all copies in your control and contact us by e-mailing to >>> secur...@mavenir.com. This message contains the views of its author and >>> may not necessarily reflect the views of Mavenir Systems, Inc. or its >>> affiliates, who employ systems to monitor email messages, but make no >>> representation that such messages are authorized, secure, uncompromised, or >>> free from computer viruses, malware, or other defects. Thank You >>> >>
