Many thanks everyone for their valuable contribution. We all started with Spark a few years ago where Scala was the talk of the town. I agree with the note that as long as Spark stayed nish and elite, then someone with Scala knowledge was attracting premiums. In fairness in 2014-2015, there was not much talk of Data Science input (I may be wrong). But the world has moved on so to speak. Python itself has been around a long time (long being relative here). Most people either knew UNIX Shell, C, Python or Perl or a combination of all these. I recall we had a director a few years ago who asked our Hadoop admin for root password to log in to the edge node. Later he became head of machine learning somewhere else and he loved C and Python. So Python was a gift in disguise. I think Python appeals to those who are very familiar with CLI and shell programming (Not GUI fan). As some members alluded to there are more people around with Python knowledge. Most managers choose Python as the unifying development tool because they feel comfortable with it. Frankly I have not seen a manager who feels at home with Scala. So in summary it is a bit disappointing to abandon Scala and switch to Python just for the sake of it.
Disclaimer: These are opinions and not facts so to speak :) Cheers, Mich On Fri, 9 Oct 2020 at 21:56, Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > I have come across occasions when the teams use Python with Spark for ETL, > for example processing data from S3 buckets into Snowflake with Spark. > > The only reason I think they are choosing Python as opposed to Scala is > because they are more familiar with Python. Since Spark is written in > Scala, itself is an indication of why I think Scala has an edge. > > I have not done one to one comparison of Spark with Scala vs Spark with > Python. I understand for data science purposes most libraries like > TensorFlow etc. are written in Python but I am at loss to understand the > validity of using Python with Spark for ETL purposes. > > These are my understanding but they are not facts so I would like to get > some informed views on this if I can? > > Many thanks, > > Mich > > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > >
