Hi Wim,
I think we are splitting the atom here but my inference to functionality was based on: 1. Spark is written in Scala, so knowing Scala programming language helps coders navigate into the source code, if something does not function as expected. 2. Given the framework using Python increases the probability for more issues and bugs because translation between these two different languages is difficult. 3. Using Scala for Spark provides access to the latest features of the Spark framework as they are first available in Scala and then ported to Python. 4. Some functionalities are not available in Python. I have seen this few times in Spark doc. There is an interesting write-up on this, although it does on touch on CI/CD aspects. Developing Apache Spark Applications: Scala vs. Python <https://www.pluralsight.com/blog/software-development/scala-vs-python> Regards, Mich On Fri, 23 Oct 2020 at 10:23, Wim Van Leuven <wim.vanleu...@highestpoint.biz> wrote: > I think Sean is right, but in your argumentation you mention that > 'functionality > is sacrificed in favour of the availability of resources'. That's where I > disagree with you but agree with Sean. That is mostly not true. > > In your previous posts you also mentioned this . The only reason we > sometimes have to bail out to Scala is for performance with certain udfs > > On Thu, 22 Oct 2020 at 23:11, Mich Talebzadeh <mich.talebza...@gmail.com> > wrote: > >> Thanks for the feedback Sean. >> >> Kind regards, >> >> Mich >> >> >> >> LinkedIn * >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >> >> >> >> >> >> *Disclaimer:* Use it at your own risk. Any and all responsibility for >> any loss, damage or destruction of data or any other property which may >> arise from relying on this email's technical content is explicitly >> disclaimed. The author will in no case be liable for any monetary damages >> arising from such loss, damage or destruction. >> >> >> >> >> On Thu, 22 Oct 2020 at 20:34, Sean Owen <sro...@gmail.com> wrote: >> >>> I don't find this trolling; I agree with the observation that 'the >>> skills you have' are a valid and important determiner of what tools you >>> pick. >>> I disagree that you just have to pick the optimal tool for everything. >>> Sounds good until that comes in contact with the real world. >>> For Spark, Python vs Scala just doesn't matter a lot, especially if >>> you're doing DataFrame operations. By design. So I can't see there being >>> one answer to this. >>> >>> On Thu, Oct 22, 2020 at 2:23 PM Gourav Sengupta < >>> gourav.sengu...@gmail.com> wrote: >>> >>>> Hi Mich, >>>> >>>> this is turning into a troll now, can you please stop this? >>>> >>>> No one uses Scala where Python should be used, and no one uses Python >>>> where Scala should be used - it all depends on requirements. Everyone >>>> understands polyglot programming and how to use relevant technologies best >>>> to their advantage. >>>> >>>> >>>> Regards, >>>> Gourav Sengupta >>>> >>>> >>>>>>
