If you look inside of the generation we generate java code and compile it with Janino. For interested folks the conversation moved over to the dev@ list
On Sat, Nov 9, 2019 at 10:37 AM Marcin Tustin <marcin.tus...@bluevoyant.com.invalid> wrote: > What do you mean by this? Spark is written in a combination of Scala and > Java, and then compiled to Java Byte Code, as is typical for both Scala and > Java. If there's additional byte code generation happening, it's java byte > code, because the platform runs on the JVM. > > On Sat, Nov 9, 2019 at 12:47 PM Bartosz Konieczny <bartkoniec...@gmail.com> > wrote: > >> *This Message originated outside your organization.* >> ------------------------------ >> Hi there, >> > >> Few days ago I got an intriguing but hard to answer question: >> "Why Spark generates Java code and not Scala code?" >> (https://github.com/bartosz25/spark-scala-playground/issues/18) >> >> Since I'm not sure about the exact answer, I'd like to ask you to confirm >> or not my thinking. I was looking for the reasons in the JIRA and the >> research paper "Spark SQL: Relational Data Processing in Spark" ( >> http://people.csail.mit.edu/matei/papers/2015/sigmod_spark_sql.pdf) but >> found nothing explaining why Java over Scala. The single task I found was >> about why Scala and not Java but concerning data types ( >> https://issues.apache.org/jira/browse/SPARK-5193) That's why I'm writing >> here. >> >> My guesses about choosing Java code are: >> - Java runtime compiler libs are more mature and prod-ready than the >> Scala's - or at least, they were at the implementation time >> - Scala compiler tends to be slower than the Java's >> https://stackoverflow.com/questions/3490383/java-compile-speed-vs-scala-compile-speed >> - Scala compiler seems to be more complex, so debugging & maintaining it >> would be harder >> - it was easier to represent a pure Java OO design than mixed FP/OO in >> Scala >> ? >> >> Thank you for your help. >> >> -- >> Bartosz Konieczny >> data engineer >> https://www.waitingforcode.com >> https://github.com/bartosz25/ >> https://twitter.com/waitingforcode >> >> -- Twitter: https://twitter.com/holdenkarau Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
