So Python is used in many of the Spark Ecosystem products, but not Streaming at this point. Is there a roadmap to include Python APIs in Spark Streaming? Anytime frame on this?
Thanks! John On Thu, May 29, 2014 at 4:19 PM, Matei Zaharia <matei.zaha...@gmail.com> wrote: > Quite a few people ask this question and the answer is pretty simple. When > we started Spark, we had two goals — we wanted to work with the Hadoop > ecosystem, which is JVM-based, and we wanted a concise programming > interface similar to Microsoft’s DryadLINQ (the first language-integrated > big data framework I know of, that begat things like FlumeJava and Crunch). > On the JVM, the only language that would offer that kind of API was Scala, > due to its ability to capture functions and ship them across the network. > Scala’s static typing also made it much easier to control performance > compared to, say, Jython or Groovy. > > In terms of usage, however, we see substantial usage of our other > languages (Java and Python), and we’re continuing to invest in both. In a > user survey we did last fall, about 25% of users used Java and 30% used > Python, and I imagine these numbers are growing. With lambda expressions > now added to Java 8 ( > http://databricks.com/blog/2014/04/14/Spark-with-Java-8.html), I think > we’ll see a lot more Java. And at Databricks I’ve seen a lot of interest in > Python, which is very exciting to us in terms of ease of use. > > Matei > > On May 29, 2014, at 1:57 PM, Benjamin Black <b...@b3k.us> wrote: > > HN is a cesspool safely ignored. > > > On Thu, May 29, 2014 at 1:55 PM, Nick Chammas <nicholas.cham...@gmail.com> > wrote: > >> I recently discovered Hacker News and started reading through older >> posts about Scala >> <https://hn.algolia.com/?q=scala#!/story/forever/0/scala>. It looks like >> the language is fairly controversial on there, and it got me thinking. >> >> Scala appears to be the preferred language to work with in Spark, and >> Spark itself is written in Scala, right? >> >> I know that often times a successful project evolves gradually out of >> something small, and that the choice of programming language may not always >> have been made consciously at the outset. >> >> But pretending that it was, why is Scala the preferred language of Spark? >> >> Nick >> >> >> ------------------------------ >> View this message in context: Why Scala? >> <http://apache-spark-user-list.1001560.n3.nabble.com/Why-Scala-tp6536.html> >> Sent from the Apache Spark User List mailing list archive >> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com. >> > > >
