I'm putting together a big data system centered around using Spark Streaming for data ingest and Spark SQL for querying the stored data. I've been investigating what options there are for implementing Spark applications using Clojure. It's been close to a decade since sparkling or flambo have received any updates and it doesn't look like either will accommodate recent distributions of Spark. I've found powderkeg an interesting option, and I like how it supports remote REPLs and the use of tranducers rather than wrapped Scala fns. However, it looks like it's also seen a few years without commits and I've heard loose talk that the developers have moved on to other pursuits.
Part of the problem seems to be Spark. The project seem unapologetic about breaking interfaces and seems willing to sacrifice third-party code that tries to track Spark's development. So my options seem to be the following: 1. Deploy an older version of Spark that's compatible with one of the above mentioned libraries. While we don't need to be bleeding edge, deploying a three year old version just to accommodate my preferred language is hard to justify. 2. Create a merge to update one of those libraries to more recent versions of Spark and be prepared to maintain it internally for the lifespan of this project. This may be vastly overestimating my personal heroics. 3. Code my own solution from scratch using Java/Scala interop, sketching out just enough of a Clojure wrapper to suit my ends. 4. Learn Scala. I realize that Spark isn't the only game in town (Onyx, for example). However, I'm working with a team of developers who are not familiar with Clojure (though I'm working to be an advocate). I choose Spark as an established solution that supports multiple languages and handles both streaming and batch processing. Any insights? Any solutions I'm overlooking? -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/clojure/259f5ff6-dd66-4688-aa80-439fed88ab39o%40googlegroups.com.
