Another option is Apache Beam. We use it quite extensively. There are a few options for Clojure wrappers (we use datasplash), and beam has libraries for a number of popular languages.
Kind Regards, Dom Parry On 10 Jul 2020, 08:22 +0200, Alex Ott <alex...@gmail.com>, wrote: > From Spark perspective, I would really advise to use Dataframe API as much as > possible, including the Spark Structured Streaming instead of Spark Streaming > - the main reason is more optimized execution of the code because of all > optimizations that Catalyst is able to make. But I really don't see libraries > that wrap dataframe API > > > On Thu, Jul 9, 2020 at 11:36 PM Tim Clemons <tclem...@gmail.com> wrote: > > > I'm putting together a big data system centered around using Spark > > > Streaming for data ingest and Spark SQL for querying the stored data. > > > I've been investigating what options there are for implementing Spark > > > applications using Clojure. It's been close to a decade since sparkling > > > or flambo have received any updates and it doesn't look like either will > > > accommodate recent distributions of Spark. I've found powderkeg an > > > interesting option, and I like how it supports remote REPLs and the use > > > of tranducers rather than wrapped Scala fns. However, it looks like it's > > > also seen a few years without commits and I've heard loose talk that the > > > developers have moved on to other pursuits. > > > > > > Part of the problem seems to be Spark. The project seem unapologetic > > > about breaking interfaces and seems willing to sacrifice third-party code > > > that tries to track Spark's development. > > > > > > So my options seem to be the following: > > > > > > 1. Deploy an older version of Spark that's compatible with one of the > > > above mentioned libraries. While we don't need to be bleeding edge, > > > deploying a three year old version just to accommodate my preferred > > > language is hard to justify. > > > > > > 2. Create a merge to update one of those libraries to more recent > > > versions of Spark and be prepared to maintain it internally for the > > > lifespan of this project. This may be vastly overestimating my personal > > > heroics. > > > > > > 3. Code my own solution from scratch using Java/Scala interop, sketching > > > out just enough of a Clojure wrapper to suit my ends. > > > > > > 4. Learn Scala. > > > > > > I realize that Spark isn't the only game in town (Onyx, for example). > > > However, I'm working with a team of developers who are not familiar with > > > Clojure (though I'm working to be an advocate). I choose Spark as an > > > established solution that supports multiple languages and handles both > > > streaming and batch processing. > > > > > > Any insights? Any solutions I'm overlooking? > > > > > > > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "Clojure" group. > > > To post to this group, send email to clojure@googlegroups.com > > > Note that posts from new members are moderated - please be patient with > > > your first post. > > > To unsubscribe from this group, send email to > > > clojure+unsubscr...@googlegroups.com > > > For more options, visit this group at > > > http://groups.google.com/group/clojure?hl=en > > > --- > > > You received this message because you are subscribed to the Google Groups > > > "Clojure" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to clojure+unsubscr...@googlegroups.com. > > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/clojure/259f5ff6-dd66-4688-aa80-439fed88ab39o%40googlegroups.com. > > > -- > With best wishes, Alex Ott > http://alexott.net/ > Twitter: alexott_en (English), alexott (Russian) > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with your > first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/clojure/CALV1_%3DJtBC02CchwoCT3%3DgHbdMBfaACRA_T6yRnZo0KCr9tACg%40mail.gmail.com. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/clojure/637c6835-db4e-4100-9c48-bd2f167b44a2%40Spark.
