Hi, I have a couple of use cases for Apache Spark applications/scripts, generally of the following form:
*General ETL use case* - more specifically a transformation of a Cassandra column family containing many events (think event sourcing) into various aggregated column families. *Streaming use case* - realtime analysis of the events as they arrive in the system. For *(1)*, I'll need to kick off the Spark application periodically. For *(2)*, just kick off the long running Spark Streaming process at boot time and let it go. /(Note - I'm using Spark Standalone as the cluster manager, so no yarn or mesos)/ I'm trying to figure out the most common / best practice deployment strategies for Spark applications. So far the options I can see are: *1) Deploying my program as a jar, and running the various tasks with spark-submit* - which seems to be the way recommended in the spark docs. Some thoughts about this strategy: * how do you start/stop tasks - just using simple bash scripts? * how is scheduling managed? - simply use cron? * any resilience? (e.g. Who schedules the jobs to run if the driver server dies?) *2) Creating a separate webapp as the driver program.* * creates a spark context programmatically to talk to the spark cluster * allowing users to kick off tasks through the http interface * using Quartz (for example) to manage scheduling * could use cluster with zookeeper election for resilience *3) Spark job server (https://github.com/ooyala/spark-jobserver)* * I don't think there's much benefit over *(2)* for me, as I don't (yet) have many teams and projects talking to Spark, and would still need some app to talk to job server anyway * no scheduling built in as far as I can see I'd like to understand the general consensus w.r.t a simple but robust deployment strategy - I haven't been able to determine one by trawling the web, as of yet. Thanks very much! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Apache-Spark-application-deployment-best-practices-tp23041.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
