You also need to ensure you're using checkpointing and support recreating the context on driver failure as described in the docs here: http://spark.apache.org/docs/latest/streaming-programming-guide.html#failure-of-the-driver-node
From: Matt Narrell <matt.narr...@gmail.com<mailto:matt.narr...@gmail.com>> Date: Thursday, August 14, 2014 at 10:34 AM To: Tobias Pfeiffer <t...@preferred.jp<mailto:t...@preferred.jp>> Cc: salemi <alireza.sal...@udo.edu<mailto:alireza.sal...@udo.edu>>, "user@spark.apache.org<mailto:user@spark.apache.org>" <user@spark.apache.org<mailto:user@spark.apache.org>> Subject: Re: spark streaming : what is the best way to make a driver highly available I'd suggest something like Apache YARN, or Apache Mesos with Marathon or something similar to allow for management, in particular restart on failure. mn On Aug 13, 2014, at 7:15 PM, Tobias Pfeiffer <t...@preferred.jp<mailto:t...@preferred.jp>> wrote: Hi, On Thu, Aug 14, 2014 at 5:49 AM, salemi <alireza.sal...@udo.edu<mailto:alireza.sal...@udo.edu>> wrote: what is the best way to make a spark streaming driver highly available. I would also be interested in that. In particular for Streaming applications where the Spark driver is running for a long time, this might be important, I think. Thanks Tobias
