There is a spark listener interface <https://spark.apache.org/docs/1.0.0/api/scala/index.html#org.apache.spark.scheduler.SparkListener> which can be used to trigger events like jobStarted, TaskGotResults etc but i don't think you can set execution time anywhere. If a task is hung, its mostly because of the GC pause (depends on your job), if you can paste the code, then probably we can tell you where the bottleneck is.
Thanks Best Regards On Tue, Dec 16, 2014 at 9:59 AM, Mohamed Lrhazi < mohamed.lrh...@georgetown.edu> wrote: > > Is that possible, if not, how would one do it from PySpark ? > > This probably does not make sense in most cases, but am writing a script > where my job involves downloading and pushing data into cassandra.. > sometimes a task hangs forever, and I dont really mind killing it.. The job > is not actually computing some result that requires all tasks to succeed. > > Thanks, > Mohamed. >
