Re: [Structured Streaming] Restarting streaming query on exception/termination

[Arun Mahadevan]

I guess you can wait for the termination, catch exception and then restart the 
query in a loop. Something like…

while (true) {
  try {
    val query = df.writeStream().
                            …
                           .start()
    query.awaitTermination()
  } catch {
    case e: StreamingQueryException => // log it
  }
}

Thanks,
Arun

From:  Priyank Shrivastava <priya...@gmail.com>
Date:  Monday, April 23, 2018 at 11:27 AM
To:  formice <51296...@qq.com>, "user@spark.apache.org" <user@spark.apache.org>
Subject:  Re: [Structured Streaming] Restarting streaming query on 
exception/termination

Thanks for the reply formice.  I think that --supervise param helps to restart 
the whole spark application - what I want to be able to do is to only restart 
the structured streaming query which terminated due to error. Also, I am 
running my app in client mode. 

Thanks,
Priyank

On Sun, Apr 22, 2018 at 8:52 PM, formice <51296...@qq.com> wrote:
standlone 
      add  config:(1)--deploy-mode cluster (2)--supervise
      example:  spark-submit  --master spark://master:7077 --deploy-mode 
cluster --supervise ......


------------------ 原始邮件 ------------------
发件人: "Priyank Shrivastava"<priya...@gmail.com>;
发送时间: 2018年4月21日(星期六) 凌晨5:45
收件人: "user"<user@spark.apache.org>;
主题: [Structured Streaming] Restarting streaming query on exception/termination

What's the right way of programmatically restarting a structured streaming 
query which has terminated due to an exception? Example code or reference would 
be appreciated.

Could it be done from within the onQueryTerminated() event handler of 
StreamingQueryListener class?

Priyank