Hi, Just noticed that Spark SQL uses spark.sql.execution.id local property (via SQLExecution.withNewExecutionId [1]) to group Spark jobs logically together while Structured Streaming uses SparkContext.setJobGroup [2] to do the same.
I think Structured Streaming is more correct as it uses what Spark Core introduced and uses in web UI (without introducing a custom solution). Why does Spark SQL introduce a custom solution based on spark.sql.execution.id local property? What's wrong with SparkContext.setJobGroup? [1] https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala#L63 [2] https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala#L265 Pozdrawiam, Jacek Laskowski ---- https://medium.com/@jaceklaskowski/ Mastering Apache Spark 2 https://bit.ly/mastering-apache-spark Follow me at https://twitter.com/jaceklaskowski --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
