I've solved this in the past by using a thread pool which runs clean up code on thread creation, to clear out stale values.
On Wednesday, September 28, 2016, Grant Digby <dig...@gmail.com> wrote: > Hi, > > We've received the following error a handful of times and once it's > occurred > all subsequent queries fail with the same exception until we bounce the > instance: > > IllegalArgumentException: spark.sql.execution.id is already set > at > org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId( > SQLExecution.scala:77) > at > org.apache.spark.sql.DataFrame.withNewExecutionId(DataFrame.scala:2086) > > ForkJoinWorkerThreads call into SQLExecution#withNewExecutionId, are > assigned an execution Id into their InheritableThreadLocal and this is > later > cleared in the finally block. > I've noted that these ForkJoinWorkerThreads can create additional > ForkJoinWorkerThreads and (as of SPARK-10563) the child threads receive a > copy of the parent's properties. > It seems that Prior to SPARK-10563, clearing the parent's executionId would > have cleared the child's, but now it's a copy of the properties the child's > executionId is never cleared leading to the above exception. > I'm yet to recreate the issue locally, whilst I've seen > ForkJoinWorkerThreads creating others and the properties being copied > across > I've not seen this from within the body of withNewExecutionId. > > Does this all sound reasonable? > Our plan for a short term work around is to allow the condition to arise > but > remove the execution.id from the thread local before throwing the > IllegalArgumentException so it succeeds on re-try. > > > > > -- > View this message in context: http://apache-spark- > developers-list.1001551.n3.nabble.com/IllegalArgumentException- > spark-sql-execution-id-is-already-set-tp19124.html > Sent from the Apache Spark Developers List mailing list archive at > Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org <javascript:;> > > -- Want to work at Handy? Check out our culture deck and open roles <http://www.handy.com/careers> Latest news <http://www.handy.com/press> at Handy Handy just raised $50m <http://venturebeat.com/2015/11/02/on-demand-home-service-handy-raises-50m-in-round-led-by-fidelity/> led by Fidelity
