viirya commented on issue #6295: URL: https://github.com/apache/datafusion-comet/issues/6295#issuecomment-6510298337
#7181 adds a kill path for native Arrow UDFs, and part of it could be reused for plan cancellation. `CometExecIterator` binds a token to the plan with `Native.setPythonUdfTask` before the plan first executes. `PythonUdfMonitor` polls `TaskContext.isInterrupted` with one thread per task and passes that token to `NativeBase.killPythonUdfs`. Two details there apply to cancelling a plan too: - The token is generated by the JVM because a task attempt id is unique only within one SparkContext. - The watch has to keep running after the task completes, until the task's plans are released. Spark marks the task completed before its completion listeners release the plan, and releasing a plan waits for a batch that is still being computed. The rest is specific to Python UDFs and would not carry over: the `hasPythonUdf` gate, the `spark.python.task.killTimeout` grace period, and `spark.comet.exec.nativeArrowPythonUDF.interruptOnKill.enabled`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
