Re: Structured Streaming and Spark Connect

2024-09-23 Thread 刘唯
Hi Anastasiia, Thanks for the email. I think you can tweak this spark config *spark.connect.session.manager.defaultSessionTimeout, *this is defined here*: * https://github.com/apache/spark/blob/343471dac4b96b43a09763d759b6c30760fb626e/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/

Re: Structured Streaming and Spark Connect

2024-09-23 Thread Mich Talebzadeh
Hi Anastasia, My take is that in its current form, Spark Connect is not suitable for running long-lived Structured Streaming queries in Standalone mode, especially with long trigger intervals. The lack of support for detached streaming queries makes it problematic for this particular use case. To