Re: Adding pause() method to pyspark.sql.streaming.StreamingQuery

2023 16:39 > *To:* Martin Andersson > *Cc:* Spark dev list > *Subject:* Re: Adding pause() method to > pyspark.sql.streaming.StreamingQuery > > > EXTERNAL SENDER. Do not click links or open attachments unless you > recognize the sender and know the content is safe. DO NOT prov

Re: Adding pause() method to pyspark.sql.streaming.StreamingQuery

Hi Mich. So it sounds like what you're really after is a way to apply new stream options in runtime without downtime? BR, Martin From: Mich Talebzadeh Sent: Tuesday, March 14, 2023 16:39 To: Martin Andersson Cc: Spark dev list Subject: Re: Adding

Re: Adding pause() method to pyspark.sql.streaming.StreamingQuery

Hi Martin, I see the major benefit of the spark stop() method in giving the ability to shut down the main topic gracefully. I have explained this in this SPIP SPIP: Shutting down spark structured streaming when the streaming process completed current process

Re: Adding pause() method to pyspark.sql.streaming.StreamingQuery

Hi Mich. I'm trying to understand, can you please provide some use-cases where it would be beneficial with a pause and how a pause would differ functionally from a stop? Best regards, Martin From: Mich Talebzadeh Sent: Thursday, March 9, 2023 17:12 To: Spark de