SPIP: Adding work load identity to Spark on Kubernetes documents (supersedes Secret Management)

Hi, I would like to propose that the current Secret Management in Spark Kubernetes documentation to include the more secure credentials Workload identity) for Spark to access Kubernetes services. Both Google Clou

Re: Graceful shutdown SPARK Structured Streaming

In an event driven architecture you have to perform an action to stop the event in a controlled manner. That ( say) streamingContext.stop() or for that matter streamingContext.stop(stopGracedully=True) won't cause any stop by itself. *You need to build in a mechanism *to tell spark streaming to pr