Two weeks ago I have published a blogpost about our experiences running 24/7 Spark Streaming applications on YARN in production: https://www.inovex.de/blog/247-spark-streaming-on-yarn-in-production/ <https://www.inovex.de/blog/247-spark-streaming-on-yarn-in-production/> Amongst others it contains a reference spark-submit command covering subtleties such as YARN, backpressure and spark.locality.wait configuration. Maybe this helps some people not getting stuck on the same issues we faced. Although the post is not targeted at Structured Streaming, the majority of configurations still hold true. Feedback is appreciated.
-- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/24-7-Spark-Streaming-on-YARN-in-Production-tp28265.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
