Short Answer: Yes Long answer: You need to understand your load characteristics to size your cluster. Most applications have 3 components to their load. A) a predictable amount of expected load. This usually changes based on time of day, and day of week The main thing is that it’s predictable. B) unpredictable spikes. The main thing about spikes that they are periods of high traffic but are short lived c) sustained high load: this is load that occurs because of some event (for example, retailers have higher load during Christmas) This might be semi-predictable: You know it’s coming, but you don’t know how big it will be. Also, unlike spikes, they will be sustained
What you want to do is size your cluster and tune your application so you can handle 120% of your predictable load without falling behind. If your load is unusually spiky, you might want to increase that to 150% or 200%. This means that when a spike hits, you have excess capacity to handle the load. Also, you want to make sure that if there is too much load, requests get queued up instead of crashing your streaming app. You don’t want an OOM because Spark tried to read 100 million records in one batch. There is a setting called maxRatePerPartition that you can set. This limits the number of records that will be read in one batch. You should set this to be to be 80% of the maximum batch size your cluster can handle. If you do this, and if you get a huge spike, your application will survive the spike, but you might have large latency until you get back to normal. If you are expecting to receive high loads for a sustained amount of time, you want to implement some sort of autoscaling that adds nodes to your cluster, and increases partitioning of the data. Autoscaling cannot react fast enough for momentary spikes, but it can prevent your system from being overwhelmed with sustained high loads From: Mich Talebzadeh <[email protected]> Date: Friday, March 26, 2021 at 1:44 PM To: "user @spark" <[email protected]> Subject: [EXTERNAL] The trigger interval in spark structured streaming CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. One thing I noticed is that when the trigger interval in foreachBatch is set to something low (in this case 2 seconds, equivalent to the batch interval that source sends data to Kafka topic (every 2 seconds) trigger(processingTime='2 seconds') Spark sends the warning that the queue is falling behind ``` batchId is 22, rows is 40 21/03/18 16:29:05 WARN org.apache.spark.sql.execution.streaming.ProcessingTimeExecutor: Current batch is falling behind. The trigger interval is 2000 milliseconds, but spent 13686 milliseconds batchId is 23, rows is 40 21/03/18 16:29:21 WARN org.apache.spark.sql.execution.streaming.ProcessingTimeExecutor: Current batch is falling behind. The trigger interval is 2000 milliseconds, but spent 15316 milliseconds batchId is 24, rows is 40 ``` So, assuming that the batch interval is somehow fixed, one needs to look at how to adjust the resources that process the topic in a timely manner. Any comments welcome [Image removed by sender.] view my Linkedln profile<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
