If you are going aggregations, you need to watermark the data. Depending on what aggrgations you are doing, state might keep accumulating till failure.
From: Eric Beabes <mailinglist...@gmail.com> Date: Thursday, January 21, 2021 at 12:19 PM To: Sean Owen <sro...@gmail.com> Cc: spark-user <user@spark.apache.org> Subject: RE: [EXTERNAL] Only one Active task in Spark Structured Streaming application CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. Yes. For this particular use case the state size could be big but I doubt if there's a leak. Maybe adding more memory would help. On Thu, Jan 21, 2021 at 5:55 PM Sean Owen <sro...@gmail.com<mailto:sro...@gmail.com>> wrote: Is your app accumulating a lot of streaming state? that's one reason something could slow down after a long time. Some memory leak in your app putting GC/memory pressure on the JVM, etc too. On Thu, Jan 21, 2021 at 5:13 AM Eric Beabes <mailinglist...@gmail.com<mailto:mailinglist...@gmail.com>> wrote: Hello, My Spark Structured Streaming application was performing well for quite some time but all of a sudden from today it has slowed down. I noticed in the Spark UI that the 'No. of Active Tasks' is 1 even though 64 Cores are available. (Please see the attached image). I don't believe there's any data skew issue related to partitioning of data. What could be the reason for this? Please advise. Thanks. --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org<mailto:user-unsubscr...@spark.apache.org>
