Hi, I am facing issues with the high number of active jobs showing in UI. I am using spark structure streaming to read data from Solace and writing back to Kafka. After 24-28Hr, I see active jobs start accumulating and it happens on random intervals after 24-28hr. And also I can see many active tasks more than 107817 and the job is up for 58 hr.
Spark Config executor 15 core 5 memory 10g driver 15g Job main flow is working absolutely fine. I can see data getting populated in Kafka. On HDFS, every minute batch data is stored. No issues at main flow. I did a lot of research but no luck. some say GC is not working or spark task scheduler is not in sync. Can anybody faced such issue in past or can guide me where to look, Thanks & Regards, Nayan Sharma *+91-8095382952* <https://www.linkedin.com/in/nayan-sharma> <http://stackoverflow.com/users/3687426/nayan-sharma?tab=profile>
