Re: Slow activation using Spark Streaming's new receiver scheduling mechanism

I do also have this problem. The total time for launching receivers seems related to the total number of executors. In my case, when I run 400 executors with 200 receivers, it takes about a minute for all receivers become active, but with 800 executors, it takes 3 minutes to activate all receivers.

Slow activation using Spark Streaming's new receiver scheduling mechanism

Hi all, My team uses Spark Streaming to implement the batch processing component of a lambda architecture with 5 min intervals. We process roughly 15 TB/day using three discrete Spark clusters and about 250 receivers per cluster. We've been having some issues migrating our platform from Spark 1