Hi, I started Spark Streaming job with 96 executors which reads from 96 Kafka partitions and applies mapWithState on the incoming DStream.
Why would it cache only 77 partitions? Do I have to allocate more memory? Currently each executor gets 10 GB and it is not clear why it can't cache all 96 partitions. [cid:5DD89010-AB5E-43B3-88F6-486FDB58CD75] Thank you!
