Are you using KafkaUtils.createDirectStream?
On Wed, Aug 3, 2016 at 9:42 AM, Soumitra Johri
wrote:
> Hi,
>
> I am running a steaming job with 4 executors and 16 cores so that each
> executor has two cores to work with. The input Kafka topic has 4 partitions.
> With this given configuration I was
Did you check the executors logs to check whether the kafka offsets pulled in
evenly over the 4 executors?
I recall a similar situation with such uneven balancing from a kafka stream,
and ended up raising the amount of resources (RAM and cores). Then it nicely
balanced out. I don’t understand t
Hi,
I am running a steaming job with 4 executors and 16 cores so that each
executor has two cores to work with. The input Kafka topic has 4 partitions.
With this given configuration I was expecting MapWithStateRDD to be evenly
distributed across all executors, how ever I see that it uses only two
