As far as I know, createStream doesn't let you specify where receivers are
run.
createDirectStream in 1.3 doesn't use long-running receivers, so it is
likely to give you more even distribution of consumers across your workers.
On Mon, Apr 13, 2015 at 11:31 AM, Laeeq Ahmed <laeeqsp...@yahoo.com.invalid>
wrote:
> Hi,
>
> I have 4 workers and I am trying to have parallel Kafka receivers, 1 on
> each worker, with the following code.
>
> val kafkaStreams = (0 to args.length - 1).map{ i =>
> KafkaUtils.createStream(ssc, zkQuorum, group, Map(args(i) ->
> 1),StorageLevel.MEMORY_ONLY).map(_._2) }
> val unifiedStream = ssc.union(kafkaStreams)
> unifiedStream.print()
>
> But I am getting receivers mostly on two workers (two on each), sometime
> on three workers. Whats wrong with the code??
>
> Regards,
> Laeeq
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
