This preferred locality is a hint to spark to schedule Kafka tasks on the preferred nodes, if Kafka and Spark are two separate cluster, obviously this locality hint takes no effect, and spark will schedule tasks following node-local -> rack-local -> any pattern, like any other spark tasks.
On Wed, Oct 14, 2015 at 8:10 PM, Rishitesh Mishra <rmis...@snappydata.io> wrote: > Hi Gerard, > I am also trying to understand the same issue. Whatever code I have seen > it looks like once Kafka RDD is constructed the execution of that RDD is > upto the task scheduler and it can schedule the partitions based on the > load on nodes. There is preferred node specified in Kafks RDD. But ASFIK it > maps to the Kafka partitions host . So if Kafka and Spark are co hosted > probably this will work. If not, I am not sure how to get data locality for > a partition. > Others, > correct me if there is a way. > > On Wed, Oct 14, 2015 at 3:08 PM, Gerard Maas <gerard.m...@gmail.com> > wrote: > >> In the receiver-based kafka streaming model, given that each receiver >> starts as a long-running task, one can rely in a certain degree of data >> locality based on the kafka partitioning: Data published on a given >> topic/partition will land on the same spark streaming receiving node until >> the receiver dies and needs to be restarted somewhere else. >> >> As I understand, the direct-kafka streaming model just computes offsets >> and relays the work to a KafkaRDD. How is the execution locality compared >> to the receiver-based approach? >> >> thanks, Gerard. >> > > > > -- > > Regards, > Rishitesh Mishra, > SnappyData . (http://www.snappydata.io/) > > https://in.linkedin.com/in/rishiteshmishra >
