Hi Folks, I am trying to speed up my spark streaming job, I found a presentation by Tathagata Das that mentions to increase value of "spark.streaming.concurrentJobs" if I have more than one output.
In my spark streaming job I am reading from Kafka using Receiver Bases approach and transforming each line of data from Kafka and storing to HBase. I do not intend to do any kind of collation at this stage. I believe this can be parallelized by creating a separate job to write a different set of lines from Kafka to HBase and hence, I set the above parameter to a value > 1. Is my above assumption that writing to HBase for each partition in the RDDs from a given DStream is an independent output operation and can be parallelized? If the assumption is correct, and I run the job - this job creates multiple (smaller) jobs but they are executed one after another, not in parallel - I am curious if there is a requirement that #Executors be >= a particular number (a calculation based on how many repartitions after unio od DSreams etc. - I don't know I am grasping at Straws here.) I would appreciate some help in this regard. Thanks in advance. -- Regards, Atul Kulkarni
