Hi Sateesh, It is interesting to know , how did you determine that the Dstream runs on a single core. Did you mean receivers?
Coming back to your question, could you not start disk io in a separate thread, so that the sceduler can go ahead and assign other tasks ? On 21 Aug 2015 16:06, "Sateesh Kavuri" <sateesh.kav...@gmail.com> wrote: > Hi, > > My scenario goes like this: > I have an algorithm running in Spark streaming mode on a 4 core virtual > machine. Majority of the time, the algorithm does disk I/O and database > I/O. Question is, during the I/O, where the CPU is not considerably loaded, > is it possible to run any other task/thread so as to efficiently utilize > the CPU? > > Note that one DStream of the algorithm runs completely on a single CPU > > Thank you, > Sateesh >
