1. Multiple output operations are processed in the order they are defined. That is because by default each one output operation is processed at a time. This *can* be parallelized using an undocumented config parameter "spark.streaming.concurrentJobs" which is by default set to 1.
2. Yes, the output operations (and the spark jobs that are involved with them) gets queued up. TD On Wed, Jul 9, 2014 at 11:22 AM, Yan Fang <yanfang...@gmail.com> wrote: > I am using the Spark Streaming and have the following two questions: > > 1. If more than one output operations are put in the same StreamingContext > (basically, I mean, I put all the output operations in the same class), are > they processed one by one as the order they appear in the class? Or they > are actually processes parallely? > > 2. If one DStream takes longer than the interval time, does a new DStream > wait in the queue until the previous DStream is fully processed? Is there > any parallelism that may process the two DStream at the same time? > > Thank you. > > Cheers, > > Fang, Yan > yanfang...@gmail.com > +1 (206) 849-4108 >
