I am not sure I fully understand your answer.
Is this code correct?
def main() {
KafkaUtils.createDirectStream[String, Array[Byte], StringDecoder,
DefaultDecoder](streamingContext, configs, topics).foreachRDD { rdd =>
Await.ready(processItems(rdd.collect()), Duration.Inf)
}
}
def processItems(items: Array[(String, Array[Byte])]): Future[Unit] = {
// start some work which may take a while and immediately return a
Future to keep track of the work
}
On Fri, Jul 8, 2016 at 12:56 PM, Sean Owen <so...@cloudera.com> wrote:
> It's no different than any other operation on an RDD. A transformation
> doesn't actually do anything by itself, so does not block. An action
> triggers computation and blocks until the action completes. You can wait
> for it with a Future, sure.
>
> On Fri, Jul 8, 2016 at 10:43 AM, Mikael Ståldal <mikael.stal...@magine.com
> > wrote:
>
>> In a Spark Streaming job, is the operation inside foreachRDD supposed to
>> synchronous / blocking?
>>
>> What if you do some asynchronous operation which returns a Future? Are
>> you then supposed to do Await on that Future?
>> --
>> [image: MagineTV]
>>
>> *Mikael Ståldal*
>> Senior software developer
>>
>> *Magine TV*
>> mikael.stal...@magine.com
>> Grev Turegatan 3 | 114 46 Stockholm, Sweden | www.magine.com
>>
>> Privileged and/or Confidential Information may be contained in this
>> message. If you are not the addressee indicated in this message
>> (or responsible for delivery of the message to such a person), you may
>> not copy or deliver this message to anyone. In such case,
>> you should destroy this message and kindly notify the sender by reply
>> email.
>>
>
>
--
[image: MagineTV]
*Mikael Ståldal*
Senior software developer
*Magine TV*
mikael.stal...@magine.com
Grev Turegatan 3 | 114 46 Stockholm, Sweden | www.magine.com
Privileged and/or Confidential Information may be contained in this
message. If you are not the addressee indicated in this message
(or responsible for delivery of the message to such a person), you may not
copy or deliver this message to anyone. In such case,
you should destroy this message and kindly notify the sender by reply
email.
