Hi Ewan,Transforms are definitions of what needs to be done - they don't
execute until and action is triggered. For what you want, I think you might
need to have an action that writes out rdds to some sort of buffered writer.
-Ashic.
From: ewan.le...@realitymine.com
To: user@spark.apache.org
Subject: Batch together RDDs for Streaming output, without delaying execution
of map or transform functions
Date: Thu, 31 Dec 2015 11:35:37 +0000
Hi all,
I’m sure this must have been solved already, but I can’t see anything obvious.
Using Spark Streaming, I’m trying to execute a transform function on a DStream
at short batch intervals (e.g. 1 second), but only write the resulting data to
disk using saveAsTextFiles in a larger batch after a longer delay (say 60
seconds).
I thought the ReceiverInputDStream window function might be a good help here,
but instead, applying it to a transformed DStream causes the transform function
to only execute at the end of the window too.
Has anyone got a solution to this?
Thanks,
Ewan
