You can get the batch start (the expected, not the exact time when the jobs
are submitted) from DStream operation "transform". There is a version of
transform that allows you specify a function with two params - the parent
RDD and the batch time at which the RDD was generated.
TD
On Thu, Nov 26, 2015 at 1:33 PM, Abhishek Anand <abhis.anan...@gmail.com>
wrote:
> Hi ,
>
> I need to use batch start time in my spark streaming job.
>
> I need the value of batch start time inside one of the functions that is
> called within a flatmap function in java.
>
> Please suggest me how this can be done.
>
> I tried to use the StreamingListener class and set the value of a variable
> inside the onBatchSubmitted function something like this :
>
> public void onBatchSubmitted(StreamingListenerBatchSubmitted
> batchSubmitted) { batchstarttime =
> batchSubmitted.batchInfo().batchTime().milliseconds();
> CommandLineArguments.BATCH_START_TIME = batchstarttime;
> }
>
>
> But, the issue is that the BATCH_START_TIME set only when the batch
> starts. I see in the worker logs that BATCH_START_TIME takes the default
> value and is not set.
>
>
> Please suggest how this can be achieved.
>
>
>
> BR,
> Abhi
>
