gt;>>>> consumerConfig.zkSessionTimeoutMs,
>>>>>> consumerConfig.zkConnectionTimeoutMs, ZKStringSerializer)
>>>>>>
>>>>>> offsetRanges.foreach { osr =>
>>>>>> val topicDirs = new ZKGroupTopicDirs(groupId, osr.topic)
>>>>>>
t;
>>>>>> Sounds like something's not set up right... can you post a minimal
>>>>>> code example that reproduces the issue?
>>>>>>
>>>>>> On Tue, Aug 25, 2015 at 1:40 PM, Susan Zhang
>>>>>> wrote:
>>
y Koeninger
>>>>>> wrote:
>>>>>>
>>>>>>> Are you actually losing messages then?
>>>>>>>
>>>>>>> On Tue, Aug 25, 2015 at 1:15 PM, Susan Zhang
>>>>>>> wrote:
>>>>>>>
>>>>>>
>>>>> On Tue, Aug 25, 2015 at 11:07 AM, Cody Koeninger >>>>>> > wrote:
>>>>>>>
>>>>>>>> Does the first batch after restart contain all the messages
>>>>>>>> received while the job was down?
>>
>>>>>>>> streaming job, wait 1 minute, then re-submit, there is somehow a
>>>>>>>> series of 0
>>>>>>>> event batches that
000 events.
>>>>>>>
>>>>>>> I see that at the beginning of the second launch, the checkpoint
>>>>>>> dirs are
>>>>>>> found and "loaded", according to console output.
>>>>>>>
>>>>
the
>>>>>> streaming job
>>>>>> would resume from checkpoint and continue processing from there
>>>>>> (without
>>>>>> seeing 0 event batches corresponding to when the job was down).
>>>>>>
>>>&
nching, there
>>>>> would
>>>>> be so many 0 event batches that the job would hang. Is this merely
>>>>> something
>>>>> to be "waited out", or should I set up some restart behavior/make a
>>>>> config
>>>>&
d hang. Is this merely
>>>> something
>>>> to be "waited out", or should I set up some restart behavior/make a
>>>> config
>>>> change to discard checkpointing if the elapsed time has been too long?
>>>>
>>>> Thanks!
&g
ng. Is this merely
>>> something
>>> to be "waited out", or should I set up some restart behavior/make a
>>> config
>>> change to discard checkpointing if the elapsed time has been too long?
>>>
>>> Thanks!
>>>
>>>
t; <
>> http://apache-spark-user-list.1001560.n3.nabble.com/file/n24450/Screen_Shot_2015-08-25_at_10.png
>> >
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Checkpointing-Res
3.nabble.com/file/n24450/Screen_Shot_2015-08-25_at_10.png
> >
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Checkpointing-Restarts-with-0-Event-Batches-tp24450.html
> Sent from the Apache Spark User List ma
user-list.1001560.n3.nabble.com/file/n24450/Screen_Shot_2015-08-25_at_10.png>
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Checkpointing-Restarts-with-0-Event-Batches-tp24450.html
Sent from the Apache Spark User
13 matches
Mail list logo
