@Stefano, Aljoscha:
Thank you for pointing that out. With the following steps I verified that
the state of the job gets restored
1. Use HDFS as state backend with env.setStateBackend(new
FsStateBackend("hdfs:///home/user/flink/KafkaWordCount"))
2. Start the job. In my case the job ID is
Hi,
yes Stefano is spot on! The state is only restored if a job is restarted
because of abnormal failure. For state that survives stopping/canceling a
job you can look at savepoints:
https://ci.apache.org/projects/flink/flink-docs-master/apis/streaming/savepoints.html
This
essentially uses the same
Hello again,
thanks for giving a shot at my advice anyway but Aljoscha is far more
knowledgeable then me regarding Flink. :)
I hope I'm not getting mixed up again but I think gracefully canceling your
job means you lose your job state. Am I right in saying that the state is
preserved in case of a
@Aljoscha:
For this word count example I am using a kafka topic as the input stream.
The problem is that when I cancel the task and restart it, the task loses
the accumulated word counts so far and start counting from 1 again. Am I
missing something basic here?
@Stefano:
I also tried to implements
My bad, thanks for pointing that out.
On Wed, Apr 20, 2016 at 1:49 PM, Aljoscha Krettek
wrote:
> Hi,
> the *withState() family of functions use the Key/Value state interface
> internally, so that should work.
>
> On Wed, 20 Apr 2016 at 12:33 Stefano Baghino <
> stefano.bagh...@radicalbit.io> wro
Hi,
the *withState() family of functions use the Key/Value state interface
internally, so that should work.
On Wed, 20 Apr 2016 at 12:33 Stefano Baghino
wrote:
> Hi Jack,
>
> it seems you correctly enabled the checkpointing by calling
> `env.enableCheckpointing`. However, your UDFs have to eithe
Hi Jack,
it seems you correctly enabled the checkpointing by calling
`env.enableCheckpointing`. However, your UDFs have to either implement the
Checkpointed interface or use the Key/Value State interface to make sure
the state of the computation is snapshotted.
The documentation explains how to d
Hi,
what seems to be the problem?
Cheers,
Aljoscha
On Wed, 20 Apr 2016 at 03:52 Jack Huang wrote:
> Hi all,
>
> I am doing a simple word count example and want to checkpoint the
> accumulated word counts. I am not having any luck getting the counts saved
> and restored. Can someone help?
>
> en
Hi all,
I am doing a simple word count example and want to checkpoint the
accumulated word counts. I am not having any luck getting the counts saved
and restored. Can someone help?
env.enableCheckpointing(1000)
env.setStateBackend(new MemoryStateBackend())
> ...
inStream
> .keyBy({s =>
