n as follows in Web UI (note that
"Persist Checkpoints Externally" is "Disabled" in the final row):
According to [2],
externalized checkpoints: You can configure periodic checkpoints to be
persisted externally. Externalized checkpoints write their meta data out to
persisten
Got it.Thank you
On Thu, Aug 22, 2019 at 8:54 PM Congxian Qiu wrote:
> Hi, Vishwas
>
> As Zhu Zhu said, you can set "state.checkpoints.num-retained"[1] to
> specify the maximum number of completed checkpoints to retain.
> maybe you can also ref the external checkpoint cleanup type[2] config for
Hi, Vishwas
As Zhu Zhu said, you can set "state.checkpoints.num-retained"[1] to specify
the maximum number of completed checkpoints to retain.
maybe you can also ref the external checkpoint cleanup type[2] config for
how to clean up the retained checkpoint[2]
[1]
https://ci.apache.org/projects/fl
Hi Vishwas,
You can configure "state.checkpoints.num-retained" to specify the max
checkpoints to retain.
By default it is 1.
Thanks,
Zhu Zhu
Vishwas Siravara 于2019年8月22日周四 上午6:48写道：
> I am also using exactly once checkpointing mode, I have a kafka source and
> sink so both support transactions
I am also using exactly once checkpointing mode, I have a kafka source and
sink so both support transactions which should allow for exactly once
processing. Is this the reason why there is only one checkpoint retained ?
Thanks,
Vishwas
On Wed, Aug 21, 2019 at 5:26 PM Vishwas Siravara
wrote:
> H
Hi peeps,
I am externalizing checkpoints in S3 for my flink job and I retain them on
cancellation. However when I look into my S3 bucket where the checkpoints
are stored there is only 1 checkpoint at any point in time . Is this the
default behavior of flink where older checkpoints are deleted when
10%3A+Unify+Checkpoints+and+Savepoints
?
Thank you,
Juan G.
From: hao gao
Date: Wednesday, 25 April 2018 at 20:25
To: Juan Gentile
Cc: "user@flink.apache.org" , Oleksandr Nitavskyi
Subject: Re: Externalized checkpoints and metadata
Hi Juan,
We modified the flink code
at 20:25
To: Juan Gentile
Cc: "user@flink.apache.org" , Oleksandr Nitavskyi
Subject: Re: Externalized checkpoints and metadata
Hi Juan,
We modified the flink code a little bit to change the flink checkpoint
structure so we can easily identify which is which
you can read my note or th
helps
Thanks
Hao
2018-04-25 6:07 GMT-07:00 Juan Gentile :
> Hello,
>
>
>
> We are trying to use externalized checkpoints, using RocksDB on Hadoop
> hdfs.
>
> We would like to know what is the proper way to resume from a saved
> checkpoint as we are currently running
Hello,
We are trying to use externalized checkpoints, using RocksDB on Hadoop hdfs.
We would like to know what is the proper way to resume from a saved checkpoint
as we are currently running many jobs in the same flink cluster.
The problem is that when we want to restart the jobs and pass the
I left a question on stackoverflow (
https://stackoverflow.com/questions/49712817/two-questions-on-flink-externalized-checkpoints)
and ask again in mail-list in case you check mail more often.
I have two questions on Flink externalized checkpoints
(Q1) I can set "state.checkpoints.dir"
roduce duplicates? Is
> it because Kafka guarantees only At-least once delivery at the moment?
>
> Thanks,
> Amara
>
>
>
> --
> View this message in context:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Duplica
> ted-data-when-using-Externali
FlinkKafkaConsumer onwards. Why does this component introduce duplicates? Is
it because Kafka guarantees only At-least once delivery at the moment?
Thanks,
Amara
--
View this message in context:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Duplicated-data-when-using-Externalized
Kafka) | | | |
|
|--| |---|
||
Thanks,
Amara
--
View this message in context:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Duplicated-data-when-using-Externalized-Checkpoints-in-a-Flink-Highly-Available-cluster-tp13301p13481.html
Sen
processing for my application? Could you
> please guide me on what I might have missed?
>
> Thanks,
> Amara
>
>
>
>
> --
> View this message in context: http://apache-flink-user-
> mailing-list-archive.2336050.n4.nabble.com/Duplicated-data-
> when-using-Externalized
? Could you
please guide me on what I might have missed?
Thanks,
Amara
--
View this message in context:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Duplicated-data-when-using-Externalized-Checkpoints-in-a-Flink-Highly-Available-cluster-tp13301p13379.html
Sent from the Apache
flink-user-mailing-list-archive.2336050.n4.nabble.com/Duplicated-data-when-using-Externalized-Checkpoints-in-a-Flink-Highly-Available-cluster-tp13301.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at
Nabble.com.
hanks,
Amara
--
View this message in context:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Duplicated-data-when-using-Externalized-Checkpoints-in-a-Flink-Highly-Available-cluster-tp13301.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at
Nabble.com.
oints are externalized checkpoints with some
more properties.
At the moment, Flink's checkpoint coordinator only retains the last
successfully completed checkpoint. This means that whenever a new
checkpoint completes then the last completed checkpoint will be discarded.
This also applies to ex
in memory. The former has the advantage that you don't
> lose the checkpoints if you shutdown your cluster. They behave similar to
> savepoints and in fact savepoints are externalized checkpoints with some
> more properties.
>
> At the moment, Flink's checkpoint coo
Hi all,
Could someone clarify the difference between externalized checkpoints[1]
and regular periodic checkpoints[2]?
Moreover, I have a question regarding the retention of checkpoints: For
regular checkpoints, does the last checkpoint discard the previous ones? If
yes, is that the case too for
.
>
> On Thu, Nov 3, 2016 at 3:23 PM, Ufuk Celebi wrote:
> > I don't need the logs. Externalized checkpoints have been configured
> > to be deleted when the job is suspended, too. When the YARN session is
> > terminated, all jobs are suspended.
> >
> > The
A fix is pending here: https://github.com/apache/flink/pull/2750
The behaviour on graceful shut down/suspension respects the
cancellation behaviour with this change.
On Thu, Nov 3, 2016 at 3:23 PM, Ufuk Celebi wrote:
> I don't need the logs. Externalized checkpoints have been configured
I don't need the logs. Externalized checkpoints have been configured
to be deleted when the job is suspended, too. When the YARN session is
terminated, all jobs are suspended.
The behaviour seems like a bug. As a work around you have to cancel
the job before you shut down the YARN session. L
be going on here?
>
>
> On Wed, 2 Nov 2016 at 21:52 Clifford Resnick wrote:
>>
>> Testing externalized checkpoints in a YARN-based cluster, configured with:
>>
>>
>> env.getCheckpointConfig.enableExternalizedCheckpoints(ExternalizedCheckpointCleanup.RETAIN_
+Ufuk
Ufuk recently worked on that, if I'm not mistaken. Do you have an Idea what
could be going on here?
On Wed, 2 Nov 2016 at 21:52 Clifford Resnick wrote:
> Testing externalized checkpoints in a YARN-based cluster, configu
Testing externalized checkpoints in a YARN-based cluster, configured with:
env.getCheckpointConfig.enableExternalizedCheckpoints(ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
I can confirm that checkpoint is retained between cancelled jobs, however it’s
deleted when the Job Manager
27 matches
Mail list logo
