delete all available flink timers on app start

Hello, I have a custom app, in which when due to some exception, the app restarts I want to cancel all registered flink timers in the initializeState method. Based on the documentation I feel like all timer state is saved in the state, so if the app restarts the timers are still active. Is there

Re: Stream in loop and not getting to sink (Parquet writer )

Can you try closing the writer? AvroParquetWriter has an internal buffer. Try doing a .close() in snapshot()( since you are checkpointing hence this method will be called) On Wed, Nov 28, 2018 at 7:33 PM Avi Levi wrote: > Thanks Rafi, > I am actually not using assignTimestampsAndWatermarks , I

Flink REST api for cancel with savepoint on yarn

Hello, I have a question about flink 1.5/1.6 REST endpoints. I was trying to see how the rest endpoints have changed wrt to cancelling with savepoint; it seems like now to cancel with savepoint one need to use POST api / jobs/:jobid/savepoints

Flink on kubernetes: taskmanager error

Hello, I am trying to run flink on a kubernetes cluster using minikube and kubectl. I am following this example , which runs a flink 1.2 cluster ok. I am interested in running flink 1.5.1, but when I modify the flink version, I start to see these

Re: Flink 1.6 release note!!

I think you are looking for this? http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Flink-1-6-features-tc20502.html 1.6 release notes as per current website: https://ci.apache.org/projects/flink/flink-docs-master/release-notes/flink-1.6.html Recently 1.5 was released: ht

Having a backoff while experiencing checkpointing failures

Hello all, Are there any recommendations on using a backoff when experiencing checkpointing failures? What we have seen is when a checkpoint starts to expire, the next checkpoint dosent care about the previous failure, and starts soon after. We experimented with *min_pause_between_checkpoints*, ho

Exception on running an Elasticpipe flink connector

Hello, We are working on a Flink ES connector, sourcing from a kafka stream, and sinking data into elasticsearch. The code works fine in intellij, but while running the code on emr(version 5.9, which uses flink 1.3.2) using flink-yarn-session, we are seeing this exception Using the parallelism pr

Re: Pending parquet file with Bucking Sink

Hi Tao, Is checkpointing enabled in your app? The pending files should be moved to non-pending files after checkpoint interval. Please take a look at this link

Re: save points through REST API not supported ?

Hi Vishal, Job cancellations can be done via a REST API: https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html#cancel-job-with-savepoint Thanks, Vipul On Wed, Dec 6, 2017 at 10:56 AM, Vishal Santoshi wrote: > One can submit jobs, upload jars, kill jobs etc very s

Re: Questions about checkpoints/savepoints

). Thanks, Vipul On Tue, Oct 24, 2017 at 11:53 PM, vipul singh wrote: > Thanks Aljoscha for the explanations. I was able to recover from the last > externalized checkpoint, by using flink run -s > > I am curious, are there any options to save the metadata file name to some > o

Re: Questions about checkpoints/savepoints

y one file. If you know > the file-path (either from the Flink dashboard or by looking at the S3 > bucket) you can restore from it. > > Best, > Aljoscha > > > On 24. Oct 2017, at 08:22, vipul singh wrote: > > Thanks Tony, that was the issue. I was thinking that when

Re: Questions about checkpoints/savepoints

wrote: > Hi, > > Did you enable externalized checkpoints? [1] > > Best, > Tony Wei > > [1] https://ci.apache.org/projects/flink/flink-docs- > release-1.3/setup/checkpoints.html#externalized-checkpoints > > 2017-10-24 13:07 GMT+08:00 vipul singh : > >> Tha

Re: Questions about checkpoints/savepoints

read-after-write consistency. > > Best, > Aljoscha > > > On 10. Oct 2017, at 03:01, vipul singh wrote: > > Thanks Stefan for the answers above. These are really helpful. > > I have a few followup questions: > >1. I see my savepoints are created in a fol

Re: Custom Sink Checkpointing errors

Thanks Stefan. I found the issue in my application. Everything is working as excepted now. Once again thanks for the help and advice. On Fri, Oct 20, 2017 at 4:51 AM, vipul singh wrote: > Thanks Stefan for the answers. The serialization is happening during the > creation of snapshot st

Re: Custom Sink Checkpointing errors

uld be my guess. Ensure to duplicate serializers whenever > different threads could work on them, e.g. processing vs checkpointing. > > Best, > Stefan > > > > > Am 20.10.2017 um 14:24 schrieb vipul singh : > > Hello all, > > I am working on a custom sink implementa

Custom Sink Checkpointing errors

Hello all, I am working on a custom sink implementation, but having weird issues with checkpointing. I am using a custom ListState to checkpoint, and it looks like this: private var checkpointMessages: ListState[Bucket] =_ My snapshot function looks like: @throws[IOException] def snapshotStat

Re: Questions about checkpoints/savepoints

Thanks Stefan for the answers above. These are really helpful. I have a few followup questions: 1. I see my savepoints are created in a folder, which has a _metadata file and another file. Looking at the code

Weird error in submitting a flink job to yarn cluster

Hello, I am working on a ParquetSink writer, which will convert a kafka stream to parquet format. I am having some weird issues in deploying this application to a yarn cluster. I am not 100% sure this falls into a flink related error, but I wanted to reach out to folks here incase it might be. I

Questions about checkpoints/savepoints

Hello, I have some confusion about checkpoints vs savepoints, and how to use them effectively in my application. I am working on an application which is relies on flink's fault tolerant mechanism to ensure exactly once semantics. I have enabled external checkpointing in my application as below:

Re: Issues in recovering state from last crash using custom sink

> > On 28. Aug 2017, at 00:28, vipul singh wrote: > > Hi all, > > I am working on a flink archiver application. In a gist this application > tries to reads a bunch of schematized messages from kafka and archives them > to s3. Due to the nature of the naming of the files, I h

Issues in recovering state from last crash using custom sink

Hi all, I am working on a flink archiver application. In a gist this application tries to reads a bunch of schematized messages from kafka and archives them to s3. Due to the nature of the naming of the files, I had to go towards a custom sink implementation. As of the current progress the applica