Re: Checkpoint metadata deleted by Flink after ZK connection issues

Great, thanks Klou! Cheers, Till On Mon, Sep 28, 2020 at 5:07 PM Kostas Kloudas wrote: > Hi all, > > I will have a look. > > Kostas > > On Mon, Sep 28, 2020 at 3:56 PM Till Rohrmann > wrote: > > > > Hi Cristian, > > > > thanks for reporting this issue. It looks indeed like a very critical > pr

Re: Flink Batch Processing

Hi Piotrek, Thank you for the reply. Flink changes are good, However Flink is changing so much that we are unable to get any good implementation examples either on Flink documents or any other website. Using HBaseInputFormat I was able to read the data as a DataSet<>, now I see that DataSet wou

Re: Reading from HDFS and publishing to Kafka

Hi, I actually have no experience running a Flink job on K8s against a kerberized HDFS so please take what I'll say with a grain of salt. The only thing you should need to do is to configure the path of your keytab and possibly some other Kerberos settings. For that check out [1] and [2].

Re: Flink Batch Processing

Hi Sunitha, here is some documentation about how to use the Hbase sink with Flink [1, 2]. [1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/connectors/hbase.html [2] https://docs.cloudera.com/csa/1.2.0/datastream-connectors/topics/csa-hbase-connector.html Cheers, Till On Tue,

Re: Streaming SQL Job Switches to FINISHED before all records processed

Hi Austin, could you share with us the exact job you are running (including the custom window trigger)? This would help us to better understand your problem. I am also pulling in Klou and Timo who might help with the windowing logic and the Table to DataStream conversion. Cheers, Till On Mon, S

Re: Flink Batch Processing

Hi Sunitha, currently, not every connector can be mixed with every API. I agree that it is confusing from time to time. The HBase connector is an InputFormat. DataSet, DataStream and Table API can work with InputFormats. The current Hbase input format might work best with Table API. If you li

Re: Poor performance with large keys using RocksDB and MapState

Thanks Yun!, I used this option, and it greatly helped 2:44 val be = new RocksDBStateBackend("file:///tmp")class MyConfig extends DefaultConfigurableOptionsFactory { override def createColumnOptions(currentOptions: ColumnFamilyOpti

Re: Apache Qpid connector.

Hi Austin, thanks for the response. Yes, the protocol is AMQP. I will try out the RabbitMQ connector with Qpid thanks, Parag On Sat, Sep 26, 2020 at 12:22 AM Austin Cawley-Edwards < austin.caw...@gmail.com> wrote: > Hey (Master) Parag, > > I don't know anything about Apache Qpid, but from the h

Savepoint incomplete when job was killed after a cancel timeout

Hi, We have a Flink job that was stopped erroneously with no available checkpoint/savepoint to restore, and are looking for some help to narrow down the problem. How we ran into this problem: We stopped the job using cancel with savepoint command (for compatibility issue), but the command tim

Re: Savepoint incomplete when job was killed after a cancel timeout

Hi Paul, could you share with us the logs of the JobManager? They might help to better understand in which order each operation occurred. How big are you expecting the size of the state to be? If it is smaller than state.backend.fs.memory-threshold, then the state data will be stored in the _meta

Re: [DISCUSS] FLIP-144: Native Kubernetes HA for Flink

For 1. I was wondering whether we can't write the leader connection information directly when trying to obtain the leadership (trying to update the leader key with one's own value)? This might be a little detail, though. 2. Alright, so we are having a similar mechanism as we have in ZooKeeper with

Re: Savepoint incomplete when job was killed after a cancel timeout

Thanks for sharing the logs with me. It looks as if the total size of the savepoint is 335kb for a job with a parallelism of 60 and a total of 120 tasks. Hence, the average size of a state per task is between 2.5kb - 5kb. I think that the state size threshold refers to the size of the per task stat

Re: Savepoint incomplete when job was killed after a cancel timeout

Hi Till, Thanks a lot for the pointer! I tried to restore the job using the savepoint in a dry run, and it worked! Guess I've misunderstood the configuration option, and confused by the non-existent paths that the metadata contains. Best, Paul Lam Till Rohrmann 于2020年9月29日周二 下午10:30写道： > Than

Re: Efficiently processing sparse events in a time windows

Thanks David, this is very helpful. I'm glad that it's not just that I had missed something obvious from the (generally very clear) documentation. I found various features that felt almost right (e.g. the priority queue behind Timers) but nothing that did the job. The temporal state idea does so

Re: Streaming SQL Job Switches to FINISHED before all records processed

Hey Till, Thanks for the reply -- I'll try to see if I can reproduce this in a small repo and share it with you. Best, Austin On Tue, Sep 29, 2020 at 3:58 AM Till Rohrmann wrote: > Hi Austin, > > could you share with us the exact job you are running (including the > custom window trigger)? Thi

Flink 1.12 snapshot throws ClassNotFoundException

Hi, I use Flink source master to build a snapshot and use the jars in my project. The goal is to avoid hacky deserialization code caused by avro 1.8 in old Flink versions since Flink 1.12 uses avro 1.10. Unfortunately, the code throws below ClassNotFoundException. I have verified that the akka-act

Re: Savepoint incomplete when job was killed after a cancel timeout

Glad to hear that your job data was not lost! Cheers, Till On Tue, Sep 29, 2020 at 7:28 PM Paul Lam wrote: > Hi Till, > > Thanks a lot for the pointer! I tried to restore the job using the > savepoint in a dry run, and it worked! > > Guess I've misunderstood the configuration option, and confus

Flink on k8s

Hi How to configure statebackend when I deploy flink on k8s , I just add the following to flink-conf.yaml, but it doesn’t work state.backend: rocksdb state.checkpoints.dir: hdfs://slave2:8020/flink/checkpoints state.savepoints.dir: hdfs://slave2:8020/flink/savepoints state.backend.incrementa

Re: Flink 1.12 snapshot throws ClassNotFoundException

Hi Lian, I suspect that it is caused by an incompatible Akka version. Flink uses Akka 2.5.21 instead of 2.5.12. Moreover, you are mixing Flink jars which use Scala 2.11 with Akka dependencies which are built against Scala 2.12. I am not an Gradle expert but can't Gradle simply pull in the transit