Re: Shaded zookeeper - curator mismatch?

Hi Filip, Could you share the version of the ZK server you are connecting to? Best, Zhanghao Chen From: Filip Karnicki Sent: Friday, March 4, 2022 23:12 To: user Subject: Shaded zookeeper - curator mismatch? Hi, I believe there's a mismatch in shaded zookeeper

Re: Question about Flink counters

Hi Shane, Flink provides a generic counter interface with a few implementations. The default implementation SimpleCounter, which is not thread-safe, is used when you calling counter(String name) on a MetricGroup. Therefore, you'll need to use your own thread-safe implementation, check out the s

Re: Controlling group partitioning with DataStream

Hi, I think you are looking for this answer from David: https://stackoverflow.com/questions/69799181/flink-streaming-do-the-events-get-distributed-to-each-task-slots-separately-acc I think then you could technically create your partitioner - though little bit cubersome - by mapping your exist

Controlling group partitioning with DataStream

Hi all, I need to be able to control which slot a keyBy group goes to, in order to compensate for a badly skewed dataset. Any recommended approach to use here? Previously (with a DataSet) I used groupBy followed by a withPartitioner, and provided my own custom partitioner. I posted this same

Question about Flink counters

Hi all, For Flink counters [1], are increment operations guaranteed to be atomic across all parallel tasks? I.e., is there a guarantee that the counter values will not be higher than expected? Thanks, Shane --- [1] https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/ops/metrics/#c

Re: Source API question around idle (expensive) SplitFetchers being shutdown.

Hi, If I understand the code correctly, the only option is to implement a custom SplitFetcherManager. There, you can either: 1) override maybeShutdownFinishedFetchers(), or 2) override createSplitFetcher() to return a custom fetcher; that fetcher would override isIdle() and return true after some

Source API question around idle (expensive) SplitFetchers being shutdown.

I am working on developing a custom source with the new Source api. What I'm noticing is that during periods of low incoming data it repeatedly will shutdown and restart the fetchers when the split assignments are empty and periodically added. I get log message such as org.apache.flink.connector

Shaded zookeeper - curator mismatch?

Hi, I believe there's a mismatch in shaded zookeeper/curator dependencies. I see that curator 4.2.0 needs zookeeper 3.5.4-beta, but it's still used in flink-shaded-zookeeper-34, which as far as I can tell is used by flink runtime 1.14.3 https://mvnrepository.com/artifact/org.apache.flink/flink-run

Re: Task Manager shutdown causing jobs to fail

Hi, Puneet~ AFAIK, that should be expected behavior that jobs on crashed TaskManager restarts. HA means there is no single point risk but Flink job still need to through failover to ensure state and data consistency. You may refer https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/ops

Re: Incremental checkpointing & RocksDB Serialization

Hi Vidya, > Why is the incremental checkpointing taking more time for the snapshot at the > end of the window duration? I guess that this is because the job is under back pressure on end of window. You can expand the checkpoint details to see whether that the async duration of each task is muc

Re: Version Upgrade of FlinkSQL (1.10 to 1.12)

Hi, Martijn, Thanks for your information. It seems that the situation is similar to what I know, I will follow FLIP-190. Also congratulations on becoming a Flink committer! Best regards, Chen Zihao Martijn Visser 于2022年3月4日周五 16:18写道： > Hi, > > Per the documentation [1] stateful upgr

[statefun] hadoop dependencies and StatefulFunctionsConfigValidator

Hi All! We're running a statefun uber jar on a shared cloudera flink cluster, the latter of which launches with some ancient protobuf dependencies because of reasons[1]. Setting the following flink-config settings on the entire cluster classloader.parent-first-patterns.additional: org.apache.fli

Re: Version Upgrade of FlinkSQL (1.10 to 1.12)

Hi, Per the documentation [1] stateful upgrades for SQL are currently not supported when upgrading from one minor version to another. There's ongoing work to improve this (via FLIP-190 [2]) but that's currently not yet available. Best regards, Martijn Visser https://twitter.com/MartijnVisser82