Re: Wrapping a Flink Function

2020-10-08 Thread Teng Fei Liao
I think any solution here is inherently fragile since future versions of flink can have different abstract classes or interfaces you won't know it has to support. But for a given release, something you can consider is a wrapper class that extends/implements the ones you support. Then, during the me

CheckpointedFunction initialization during checkpoint

2020-09-28 Thread Teng Fei Liao
Hey all, I've been trying to debug a job recovery performance issue and I'm noticing some interesting events in the timeline that seem unexpected to me. Here's a brief outline of the first checkpoint following a job restart: 1. All tasks are deployed and transition into the RUNNING state. 2. I se

Re: Flink not restoring from checkpoint when job manager fails even with HA through zookeeper

2020-06-06 Thread Teng Fei Liao
It seems like the JobManager is treating this as a job failure. A FAILED JobStatus is a globally terminal state so everything gets deleted with zookeeper HA. https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/common/JobStatus.java#L39 https://github.com/apache

[jira] [Created] (FLINK-17355) Exactly once kafka checkpointing sensitive to single node failures

2020-04-23 Thread Teng Fei Liao (Jira)
Teng Fei Liao created FLINK-17355: - Summary: Exactly once kafka checkpointing sensitive to single node failures Key: FLINK-17355 URL: https://issues.apache.org/jira/browse/FLINK-17355 Project: Flink

[jira] [Created] (FLINK-17195) Worker threads for source function have wrong ClassLoader

2020-04-16 Thread Teng Fei Liao (Jira)
Teng Fei Liao created FLINK-17195: - Summary: Worker threads for source function have wrong ClassLoader Key: FLINK-17195 URL: https://issues.apache.org/jira/browse/FLINK-17195 Project: Flink