Re: Flink Job Manager Recovery from EKS Node Terminations

2023-01-11 Thread Yang Wang
First, JobManager does not store any persistent data to local when the Kubernetes HA + S3 used. It means that you do not need to mount a PV for JobMananger deployment. Secondly, node failures or terminations should not cause the CrashLoopBackOff status. One possible reason I could imagine is a bug

PyFlink raises KeyError at on_evict method of CachedMapState

2023-01-11 Thread Ru Zhang
Hello, I am an user of pyflink. The pseudo code of my pipeline is as follows: ``` if not map_state.contains(data[“key”]): processing(data) map_state.put(data[“key”], 0) ``` However, it raises a KeyError: File "/opt/venv/lib64/python3.7/site-packages/pyflink/fn_execution/state_impl.py"

Lookup join or enrichment join against a changelog stream in Apache Flink Table API

2023-01-11 Thread Colin Williams
I'm interested in doing a "lookup join" or "enrichment join" against a "changelog stream" read by "upsert-kafka". I am wondering if this is possible against the table API. I found https://github.com/fhueske/flink-sql-demo#enrichment-join-against