[Flink-Kubernetes-Operator] Beam example application getting into "State: RECONCILING" after minikube restart

2023-02-18 Thread Ritesh Nadhani
Hello This is my first setup with Apache Beam and trying to use the flink runner for it. Following the example of https://github.com/apache/flink-kubernetes-operator/tree/main/examples/flink-beam-example I got my app running inside my local minikube. The app is pretty stateless - minimal transform

Re: KeyedProcessFunction within an iteration

2023-02-18 Thread Lorenzo Nicora
Hi Zhipeng IterativeStreams does have keyBy() methods, but they all throw UnsupportedOperationException [1] For some context: the whole thing is to do message enrichment with asyncIO, caching the enrichment info in state (with TTL). I am using an iteration as RichAsyncFunction does not support st

Re: KeyedProcessFunction within an iteration

2023-02-18 Thread Zhipeng Zhang
Hi Lorenzo, Could you provide some code example to reproduce your question? As I understand, IterativeStream#keyBy is supported since it is a subclass of DataStream. Moreover, we have implemented an unified iteration module for Flink [1] in Flink ML [2], which relies on Flink 1.15.2. Probably you

KeyedProcessFunction within an iteration

2023-02-18 Thread Lorenzo Nicora
Hi all, I am trying to implement an iterative streaming job that processes the loop with a KeyedProcessFunction. I need a KeyedProcessFunction to use keyed state and to emit a side-output (that after further transformations becomes the feedback) Problem is IterativeStream.process() only accepts