Re: Dynamic rescaling in Flink

Hi Prasanna, IIUC, your screenshot shows the scaling feature of an EMR cluster, not Flink. Let me try to better understand your question. Which kind of rescaling do you need? - If you deploy a long running streaming job, and want it to dynamically rescale based on the real-time incoming data stre

Re: Dynamic rescaling in Flink

Thanks Xintong and Yu Yang for the replies, I see AWS provides deploying Flink on EMR out of the box. There they have an option of EMR cluster scaling based on the load. Is this not equal to dynamic rescaling ? [image: Screen Shot 2020-06-15 at 9.23.24 AM.png] https://docs.aws.amazon.com/emr/l

Re: The Flink job recovered with wrong checkpoint state.

Hi Thomas The answer is yes. Without high availability, once the job manager is down and even the job manager is relaunched via YARN, the job graph and last checkpoint would not be recovered. Best Yun Tang From: Thomas Huang Sent: Sunday, June 14, 2020 22:58 To

How do I backfill time series data?

Hello Flink community. I need help. Thus far, Flink has proven very useful to me. I am using it for stream processing of time-series data. For the scope of this mailing list, let's say the time-series has the fields: name: String, value: double, and timestamp: Instant. I named the time series: t

Re: Shared state between two process functions

Hi Jaswin, Currently the state belongs to single operators, thus it should be not possible to share states between different operators. Could you also share the original problem want to solve by sharing states ? Best, Yun --Original Mail -- Sender:Jaswin S

The Flink job recovered with wrong checkpoint state.

Hi Flink Community, Currently, I'm using yarn-cluster mode to submit flink job on yarn, and I haven't set high availability configuration (zookeeper), but set restart strategy: env.getConfig.setRestartStrategy(RestartStrategies.fixedDelayRestart(10, 3000)) the attempt time is 10 and the wait

Shared state between two process functions

Hi, Is it possible to create the shared state(MapState) between two different keyedProcessFunction? If it's possible, how can we do that in flink? Thanks, Jaswin

Re: Incremental state

Hi Can process function[1] can meet your needs here?, you can do the TTL logic using timers in process functions. [1] https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/stream/operators/process_function.html Best, Congxian Timo Walther 于2020年6月10日周三 下午9:36写道： > Hi Annemarie, > >

Re: MapState in flink

Hi Could you please share why you need `MapState` instead of `MapState>` Best, Congxian Oytun Tez 于2020年6月14日周日 上午3:39写道： > Correct me @everyone if I'm wrong, but you cannot keep State inside State > in that way. So change your signature to: MapState> > > The underlying mechanism wouldn't mak