Re: Which flink version is compatible with beam

Hi, It’s not working with flink 1.14 and beam 2.44 or 2.43 with python 3.10. Please suggest. Get Outlook for iOS From: Yaroslav Tkachenko Sent: Friday, January 27, 2023 10:53:49 PM To: P Singh Cc: user@flink.apache.org Subject: Re: Which

Rescale Flink - Adjust Parallelism based on TPS

Hello Team, I would like to understand if there's a way flink can rescale automatically by adjusting parallelism based on incoming traffic. For example, when a job starts incoming TPS is around 100 which can be processed with parallelism 4, and when TPS increased 1000 I might need to rescale fl

Re: "Error while retrieving the leader gateway" when using Kubernetes HA

We never encountered this problem before but also we don't configure those settings. Can you simply try: high-availability: kubernetes And remove the other configs? I think that can only cause problems and should not achieve anything :) Gyula On Fri, Jan 27, 2023 at 6:44 PM Anton Ippolitov via

Re: Flink application mode/S3A Exception after upgrading from to Flink 1.16.0

Thank you Gabor. I will test with a downgraded version to see how it goes. On Fri, Jan 27, 2023 at 11:51 AM Gabor Somogyi wrote: > The min supported version was 2.8.5 but in 1.17 it's gonna be 2.10.1 so > one can downgrade. > > G > > On Fri, Jan 27, 2023, 20:42 Leon Xu wrote: > >> Thank you Ma

Re: Flink application mode/S3A Exception after upgrading from to Flink 1.16.0

The min supported version was 2.8.5 but in 1.17 it's gonna be 2.10.1 so one can downgrade. G On Fri, Jan 27, 2023, 20:42 Leon Xu wrote: > Thank you Mate. > Yeah this looks like the root cause. A follow-up question, do you know if > Flink 1.16 will have a hard dependency on Hadoop 3.3.x? or can

Re: Flink application mode/S3A Exception after upgrading from to Flink 1.16.0

Thank you Mate. Yeah this looks like the root cause. A follow-up question, do you know if Flink 1.16 will have a hard dependency on Hadoop 3.3.x? or can we actually use a older version of hadoop to work around this issue. Thanks Leon On Fri, Jan 27, 2023 at 10:28 AM Máté Czagány wrote: > Hi Leo

Re: Flink application mode/S3A Exception after upgrading from to Flink 1.16.0

Hi Leon, It seems to me like S3 cannot be used as YARN resource storage starting with Hadoop 3.3.2 In HADOOP-17139 S3AFileSystem#copyFromLocalFile was refactored and expects the local source Hadoop Path object to have a scheme specified which the YarnClusterDescriptor uploading the local files wo

Multiple Window Streams to same Kinesis Sink

I'm trying to sink two Window Streams to the same Kinesis Sink. When I do this, no results are making it to the sink (code below). If I remove one of the windows from the Job, results do get published. Adding another stream to the sink seems to void both. How can I have results from both Window

"Error while retrieving the leader gateway" when using Kubernetes HA

Hi everyone, I've been experimenting with Kubernetes HA and the Kubernetes Operator and ran into the following issue which is happening regularly on TaskManagers with Flink 1.16: Error while retrieving the leader gateway. Retrying to connect to akka.tcp://flink@SERVICE-NAME-HERE.NAMESPACE-HERE:61

Re: Which flink version is compatible with beam

Hi! According to this https://beam.apache.org/documentation/runners/flink/#flink-version-compatibility, 1.14 is the latest supported version. On Fri, Jan 27, 2023 at 9:19 AM P Singh wrote: > Hi Team, > > I am trying to run apache beam pipeline on flink cluster. I have set up > kubernetes locally

Which flink version is compatible with beam

Hi Team, I am trying to run apache beam pipeline on flink cluster. I have set up kubernetes locally with flink1.16and apache/beam_python3.10_sdk:2.44.0. When I submit the job using like python file.py Job is just hang not able to see on flink UI or logs. Can you please suggest compatible vers