Re: [ANNOUNCE] FRocksDB 6.20.3-ververica-2.0 released

2023-01-31 Thread Leonard Xu
Thanks Yanfei for driving the release ! ! Best, Leonard > On Jan 31, 2023, at 3:43 PM, Yun Tang wrote: > > Thanks Yuanfei for driving the frocksdb release! > > Best > Yun Tang > From: Yuan Mei > Sent: Tuesday, January 31, 2023 15:09 > To: Jing Ge > Cc: Yanfei Lei ; d...@flink.apache.org >

Re: "Error while retrieving the leader gateway" when using Kubernetes HA

2023-01-31 Thread Gyula Fóra
There is also a pending fix for the standalone + k8s HA case : https://github.com/apache/flink-kubernetes-operator/pull/518 You could maybe try and review the fix :) Gyula On Tue, Jan 31, 2023 at 8:36 AM Yang Wang wrote: > I assume you are using the standalone mode. Right? > > For the native K

Flink 1.16 arm64/v8 docker images

2023-01-31 Thread Roberts, Ben (Senior Developer) via user
Hi, Is it possible for the arm64/v8 architecture images to be published for >1.16 flink-docker (apache/flink)? I’m aware that the official docker flink image is now published in the arm64 arch, but that image doesn’t include a JDK, so it’d be super helpful to have the apache/flink images publi

Re: "Error while retrieving the leader gateway" when using Kubernetes HA

2023-01-31 Thread Anton Ippolitov via user
I am using the Standalone Mode indeed, should've mentioned it right away. This fix looks exactly like what I need, thank you!! On Tue, Jan 31, 2023 at 9:16 AM Gyula Fóra wrote: > There is also a pending fix for the standalone + k8s HA case : > https://github.com/apache/flink-kubernetes-operator/

Re: "Error while retrieving the leader gateway" when using Kubernetes HA

2023-01-31 Thread Gyula Fóra
Thanks @Anton Ippolitov At this stage I would highly recommend the native mode if you have the liberty to try that. I think that has better production characteristics and will work out of the box with the autoscaler. (the standalone mode won't) Gyula On Tue, Jan 31, 2023 at 10:41 AM Anton Ippoli

Re: beam + flink + k8

2023-01-31 Thread Jan Lukavský
Hi, can you please share the also the script itself? I'd say that the problem is that the flink jobmanager is not accessible through localhost:8081, because it runs inside the minikube. You need to expose it outside of the minikube via [1], or run the script from pod inside the minikube and a

Re: "Error while retrieving the leader gateway" when using Kubernetes HA

2023-01-31 Thread Anton Ippolitov via user
Makes sense, thank you! On Tue, Jan 31, 2023 at 10:48 AM Gyula Fóra wrote: > Thanks @Anton Ippolitov > At this stage I would highly recommend the native mode if you have the > liberty to try that. > I think that has better production characteristics and will work out of > the box with the autos

Re: Using pyflink from flink distribution

2023-01-31 Thread Andrew Otto
Great, thank you so much for your responses. It all makes sense now. :) On Mon, Jan 30, 2023 at 10:41 PM Dian Fu wrote: > >> What is the reason for including > opt/python/{pyflink.zip,cloudpickle.zip,py4j.zip} in the base > distribution then? Oh, a guess: to make it easier for TaskManagers to

Re: beam + flink + k8

2023-01-31 Thread Jan Lukavský
The script looks good to me, did you run the SDK harness? External environment needs the SDK harness to be run externally, see [1]. Generally, the best option is DOCKER, but that usually does not work in k8s. For this, you might try PROCESS environment and build your own docker image for flink,

Extremely long startup time for exactly_once kafka sink

2023-01-31 Thread Bobby Richard
When enabling exactly_once on my kafka sink, I am seeing extremely long initialization times (over 1 hour), especially after restoring from a savepoint. In the logs I see the job constantly initializing thousands of kafka producers like this: 2023-01-31 14:39:58,150 INFO org.apache.kafka.clients.p

Re: Custom catalog implementation - getting table schema for computed columns

2023-01-31 Thread yuxia
Hi. Just FYI, I have seen some catalogs are still use deprecated TableSchema in flink hive, Iceberg, etc connector. But it's in Flink plan to drop the deprecated table schema [1]. In long term, seems use new schema api is a better choice. If it's for the case of Catalog's createTable method,

Re: Custom catalog implementation - getting table schema for computed columns

2023-01-31 Thread yuxia
HI, > about the question can I assume that ResolvedCatalogTable will be always a > runtime type. Sorry for I don't really understand your question , why do you have such assumtion? Best regards, Yuxia 发件人: "Krzysztof Chmielewski" 收件人: "User" 发送时间: 星期六, 2023年 1 月 21日 上午 3:13:12 主题: R

How to add permission validation? flink reads and writes hive table。

2023-01-31 Thread melin li
flink supports both sql and jar types.How can we implement a unified access check in flink? spark supports extensions; flink lacks extensions.

RE: Processing watermarks in a broadcast connected stream

2023-01-31 Thread Schwalbe Matthias
Good Morning Sajjad, I’ve once had a similar problem. As you’ve found out, directly using KeyedBroadcastProcessFunction is a little tricky. What I ended up with instead is to use the rather new @PublicEvolving MultipleInputStreamOperator. It allows you to connect and process any (reasonable) num

Re: How to add permission validation? flink reads and writes hive table。

2023-01-31 Thread yuxia
HI, melin li. Could you please explain a bit more about unified access check in flink? Best regards, Yuxia 发件人: "melin li" 收件人: "User" 发送时间: 星期三, 2023年 2 月 01日 下午 2:39:15 主题: How to add permission validation? flink reads and writes hive table。 flink supports both sql and jar types.H

Reducing Checkpoint Count for Chain Operator

2023-01-31 Thread Talat Uyarer via user
Hi, We have a job that is reading from kafka and writing some endpoints. The job does not have any shuffling steps. I implement it with multiple steps. Flink chained those operators in one operator in submission time. However I see all operators are doing checkpointing. Is there any way to crea