Re: HybridSource: how to specify different watermark strategies for underlying sources

2023-01-17 Thread David Christle via user
Hi Martijn, That's correct. We have a forked version of the PR from Ryan Skraba ( https://github.com/apache/flink/pull/18823), which itself is originally based on the PR from Jakob Edding ( https://github.com/apache/flink/pull/15152), and have incorporated an old PR (https://github.com/apache/flin

Re: Apache Beam MinimalWordCount on Flink on Kubernetes using Flink Kubernetes Operator on GCP

2023-01-17 Thread Yang Wang
The "JAR file does not exist" exception comes from the JobManager side, not on the client. Please be aware that the local:// scheme in the jarURI means the path in the JobManager pod. You could use an init-container to download your user jar and mount it to the JobManager main-container. Refer to

Apache Beam MinimalWordCount on Flink on Kubernetes using Flink Kubernetes Operator on GCP

2023-01-17 Thread Lee Parayno
I have a Kubernetes cluster in GCP running the Flink Kubernetes Operator. I'm trying to package a project with the Apache Beam MinimalWordCount using the Flink Runner as a FlinkDeployment to the Kubernetes Cluster Job Docker image created with this Dockerfile: FROM flink ENV FLINK_CLASSPATH /op

Re: Cluster uploading and running a jar itself

2023-01-17 Thread Martijn Visser
Hi Theo, Flink doesn't do that. The only occurrence I could find on the web is this from https://github.com/LandGrey/flink-unauth-rce/blob/master/flink-unauth-rce.py. Could it be that someone is accessing your cluster (perhaps via the public internet)? Best regards, Martijn Op ma 16 jan. 2023 o

Re: HybridSource: how to specify different watermark strategies for underlying sources

2023-01-17 Thread Martijn Visser
Hi David, While I don't immediately have an answer to your question, I was triggered by your inclusion of "an unbounded FLIP-27 PubSub source". Do you mean your own (forked) version of GCP using the new interfaces, since the current GCP connector is not a FLIP-27 compatible source? Best regards,

[SUMMARY] Flink 1.17 Release Sync 1/17/2023

2023-01-17 Thread Qingsheng Ren
Hi devs and users, I'd like to share some highlights from the release sync on 1/17/2023. - CI & Performance: totally 5 blocker issues. Owners should have been pinged. - FLIP-272 [1] has finished and there will be a blog post before 1.17 release. - PR about publishing SBOM [2] has been merged, a