Re: Flink in HA mode causing JM Failure

Hi amenreet, According to the error message, I think you can log in the jm pod after it starts, and check access permissions for the directory `file:///opt/flink/pm/ha` Best, Shammon FY On Fri, Jul 7, 2023 at 6:04 PM amenreet sodhi wrote: > Hi Shammon > > I am using an external NFS mount whic

Java 21 for flink

Hi all, Java 21 plans to support light weight thread called fiber based on Project LOOM which will increase the concurrency to great extent. Is there any plan for flink to leverage it? Thanks, Prasanna.

Stateful job checkpoint size increases continuously

Hello, I am running a Flink stateful job, where the checkpoint size increases continuously over time (200+ MB). The actual State size should be in < 1 MB.  The source is a Kafka Topic.  The number of keys in the Topic is < 1000 (confirmed by inspecting the Topic).  Each Key needs to store a set

Re: Application Mode Start Using Restful API

Hi Xiao, I don't believe it's possible for the application mode. On Fri, Jul 7, 2023 at 12:20 PM Xiao Ma wrote: > Hi Alexey, > > Thank you for the reply. > > So the link pasted is to start a job though the kubectl command. Is there > a way to start an Application Mode job through the RESTful Ja

Re: Application Mode Start Using Restful API

Hi Alexey, Thank you for the reply. So the link pasted is to start a job though the kubectl command. Is there a way to start an Application Mode job through the RESTful Java API, instead of the command line? Thank you *Xiao Ma* On Fri, Jul 7, 2023 at 3:46 AM Alexey Novakov wrote: > Hi Xiao.

Re: Getting rid of getCheckpointLock() method

Hi Sanket, Have you read the release notes for Flink 1.11 at https://nightlies.apache.org/flink/flink-docs-release-1.11/release-notes/flink-1.11.html#removal-of-deprecated-streamtaskgetcheckpointlock-flink-12484 ? Given that Flink 1.11 is a version that's no longer supported in the Flink community

Re: Dealing with incompatible state in Temporal Left Join on Schema Evolution (Table API)

Hi Martijn, Indeed, I was thinking along the lines of the underlying operators changing, which does not happen in this case (The docs here: https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/table/concepts/overview/#stateful-upgrades-and-evolution specify that it depends on the un

Re: Dealing with incompatible state in Temporal Left Join on Schema Evolution (Table API)

Hi Michael, A change in your schema would also result in a change in the columns that you are selecting with your SELECT * query, would it not? Given that your compiled plan contains the fields you're selecting, it would mean that your compiled plan changes when you have a different schema. After

Re: Dealing with incompatible state in Temporal Left Join on Schema Evolution (Table API)

Hi Martijn, Thanks for the reply. I understand that this is generally the case for the Table/SQL API, though I didn't expect a change in the schema should effect a change in the compiled plan, especially since we are not changing the join keys, etc. I was wondering if there is a way to deal with

Re: Error upgrading operator CRD

Additionally, when I try the next step I get another unexpected error: ✗ helm -n flink template flink-kubernetes-operator flink-operator-repo/flink-kubernetes-operator Error: failed to fetch https://downloads.apache.org/flink/flink-kubernetes-operator-1.0.1/flink-kubernetes-operator-1.0.1-helm.tgz

Error upgrading operator CRD

Somehow I was able in the past to upgrade the CRD when I upgraded the operator to 1.2 and 1.3, but trying now to upgrade to 1.4 I am getting the following error: ✗ kubectl replace -f helm/flink-kubernetes-operator/crds/flinksessionjobs.flink.apache.org-v1.yml error: the path "helm/flink-kubernete

Re: Dealing with incompatible state in Temporal Left Join on Schema Evolution (Table API)

Hi Michael, In the current Table API/SQL, there's no guarantee that a change to either the query or the Flink version won't lead to state incompatibility. That's also documented at https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/upgrading/#table-api--sql Best regards, Martijn

Re: Flink in HA mode causing JM Failure

Hi amenreet, Maybe you can try to use hdfs or s3 for `high-availability.storageDir`, I found your current job is using a local file which is started with `file:///`. Best, Shammon FY On Fri, Jul 7, 2023 at 4:20 PM amenreet sodhi wrote: > Hi All, > I am deploying Flink cluster on Kubernetes in

Flink in HA mode causing JM Failure

Hi All, I am deploying Flink cluster on Kubernetes in HA mode. But i noticed, whenever i deploy Flink cluster for first time on K8s cluster, it is not able to populate the cluster configmap, and due to which JM fails with the following exception: 2023-07-06 16:46:11,428 ERROR org.apache.flink.runt

Re: Application Mode Start Using Restful API

Hi Xiao. I think you need to look at this example: https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/try-flink-kubernetes-operator/quick-start/#submitting-a-flink-job Best regards, Alexey On Thu, Jul 6, 2023 at 7:02 PM Xiao Ma wrote: > Hi Alexey, > > Yes, sending a RE

Dealing with incompatible state in Temporal Left Join on Schema Evolution (Table API)

Hi all, I had a question about how to allow a migration of state in a temporal left join. Let me first describe what we are doing: - We have an input stream (Kinesis stream) of device data and we are performing a left temporal join on this stream to enrich it with metadata about the device as the