Re: How to know when a pipeline ends

2023-05-11 Thread Weihua Hu
Hi, Luke You can enable "execution.attached", then env.execute() will wait until the job is finished. [1] https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#execution-attached Best, Weihua On Fri, May 12, 2023 at 8:59 AM Shammon FY wrote: > Hi Luke, > > Maybe you ca

Re: How to know when a pipeline ends

2023-05-11 Thread Shammon FY
Hi Luke, Maybe you can get 'JobClient' after submit the job and check the job status with 'JobClient.getJobStatus()' Best, Shammon FY On Fri, May 12, 2023 at 2:58 AM Luke Xiong wrote: > Hi, > > My flink job needs to do something when the pipeline execution has ended. > The job code is like th

How to know when a pipeline ends

2023-05-11 Thread Luke Xiong
Hi, My flink job needs to do something when the pipeline execution has ended. The job code is like this: createSomeStream().applySomeOperators(); env.execute(jobName); doSomeCleanupTasks(); It looks like doSomeCleanupTasks() can be called while the pipeline is still running. The job is for proce

Issues using PyFlink

2023-05-11 Thread Jill Cardamon
Hi! I'm new to Flink, and I have been trying to run a simple python flink script that consumes messages from Kafka as well as the examples locally with a few issues. 1. When I run the word count example using `./flink-1.17.0/bin/flink run --python flink-1.17.0/examples/python/datastream/word_coun

Re: Flink compatibility with JDK17

2023-05-11 Thread amenreet sodhi
Hi Martijn, Thanks a lot for sharing this information. Best regards, Amenreet Singh Sodhi On Thu, May 11, 2023, 11:43 PM Martijn Visser wrote: > Hi Amenreet Singh Sodhi, > > Flink is compatible with JDK8 and JDK11, not with JDK17. You can find the > Jira issue that tracks compatibility at > ht

Re: Flink compatibility with JDK17

2023-05-11 Thread Martijn Visser
Hi Amenreet Singh Sodhi, Flink is compatible with JDK8 and JDK11, not with JDK17. You can find the Jira issue that tracks compatibility at https://issues.apache.org/jira/browse/FLINK-15736. The biggest problem is the Kryo serializer that's currently being used. That doesn't work with JDK17, but up

Flink compatibility with JDK17

2023-05-11 Thread amenreet sodhi
Hi, I wanted to confirm if Apache Flink is compatible with JDK17. If not, when can we expect this to happen? What major issues will I face if I try to compile it with JDK17? Thanks Regards Amenreet Singh Sodhi

Re: Flink Sql erroring at runtime

2023-05-11 Thread neha goyal
Hi Hang and community, There is a correction in my earlier email. The issue comes when I use the UPPER or TRIM function with IF. Looks like there is a bug with Flink 1.16's IF operator. If I use UPPER or TRIM functions(there might be more such functions), I am getting the exception. These functions

Re: Flink Job Restarts if the metadata already exists for some Checkpoint

2023-05-11 Thread Weihua Hu
Hi, checkpoints are only used in failover for one job. Once a job is cancelled, the related checkpoint-count metadata (stored on HA) will be removed. But the checkpoint data could be retained if you configured it. IIUC, the redeploy/update job will cancel the old job and then start a new one. The