Re: Jobmanager trying to be registered for Zombie Job

Hi Peter, based on our analysis the issue already existed before 1.15, yes. We couldn't come up with any other reasoning. It was just never reported... or noticing an older ticket. Matthias On Mon, Apr 25, 2022 at 6:21 PM Peter Schrott wrote: > Hi Matthias, > > You are welcome & thanks a lot fo

Re: Random incorrect checkpoint existence check

Hi, Chen-che, I think it may be similar to FLINK-12381 . You could adopt the suggestion like setting the job-id as you could see the comment below the ticket. I think you could also share your environment in this ticket to let us know more informat

Re: Unit testing PyFlink SQL project

Great to hear! Regards, Dian On Tue, Apr 26, 2022 at 4:11 AM John Tipper wrote: > Hi Dian, > > I've tried this and it works nicely, on both MacOS and Windows, thank you > very much indeed for your help. > > Kind regards, > > John > -- > *From:* Dian Fu > *Sent:* 25

Re: AvroRowDeserializationSchema

Hi Quynh, The same code in my last reply showed how to set the UID for the source operator generated using Table API. I meant that you could firstly create a source using Table API, then convert it to a DataStream API and set uid for the source operator using the same code above, then perform oper

RE: UUID on TableAPI

Ok, thanks for the clarification.   Sent from Mail for Windows From: Francis ConroySent: Tuesday, April 26, 2022 7:26 AMTo: lan tranCc: user@flink.apache.orgSubject: Re: UUID on TableAPI Hi Quynh,  My understanding is mostly based on the documentation I linked in the first reply. If the flink versi

Re: UUID on TableAPI

Hi Quynh, My understanding is mostly based on the documentation I linked in the first reply. If the flink version and the query both remain the same then you can restart a job from a savepoint, this means that it might be workable for running a low-criticality job on say an AWS spot instance. Tha

RE: UUID on TableAPI

Hi Francis,Thanks for the reply. However, can you elaborate more on the part ‘work for cases where you wish to pause/resume a job’ ? Is there another way besides savepoint having this mechanism that can implement in Table API ? Best,Quynh  Sent from Mail for Windows From: Francis ConroySent: Tuesda

Re: UUID on TableAPI

Hi Quynh, Set the savepoint UUID? Is that what you mean? The savepoint UUIDs are issued dynamically when you request them, flink won't know automatically what the last savepoint was, but you can start a new job and restore from a savepoint by passing in the UUID. All that said there are limitatio

Re: flink-stop command fails with ` Operation not found under key`

Hello Huweihua (sorry for dup email, ended up clicking reply instead of reply-all), Thanks for reaching out and having a look at it. The job managers have the same logs I shared before, another example below. {"instant":{"epochSecond":1650920964,"nanoOfSecond":64800},"thread":"flink-akka.acto

Re: Unit testing PyFlink SQL project

Hi Dian, I've tried this and it works nicely, on both MacOS and Windows, thank you very much indeed for your help. Kind regards, John From: Dian Fu Sent: 25 April 2022 02:42 To: John Tipper Cc: user@flink.apache.org Subject: Re: Unit testing PyFlink SQL proje

Re: Jobmanager trying to be registered for Zombie Job

Hi Matthias, You are welcome & thanks a lot for your help too! It's not quite clear to me, the bug was already there since 1.13.6 but not reported yet (FLINK-27354 is a new ticket)? Best, Peter On Mon, Apr 25, 2022 at 5:48 PM Matthias Pohl wrote: > Thanks again, Peter for sharing your logs.

Re: Jobmanager trying to be registered for Zombie Job

Thanks again, Peter for sharing your logs. I looked into the issue with the help of Chesnay. Essentially, it's FLINK-27354 [1] that is causing this issue. We couldn't come up with a reason why it should have popped up just now with 1.15. The bug itself is already present in 1.14. You can find more

Re: OOM errors cause by the new KafkaSink API

Hi Huweihua, Thanks for the reply. Yes, we increased memory first. But we are still curious about the memory increasing with the new Kafka APIs/Serilizers. On Mon, Apr 25, 2022 at 8:38 PM huweihua wrote: > Hi, > > You can try to increase the memory of TaskManager. > If there is persistent OOM,

RE: AvroRowDeserializationSchema

Hi Dian, Thank again for fast response. As your suggestion above, we can apply to set the UID for only for the DataStream state (as you suggest to convert from table to data stream). However, at the first phase which is collecting the data from Kafka ( having Debezium format), the UID cannot be set

Re: AvroRowDeserializationSchema

Hi Quynh, You could try the following code (also it may be a little hacky): ``` def set_uid_for_source(ds: DataStream, uid: str): transformation = ds._j_data_stream.getTransformation() source_transformation = transformation while not source_transformation.getInputs().isEmpty(): source_transformat

Re: OOM errors cause by the new KafkaSink API

Hi, You can try to increase the memory of TaskManager. If there is persistent OOM, you can dump the memory and check which part is taking up memory. > 2022年4月25日 上午11:44，Hua Wei Chen 写道： > > Hi all, > > Due to FlinkKafkaConsumer and FlinkKafkaProducer will be depreciated at Flink > 1.15[1]

Re: Jobmanager trying to be registered for Zombie Job

Thanks Peter, we're looking into it... On Mon, Apr 25, 2022 at 11:54 AM Peter Schrott wrote: > Hi, > > sorry for the late reply. It took me quite some time to get the logs out > of the system. I have attached them now. > > Its logs of 2 jobmanagers and 2 taskamangers. It can be seen on jm 1 that

Re: how to setup working dir in Flink operator

On Mon, Apr 25, 2022 at 05:15:58PM +0800, Yang Wang wrote: > Using the pod template to configure the local SSD(via host-path or local > PV) is the correct way. > After that, either "java.io.tmpdir" or "process.taskmanager.working-dir" in > CR should take effect. > > Maybe you need to share the com

Re: how to setup working dir in Flink operator

Using the pod template to configure the local SSD(via host-path or local PV) is the correct way. After that, either "java.io.tmpdir" or "process.taskmanager.working-dir" in CR should take effect. Maybe you need to share the complete pod yaml and logs of failed TaskManager. nit: if the TaskManager

Re: Write to Aliyun OSS via FileSystem connector hang Job Master on Finishing

Thanks Guowei. Good to know it may have been fixed in another way. I'll have a try. BTW, which version is this kind of sink supported from? Does it also work for batch mode? On Mon, Apr 25, 2022 at 2:42 PM Guowei Ma wrote: > Hi > > Afaik the commit files action happens at the committer operat