Re: Flink - Pod Identity

Austin - In my case the set up is such that services are deployed on Kubernetes with Docker, running on EKS. There is also an istio service mesh. So all the services communicate and access AWS resources like S3 using the service account. Service account is associated with IAM roles. I have verifie

Re: ARM support

Thanks for all of this info. Highly appreciated! On Thu, Apr 1, 2021 at 1:17 AM Guowei Ma wrote: > Hi, Rex > > I think that Flink does not have an official release that supports the arm > architecture. There are some efforts and discussion [1][2][3] about > supporting the architecture. I think y

Re: Proper way to get DataStream

Hi, @Arvid Heise-4, @Matthias I'm very appreciate for your attention, guys. And sorry for my late reply. Yes, Arvid, you are right, the second way in fact works. I coppied schema from Schema Registry using it's API and created the .avsc format file. And thanks again for explaining me why the firs

Re: DataStream from kafka topic

Thank you all very much! The problem is solved using ConfluentRegistryAvroDeserializationSchema.forGeneric(schema, "http://xxx.xx.xxx.xx:8081";) method. http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Proper-way-to-get-DataStream-lt-GenericRecord-gt-td42640.html

Meaning of checkpointStartDelayNanos

Hi team, I'm a little confused by the meaning of *checkpointStartDelayNanos*, I do not understand what time it exactly means, but it seems it's a quite important indicator for checkpoint/backpresure. The explanation of it on metrics page

Re: Meaning of checkpointStartDelayNanos

I found its meaning in the code . It means the delay of checkpoint action when the checkpoint barrier comes to the cur

Re: Re: Meaning of checkpointStartDelayNanos

Hi Kai, Yes, you are basically right, one minor point is that the start time is taken as the time that the checkpoint get intiated in the JM side. Best, Yun --Original Mail -- Sender:Kai Fu Send Date:Mon Apr 5 09:31:58 2021 Recipients:user Subject:Re: Meanin

Re: Questions about checkpointAlignmentTime in unaligned checkpoint

Hi Kai, Under unaligned checkpoint settings, there are still alignment process. Although the task could snapshot the state of the operators on received the first barrier and emit barriers to the following tasks, it still need to wait till all the barriers to be received before finalize the check

Re: Union of more then two streams

Hi, With a.connect(b).coprocess(xx).connect(c).coprocess(xx), there would create two operators, the first operators would union a and b and output the enriched data, and then .connect(c).coprocess(xx) would pass-throught the already enriched data and enrich the record from c. Since the two oper

Re: Questions about checkpointAlignmentTime in unaligned checkpoint

Hi Yun, Thank you for the explanation, it clarifies a lot. *-- Best wishes* *Kai* On Mon, Apr 5, 2021 at 12:13 PM Yun Gao wrote: > Hi Kai, > > Under unaligned checkpoint settings, there are still alignment process. > Although > the task could snapshot the state of the operators on received the

Re: Re: Meaning of checkpointStartDelayNanos

Thank you for the clarification Yun, it helps. *-- Best wishes* *Kai* On Mon, Apr 5, 2021 at 12:03 PM Yun Gao wrote: > Hi Kai, > > Yes, you are basically right, one minor point is that the start time is > taken as the time that the checkpoint get intiated in the JM side. > > Best, > Yun > > >

Re: [External] : Union of more then two streams

Hello BB, Just want to share you some of my immature ideas. Maybe some experts can give you better solutions and advice. 1. DataStream based solution: * To do a union, as you already know, you must have the datastream to be of the same format. Otherwise, you can’t do it. There is a wo

Re: [External] : Re: Need help with executing Flink CLI for native Kubernetes deployment

Hello Yang, I am just following up the previous email to see if you got some time to reply. I also took a deeper look into lyft k8s operator recently. It seems it doesn’t support HA natively. It still needs the help of ZooKeeper. In terms of this, native k8s is better. Any other ideas? Thanks fo