Datetime format

2022-03-27 Thread lan tran
Hi team, So basically, when I use Flink Table API to generate the files and store in S3. The format files will be like this part-0d373ee1-d594-40b1-a3cf-8fa895260980-0-0. So my question is is there any way that we can config this files names (by adding the last_modified_value) to this files name ?B

Date time convert

2022-03-27 Thread lan tran
Hi team,Currently, I was facing this situation, I have the format string datetime like this "2018-03-29T07:39:49.722594Z". So how can I convert this into timestamp with local time zone ?My current solution as cast as belowTO_TIMESTAMP(REPLACE(parcel.picked_up_date, 'T', ' '),'-MM-dd HH:mm:ss%z'

Where is the "Partitioned All Cache" doc?

2022-03-27 Thread dz902
Hi, I've read some docs (https://help.aliyun.com/document_detail/182011.html) stating Flink optimization technique using: - partitionedJoin = 'true' - cache = 'ALL' - blink.partialAgg.enabled=true However I could not find any official doc references. Are these supported at all? Also "partitione

Re: JobManager failed to renew it's leadership (K8S HA)

2022-03-27 Thread Yang Wang
Could you please verify whether the JobManager is going through a long full GC or the Kubernetes APIServer is working well at that moment? We are using Kubernetes HA service in the production and it seems stable without your issue. Best, Yang marco andreas 于2022年3月27日周日 18:35写道: > > Hello, >

Re: Deploy a Flink session cluster natively on K8s with multi AZ

2022-03-27 Thread Yang Wang
> In the example, we can pass args in the command, is there a way to do it by using the flink-conf.yaml? Yes. All the changes in the $FLINK_HOME/conf/flink-conf.yaml at the client side will also be picked up when deploying a native K8s cluster. For your use case, I am also suggesting the flink-ku

JobManager failed to renew it's leadership (K8S HA)

2022-03-27 Thread marco andreas
Hello, Does anyone have the same issue or have an idea why the jobmanager fails to renew its leadership when using kubernetes ha service. Configuration : kubernetes.namespace: flink-ps-flink-dev high-availability.kubernetes.leader-election.lease-duration: 200 s high-availability.kubernetes.leader

Re: Deploy a Flink session cluster natively on K8s with multi AZ

2022-03-27 Thread Gyula Fóra
Hi! I think the Flink Kubernetes Operator ( https://github.com/apache/flink-kubernetes-operator) project is exactly what you are looking for. This is a relatively new addition to Flink that supports k8s application and session deployments with lifecycle management through kubernetes native toolin