Re: [ANNOUNCE] Apache Flink 1.10.3 released

2021-01-31 Thread Zhu Zhu
Thanks Xintong for being the release manager and everyone who helped with the release! Cheers, Zhu Dian Fu 于2021年1月29日周五 下午5:56写道: > Thanks Xintong for driving this release! > > Regards, > Dian > > 在 2021年1月29日,下午5:24,Till Rohrmann 写道: > > Thanks Xintong for being our release manager. Well don

Re: Configuring ephemeral storage limits when using Native Kubernetes

2021-01-31 Thread Yang Wang
Thanks for testing the pod template. I really hope to get more feedbacks from your use case. Best, Yang Emilien Kenler 于2021年2月1日周一 上午9:45写道: > Hello, > > I think this would solve our problem. > We are also looking at supporting affinity rules, and it would also cover > it. > > I'm going to try

Re: Flink 1.11 job hit error "Job leader lost leadership" or "ResourceManager leader changed to new address null"

2021-01-31 Thread Xintong Song
Hi Colletta, This error is kind of expected if the JobMaster / ResourceManager does not maintain a stable connection to the ZooKeeper service, which may be caused by network issues, GC pause, or unstable ZK service availability. By "similar issue", what I meant is I'm not aware of any issue relat

Re: Flink running in k8s pod - pod is able to access S3 bucket, flink does not

2021-01-31 Thread OranShuster
K8s is self managed on ec2 nodes After submitting the job and getting an exception I checked: 1.ssh into the machine and verify using the cli the pod has access. 2. In the job main method I instantiate a s3 client from the sdk (once with default credential chain and once with access key and secret)

Re: Configuring ephemeral storage limits when using Native Kubernetes

2021-01-31 Thread Emilien Kenler
Hello, I think this would solve our problem. We are also looking at supporting affinity rules, and it would also cover it. I'm going to try to find some time this week to try your patch. Thanks From: Yang Wang Sent: Friday, January 29, 2021 6:20 PM To: Emilien K

Re: What is "Processed (persisted) in-flight data"

2021-01-31 Thread Rex Fenley
Oh, nvm, that's the "Persisted" part which is documented as "*Persisted in-flight data*: The number of bytes persisted during the alignment (time between receiving the first and the last checkpoint barrier) over all acknowledged subtasks. This is > 0 only if the unaligned checkpoints are enabled."

Re: What is "Processed (persisted) in-flight data"

2021-01-31 Thread Rex Fenley
Got it, thanks! What is the 0 B part of that? On Sun, Jan 31, 2021 at 3:43 AM Arvid Heise wrote: > Processed in-flight data is the size of data that is processed between the > first and last checkpoint barrier in aligned checkpointing. [1] > > [1] > https://ci.apache.org/projects/flink/flink-doc

Re: Flink running in k8s pod - pod is able to access S3 bucket, flink does not

2021-01-31 Thread Itamar Syn-Hershko
Hi Oran, How is that k8s deployed? Are you sure all nodes have the same IAM role? can you try and see if this is fixed by granting permissions to that bucket to the IAM role in use? On Sun, Jan 31, 2021 at 5:15 PM OranShuster wrote: > I made some more tests and the issue is still not resolved

Re: Flink running in k8s pod - pod is able to access S3 bucket, flink does not

2021-01-31 Thread OranShuster
I made some more tests and the issue is still not resolved Since the submitted job main method is executed before the execution graph is submitted i added the aws sdk as an dependency and used it to upload files to the bucket in the main method Once with the default credentials provider, this wor

Re: What is "Processed (persisted) in-flight data"

2021-01-31 Thread Arvid Heise
Processed in-flight data is the size of data that is processed between the first and last checkpoint barrier in aligned checkpointing. [1] [1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/monitoring/checkpoint_monitoring.html#history-tab On Sun, Jan 31, 2021 at 7:45 AM Rex Fenley w