Re: adding core-site xml to flink1.11

2020-10-28 Thread Shachar Carmeli
" and a "per cluster" > configuration. > > I filed a JIRA ticket for a feature request: > https://issues.apache.org/jira/browse/FLINK-19828 > > > On Tue, Oct 27, 2020 at 10:54 AM Shachar Carmeli > wrote: > > > Hi, > > Thank you for your reply, > >

Re: adding core-site xml to flink1.11

2020-10-27 Thread Shachar Carmeli
are intentionally ignoring xml > configurations from the classpath. > You can configure the filesystem in the flink-conf.yaml file. > > > On Sun, Oct 25, 2020 at 7:56 AM Shachar Carmeli > wrote: > > > Hi, > > I'm trying to define filesystem to flink

adding core-site xml to flink1.11

2020-10-24 Thread Shachar Carmeli
Hi, I'm trying to define filesystem to flink 1.11 using core-site.xml I tried adding in the flink-conf.yaml env.hadoop.conf.dir and I see it is added to the classpath also adding environment variable HADOOP_CONF_DIR didn't help The flink 1.11.2 is running on docker using kubernetes I added hado

Re: Flink incremental checkpointing - how long does data is kept in the share folder

2020-04-21 Thread Shachar Carmeli
ile but are related to the checkpoint? How to > judge that they are related to specific checkpoint? > > BTW, my name is "Yun" which means cloud in Chinese, not the delicious "Yum" 🙂 > > Best > Yun Tang > > From: Shachar Car

Re: Flink incremental checkpointing - how long does data is kept in the share folder

2020-04-20 Thread Shachar Carmeli
[1] > https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/checkpoints.html#directory-structure > > Best > Yun Tang > > > From: Shachar Carmeli > Sent: Sunday, April 12, 2020 15:32 > To: user@flink.apache.org > Subject:

Re: Flink incremental checkpointing - how long does data is kept in the share folder

2020-04-12 Thread Shachar Carmeli
ed. You could follow this to debug > or maybe I could write a tool to help know what files could be deleted later. > > [1] > https://github.com/apache/flink/blob/693cb6adc42d75d1db720b45013430a4c6817d4a/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/Checkpoints.j

Flink incremental checkpointing - how long does data is kept in the share folder

2020-04-07 Thread Shachar Carmeli
We are using Flink 1.6.3 and keeping the checkpoint in CEPH ,retaining only one checkpoint at a time , using incremental and using rocksdb. We run windows with lateness of 3 days , which means that we expect that no data in the checkpoint share folder will be kept after 3-4 days ,Still We see t

Flink 1.5 job distribution over cluster nodes

2018-07-18 Thread Shachar Carmeli
Hi, We have 4 jobs with parallelism 3 that are running over 3 task manager with 4 slots per each , each task manager runs on a different VM , On Flink 1.3.2 the jobs were evenly distributed per node each job took one task slot of each task manager . After upgrading to flink 1.5 , each job is r