Fwd: RocksDB state on HDFS seems not being cleanned up

2019-11-17 Thread shuwen zhou
Forward to user group again since mail server was rejecting for last time -- Forwarded message - From: shuwen zhou Date: Wed, 13 Nov 2019 at 13:33 Subject: Re: RocksDB state on HDFS seems not being cleanned up To: Yun Tang Cc: user Hi Yun, After my investigation, I found out t

Re: Keyed raw state - example

2019-11-17 Thread Congxian Qiu
Hi Currently, I think you can ref the implementation of timerservice[1] which used the raw keyed state, the snapshot happens in AbstractStreamOperator#snapshotState(), for using Raw State you need to implement a new operator[2]. There is an issue wants to give some example for raw state[2] [1]

Re: Apache Airflow - Question about checkpointing and re-run a job

2019-11-17 Thread Congxian Qiu
Hi Yes, checkpoint data locates under jobid dir. you can try to restore from the retained checkpoint[1] [1] https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/state/checkpoints.html#resuming-from-a-retained-checkpoint Best, Congxian M Singh 于2019年11月18日周一 上午2:54写道: > Folks - Please

Weekly Community Update 2019/46

2019-11-17 Thread Konstantin Knauf
Dear community, happy to share this week's community update. While slowly approaching the planned feature freeze for Flink 1.10, things have calmed down a bit on the dev mailing list. Stil there are couple of interesting topics to cover. This week's digest includes an update on the Flink 1.8.3 rel

Re: Apache Airflow - Question about checkpointing and re-run a job

2019-11-17 Thread M Singh
Folks - Please let me know if you have any advice on this question.  Thanks On Saturday, November 16, 2019, 02:39:18 PM EST, M Singh wrote: Hi: I have a Flink job and sometimes I need to cancel and re run it.  From what I understand the checkpoints for a job are saved under the job id

Hdfs -> hdfs ends up with .progress files

2019-11-17 Thread Johan Rask
Hi! I am quite new to Apache Flink but we have been evaluating it for some weeks to read from kafka and transform and write to hdfs. With kafka to hdfs with exactly-once configured this works as expected but when we replace the source with kafka to hdfs files hangs in .in-progress. We first exper