AW: number of files in checkpoint directory grows endlessly

; step...@data-artisans.com Betreff: Re: number of files in checkpoint directory grows endlessly Hi Bernd, Does this directory contains approximately 300 files on task executor? The 'transactionState' seems to be ok and compacted. It takes most of data size, up to 100Mb. Do you use any Fl

Re: number of files in checkpoint directory grows endlessly

>> >> rocksdb.merge.operatorMerge By >> TTLrocksdb.num.data.blocksrocksdb.num.entriesrocksdb.prefix.extractor.namenullptrocksdb.property.collectors[]rocksdb.raw.key.sizeArocksdb.r

Re: number of files in checkpoint directory grows endlessly

ndet: Donnerstag, 6. Dezember 2018 12:07 > An: Winterstein, Bernd > Cc: Yun Tang; Kostas Kloudas; user; Stefan Richter; Till Rohrmann; Stephan > Ewen > Betreff: Re: number of files in checkpoint directory grows endlessly > > Hi Bernd, > > Thanks for sharing the code. >

AW: number of files in checkpoint directory grows endlessly

Betreff: Re: number of files in checkpoint directory grows endlessly Hi Bernd, Thanks for sharing the code. I understand your TTL requirement. It definitely makes sense for your application. My recommendation is still to try running job with original backend without TtlDb modification to narrow

Re: number of files in checkpoint directory grows endlessly

2018 15:25 An: Winterstein, Bernd Cc: Yun Tang; Kostas Kloudas; user; Stefan Richter; Till Rohrmann; step...@data-artisans.com<mailto:step...@data-artisans.com> Betreff: Re: number of files in checkpoint directory grows endlessly Hi Bernd, Thanks for sharing the code. Could you try the

Re: number of files in checkpoint directory grows endlessly

ndet: Mittwoch, 5. Dezember 2018 15:25 > An: Winterstein, Bernd > Cc: Yun Tang; Kostas Kloudas; user; Stefan Richter; Till Rohrmann; > step...@data-artisans.com > Betreff: Re: number of files in checkpoint directory grows endlessly > > Hi Bernd, > > Thanks for sharing the code

Re: number of files in checkpoint directory grows endlessly

> Gesendet: Dienstag, 4. Dezember 2018 13:55 > An: Winterstein, Bernd; and...@data-artisans.com > Cc: k.klou...@data-artisans.com; user@flink.apache.org; > s.rich...@data-artisans.com; t...@data-artisans.com; step...@data-artisans.com > Betreff: Re: number of files in checkpoint dir

AW: number of files in checkpoint directory grows endlessly

ns.com; user@flink.apache.org; s.rich...@data-artisans.com; t...@data-artisans.com; step...@data-artisans.com Betreff: Re: number of files in checkpoint directory grows endlessly Hi Bernd, RocksDBStateBackend would not use default column family, but put state's entries into its newly-creat

Re: number of files in checkpoint directory grows endlessly

om; step...@data-artisans.com Subject: AW: number of files in checkpoint directory grows endlessly Sorry for the late answer. I haven’t been in the office. The logs show no problems. The files that remain in the shared subfolder are almost all 1121 bytes. Except the files from the latest checkpoint (

AW: number of files in checkpoint directory grows endlessly

().setUseSnapshotCompression(true); } Von: Andrey Zagrebin [mailto:and...@data-artisans.com] Gesendet: Donnerstag, 29. November 2018 15:38 An: Winterstein, Bernd Cc: Kostas Kloudas; user; Stefan Richter; Till Rohrmann; Stephan Ewen Betreff: Re: number of files in checkpoint directory grows endlessly Could

Re: number of files in checkpoint directory grows endlessly

gt; Cc: Kostas Kloudas; user; s.rich...@data-artisans.com >> <mailto:s.rich...@data-artisans.com>; t...@data-artisans.com >> <mailto:t...@data-artisans.com>; step...@data-artisans.com >> <mailto:step...@data-artisans.com> >> Betreff: Re: number of files in che

Re: number of files in checkpoint directory grows endlessly

quot; directory next to the chk-xxx directory. > > > -Ursprüngliche Nachricht- > Von: Andrey Zagrebin [mailto:and...@data-artisans.com] > Gesendet: Donnerstag, 29. November 2018 10:39 > An: Kostas Kloudas > Cc: Winterstein, Bernd; user; Stefan Richter; Till Rohrmann; S

AW: number of files in checkpoint directory grows endlessly

: Re: number of files in checkpoint directory grows endlessly Hi Bernd, Did you change 'state.checkpoints.num-retained’ in flink-conf.yaml? By default, only one checkpoint should be retained. Which version of Flink do you use? Can you check Job Master logs whether you see there warning

Re: number of files in checkpoint directory grows endlessly

Hi Bernd, Did you change 'state.checkpoints.num-retained’ in flink-conf.yaml? By default, only one checkpoint should be retained. Which version of Flink do you use? Can you check Job Master logs whether you see there warning like this: `Fail to subsume the old checkpoint`? Best, Andrey > On 29

Re: number of files in checkpoint directory grows endlessly

Hi Bernd, I think the Till, Stefan or Stephan (cc'ed) are the best to answer your question. Cheers, Kostas

number of files in checkpoint directory grows endlessly

I have a flink job running with the following settings: * CheckpointingMode.EXACTLY_ONCE * RocksDB backend (Modified with TtlDB usage) * CheckpointConfig.ExternalizedCheckpointCleanup.DELETE_ON_CANCELLATION * 60 sec interval * Asnyc snapshots * Incremental checkp