Hi, I have enabled state.backend.changelog And state.backend.changelog.storage is set to filesystem
I wanted to know for how long the changelog files are retained as I was storing the data to HDFS and I was quickly running out of space. What should be the ideal size of disk storage I should allocate for the HDFS ? I am performing incremental checkpointing using rocksdb and using HDFS as the store. A checkpoint state can easily go to 50 GB. Thanks Sachin
