Hi, We are using flink1.3.2 and trying to explore rocksdb state backend and checkpointing. Data source is Kafka and checkpointing enabled in Flink. We have few doubts regarding the same:
- What is the exact difference between checkpoint and state backend? - Is the data stored in rocksdb checkpoints incremental(it keeps all past data also in newer file)? New checkpoint is created after defined interval and does it contains the previous checkpoint's data? Our use case demands all the checkpoint data to be in a single db, but when we manually restart the job it's id changes and new directory gets created(new metadata file in case of savepoints). - What data does rocksdb stores inside in case of checkpoints? We are interested in knowing whether it stores actual aggregations or it stores the offsets metadata for an aggregation window? - If we run aggregations on past data, then will it take help of state backend to not run aggregations again and give results by querying the state backend, saving the processing time? -- *Ashwin Sinha *| Data Engineer ashwin.si...@go-mmt.com <shivam.sha...@go-mmt.com> | 9452075361 <https://www.makemytrip.com/> <https://www.goibibo.com/> <https://www.redbus.in/>2nd floor, Tower B Divyashree Technopolis Yemalur, Bangalore, Karnataka 560025, India <https://www.redbus.in/> -- ::DISCLAIMER:: ---------------------------------------------------------------------------------------------------------------------------------------------------- This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from your system.
