RE: [EXT] Re: Reusing savepoints from a streaming job in a batch job

Hi Tobi, State in batch mode works in a single-key-a-time fashion. That makes state to keep fast and small. However, although streaming and batch mode on streaming API are in general equivalent, batch mode covers only a subset of algorithm possible in streaming mode. Elaborate timers and waterma

Re: [ANNOUNCE] RocksDB Version Upgrade and Performance

Hi Yun, Thanks for your detailed description about the progress of Flink and RocksDB's community. There're more than 1,200 jobs using RocksDB as the state backend at Bytedance, and we do met several problems mentioned in the JIRA issues you referred: (1) Memory Management: for large-scale jobs(10

Inspecting SST state of rocksdb

Hi team, I'm trying to inspect SST files of flink's state with sst related tools like sst_dump, ldb in wiki . But it seems I'm getting meaningless results as shown below. The tools I'm using are from RocksDB's trunk and

Re: KafkaDeserializationSchema.open() is not called after task state change

Hi! This does not sound like an expected behavior. However there might be a lot of reasons causing some values to be uninitialized (for example, once I've met a bug that a thread is created and runs in the open method before some values are initialized). You can always add some log at the beginnin

Dynamic Cluster/Index Routing for Elasticsearch Sink

Hi folks, I have a use-case that I wanted to initially pose to the mailing list as I’m not terribly familiar with the Elasticsearch connector to ensure I’m not going down the wrong path trying to accomplish this in Flink (or if something downstream might be a better option). Basically, I have

KafkaDeserializationSchema.open() is not called after task state change

Hi, I have a class that implements KafkaDeserializationSchema and overrides the open() method. When deploying my job, everything seems to work as expected. But, when my task hit an exception and switch to CANCELED -> CREATED -> DEPLOYING, it seems that the open method is not called, and I am endin