Hello, I have a streaming application that has an operator based on the KeyedCoProcessFunction. The operator has a MapState object. I store some data in this operator with a fixed ttl. I would like to monitor the size/count of this state over time since its related to some operational metrics we want to track. Seems like a simple thing to do; but I havent come up with a way to do so
Given that iterating over the complete map is an expensive operation, I only plan to do so periodically. The first issue is that , the stream is keyed, so any time i do a count of the mapstate, i dont get the complete size of the state object, but only count pertaining to the specific key of partition. Is there a way around this ? Secondly, is there a way to monitor rocksdb usage over time. I can find managed memory metrics. but this does not include disk space rocksdb used. is there a way to get this from standard flink metrics; either task manager or job manager ?
