Here is what the documentation on RocksDBStateBackend <https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/ops/state/state_backends/#the-embeddedrocksdbstatebackend> says:
The EmbeddedRocksDBStateBackend holds in-flight data in a RocksDB database that is (per default) stored in the TaskManager local data directories. Unlike storing java objects in HashMapStateBackend, data is stored as serialized byte arrays, which are mainly defined by the type serializer, *resulting in key comparisons being byte-wise instead of using Java’s hashCode() and equals() methods.* This means that if your keys are not byte-wise equivalent, they won't be matched. On Wed, Jun 30, 2021 at 7:37 PM Thomas Breloff <t...@ec.ai> wrote: > Hello, > > I am having trouble with a Flink job which is configured using a RocksDB > state backend. > > > > Tl;dr: How can I debug the key serialization for RocksDB MapState for a > deployed Flink job? > > > > Details: > > > > When I “put” a key/value pair into a MapState, and then later try to “get” > using a key which has the same hashCode/equals as what I put in, I get back > “null”. > > > > Some things I have verified: > > > > - Looping over the “keys()” or “entries()” of the MapState contains > the expected key (which matches both hashCode and equals) > - If I “put” the same key that I’m attempting to “get” with, and then > look at the “entries”, then both of the keys appear in the map. > > > > I think I understand that the RocksDB version of MapState will use the > serialized keys, however I have tested what I think is the serializer and > it returns the same serialization for both objects. > > > > How can I find out the serialized values that are being used for key > comparison? Can you recommend any possible solutions or debugging > strategies that would help? > > > > Thank you, > > Tom > -- Best Regards, Yuval Itzchakov.
