Hello, I have some questions regarding best practices to deal with ever expanding state with KeyBy(). In my input stream I will continue to see new keys. And I am using Keyed state. How do I keep the total state in limit? After reading the flink documentation and some blogs I am planning to use following :
* When I know I have seen the “last” element associated with a key, I can manually clear the state * I can also use the TTL on state and expire it and garbage collect it (with next full snapshot). This is useful when I never see the “last” element. Is that the right strategy? Also if an element arrives late (after the state has been cleared), how do I detect that the state has been cleared/expired so I can skip these late elements ? Is there an API that will give you the hint about cleared/expired state? Thanks. Ajay
