Hi I am looking for some advice on how to solve the following problem I'd like to keep track of the all time last n events received for a user. An event on average takes up 500 bytes and here will be ten's of millions of users for which we need to keep this information. The list of events will be stored in cassandra for serving by an api
One way I can think of to implement this , is to use a global window per user with a count evictor. The problem I see with this is that the state would forever remain on the worker nodes, in our case, in rocks db. This means there would be a *lot* of state to include for savepoints and checkpoints. This would make such a job very unwieldy to operate. Is it possible to evict state from global state after some period of inactivity. and then reinitalize the global state with data loaded from cassandra when a new event arrives ? Or is there an obvious better way to tackle this problem that i am missing Any pointers would be greatly appreciated
