Hi Navneeth, What is the purpose of holding these user records? Is it to join against other streams, or emit to data storage?
Cheers, Liam Clarke-Hutchinson On Mon, 15 Feb. 2021, 9:08 pm Navneeth Krishnan, <reachnavnee...@gmail.com> wrote: > Hi All, > > I have a question about how I can use window stores to achieve this use > case. Thanks for all the help. > > A user record will be created when the user first logins and the records > needs to be cleaned up after 10 mins of inactivity. Thus for each user > there will be a TTL but the TTL value will be updated each time when the > user is active before he becomes inactive for the entire 10 min period. We > are currently using PAPI for all our topologies and I was thinking of > implementing it using a punctuator. > > My initial logic was to have a KV store with each user as key and TTL as > the value and run a scheduled task every minute that looks at all the > records which have TTL value lesser than the timestamp. But the problem in > this approach was performance. When there are more than 1M records it takes > more than a few seconds to complete this task. > > Next approach is to have a window store and a KV store. Window store will > have each user and corresponding TTL rounded to the nearest minute. Then > find all keys between the current time and current time - 1min. Then > iterate these keys and use the KV store to find if the TTL value is still > the same or if we have received any updates after that. If not then the > user will be evicted. > > What would be a better and much more scalable solution for this. > > Thanks >
