It’s just for emitting to data storage. There is no join here. Thanks
On Mon, Feb 15, 2021 at 1:42 AM Liam Clarke-Hutchinson < liam.cla...@adscale.co.nz> wrote: > Hi Navneeth, > > What is the purpose of holding these user records? Is it to join against > other streams, or emit to data storage? > > Cheers, > > Liam Clarke-Hutchinson > > > > On Mon, 15 Feb. 2021, 9:08 pm Navneeth Krishnan, <reachnavnee...@gmail.com > > > wrote: > > > Hi All, > > > > I have a question about how I can use window stores to achieve this use > > case. Thanks for all the help. > > > > A user record will be created when the user first logins and the records > > needs to be cleaned up after 10 mins of inactivity. Thus for each user > > there will be a TTL but the TTL value will be updated each time when the > > user is active before he becomes inactive for the entire 10 min period. > We > > are currently using PAPI for all our topologies and I was thinking of > > implementing it using a punctuator. > > > > My initial logic was to have a KV store with each user as key and TTL as > > the value and run a scheduled task every minute that looks at all the > > records which have TTL value lesser than the timestamp. But the problem > in > > this approach was performance. When there are more than 1M records it > takes > > more than a few seconds to complete this task. > > > > Next approach is to have a window store and a KV store. Window store will > > have each user and corresponding TTL rounded to the nearest minute. Then > > find all keys between the current time and current time - 1min. Then > > iterate these keys and use the KV store to find if the TTL value is still > > the same or if we have received any updates after that. If not then the > > user will be evicted. > > > > What would be a better and much more scalable solution for this. > > > > Thanks > > >
