Hey Navneeth, So to understand your problem better - do you only want to stream users active within 10 minutes to storage?
Cheers, Liam On Tue, Feb 16, 2021 at 9:50 AM Navneeth Krishnan <reachnavnee...@gmail.com> wrote: > It’s just for emitting to data storage. There is no join here. > > Thanks > > On Mon, Feb 15, 2021 at 1:42 AM Liam Clarke-Hutchinson < > liam.cla...@adscale.co.nz> wrote: > > > Hi Navneeth, > > > > What is the purpose of holding these user records? Is it to join against > > other streams, or emit to data storage? > > > > Cheers, > > > > Liam Clarke-Hutchinson > > > > > > > > On Mon, 15 Feb. 2021, 9:08 pm Navneeth Krishnan, < > reachnavnee...@gmail.com > > > > > wrote: > > > > > Hi All, > > > > > > I have a question about how I can use window stores to achieve this use > > > case. Thanks for all the help. > > > > > > A user record will be created when the user first logins and the > records > > > needs to be cleaned up after 10 mins of inactivity. Thus for each user > > > there will be a TTL but the TTL value will be updated each time when > the > > > user is active before he becomes inactive for the entire 10 min period. > > We > > > are currently using PAPI for all our topologies and I was thinking of > > > implementing it using a punctuator. > > > > > > My initial logic was to have a KV store with each user as key and TTL > as > > > the value and run a scheduled task every minute that looks at all the > > > records which have TTL value lesser than the timestamp. But the problem > > in > > > this approach was performance. When there are more than 1M records it > > takes > > > more than a few seconds to complete this task. > > > > > > Next approach is to have a window store and a KV store. Window store > will > > > have each user and corresponding TTL rounded to the nearest minute. > Then > > > find all keys between the current time and current time - 1min. Then > > > iterate these keys and use the KV store to find if the TTL value is > still > > > the same or if we have received any updates after that. If not then the > > > user will be evicted. > > > > > > What would be a better and much more scalable solution for this. > > > > > > Thanks > > > > > >
