Hi Liam, The use case is stream all data and send it to storage after processing. Also when the user is inactive for a 10 min period then send a special event that marks the user as inactive. I'm trying to implement the special event here.
Thanks On Tue, Feb 16, 2021 at 1:18 AM Liam Clarke-Hutchinson < liam.cla...@adscale.co.nz> wrote: > Hey Navneeth, > > So to understand your problem better - do you only want to stream users > active within 10 minutes to storage? > > Cheers, > > Liam > > On Tue, Feb 16, 2021 at 9:50 AM Navneeth Krishnan < > reachnavnee...@gmail.com> > wrote: > > > It’s just for emitting to data storage. There is no join here. > > > > Thanks > > > > On Mon, Feb 15, 2021 at 1:42 AM Liam Clarke-Hutchinson < > > liam.cla...@adscale.co.nz> wrote: > > > > > Hi Navneeth, > > > > > > What is the purpose of holding these user records? Is it to join > against > > > other streams, or emit to data storage? > > > > > > Cheers, > > > > > > Liam Clarke-Hutchinson > > > > > > > > > > > > On Mon, 15 Feb. 2021, 9:08 pm Navneeth Krishnan, < > > reachnavnee...@gmail.com > > > > > > > wrote: > > > > > > > Hi All, > > > > > > > > I have a question about how I can use window stores to achieve this > use > > > > case. Thanks for all the help. > > > > > > > > A user record will be created when the user first logins and the > > records > > > > needs to be cleaned up after 10 mins of inactivity. Thus for each > user > > > > there will be a TTL but the TTL value will be updated each time when > > the > > > > user is active before he becomes inactive for the entire 10 min > period. > > > We > > > > are currently using PAPI for all our topologies and I was thinking of > > > > implementing it using a punctuator. > > > > > > > > My initial logic was to have a KV store with each user as key and TTL > > as > > > > the value and run a scheduled task every minute that looks at all the > > > > records which have TTL value lesser than the timestamp. But the > problem > > > in > > > > this approach was performance. When there are more than 1M records it > > > takes > > > > more than a few seconds to complete this task. > > > > > > > > Next approach is to have a window store and a KV store. Window store > > will > > > > have each user and corresponding TTL rounded to the nearest minute. > > Then > > > > find all keys between the current time and current time - 1min. Then > > > > iterate these keys and use the KV store to find if the TTL value is > > still > > > > the same or if we have received any updates after that. If not then > the > > > > user will be evicted. > > > > > > > > What would be a better and much more scalable solution for this. > > > > > > > > Thanks > > > > > > > > > >
