Re: Question about session_aggregate.merging-window-set.rocksdb_estimate-num-keys

2021-03-16 Thread Vishal Santoshi
4) low number of samples > > [1] > https://github.com/ververica/frocksdb/blob/49bc897d5d768026f1eb816d960c1f2383396ef4/db/version_set.cc#L919-L924 > > > > Best > Yun Tang > ------ > *From:* Vishal Santoshi > *Sent:* Monday, March 15, 2021 5:48

Re: Question about session_aggregate.merging-window-set.rocksdb_estimate-num-keys

2021-03-15 Thread Yun Tang
[1] https://github.com/ververica/frocksdb/blob/49bc897d5d768026f1eb816d960c1f2383396ef4/db/version_set.cc#L919-L924 Best Yun Tang From: Vishal Santoshi Sent: Monday, March 15, 2021 5:48 To: user Subject: Re: Question about session_aggregate.merging-window-set.r

Re: Question about session_aggregate.merging-window-set.rocksdb_estimate-num-keys

2021-03-14 Thread Vishal Santoshi
All I can think is, that any update on a state key, which I do in my ProcessFunction, creates an update ( essentially an append on rocksdb ) which does render the previous value for the key, a tombstone , but that need not reflect on the count ( as double or triple counts ) atomically, thus the c

Re: Question about session_aggregate.merging-window-set.rocksdb_estimate-num-keys

2021-03-14 Thread Vishal Santoshi
The reason I ask is that I have a "Process Window Function" on that Session Window and I keep key scoped Global State. I maintain a TTL on that state ( that is outside the Window state ) that is roughly the current WM + lateness. I would imagine that keys for that custom state are *roughly* eq

Question about session_aggregate.merging-window-set.rocksdb_estimate-num-keys

2021-03-14 Thread Vishal Santoshi
Hey folks, Was looking at this very specific metric "session_aggregate.merging-window-set.rocksdb_estimate-num-keys". Does this metric also represent session windows ( it is a session window ) that have lateness on them ? In essence if the session window was closed but has a lateness of a f