I think so, I just wanted to bring it up again because the question was raised.
> On 8. Apr 2019, at 22:56, Elias Levy wrote:
>
> Hasn't this been always the end goal? It's certainly what we have been
> waiting on for job with very large TTLed state. Beyond timer storage,
> timer processing to
Hasn't this been always the end goal? It's certainly what we have been
waiting on for job with very large TTLed state. Beyond timer storage,
timer processing to simply expire stale data that may not be accessed
otherwise is expensive.
On Mon, Apr 8, 2019 at 7:11 AM Aljoscha Krettek wrote:
> I
I had a discussion with Andrey and now think that also the case
event-time-timestamp/watermark-cleanup is a valid case. If you don’t need this
for regulatory compliance but just for cleaning up old state, in case where you
have re-processing of old data.
I think the discussion about whether to
Hi all,
For GDPR: I am not sure about the regulatory requirements of GDPR but I
would assume that the time for deletion starts counting from the time an
organisation received the data (i.e. the wall-clock ingestion time of the
data), and not the "event time" of the data. In other case, an organisa
Oh boy, this is an interesting pickle.
For *last-access-timestamp*, I think only *event-time-of-current-record* makes
sense. I’m looking at this from a GDPR/regulatory compliance perspective. If
you update a state, by say storing the event you just received in state, you
want to use the exact t
Hi Andrey,
I agree with Elias. This would be the most natural behavior. I wouldn't add
additional slightly different notions of time to Flink.
As I can also see a use case for the combination
* Timestamp stored: Event timestamp
* Timestamp to check expiration: Processing Time
we could (maybe in
My 2c:
Timestamp stored with the state value: Event timestamp
Timestamp used to check expiration: Last emitted watermark
That follows the event time processing model used elsewhere is Flink. E.g.
events are segregated into windows based on their event time, but the
windows do not fire until the
Hi All,
As you might have already seen there is an effort tracked in FLINK-12005
[1] to support event time scale for state with time-to-live (TTL) [2].
While thinking about design, we realised that there can be multiple options
for semantics of this feature, depending on use case. There is also
so
