Hi Ananth,
> if it detects significant backlog, skip over and consume from the latest
offset, and schedule a separate backfill task for the backlogged 20 minutes?
Fabian is right, there is no built-in operators for this.
If you don't care about Watermark, I think we can implement it with a
custom
Hi Ananth,
You can certainly do this with Flink, but there are no built-in operators
for this.
What you probably want to do is to compare the timestamp of the event with
the current processing time and drop the record if it is too old.
If the timestamp is encoded in the record, you can do this in
I'm writing a Flink connector to write a stream of events from Kafka to
Elastic Search. It is a typical metrics ingestion pipeline, where the
latest metrics preferred over the stale data.
What I mean by that, let's assume there was an outage of Elastic Search
cluster for about 20 minutes, all the m
