Yep, that should also work!
-John
On Mon, Sep 17, 2018 at 8:36 AM David Espinosa wrote:
> Thank you all for your responses!
> I also asked this on the confluent slack channel (
> https://confluentcommunity.slack.com) and I got this approach:
>
>1. Query the partitions' high watermark offset
Thank you all for your responses!
I also asked this on the confluent slack channel (
https://confluentcommunity.slack.com) and I got this approach:
1. Query the partitions' high watermark offset
2. Set the consumer to consume from beginning
3. Break out when you've reached the high offset
Specifically, you can monitor the "records-lag-max" (
https://docs.confluent.io/current/kafka/monitoring.html#fetch-metrics)
metric. (or the more granular one per partition).
Once this metric goes to 0, you know that you've caught up with the tail of
the log.
Hope this helps,
-John
On Fri, Sep 1
Using Kafka Streams this is a little tricky.
The API itself has no built-in mechanism to do this. You would need to
monitor the lag of the application, and if the lag is zero (assuming you
don't write new data into the topic in parallel), terminate the application.
-Matthias
On 9/14/18 4:19 AM,
Spin up a consumer, subscribe to EOF events, assign all partitions from the
beginning, and keep polling until all partitions has reached EOF.
Though, if you have concurrent writers, new messages may be appended after you
observe EOF on a partition, so you are never guaranteed to have read all
me
