The offset API is one of the known gaps in the new consumer. There is a JIRA (KAFKA-1332), but we might need a KIP to make that change now that the API is released. For now, Gwen's suggestion is the only way to do it.
-Jason On Thu, Jan 21, 2016 at 8:22 PM, Gwen Shapira <g...@confluent.io> wrote: > Hi Robert! > > Jason is the expert, and I hope he'll respond soon. > > Meanwhile: I think that you can do what you are trying to do by: > 1. call position() to get the current position you are consuming > 2. call seekToEnd() and then position(), which will give you the last > position at the point in which you called seekToEnd() > 3. Use another seek() call to go to the offset you really want to get to. > > I don't think there are current plans to add getOffsetsBefore, but maybe we > need it for the use-case you specified. > I think the developer mailing list (or a JIRA) will be a better place for > an API discussion. > > Gwen > > On Wed, Jan 20, 2016 at 3:09 AM, Robert Metzger <rmetz...@apache.org> > wrote: > > > Hi, > > > > I'm currently looking into implementing a load shedding strategy into > > Flink's Kafka consumer. > > > > Therefore, I would like to allow users to request the latest offset of > the > > subscribed TopicPartitions, so that they can > > a) determine the lag > > b) maybe set the next fetch offset to the latest offset (or anything in > > between) > > > > With the SimpleConsumer, there is the getOffsetsBefore() method, which > > allows to request the largest offset. > > For the new KafkaConsumer, I only found the seekToEnd() and > > seekToBeginning() methods, but they don't allow to get the actual end or > > beginning. > > > > > > Are there plans to add such an API to the KafkaConsumer again? > > Can you suggest any workarounds? > > > > Regards, > > Robert > > >
