If you use the Java API you can do SimpleConsumer.getOffsetsBefore(topic, partition, time, 1) which will return a long offset value
The time parameter can be kafka.api.OffsetRequest.EarliestTime() or kafka.api.OffsetRequest.LatestTime() based on your application need. On May 24, 2013, at 2:03 PM, Sining Ma <sinin...@aol.com> wrote: > Thanks Suyog > Could you explain more about OffsetRequest? > I find new > OffsetRequest(topic: String, partition: Int, time: Long, maxNumOffsets: Int) > in Kafka api. > > > How can I send this request? And where can I receive a response from this > OffsetRequest? > Could you give me an example for this API? > > > > -- Regards > Sining Ma > > > > > > -----Original Message----- > From: Suyog Rao <su...@loggly.com> > To: users <users@kafka.apache.org> > Sent: Fri, May 24, 2013 1:32 pm > Subject: Re: OffsetOutOfRangeException and getOffsetsBefore > > > Since you are using the simple consumer you will need to handle the > OffsetOutOfRange Exception in your code. This happens when your consumer > queries > for an offset which is no longer persisted in Kafka (The logs have been > deleted > based on the retention policy). Ideally when this happens, the consumer > should > issue an OffsetRequest to get the latest/earliest offset currently available > on > the server. You can then use this in your FetchRequest. Whether you use the > latest or earliest depends on your application needs. In essence, you are > resetting the consumer to catch up with the server. > > On May 24, 2013, at 11:29 AM, Sining Ma <sinin...@aol.com> wrote: > >> Hi, >> We are currently using kafka-0.7.1 right now. >> I have two questions: >> 1. We use SimpleConsumer to aggregate messages to log files and there is no > zookeeper. Sometimes we can see kafka.common.OffsetOutOfRangeException. >> And this exception happens when we start our consumer program. We do not >> know > the reason why this happens. >> How can I get a valid latest message offset in kafka-0.7.1 when this >> exception > happens? >> 2. Before we start consumer, we call getOffsetsBefore function to get a list > of valid offsets (up to maxSize) before the given time. >> How can we interpret this list? >> For example, this function returns an array [offset1, offset2]. >> Does this mean from offset1 to offset2 are valid, and offset2 to current > offset are valid? We are confused about the meaning of this array. >> >> >> >> -- Regards >> Sining Ma >> >> > > >
