If you use the Java API you can do SimpleConsumer.getOffsetsBefore(topic, 
partition, time, 1) which will return a long offset value

The time parameter can be kafka.api.OffsetRequest.EarliestTime() or 
kafka.api.OffsetRequest.LatestTime() based on your application need.


On May 24, 2013, at 2:03 PM, Sining Ma <sinin...@aol.com> wrote:

> Thanks Suyog
> Could you explain more about OffsetRequest?
> I find new
> OffsetRequest(topic: String, partition: Int, time: Long, maxNumOffsets: Int)
> in Kafka api.
> 
> 
> How can I send this request? And where can I receive a response from this 
> OffsetRequest?
> Could you give me an example for this API?
> 
> 
> 
> -- Regards    
>   Sining Ma
> 
> 
> 
> 
> 
> -----Original Message-----
> From: Suyog Rao <su...@loggly.com>
> To: users <users@kafka.apache.org>
> Sent: Fri, May 24, 2013 1:32 pm
> Subject: Re: OffsetOutOfRangeException and getOffsetsBefore
> 
> 
> Since you are using the simple consumer you will need to handle the 
> OffsetOutOfRange Exception in your code. This happens when your consumer 
> queries 
> for an offset which is no longer persisted in Kafka (The logs have been 
> deleted 
> based on the retention policy). Ideally when this happens, the consumer 
> should 
> issue an OffsetRequest to get the latest/earliest offset currently available 
> on 
> the server. You can then use this in your FetchRequest. Whether you use the 
> latest or earliest depends on your application needs. In essence, you are 
> resetting the consumer to catch up with the server.
> 
> On May 24, 2013, at 11:29 AM, Sining Ma <sinin...@aol.com> wrote:
> 
>> Hi,
>> We are currently using kafka-0.7.1 right now.
>> I have two questions:
>> 1. We use SimpleConsumer to aggregate messages to log files and there is no 
> zookeeper. Sometimes we can see kafka.common.OffsetOutOfRangeException. 
>> And this exception happens when we start our consumer program. We do not 
>> know 
> the reason why this happens.
>> How can I get a valid latest message offset in kafka-0.7.1 when this 
>> exception 
> happens?
>> 2. Before we start consumer, we call getOffsetsBefore function to get a list 
> of valid offsets (up to maxSize) before the given time.
>> How can we interpret this list? 
>> For example, this function returns an array [offset1, offset2].
>> Does this mean from offset1 to offset2 are valid, and offset2 to current 
> offset are valid? We are confused about the meaning of this array.
>> 
>> 
>> 
>> -- Regards   
>>  Sining Ma
>> 
>> 
> 
> 
> 

Reply via email to