Hm, alright. Haven't really used the method to anything besides getting first and last offset (using -1 and -2 as timestamps IIRC) of a topic+partition combination.
Maybe someone else can shed some light on this? Cheers, Mathias On 17 December 2012 19:51, Jason Huang <jason.hu...@icare.com> wrote: > Mathias, > > Thanks for response. I am not sure if this timestamp is the Unix time > or not. I've tried the following: > > Create 3 messages of the same topic, at the same partition like this: > 1355769714152: Jason has a new message 1 > 1355769964900: Jason has a new message 2 > 1355769980296: Jason has a new message 3 > > I then tried to call getOffsetsBefore with a timestamp = 1355769964999 > (99 milliseconds after the timestamp in message two above), hoping to > get some offset but the long array returned by the call is empty. > > Some google search found that getOffsetsBefore is based on the mime of > the log segments. In other words, if I only have one log file > 00000000000000000000.kafka in the topic directory (log/topic-0), then > the offset array returned by this call will always be 0? > > If so, this API is probably not designed for my use case. > > thanks, > > Jason > > > On Mon, Dec 17, 2012 at 1:40 PM, Mathias Söderberg > <mathias.soederb...@gmail.com> wrote: > > The SimpleConsumer API [1] has a method called getOffsetsBefore which > takes > > a topic, partition, timestamp (UNIX I assume since it's a long) and > integer > > limit on how many offsets to get. > > > > Might not solve your problem *exactly*, but could be useful, unless > you're > > using the ConsumerConnector? > > > > [1]: http://people.apache.org/~joestein/kafka-0.7.1-incubating-docs/ > > > > > > On 17 December 2012 19:23, Jason Huang <jason.hu...@icare.com> wrote: > > > >> Hello, > >> > >> Is it possible to fetch messages from the Kafka message queue since a > >> specific time? For example, a user may subscribe to a topic and the > >> producer will continuously publish messages related to this topic. The > >> first time this user logs in, we will fetch all the messages from the > >> beginning. However, the next time this user logs in, we want to only > >> fetch the "new" messages. In other words, messages since the user's > >> last log out time. > >> > >> Is there any API in Kafka that allows us to do that? I am not sure if > >> Kafka actually stores a timestamp with each message as the message's > >> meta data. If not, is there any way to fetch the offset related to the > >> user's last log out time? > >> > >> One way that I can think of to do this is to store the offset of the > >> last message this user consumers before he logged out of the system > >> (persist this offset at a DB). The next time this user logs in, we > >> will read the DB to get that offset and start from there to fetch > >> messages. However, if there is a better way to do this in Kafka, then > >> it will save me the work to write/read from the DB. > >> > >> thanks! > >> > >> Jason > >> >
