Mathias, Thanks for response. I am not sure if this timestamp is the Unix time or not. I've tried the following:
Create 3 messages of the same topic, at the same partition like this: 1355769714152: Jason has a new message 1 1355769964900: Jason has a new message 2 1355769980296: Jason has a new message 3 I then tried to call getOffsetsBefore with a timestamp = 1355769964999 (99 milliseconds after the timestamp in message two above), hoping to get some offset but the long array returned by the call is empty. Some google search found that getOffsetsBefore is based on the mime of the log segments. In other words, if I only have one log file 00000000000000000000.kafka in the topic directory (log/topic-0), then the offset array returned by this call will always be 0? If so, this API is probably not designed for my use case. thanks, Jason On Mon, Dec 17, 2012 at 1:40 PM, Mathias Söderberg <mathias.soederb...@gmail.com> wrote: > The SimpleConsumer API [1] has a method called getOffsetsBefore which takes > a topic, partition, timestamp (UNIX I assume since it's a long) and integer > limit on how many offsets to get. > > Might not solve your problem *exactly*, but could be useful, unless you're > using the ConsumerConnector? > > [1]: http://people.apache.org/~joestein/kafka-0.7.1-incubating-docs/ > > > On 17 December 2012 19:23, Jason Huang <jason.hu...@icare.com> wrote: > >> Hello, >> >> Is it possible to fetch messages from the Kafka message queue since a >> specific time? For example, a user may subscribe to a topic and the >> producer will continuously publish messages related to this topic. The >> first time this user logs in, we will fetch all the messages from the >> beginning. However, the next time this user logs in, we want to only >> fetch the "new" messages. In other words, messages since the user's >> last log out time. >> >> Is there any API in Kafka that allows us to do that? I am not sure if >> Kafka actually stores a timestamp with each message as the message's >> meta data. If not, is there any way to fetch the offset related to the >> user's last log out time? >> >> One way that I can think of to do this is to store the offset of the >> last message this user consumers before he logged out of the system >> (persist this offset at a DB). The next time this user logs in, we >> will read the DB to get that offset and start from there to fetch >> messages. However, if there is a better way to do this in Kafka, then >> it will save me the work to write/read from the DB. >> >> thanks! >> >> Jason >>
