The SimpleConsumer API [1] has a method called getOffsetsBefore which takes a topic, partition, timestamp (UNIX I assume since it's a long) and integer limit on how many offsets to get.
Might not solve your problem *exactly*, but could be useful, unless you're using the ConsumerConnector? [1]: http://people.apache.org/~joestein/kafka-0.7.1-incubating-docs/ On 17 December 2012 19:23, Jason Huang <jason.hu...@icare.com> wrote: > Hello, > > Is it possible to fetch messages from the Kafka message queue since a > specific time? For example, a user may subscribe to a topic and the > producer will continuously publish messages related to this topic. The > first time this user logs in, we will fetch all the messages from the > beginning. However, the next time this user logs in, we want to only > fetch the "new" messages. In other words, messages since the user's > last log out time. > > Is there any API in Kafka that allows us to do that? I am not sure if > Kafka actually stores a timestamp with each message as the message's > meta data. If not, is there any way to fetch the offset related to the > user's last log out time? > > One way that I can think of to do this is to store the offset of the > last message this user consumers before he logged out of the system > (persist this offset at a DB). The next time this user logs in, we > will read the DB to get that offset and start from there to fetch > messages. However, if there is a better way to do this in Kafka, then > it will save me the work to write/read from the DB. > > thanks! > > Jason >
