Technically this is possible with the existing server and protocol and could be implemented using the "low level" network client. The high-level client doesn't really allow this. This would be a good thing to think about as we start on the redesign of that client. I don't think it has to be terribly convenient because it is not a very common use case, but just being able to read chunks and reverse them in memory would allow this, so I think the only thing needed is the ability to set your offset easily.
-Jay On Wed, Feb 5, 2014 at 2:41 PM, Jonathan Colt < jonathan.c...@jivesoftware.com> wrote: > Hi, > > Here is a use case that we would like to see kafka's client support in the > future. Currently reading a topic is FIFO. It would be awesome to read a > topic in LIFO order. Put another way we would like to be able to read a > topic in reverse. > > > Why? Basically we have per user streams which we have to keep up to date. > When an event comes into the system it is determined to be cheap enough to > fan out the user stream updates at write time or to expensive and thus > deferred until read time. This is where kafka comes into play. We push all > the events that were to expensive to fan out at write time onto a kafka > topic. When a given user logs on we read through this topic and update the > users stream. If it has been a while since the user has logged in it takes > a while to catch up. If we could read a topic in reverse order we could > catch the most recent events in the stream first. Which in our case is what > the user cares about the most. Once the 20 most resent items have been > added to a user's stream we can return the head of the stream and continue > the catch up process in the background. > > > Thanks, > > Jonathan > >
