Hi Kevin, The current high-level Scala consumer does not have a rewind function but if the other client is able to notify you through WebSocket periodically as long as it is not disconnected, then what you can do is let your Java app to buffer messages even after pushing them to the WebSocket, and only flush them and commit back to Kafka after you got notified from the WebSocket that they have been successfully consumed.
Guozhang On Wed, May 20, 2015 at 5:48 AM, Kevin Sjöberg <kevin.sjob...@fortnox.se> wrote: > Thanks for the link, Sharninder. I'm not entirely sure this is what I'm > looking for. I do not know whether or not the client actually received the > messages or not. > > The Java-application consumes the messages from Kafka and then push them to > Pusher (https://pusher.com/). The client, connected to the Pusher > WebSocket-server (i.e., not provided by me), then get the messages I sent > them. The scenario that could happen is that the client loses the > connection to the WebSocket server and may miss messages that were pushed > during the timeout. In this case, they want to tell me that they lost the > connection and start to get messages from a given point in time again. The > only way they interact with me is connecting to the WebSocket server again > with an optional message that I can get notified about. They do not connect > to my Java-application directly. > > 2015-05-20 14:24 GMT+02:00 Sharninder <sharnin...@gmail.com>: > > > > I have a backend service written in PHP. This service pushes messages > to > > > Apache Kafka (over the topic "posts") when posts are created, read and > > > removed. I also have a backend service written in Java. This service > > > consumes messages from Apache Kafka (for the "posts" topic) and push > them > > > out over WebSockets to whoever is listening. This works out great. > > > > > > The problem I have is as follows. Say, for some reason, the WebSocket > > > client timeout and because of this he or she does not get some of the > > > messages. The messages are still being consumed but the client isn't > > there > > > to acknowledge them. Now, the client want "rewind" and get all the > > messages > > > that they missed may have missed. This is where I'm stuck. > > > > > > Ideally, the client would provide a timestamp for when they last got a > > > message and I'd then use that timestamp to get messages from that point > > and > > > onwards. I'm not sure if this is even possible, it's just my thoughts. > > > > > > > > Something like this was discussed a few weeks back on the list. But, > > essentially what you need to do is turn off auto commit and only commit > > when you're sure the message has been acknowledged by the next stage. > > > > I think the link is this -> > > > > > http://ingest.tips/2014/10/12/kafka-high-level-consumer-frequently-missing-pieces/ > > > > Regards, > > Sharninder > > > > > > -- > Kevin Sjöberg > > -- > Du skapar din faktura, vi skickar den och ser till att kunden betalar. Vi > sköter allt, hela vägen tills fakturan är betald! > Läs om Nox Finans här > < > https://www.fortnox.se/nox/?utm_source=signature&utm_medium=email&utm_campaign=noxfinans&utm_term=fakturahantering > > > . > -- -- Guozhang
