Hi Guozhang, Thanks for your input. That would be a great solution if it wasn't that more than one client could be connected via WebSocket. Hence, a single client can only tell what they've consumed and the consumer is shared for everyone connected.
Is there any other way I can go about this problem? Really would appreciate any input. :) 2015-05-20 18:37 GMT+02:00 Guozhang Wang <wangg...@gmail.com>: > Hi Kevin, > > The current high-level Scala consumer does not have a rewind function but > if the other client is able to notify you through WebSocket periodically as > long as it is not disconnected, then what you can do is let your Java app > to buffer messages even after pushing them to the WebSocket, and only flush > them and commit back to Kafka after you got notified from the WebSocket > that they have been successfully consumed. > > Guozhang > > On Wed, May 20, 2015 at 5:48 AM, Kevin Sjöberg <kevin.sjob...@fortnox.se> > wrote: > > > Thanks for the link, Sharninder. I'm not entirely sure this is what I'm > > looking for. I do not know whether or not the client actually received > the > > messages or not. > > > > The Java-application consumes the messages from Kafka and then push them > to > > Pusher (https://pusher.com/). The client, connected to the Pusher > > WebSocket-server (i.e., not provided by me), then get the messages I sent > > them. The scenario that could happen is that the client loses the > > connection to the WebSocket server and may miss messages that were pushed > > during the timeout. In this case, they want to tell me that they lost the > > connection and start to get messages from a given point in time again. > The > > only way they interact with me is connecting to the WebSocket server > again > > with an optional message that I can get notified about. They do not > connect > > to my Java-application directly. > > > > 2015-05-20 14:24 GMT+02:00 Sharninder <sharnin...@gmail.com>: > > > > > > I have a backend service written in PHP. This service pushes messages > > to > > > > Apache Kafka (over the topic "posts") when posts are created, read > and > > > > removed. I also have a backend service written in Java. This service > > > > consumes messages from Apache Kafka (for the "posts" topic) and push > > them > > > > out over WebSockets to whoever is listening. This works out great. > > > > > > > > The problem I have is as follows. Say, for some reason, the WebSocket > > > > client timeout and because of this he or she does not get some of the > > > > messages. The messages are still being consumed but the client isn't > > > there > > > > to acknowledge them. Now, the client want "rewind" and get all the > > > messages > > > > that they missed may have missed. This is where I'm stuck. > > > > > > > > Ideally, the client would provide a timestamp for when they last got > a > > > > message and I'd then use that timestamp to get messages from that > point > > > and > > > > onwards. I'm not sure if this is even possible, it's just my > thoughts. > > > > > > > > > > > Something like this was discussed a few weeks back on the list. But, > > > essentially what you need to do is turn off auto commit and only commit > > > when you're sure the message has been acknowledged by the next stage. > > > > > > I think the link is this -> > > > > > > > > > http://ingest.tips/2014/10/12/kafka-high-level-consumer-frequently-missing-pieces/ > > > > > > Regards, > > > Sharninder > > > > > > > > > > > -- > > Kevin Sjöberg > > > > -- > > Du skapar din faktura, vi skickar den och ser till att kunden betalar. Vi > > sköter allt, hela vägen tills fakturan är betald! > > Läs om Nox Finans här > > < > > > https://www.fortnox.se/nox/?utm_source=signature&utm_medium=email&utm_campaign=noxfinans&utm_term=fakturahantering > > > > > . > > > > > > -- > -- Guozhang > -- Kevin Sjöberg -- Du skapar din faktura, vi skickar den och ser till att kunden betalar. Vi sköter allt, hela vägen tills fakturan är betald! Läs om Nox Finans här <https://www.fortnox.se/nox/?utm_source=signature&utm_medium=email&utm_campaign=noxfinans&utm_term=fakturahantering> .
