Hmm, that would be a problem especially if your WebSocket clients can come and go.
I am actually wondering if your WebSocket is needed for any other usage beside this? If not, then you may probably get rid of the server and just let all clients directly consume from Kafka, where each one of them maintain their own status like offsets. Guozhang On Mon, May 25, 2015 at 12:09 AM, Kevin Sjöberg <kevin.sjob...@fortnox.se> wrote: > Hi Guozhang, > > Thanks for your input. That would be a great solution if it wasn't that > more than one client could be connected via WebSocket. Hence, a single > client can only tell what they've consumed and the consumer is shared for > everyone connected. > > Is there any other way I can go about this problem? Really would appreciate > any input. :) > > 2015-05-20 18:37 GMT+02:00 Guozhang Wang <wangg...@gmail.com>: > > > Hi Kevin, > > > > The current high-level Scala consumer does not have a rewind function but > > if the other client is able to notify you through WebSocket periodically > as > > long as it is not disconnected, then what you can do is let your Java app > > to buffer messages even after pushing them to the WebSocket, and only > flush > > them and commit back to Kafka after you got notified from the WebSocket > > that they have been successfully consumed. > > > > Guozhang > > > > On Wed, May 20, 2015 at 5:48 AM, Kevin Sjöberg <kevin.sjob...@fortnox.se > > > > wrote: > > > > > Thanks for the link, Sharninder. I'm not entirely sure this is what I'm > > > looking for. I do not know whether or not the client actually received > > the > > > messages or not. > > > > > > The Java-application consumes the messages from Kafka and then push > them > > to > > > Pusher (https://pusher.com/). The client, connected to the Pusher > > > WebSocket-server (i.e., not provided by me), then get the messages I > sent > > > them. The scenario that could happen is that the client loses the > > > connection to the WebSocket server and may miss messages that were > pushed > > > during the timeout. In this case, they want to tell me that they lost > the > > > connection and start to get messages from a given point in time again. > > The > > > only way they interact with me is connecting to the WebSocket server > > again > > > with an optional message that I can get notified about. They do not > > connect > > > to my Java-application directly. > > > > > > 2015-05-20 14:24 GMT+02:00 Sharninder <sharnin...@gmail.com>: > > > > > > > > I have a backend service written in PHP. This service pushes > messages > > > to > > > > > Apache Kafka (over the topic "posts") when posts are created, read > > and > > > > > removed. I also have a backend service written in Java. This > service > > > > > consumes messages from Apache Kafka (for the "posts" topic) and > push > > > them > > > > > out over WebSockets to whoever is listening. This works out great. > > > > > > > > > > The problem I have is as follows. Say, for some reason, the > WebSocket > > > > > client timeout and because of this he or she does not get some of > the > > > > > messages. The messages are still being consumed but the client > isn't > > > > there > > > > > to acknowledge them. Now, the client want "rewind" and get all the > > > > messages > > > > > that they missed may have missed. This is where I'm stuck. > > > > > > > > > > Ideally, the client would provide a timestamp for when they last > got > > a > > > > > message and I'd then use that timestamp to get messages from that > > point > > > > and > > > > > onwards. I'm not sure if this is even possible, it's just my > > thoughts. > > > > > > > > > > > > > > Something like this was discussed a few weeks back on the list. But, > > > > essentially what you need to do is turn off auto commit and only > commit > > > > when you're sure the message has been acknowledged by the next stage. > > > > > > > > I think the link is this -> > > > > > > > > > > > > > > http://ingest.tips/2014/10/12/kafka-high-level-consumer-frequently-missing-pieces/ > > > > > > > > Regards, > > > > Sharninder > > > > > > > > > > > > > > > > -- > > > Kevin Sjöberg > > > > > > -- > > > Du skapar din faktura, vi skickar den och ser till att kunden betalar. > Vi > > > sköter allt, hela vägen tills fakturan är betald! > > > Läs om Nox Finans här > > > < > > > > > > https://www.fortnox.se/nox/?utm_source=signature&utm_medium=email&utm_campaign=noxfinans&utm_term=fakturahantering > > > > > > > . > > > > > > > > > > > -- > > -- Guozhang > > > > > > -- > Kevin Sjöberg > > -- > Du skapar din faktura, vi skickar den och ser till att kunden betalar. Vi > sköter allt, hela vägen tills fakturan är betald! > Läs om Nox Finans här > < > https://www.fortnox.se/nox/?utm_source=signature&utm_medium=email&utm_campaign=noxfinans&utm_term=fakturahantering > > > . > -- -- Guozhang
