Bart,
Thanks for providing your observations and conclusions. Stay tuned on
further discussions on adding dynamic subscriptions in Streams.
Guozhang
On Wed, Aug 16, 2017 at 1:37 AM, Bart Vercammen wrote:
> Hi Guozhang,
>
> In the end I opted for a native Kafka consumer/producer application i
Hi Guozhang,
In the end I opted for a native Kafka consumer/producer application instead
of using Kafka streams for this.
The overhead in creating new streams applications for each update of the
metadata was a bit to cumbersome.
But still, the issue remains that, although this works (thanks for th
HI Guozhang,
Thanks for your swift feedback.
Using your "Pipe App" example might actually be a neat work-around.
I'll see if I can work out a simple prototype for this on our platform.
The only downside of this is that I will double the message-load on the
platform (from source-topics to processi
Hello Bart,
Thanks for your detailed explanation. I saw your motivation now and it
indeed validates itself as a single application that dynamically change
subscriptions.
As I mentioned Streams today do not have a native support for dynamically
changing subscriptions. That being said, If you would
Hi Guozhang,
For the use-cases I have in mind, the offset of the source topics is
irrelevant to the state stored by the streams application.
So, when topic 'A' gets dropped and topic 'B' is added, I would prefer the
application to start reading from 'latest' but that is actually not *that*
import
Hello Bart,
Before we talk about dynamic subscription in Streams I'd like to ask some
more questions about your usage.
Let's say the current running application is reading from topic A, and have
just reached offset 100; now from the global state there is a notification
saying "stop on topic A, st
Hi,
I have a question basically on how it would be the best way to implement
something within Kafka Streams. The thing I would like to do: "dynamically
update the subscription pattern of the source topics.
The reasoning behind this (in my project):
meta data about the source topics is evented on
