Hi Guozhang Just a small question, why can't this be checked when trying to instantiate KafkaStreams? The Topology should know all topics and the existence of the topics could be verified with the AdminClient. This would allow to fail fast similar to when the state directory is not available. Or am I missing something? best regards Patrik
On Mon, 25 Mar 2019 at 23:15, Guozhang Wang <wangg...@gmail.com> wrote: > Hello Murilo, > > Just to give some more background to John's message and KAFKA-7970 here. > The main reason of trickiness is around the scenario of "topics being > partially available", e.g. say your application is joining to topics A and > B, while topicA exists but topicB does not (it is surprisingly common due > to either human errors, or topic creation race conditions, etc). Then you > have a few options at hand: > > 1. Just start the app normally, which will only process data from topicA > and none from topicB. When topicB is created later the app will > auto-rebalance to get the data (this is guaranteed by Streams itself). > However before this is true the join operator would see no data from topicB > to join to while proceeding. This behavior is actually the case before > Kafka version 2.0 and many users complained about it. > > 2. Does not start the app at all, notify the users that some topics are > missing and stop. This is what we changed in KAFKA-5037. > > 3. We can also, let app to stay up and running, but does not process any > data at all until all topics subscribed become available, etc etc. > > > Now depending on user's motivation cases and preferences, they may prefer > any of these options. The reason we chose to do 2) is to give users some > control upon the event and do their wrapping logic on top of it (e.g. like > John suggested). Hope this helps some clarifications for you. > > > Guozhang > > > On Mon, Mar 25, 2019 at 12:23 PM John Roesler <j...@confluent.io> wrote: > > > Hi, Murlio, > > > > I found https://issues.apache.org/jira/browse/KAFKA-7970, which sounds > > like > > the answer is currently "yes". Unfortunately, it is still tricky to > handle > > this case, although the situation may improve soon. > > > > In the mean time, you can try to work around it with the StateListener. > > When Streams has a successful start-up, you'll see it transition from > > REBALANCING to RUNNING, so if you see it transition to PENDING_SHUTDOWN, > > NOT_RUNNING, or ERROR before you see "oldState: REBALANCING && newState: > > RUNNING", you know that Streams did not have a successful startup. It > > sounds like you can't determine programmatically *why* this happened, but > > you can log a warning or error and then create a new the KafkaStreams > > object and try starting it again. > > > > I hope this helps, and feel free to comment on that ticket to add your > own > > perspective to the issue! > > > > Thanks, > > -John > > > > On Fri, Mar 22, 2019 at 3:25 PM Murilo Tavares <murilo...@gmail.com> > > wrote: > > > > > Hi > > > After some research, I've come to a few discussions, and they all tell > me > > > that Kafka Streams require the topics to be created before starting the > > > application. > > > Nevertheless, I'd like my application to keep retrying if a topic does > > not > > > exist. > > > I've seen this thread: > > > https://groups.google.com/forum/#!topic/confluent-platform/nmfrnAKCM3c > , > > > which is pretty old, and I'd like to know if it's still hard to catch > > that > > > Exception in my app. > > > > > > Thanks > > > Murilo > > > > > > > > -- > -- Guozhang >
