Hi Paul
On Thu, Nov 21, 2024 at 6:06 PM Brebner, Paul <paul.breb...@netapp.com.invalid> wrote: > Hi John, > > I’m not a Kafka streams expert but have experimented a few times – I > recall that Kafka Streams does need to create/use “internal topics” – and > security has to be set on clients correctly from memory. > This may help? > https://kafka.apache.org/23/documentation/streams/developer-guide/manage-topics > And this > > https://kafka.apache.org/23/documentation/streams/developer-guide/security.html#streams-developer-guide-security Thanks, yeah, I've seen these two. The topics I'm referring to are the internal topics around state store, and other similar internal use cases as found at https://github.com/apache/kafka/blob/trunk/docs/streams/developer-guide/dsl-topology-naming.html#L83-L95 , which are generally (I believe) considered the non-user topics. To give a little more context, we run our production cluster with tight security and prefer to disable the auto create topic setting, but for CI purposes we're pretty open, hence didn't notice this error until deployment. My hunch is that these internal topics require the auto create setting set to true, but wanted to confirm before begging to flip it on (and do the necessary security reviews and such). John > > > Regards, Paul Brebner, NetApp > > From: John D. Ament <johndam...@apache.org> > Date: Friday, 22 November 2024 at 8:46 am > To: users@kafka.apache.org <users@kafka.apache.org> > Subject: Explicitly creating topology topics in a streams app > [You don't often get email from johndam...@apache.org. Learn why this is > important at https://aka.ms/LearnAboutSenderIdentification ] > > EXTERNAL EMAIL - USE CAUTION when clicking links or attachments > > > > > Hi, > > When I look at most stream examples I notice that they tend to include the > broker setting: > > auto.create.topics.enable=true > > My understanding is that this isn't usually recommended for production > environments, so we have it off. We have started to play with Kafka > Streams apps a bit but noticed that they fail in our production > environments with errors around "UNKNOWN_TOPIC_OR_PARTITION". I suspect > it's related to not having auto create enabled. > > Is there an option to force the client to create the topology topics that > they use for Kafka Streams apps? Or is auto creation required? > > Thanks, > > John >
