Thanks for the answers, Matthias.
You mention a metadata refresh interval. I see Kafka producers and consumers have a property called metadata.max.age.ms which sounds similar. From the documentation and looking at the Javadoc for Kafka streams it is not clear to me how I can affect KafkaStreams' discovery of topics and partitions. It is by configuring consumers using the Properties/StreamsConfig object passed to KafkaStreams' constructor? I.e. something like props.put(StreamsConfig.CONSUMER_PREFIX + "metadata.max.age.ms", 10000) .. KafkaStreams streams = new KafkaStreams(blah, props) Thanks, Neil ________________________________ From: Matthias J. Sax <matth...@confluent.io> Sent: 28 February 2017 22:26:39 To: users@kafka.apache.org Subject: Re: Kafka streams questions Adding partitions: You should not add partitions at runtime -- it might break the semantics of your application because is might "mess up" you hash partitioning. Cf. https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowtoscaleaStreamsapp,i.e.,increasenumberofinputpartitions? If you are sure, that this is not a concern, because you don't do any stateful operation (or some _manual_ re-partitioning within your application before any key-based operation), than Streams should pick up added partitions automatically. This can take multiple minutes depending on your metadata refresh interval (default is 5 minutes). About rewinding consumer partition offsets: There is no tool to do this right now. But there is a KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-122%3A+Add+Reset+Consumer+Group+Offsets+tooling For now, you could write you own little Java program that manipulate the offsets before you start your Streams application. However, be aware the this will result in duplicate processing as there is currently no way to reset your state stores. -Matthias On 2/28/17 1:31 PM, Neil Moore wrote: > Hello, > > > I have a few questions that I couldn't find answers to in the documentation: > > > * Can added partitions be auto-discovered by kafka-streams? In my > informal tests I have had to restart the stream nodes. > * Is it possible to rewind the consumer for a particular > topic-partitions. e.g. if I have a Processor handling a topic-partition can I > rewind that a to an arbitrary offset? I saw that > kafka-streams-application-reset allows all partitions to be reset, but I'd > like to decide per partition. > > If anybody can shed any light on these issues I'd be most grateful. > > Thanks. >
