Adding partitions: You should not add partitions at runtime -- it might break the semantics of your application because is might "mess up" you hash partitioning. Cf. https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowtoscaleaStreamsapp,i.e.,increasenumberofinputpartitions?
If you are sure, that this is not a concern, because you don't do any stateful operation (or some _manual_ re-partitioning within your application before any key-based operation), than Streams should pick up added partitions automatically. This can take multiple minutes depending on your metadata refresh interval (default is 5 minutes). About rewinding consumer partition offsets: There is no tool to do this right now. But there is a KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-122%3A+Add+Reset+Consumer+Group+Offsets+tooling For now, you could write you own little Java program that manipulate the offsets before you start your Streams application. However, be aware the this will result in duplicate processing as there is currently no way to reset your state stores. -Matthias On 2/28/17 1:31 PM, Neil Moore wrote: > Hello, > > > I have a few questions that I couldn't find answers to in the documentation: > > > * Can added partitions be auto-discovered by kafka-streams? In my > informal tests I have had to restart the stream nodes. > * Is it possible to rewind the consumer for a particular > topic-partitions. e.g. if I have a Processor handling a topic-partition can I > rewind that a to an arbitrary offset? I saw that > kafka-streams-application-reset allows all partitions to be reset, but I'd > like to decide per partition. > > If anybody can shed any light on these issues I'd be most grateful. > > Thanks. >
signature.asc
Description: OpenPGP digital signature
