Yes, that is the parameter I was referring, too. And yes, you can set consumer/producer config via StreamsConfig. However, it's recommended to use
> props.put(StreamsConfig.consumerPrefix("consumer.parameter.name"), value);
-Matthias
On 3/6/17 6:48 AM, Neil Moore wrote:
> Thanks for the answers, Matthias.
>
>
> You mention a metadata refresh interval. I see Kafka producers and consumers
> have a property called metadata.max.age.ms which sounds similar. From the
> documentation and looking at the Javadoc for Kafka streams it is not clear to
> me how I can affect KafkaStreams' discovery of topics and partitions. It is
> by configuring consumers using the Properties/StreamsConfig object passed to
> KafkaStreams' constructor? I.e. something like
>
>
> props.put(StreamsConfig.CONSUMER_PREFIX + "metadata.max.age.ms", 10000)
>
> ..
>
> KafkaStreams streams = new KafkaStreams(blah, props)
>
>
> Thanks,
>
>
> Neil
>
> ________________________________
> From: Matthias J. Sax <matth...@confluent.io>
> Sent: 28 February 2017 22:26:39
> To: users@kafka.apache.org
> Subject: Re: Kafka streams questions
>
> Adding partitions:
>
> You should not add partitions at runtime -- it might break the semantics
> of your application because is might "mess up" you hash partitioning.
> Cf.
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowtoscaleaStreamsapp,i.e.,increasenumberofinputpartitions?
>
>
> If you are sure, that this is not a concern, because you don't do any
> stateful operation (or some _manual_ re-partitioning within your
> application before any key-based operation), than Streams should pick up
> added partitions automatically. This can take multiple minutes depending
> on your metadata refresh interval (default is 5 minutes).
>
>
> About rewinding consumer partition offsets:
>
> There is no tool to do this right now. But there is a KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-122%3A+Add+Reset+Consumer+Group+Offsets+tooling
>
> For now, you could write you own little Java program that manipulate the
> offsets before you start your Streams application. However, be aware the
> this will result in duplicate processing as there is currently no way to
> reset your state stores.
>
>
> -Matthias
>
>
>
>
> On 2/28/17 1:31 PM, Neil Moore wrote:
>> Hello,
>>
>>
>> I have a few questions that I couldn't find answers to in the documentation:
>>
>>
>> * Can added partitions be auto-discovered by kafka-streams? In my
>> informal tests I have had to restart the stream nodes.
>> * Is it possible to rewind the consumer for a particular
>> topic-partitions. e.g. if I have a Processor handling a topic-partition can
>> I rewind that a to an arbitrary offset? I saw that
>> kafka-streams-application-reset allows all partitions to be reset, but I'd
>> like to decide per partition.
>>
>> If anybody can shed any light on these issues I'd be most grateful.
>>
>> Thanks.
>>
>
>
signature.asc
Description: OpenPGP digital signature
