I had enabled eos through streams config and as explained in the documentation, I have not added anything else other than following config.
streamsConfiguration.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, StreamsConfig.EXACTLY_ONCE); As explained by you, I think producer idempotence and retries would be automatically picked up. I was thinking if there is someway i can print the configs picked up the streams app for e.g. the current retries. Do i need to add any other configuration. -Sameer. On Tue, Oct 3, 2017 at 11:59 AM, Sameer Kumar <sam.kum.w...@gmail.com> wrote: > I had enabled eos through streams config. > > > On Fri, Sep 29, 2017 at 11:12 PM, Matthias J. Sax <matth...@confluent.io> > wrote: > >> That's correct: If EOS is enabled, we enforce some producer configs: >> >> https://github.com/apache/kafka/blob/0.11.0.1/streams/src/ >> main/java/org/apache/kafka/streams/StreamsConfig.java#L678-L688 >> >> https://github.com/apache/kafka/blob/0.11.0.1/streams/src/ >> main/java/org/apache/kafka/streams/StreamsConfig.java#L691 >> >> https://github.com/apache/kafka/blob/0.11.0.1/streams/src/ >> main/java/org/apache/kafka/streams/StreamsConfig.java#L493-L496 >> >> >> Note, that by default we set retries to Integer.MAX_VALUE but we do not >> enforce this setting (as pointed out by Damian already). So you could >> overwrite it with a smaller value (what is of course not recommended). >> >> I was not sure though, if you enabled EOS or just enabled idempotency >> only for the producer -- what you can easily do by providing the >> corresponding producer configs. >> >> If you did enable EOS in StreamsConfig, Producer will take care of >> OutOfOrderSequenceException in general. >> >> However, there are scenarios for which Producer cannot handle it. For >> those case, OutOfOrderSequenceException indicates that there has been >> data loss on the broker, ie, a previously acknowledged message no longer >> exists. For most part, this should only occur in rare situations >> (simultaneous power outages, multiple disk losses, software bugs >> resulting in data corruption, etc.). >> >> >> -Matthias >> >> On 9/29/17 7:55 AM, Damian Guy wrote: >> > You can set ProducerConfig.RETRIES_CONFIG in your StreamsConfig, i.e, >> > >> > Properties props = new Properties(); >> > props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE); >> > ... >> > >> > On Fri, 29 Sep 2017 at 13:17 Sameer Kumar <sam.kum.w...@gmail.com> >> wrote: >> > >> >> I guess once stream app are enabled exactly-once, producer idempotence >> get >> >> enabled by default and so do the retries. I guess producer retries are >> >> managed internally and not exposed through streamconfig. >> >> >> >> https://kafka.apache.org/0110/documentation/#streamsconfigs >> >> >> >> -Sameer. >> >> >> >> On Thu, Sep 28, 2017 at 12:12 AM, Matthias J. Sax < >> matth...@confluent.io> >> >> wrote: >> >> >> >>> An OutOfOrderSequenceException should only occur if a idempotent >> >>> producer gets out of sync with the broker. If you set >> >>> `enable.idempotence = true` on your producer, you might want to set >> >>> `retries = Integer.MAX_VALUE`. >> >>> >> >>> -Matthias >> >>> >> >>> On 9/26/17 11:30 PM, Sameer Kumar wrote: >> >>>> Hi, >> >>>> >> >>>> I again received this exception while running my streams app. I am >> >> using >> >>>> Kafka 11.0.1. After restarting my app, this error got fixed. >> >>>> >> >>>> I guess this might be due to bad network. Any pointers. Any config >> >>>> wherein I can configure it for retries. >> >>>> >> >>>> Exception trace is attached. >> >>>> >> >>>> Regards, >> >>>> -Sameer. >> >>> >> >>> >> >> >> > >> >> >
