Re: Global/Restore consumer use auto.offset.reset = none vs. OffsetOutOfRangeException

2018-10-04 Thread Patrik Kleindl
Saw it, thank you. Best regards Patrik > Am 04.10.2018 um 17:11 schrieb Matthias J. Sax : > > Thanks for the details. > > Seems like a fair assumption. I created a jira to track it: > https://issues.apache.org/jira/browse/KAFKA-7480 > > For now, there is not much you can do, because Streams har

Re: Global/Restore consumer use auto.offset.reset = none vs. OffsetOutOfRangeException

2018-10-04 Thread Matthias J. Sax
Thanks for the details. Seems like a fair assumption. I created a jira to track it: https://issues.apache.org/jira/browse/KAFKA-7480 For now, there is not much you can do, because Streams hard codes to set the policy to "none". Thus, a manual restart (that is gladly working as you confirmed) it c

Re: Global/Restore consumer use auto.offset.reset = none vs. OffsetOutOfRangeException

2018-10-04 Thread Patrik Kleindl
Hello Matthias Thanks for looking into this. A restart has worked, I can confirm that. Before this problem happened we had some cluster issues which are still being looked into, there were some leader changes and some offset commit failures. The consumer should not have lagged that much behind, but

Re: Global/Restore consumer use auto.offset.reset = none vs. OffsetOutOfRangeException

2018-10-03 Thread Matthias J. Sax
I double checked the code and discussed with a colleague. There are two places when we call `globalConsumer.poll()` 1. On startup, when we need to bootstrap the store. In this case, we catch the exception and handle it. 2. During regular processing. In this case, we don't catch the exception. Th

Re: Global/Restore consumer use auto.offset.reset = none vs. OffsetOutOfRangeException

2018-10-03 Thread Patrik Kleindl
Hello Matthias Thank you for the explanation. Version used is 2.0.0-cp1 The stacktrace: 2018-10-02 10:51:52,575 ERROR [org.apache.kafka.streams.processor.internals.GlobalStreamThread$StateConsumer] (...-077dce27-40fe-47bd-86dd-1615395782af-GlobalStreamThread) - [short-component-name:; transaction

Re: Global/Restore consumer use auto.offset.reset = none vs. OffsetOutOfRangeException

2018-10-02 Thread Matthias J. Sax
It is by design to set the reset policy to "none" (https://issues.apache.org/jira/browse/KAFKA-6121), and not allowed by design to overwrite this (there might be a workaround for you though). However, Streams should not die but catch the exception and recover from it automatically. What version do

Global/Restore consumer use auto.offset.reset = none vs. OffsetOutOfRangeException

2018-10-02 Thread Patrik Kleindl
Hi We had several incidents where a streams application crashed while maintaining a global state store. Updating global state failed. You can restart KafkaStreams to recover from this error.: org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Offsets out of range with no configured reset