It is by design to set the reset policy to "none" (https://issues.apache.org/jira/browse/KAFKA-6121), and not allowed by design to overwrite this (there might be a workaround for you though). However, Streams should not die but catch the exception and recover from it automatically.
What version do you use? Can you share the full stack trace to see why Streams failed to recover from this exception? -Matthias On 10/2/18 4:54 AM, Patrik Kleindl wrote: > Hi > > We had several incidents where a streams application crashed while > maintaining a global state store. > Updating global state failed. You can restart KafkaStreams to recover from > this error.: org.apache.kafka.clients.consumer.OffsetOutOfRangeException: > Offsets out of range with no configured reset policy for partitions: ... > > As we never set this to none I checked the code and found that > StreamsConfig getGlobalConsumerConfigs and getRestoreConsumerConfigs both > set this explicitely: > baseConsumerProps.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "none"); > > The logs confirms this: > 2018-10-02 11:07:06,057 INFO [org.apache.kafka.common.utils.AppInfoParser] > (ServerService Thread Pool -- 70) - [short-component-name:; > transaction-id:; user-id:; creation-time:] Kafka version : 2.0.0-cp1 > 2018-10-02 11:07:06,057 INFO [org.apache.kafka.common.utils.AppInfoParser] > (ServerService Thread Pool -- 70) - [short-component-name:; > transaction-id:; user-id:; creation-time:] Kafka commitId : > a8c648ff08b9235d > 2018-10-02 11:07:06,104 INFO > [org.apache.kafka.clients.consumer.ConsumerConfig] (ServerService Thread > Pool -- 72) - [short-component-name:; transaction-id:; user-id:; > creation-time:] ConsumerConfig values: > auto.commit.interval.ms = 5000 > auto.offset.reset = none > bootstrap.servers = [...] > check.crcs = true > client.id = ...-3f809a8a-3915-4ae4-9a37-f7a392e3dff3-global-consumer > > ... > > 2018-10-02 11:07:06,418 INFO > [org.apache.kafka.streams.processor.internals.StreamThread] (ServerService > Thread Pool -- 72) - [short-component-name:; transaction-id:; user-id:; > creation-time:] stream-thread > [...-3f809a8a-3915-4ae4-9a37-f7a392e3dff3-StreamThread-1] Creating restore > consumer client > 2018-10-02 11:07:06,419 INFO > [org.apache.kafka.clients.consumer.ConsumerConfig] (ServerService Thread > Pool -- 72) - [short-component-name:; transaction-id:; user-id:; > creation-time:] ConsumerConfig values: > auto.commit.interval.ms = 5000 > auto.offset.reset = none > bootstrap.servers = [...] > check.crcs = true > client.id = > ...-3f809a8a-3915-4ae4-9a37-f7a392e3dff3-StreamThread-1-restore-consumer > > Is this intentional and if yes, why can this not use the default policy and > recover? > > best regards > > Patrik >
signature.asc
Description: OpenPGP digital signature
