Re: Global/Restore consumer use auto.offset.reset = none vs. OffsetOutOfRangeException

2018-10-03 Thread Patrik Kleindl
Hello Matthias Thank you for the explanation. Version used is 2.0.0-cp1 The stacktrace: 2018-10-02 10:51:52,575 ERROR [org.apache.kafka.streams.processor.internals.GlobalStreamThread$StateConsumer] (...-077dce27-40fe-47bd-86dd-1615395782af-GlobalStreamThread) - [short-component-name:; transaction

Re: Global/Restore consumer use auto.offset.reset = none vs. OffsetOutOfRangeException

2018-10-03 Thread Matthias J. Sax
I double checked the code and discussed with a colleague. There are two places when we call `globalConsumer.poll()` 1. On startup, when we need to bootstrap the store. In this case, we catch the exception and handle it. 2. During regular processing. In this case, we don't catch the exception. Th

Re: manually trigger log compaction

2018-10-03 Thread Brett Rann
There's no way to "trigger compaction" per se, but you can use multiple configs to compel it. I've documented it in this ticket: https://issues.apache.org/jira/browse/KAFKA-7137 Essentially you can set min.cleanable.dirty.ratio=0 and segment.ms to make sure a new segment is rolled (compaction won

Re: kafka-2.0.0-src/bin/kafka-topics.sh The shell file is empty;

2018-10-03 Thread Brett Rann
Looks fine to me: from: https://www.apache.org/dyn/closer.cgi?path=/kafka/2.0.0/kafka_2.11-2.0.0.tgz tmp/kafka_2.11-2.0.0/bin% head -2 kafka-topics.sh #!/bin/bash # Licensed to the Apache Software Foundation (ASF) under one or more Also from this src it's fine in there too: https://www.apache.org