Exception while doing the join, cant decipher more on this. Has anyone faced it. complete exception trace attached.
2017-08-14 11:15:55 ERROR ConsumerCoordinator:269 - User provided listener org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener for group c-7-a34 failed on partition assignment org.apache.kafka.streams.errors.ProcessorStateException: Error opening store KSTREAM-JOINTHIS-0000000018-store-201708140520 at location /data/streampoc/c-7-a34/0_4/KSTREAM-JOINTHIS-0000000018-store/KSTREAM-JOINTHIS-0000000018-store-201708140520 at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:198) at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:165) at org.apache.kafka.streams.state.internals.Segment.openDB(Segment.java:40) at org.apache.kafka.streams.state.internals.Segments.getOrCreateSegment(Segments.java:86) at org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore.put(RocksDBSegmentedBytesStore.java:81) at org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore$1.restore(RocksDBSegmentedBytesStore.java:113) at org.apache.kafka.streams.processor.internals.StateRestorer.restore(StateRestorer.java:55) at org.apache.kafka.streams.processor.internals.StoreChangelogReader.processNext(StoreChangelogReader.java:216) at org.apache.kafka.streams.processor.internals.StoreChangelogReader.restorePartition(StoreChangelogReader.java:186) at org.apache.kafka.streams.processor.internals.StoreChangelogReader.restore(StoreChangelogReader.java:151) at org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:184) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:363) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:310) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297) at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078) at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) at org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:582) at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553) at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:527) Caused by: org.rocksdb.RocksDBException: at org.rocksdb.RocksDB.open(Native Method) at org.rocksdb.RocksDB.open(RocksDB.java:231) at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:191) ... 19 more -sameer.
2017-08-14 11:15:55 ERROR ConsumerCoordinator:269 - User provided listener org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener for group c-7-a34 failed on partition assignment org.apache.kafka.streams.errors.ProcessorStateException: Error opening store KSTREAM-JOINTHIS-0000000018-store-201708140520 at location /data/streampoc/c-7-a34/0_4/KSTREAM-JOINTHIS-0000000018-store/KSTREAM-JOINTHIS-0000000018-store-201708140520 at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:198) at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:165) at org.apache.kafka.streams.state.internals.Segment.openDB(Segment.java:40) at org.apache.kafka.streams.state.internals.Segments.getOrCreateSegment(Segments.java:86) at org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore.put(RocksDBSegmentedBytesStore.java:81) at org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore$1.restore(RocksDBSegmentedBytesStore.java:113) at org.apache.kafka.streams.processor.internals.StateRestorer.restore(StateRestorer.java:55) at org.apache.kafka.streams.processor.internals.StoreChangelogReader.processNext(StoreChangelogReader.java:216) at org.apache.kafka.streams.processor.internals.StoreChangelogReader.restorePartition(StoreChangelogReader.java:186) at org.apache.kafka.streams.processor.internals.StoreChangelogReader.restore(StoreChangelogReader.java:151) at org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:184) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:363) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:310) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297) at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078) at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) at org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:582) at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553) at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:527) Caused by: org.rocksdb.RocksDBException: at org.rocksdb.RocksDB.open(Native Method) at org.rocksdb.RocksDB.open(RocksDB.java:231) at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:191) ... 19 more 2017-08-14 11:15:55 INFO StreamThread:193 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-8] partition assignment took 1797 ms. current active tasks: [0_56, 0_41, 0_26, 0_11] current standby tasks: [] 2017-08-14 11:15:55 ERROR ConsumerCoordinator:269 - User provided listener org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener for group c-7-a34 failed on partition assignment org.apache.kafka.streams.errors.ProcessorStateException: Error opening store KSTREAM-JOINOTHER-0000000019-store-201708140520 at location /data/streampoc/c-7-a34/0_11/KSTREAM-JOINOTHER-0000000019-store/KSTREAM-JOINOTHER-0000000019-store-201708140520 at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:198) at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:165) at org.apache.kafka.streams.state.internals.Segment.openDB(Segment.java:40) at org.apache.kafka.streams.state.internals.Segments.getOrCreateSegment(Segments.java:86) at org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore.put(RocksDBSegmentedBytesStore.java:81) at org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore$1.restore(RocksDBSegmentedBytesStore.java:113) at org.apache.kafka.streams.processor.internals.StateRestorer.restore(StateRestorer.java:55) at org.apache.kafka.streams.processor.internals.StoreChangelogReader.processNext(StoreChangelogReader.java:216) at org.apache.kafka.streams.processor.internals.StoreChangelogReader.restorePartition(StoreChangelogReader.java:186) at org.apache.kafka.streams.processor.internals.StoreChangelogReader.restore(StoreChangelogReader.java:151) at org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:184) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:363) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:310) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297) at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078) at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) at org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:582) at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553) at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:527) Caused by: org.rocksdb.RocksDBException: at org.rocksdb.RocksDB.open(Native Method) at org.rocksdb.RocksDB.open(RocksDB.java:231) at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:191) ... 19 more 2017-08-14 11:15:55 INFO StreamThread:1040 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-11] Shutting down 2017-08-14 11:15:55 INFO StreamThread:1040 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-8] Shutting down 2017-08-14 11:15:55 INFO StreamThread:193 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-3] partition assignment took 1803 ms. current active tasks: [0_51, 0_36, 0_21, 0_6] current standby tasks: [] 2017-08-14 11:15:55 ERROR ConsumerCoordinator:269 - User provided listener org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener for group c-7-a34 failed on partition assignment org.apache.kafka.streams.errors.ProcessorStateException: Error opening store KSTREAM-JOINTHIS-0000000018-store-201708140520 at location /data/streampoc/c-7-a34/0_21/KSTREAM-JOINTHIS-0000000018-store/KSTREAM-JOINTHIS-0000000018-store-201708140520 at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:198) at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:165) at org.apache.kafka.streams.state.internals.Segment.openDB(Segment.java:40) at org.apache.kafka.streams.state.internals.Segments.getOrCreateSegment(Segments.java:86) at org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore.put(RocksDBSegmentedBytesStore.java:81) at org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore$1.restore(RocksDBSegmentedBytesStore.java:113) at org.apache.kafka.streams.processor.internals.StateRestorer.restore(StateRestorer.java:55) at org.apache.kafka.streams.processor.internals.StoreChangelogReader.processNext(StoreChangelogReader.java:216) at org.apache.kafka.streams.processor.internals.StoreChangelogReader.restorePartition(StoreChangelogReader.java:186) at org.apache.kafka.streams.processor.internals.StoreChangelogReader.restore(StoreChangelogReader.java:151) at org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:184) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:363) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:310) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297) at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078) at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) at org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:582) at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553) at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:527) Caused by: org.rocksdb.RocksDBException: at org.rocksdb.RocksDB.open(Native Method) at org.rocksdb.RocksDB.open(RocksDB.java:231) at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:191) ... 19 more 2017-08-14 11:15:55 INFO StreamThread:1040 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-3] Shutting down 2017-08-14 11:15:55 INFO StreamThread:193 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-5] partition assignment took 1830 ms. current active tasks: [0_54, 0_39, 0_24, 0_9] current standby tasks: [] 2017-08-14 11:15:55 ERROR ConsumerCoordinator:269 - User provided listener org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener for group c-7-a34 failed on partition assignment org.apache.kafka.streams.errors.ProcessorStateException: Error opening store KSTREAM-JOINOTHER-0000000019-store-201708140520 at location /data/streampoc/c-7-a34/0_39/KSTREAM-JOINOTHER-0000000019-store/KSTREAM-JOINOTHER-0000000019-store-201708140520 at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:198) at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:165) at org.apache.kafka.streams.state.internals.Segment.openDB(Segment.java:40) at org.apache.kafka.streams.state.internals.Segments.getOrCreateSegment(Segments.java:86) at org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore.put(RocksDBSegmentedBytesStore.java:81) at org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore$1.restore(RocksDBSegmentedBytesStore.java:113) at org.apache.kafka.streams.processor.internals.StateRestorer.restore(StateRestorer.java:55) at org.apache.kafka.streams.processor.internals.StoreChangelogReader.processNext(StoreChangelogReader.java:216) at org.apache.kafka.streams.processor.internals.StoreChangelogReader.restorePartition(StoreChangelogReader.java:186) at org.apache.kafka.streams.processor.internals.StoreChangelogReader.restore(StoreChangelogReader.java:151) at org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:184) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:363) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:310) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297) at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078) at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) at org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:582) at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553) at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:527) Caused by: org.rocksdb.RocksDBException: at org.rocksdb.RocksDB.open(Native Method) at org.rocksdb.RocksDB.open(RocksDB.java:231) at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:191) ... 19 more 2017-08-14 11:15:55 INFO StreamThread:193 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-6] partition assignment took 1830 ms. current active tasks: [0_50, 0_35, 0_20, 0_5] current standby tasks: [] 2017-08-14 11:15:55 ERROR ConsumerCoordinator:269 - User provided listener org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener for group c-7-a34 failed on partition assignment org.apache.kafka.streams.errors.ProcessorStateException: Error opening store KSTREAM-JOINOTHER-0000000019-store-201708140520 at location /data/streampoc/c-7-a34/0_5/KSTREAM-JOINOTHER-0000000019-store/KSTREAM-JOINOTHER-0000000019-store-201708140520 at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:198) at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:165) at org.apache.kafka.streams.state.internals.Segment.openDB(Segment.java:40) at org.apache.kafka.streams.state.internals.Segments.getOrCreateSegment(Segments.java:86) at org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore.put(RocksDBSegmentedBytesStore.java:81) at org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore$1.restore(RocksDBSegmentedBytesStore.java:113) at org.apache.kafka.streams.processor.internals.StateRestorer.restore(StateRestorer.java:55) at org.apache.kafka.streams.processor.internals.StoreChangelogReader.processNext(StoreChangelogReader.java:216) at org.apache.kafka.streams.processor.internals.StoreChangelogReader.restorePartition(StoreChangelogReader.java:186) at org.apache.kafka.streams.processor.internals.StoreChangelogReader.restore(StoreChangelogReader.java:151) at org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:184) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:363) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:310) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297) at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078) at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) at org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:582) at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553) at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:527) Caused by: org.rocksdb.RocksDBException: at org.rocksdb.RocksDB.open(Native Method) at org.rocksdb.RocksDB.open(RocksDB.java:231) at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:191) ... 19 more 2017-08-14 11:15:55 INFO StreamThread:193 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-10] partition assignment took 1831 ms. current active tasks: [0_53, 0_38, 0_23, 0_8] current standby tasks: [] 2017-08-14 11:15:55 ERROR ConsumerCoordinator:269 - User provided listener org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener for group c-7-a34 failed on partition assignment org.apache.kafka.streams.errors.ProcessorStateException: Error opening store KSTREAM-JOINOTHER-0000000019-store-201708140520 at location /data/streampoc/c-7-a34/0_8/KSTREAM-JOINOTHER-0000000019-store/KSTREAM-JOINOTHER-0000000019-store-201708140520 at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:198) at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:165) at org.apache.kafka.streams.state.internals.Segment.openDB(Segment.java:40) at org.apache.kafka.streams.state.internals.Segments.getOrCreateSegment(Segments.java:86) at org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore.put(RocksDBSegmentedBytesStore.java:81) at org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore$1.restore(RocksDBSegmentedBytesStore.java:113) at org.apache.kafka.streams.processor.internals.StateRestorer.restore(StateRestorer.java:55) at org.apache.kafka.streams.processor.internals.StoreChangelogReader.processNext(StoreChangelogReader.java:216) at org.apache.kafka.streams.processor.internals.StoreChangelogReader.restorePartition(StoreChangelogReader.java:186) at org.apache.kafka.streams.processor.internals.StoreChangelogReader.restore(StoreChangelogReader.java:151) at org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:184) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:363) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:310) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297) at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078) at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) at org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:582) at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553) at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:527) Caused by: org.rocksdb.RocksDBException: at org.rocksdb.RocksDB.open(Native Method) at org.rocksdb.RocksDB.open(RocksDB.java:231) at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:191) ... 19 more 2017-08-14 11:15:55 INFO StreamThread:1040 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-10] Shutting down 2017-08-14 11:15:55 INFO StreamThread:1040 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-6] Shutting down 2017-08-14 11:15:55 INFO StreamThread:1040 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-5] Shutting down 2017-08-14 11:15:55 INFO StateDirectory:220 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-2] Deleting obsolete state directory 0_40 for task 0_40 as cleanup delay of 9223372036854775807 ms has passed 2017-08-14 11:15:55 INFO StateDirectory:220 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-15] Deleting obsolete state directory 0_24 for task 0_24 as cleanup delay of 9223372036854775807 ms has passed 2017-08-14 11:15:55 INFO StateDirectory:220 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-15] Deleting obsolete state directory 0_10 for task 0_10 as cleanup delay of 9223372036854775807 ms has passed 2017-08-14 11:15:55 INFO StateDirectory:220 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-15] Deleting obsolete state directory 0_22 for task 0_22 as cleanup delay of 9223372036854775807 ms has passed 2017-08-14 11:15:55 INFO StreamThread:1040 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-7] Shutting down 2017-08-14 11:15:55 INFO StateDirectory:220 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-4] Deleting obsolete state directory 0_54 for task 0_54 as cleanup delay of 9223372036854775807 ms has passed 2017-08-14 11:15:55 INFO KafkaProducer:972 - Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. 2017-08-14 11:15:55 INFO StreamThread:1407 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-5] Removing all active tasks [0_54, 0_39, 0_24, 0_9] 2017-08-14 11:15:55 INFO StreamThread:1421 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-5] Removing all standby tasks [] 2017-08-14 11:15:55 INFO StreamThread:1072 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-5] Stream thread shutdown complete 2017-08-14 11:15:55 WARN StreamThread:978 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-5] Unexpected state transition from ASSIGNING_PARTITIONS to DEAD. 2017-08-14 11:15:55 INFO KafkaStreams:229 - stream-client [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241] State transition from REBALANCING to PENDING_SHUTDOWN. 2017-08-14 11:15:55 INFO StreamThread:900 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-1] Informed thread to shut down 2017-08-14 11:15:55 INFO StreamThread:980 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-1] State transition from RUNNING to PENDING_SHUTDOWN. 2017-08-14 11:15:55 INFO StreamThread:900 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-2] Informed thread to shut down 2017-08-14 11:15:55 INFO StreamThread:980 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-2] State transition from RUNNING to PENDING_SHUTDOWN. 2017-08-14 11:15:55 INFO StreamThread:900 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-3] Informed thread to shut down 2017-08-14 11:15:55 INFO StreamThread:980 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-3] State transition from ASSIGNING_PARTITIONS to PENDING_SHUTDOWN. 2017-08-14 11:15:55 INFO StreamThread:900 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-4] Informed thread to shut down 2017-08-14 11:15:55 INFO StreamThread:980 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-4] State transition from RUNNING to PENDING_SHUTDOWN. 2017-08-14 11:15:55 INFO StreamThread:900 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-5] Informed thread to shut down 2017-08-14 11:15:55 WARN StreamThread:978 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-5] Unexpected state transition from DEAD to PENDING_SHUTDOWN. 2017-08-14 11:15:55 INFO StreamThread:900 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-6] Informed thread to shut down 2017-08-14 11:15:55 INFO StreamThread:980 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-6] State transition from ASSIGNING_PARTITIONS to PENDING_SHUTDOWN. 2017-08-14 11:15:55 INFO StreamThread:900 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-7] Informed thread to shut down 2017-08-14 11:15:55 INFO StreamThread:980 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-7] State transition from RUNNING to PENDING_SHUTDOWN. 2017-08-14 11:15:55 INFO StreamThread:900 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-8] Informed thread to shut down 2017-08-14 11:15:55 INFO StreamThread:980 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-8] State transition from ASSIGNING_PARTITIONS to PENDING_SHUTDOWN. 2017-08-14 11:15:55 INFO StreamThread:900 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-9] Informed thread to shut down 2017-08-14 11:15:55 INFO StreamThread:980 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-9] State transition from RUNNING to PENDING_SHUTDOWN. 2017-08-14 11:15:55 INFO StreamThread:900 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-10] Informed thread to shut down 2017-08-14 11:15:55 INFO StreamThread:980 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-10] State transition from ASSIGNING_PARTITIONS to PENDING_SHUTDOWN. 2017-08-14 11:15:55 INFO StreamThread:900 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-11] Informed thread to shut down 2017-08-14 11:15:55 INFO StreamThread:980 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-11] State transition from ASSIGNING_PARTITIONS to PENDING_SHUTDOWN. 2017-08-14 11:15:55 INFO StreamThread:900 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-12] Informed thread to shut down 2017-08-14 11:15:55 INFO StreamThread:980 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-12] State transition from RUNNING to PENDING_SHUTDOWN. 2017-08-14 11:15:55 INFO StreamThread:900 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-13] Informed thread to shut down 2017-08-14 11:15:55 INFO StreamThread:980 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-13] State transition from RUNNING to PENDING_SHUTDOWN. 2017-08-14 11:15:55 INFO StreamThread:900 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-14] Informed thread to shut down 2017-08-14 11:15:55 INFO StreamThread:980 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-14] State transition from RUNNING to PENDING_SHUTDOWN. 2017-08-14 11:15:55 INFO StreamThread:900 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-15] Informed thread to shut down 2017-08-14 11:15:55 INFO StreamThread:980 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-15] State transition from RUNNING to PENDING_SHUTDOWN. 2017-08-14 11:15:56 INFO StateDirectory:220 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-1] Deleting obsolete state directory 0_51 for task 0_51 as cleanup delay of 9223372036854775807 ms has passed 2017-08-14 11:15:56 INFO StateDirectory:220 - stream-thread [c-7-a34-2b29e007-c317-416a-9819-a869ddfc8241-StreamThread-1] De