hi all, Faced this exception yesterday, any possible reasons for the same. At the same time, one of the machines was restarted in my Kafka Streams cluster and hence the job ended there. Detailed exception trace is attached.
I am using Kafka 1.0.0. 2017-11-28 00:07:38 ERROR Kafka010Base:46 - Exception caught in thread c-7-aq23-000647df-ff25-48de-b92f-02f43988353e-StreamThread-6 java.lang.IllegalStateException: Record's partition does not belong to this partition-group. at org.apache.kafka.streams.processor.internals.PartitionGroup.numBuffered(PartitionGroup.java:156) at org.apache.kafka.streams.processor.internals.StreamTask.addRecords(StreamTask.java:545) at org.apache.kafka.streams.processor.internals.StreamThread.addRecordsToTasks(StreamThread.java:920) at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:821) at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:774) at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:744) Regards, -Sameer.
2017-11-28 00:07:37 ERROR Kafka010Base:46 - Exception caught in thread c-7-aq23-000647df-ff25-48de-b92f-02f43988353e-StreamThread-3 org.apache.kafka.streams.errors.StreamsException: Exception caught in process. taskId=0_4, processor=KSTREAM-SOURCE-0000000015, topic=c-7-aq23-KSTREAM-BRANCHCHILD-0000000008-repartition, partition=4, offset=7697520 at org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:232) at org.apache.kafka.streams.processor.internals.AssignedTasks.process(AssignedTasks.java:403) at org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:317) at org.apache.kafka.streams.processor.internals.StreamThread.processAndMaybeCommit(StreamThread.java:942) at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:822) at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:774) at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:744) Caused by: org.apache.kafka.common.KafkaException: Cannot perform send because at least one previous transactional or idempotent request has failed with errors. at org.apache.kafka.clients.producer.internals.TransactionManager.failIfNotReadyForSend(TransactionManager.java:278) at org.apache.kafka.clients.producer.internals.TransactionManager.maybeAddPartitionToTransaction(TransactionManager.java:263) at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:804) at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:760) at org.apache.kafka.streams.processor.internals.RecordCollectorImpl.send(RecordCollectorImpl.java:100) at org.apache.kafka.streams.state.internals.StoreChangeLogger.logChange(StoreChangeLogger.java:59) at org.apache.kafka.streams.state.internals.ChangeLoggingWindowBytesStore.put(ChangeLoggingWindowBytesStore.java:68) at org.apache.kafka.streams.state.internals.ChangeLoggingWindowBytesStore.put(ChangeLoggingWindowBytesStore.java:33) at org.apache.kafka.streams.state.internals.MeteredWindowStore.put(MeteredWindowStore.java:96) at org.apache.kafka.streams.state.internals.MeteredWindowStore.put(MeteredWindowStore.java:89) at org.apache.kafka.streams.kstream.internals.KStreamJoinWindow$KStreamJoinWindowProcessor.process(KStreamJoinWindow.java:64) at org.apache.kafka.streams.processor.internals.ProcessorNode$1.run(ProcessorNode.java:46) at org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:208) at org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:124) at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:85) at org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:80) at org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:216) ... 6 more Caused by: org.apache.kafka.common.errors.ProducerFencedException: Producer attempted an operation with an old epoch. Either there is a newer producer with the same transactionalId, or the producer's transaction has been expired by the broker. 2017-11-28 00:07:38 ERROR Kafka010Base:46 - Exception caught in thread c-7-aq23-000647df-ff25-48de-b92f-02f43988353e-StreamThread-9 java.lang.IllegalStateException: Record's partition does not belong to this partition-group. at org.apache.kafka.streams.processor.internals.PartitionGroup.numBuffered(PartitionGroup.java:156) at org.apache.kafka.streams.processor.internals.StreamTask.addRecords(StreamTask.java:545) at org.apache.kafka.streams.processor.internals.StreamThread.addRecordsToTasks(StreamThread.java:920) at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:821) at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:774) at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:744) 2017-11-28 00:07:38 INFO AdvertiserCountables:84 - (-aq23)AdvCountables key k=LI_1644295_20171127_C,v=2950 2017-11-28 00:07:38 INFO AdvertiserCountables:84 - (-aq23)AdvCountables key k=LI_2421451_20171127_I,v=177960 2017-11-28 00:07:38 INFO AdvertiserCountables:84 - (-aq23)AdvCountables key k=LI_1553638_20171127_I,v=49710 2017-11-28 00:07:38 INFO AdvertiserCountables:84 - (-aq23)AdvCountables key k=LI_2301832_20171128_I,v=216 2017-11-28 00:07:38 INFO AdvertiserCountables:84 - (-aq23)AdvCountables key k=LI_1532271_20171127_I,v=55769 2017-11-28 00:07:38 INFO AdvertiserCountables:84 - (-aq23)AdvCountables key k=LI_1532278_20171128_I,v=935 2017-11-28 00:07:38 INFO AdvertiserCountables:84 - (-aq23)AdvCountables key k=LI_1516166_20171127_I,v=62855 2017-11-28 00:07:38 ERROR Kafka010Base:46 - Exception caught in thread c-7-aq23-000647df-ff25-48de-b92f-02f43988353e-StreamThread-11 java.lang.IllegalStateException: Record's partition does not belong to this partition-group. at org.apache.kafka.streams.processor.internals.PartitionGroup.numBuffered(PartitionGroup.java:156) at org.apache.kafka.streams.processor.internals.StreamTask.addRecords(StreamTask.java:545) at org.apache.kafka.streams.processor.internals.StreamThread.addRecordsToTasks(StreamThread.java:920) at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:821) at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:774) at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:744) 2017-11-28 00:07:38 ERROR Kafka010Base:46 - Exception caught in thread c-7-aq23-000647df-ff25-48de-b92f-02f43988353e-StreamThread-6 java.lang.IllegalStateException: Record's partition does not belong to this partition-group. at org.apache.kafka.streams.processor.internals.PartitionGroup.numBuffered(PartitionGroup.java:156) at org.apache.kafka.streams.processor.internals.StreamTask.addRecords(StreamTask.java:545) at org.apache.kafka.streams.processor.internals.StreamThread.addRecordsToTasks(StreamThread.java:920) at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:821) at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:774) at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:744) 2017-11-28 00:07:56 ERROR Kafka010Base:46 - Exception caught in thread c-7-aq23-000647df-ff25-48de-b92f-02f43988353e-StreamThread-10 org.apache.kafka.streams.errors.ProcessorStateException: Error opening store ADV_LI-aq23 at location /data/streampoc/c-7-aq23/1_36/rocksdb/ADV_LI-aq23 at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:204) at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:174) at org.apache.kafka.streams.state.internals.RocksDBStore.init(RocksDBStore.java:185) at org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.init(ChangeLoggingKeyValueBytesStore.java:40) at org.apache.kafka.streams.state.internals.CachingKeyValueStore.init(CachingKeyValueStore.java:59) at org.apache.kafka.streams.state.internals.InnerMeteredKeyValueStore.init(InnerMeteredKeyValueStore.java:160) at org.apache.kafka.streams.state.internals.MeteredKeyValueBytesStore.init(MeteredKeyValueBytesStore.java:104) at org.apache.kafka.streams.processor.internals.AbstractTask.initializeStateStores(AbstractTask.java:224) at org.apache.kafka.streams.processor.internals.StreamTask.initialize(StreamTask.java:163) at org.apache.kafka.streams.processor.internals.AssignedTasks.initializeNewTasks(AssignedTasks.java:122) at org.apache.kafka.streams.processor.internals.TaskManager.updateNewAndRestoringTasks(TaskManager.java:260) at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:813) at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:774) at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:744) Caused by: org.rocksdb.RocksDBException: While lock file: /data/streampoc/c-7-aq23/1_36/rocksdb/ADV_LI-aq23/LOCK: No locks available at org.rocksdb.RocksDB.open(Native Method) at org.rocksdb.RocksDB.open(RocksDB.java:231) at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:197) ... 13 more 2017-11-28 00:08:00 ERROR Kafka010Base:46 - Exception caught in thread c-7-aq23-000647df-ff25-48de-b92f-02f43988353e-StreamThread-1 org.apache.kafka.streams.errors.StreamsException: Exception caught in process. taskId=0_37, processor=KSTREAM-SOURCE-0000000012, topic=c-7-aq23-KSTREAM-BRANCHCHILD-0000000007-repartition, partition=37, offset=24786550 at org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:232) at org.apache.kafka.streams.processor.internals.AssignedTasks.process(AssignedTasks.java:403) at org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:317) at org.apache.kafka.streams.processor.internals.StreamThread.processAndMaybeCommit(StreamThread.java:942) at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:822) at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:774) at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:744) Caused by: org.apache.kafka.common.KafkaException: Cannot perform send because at least one previous transactional or idempotent request has failed with errors. at org.apache.kafka.clients.producer.internals.TransactionManager.failIfNotReadyForSend(TransactionManager.java:278) at org.apache.kafka.clients.producer.internals.TransactionManager.maybeAddPartitionToTransaction(TransactionManager.java:263) at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:804) at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:760) at org.apache.kafka.streams.processor.internals.RecordCollectorImpl.send(RecordCollectorImpl.java:100) at org.apache.kafka.streams.state.internals.StoreChangeLogger.logChange(StoreChangeLogger.java:59) at org.apache.kafka.streams.state.internals.ChangeLoggingWindowBytesStore.put(ChangeLoggingWindowBytesStore.java:68) at org.apache.kafka.streams.state.internals.ChangeLoggingWindowBytesStore.put(ChangeLoggingWindowBytesStore.java:33) at org.apache.kafka.streams.state.internals.MeteredWindowStore.put(MeteredWindowStore.java:96) at org.apache.kafka.streams.state.internals.MeteredWindowStore.put(MeteredWindowStore.java:89) at org.apache.kafka.streams.kstream.internals.KStreamJoinWindow$KStreamJoinWindowProcessor.process(KStreamJoinWindow.java:64) at org.apache.kafka.streams.processor.internals.ProcessorNode$1.run(ProcessorNode.java:46) at org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:208) at org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:124) at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:85) at org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:80) at org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:216) ... 6 more Caused by: org.apache.kafka.common.errors.ProducerFencedException: Producer attempted an operation with an old epoch. Either there is a newer producer with the same transactionalId, or the producer's transaction has been expired by the broker.