hi all,

Faced this exception yesterday, any possible reasons for the same. At the
same time, one of the machines was restarted in my Kafka Streams cluster
and hence the job ended there.
Detailed exception trace is attached.

I am using Kafka 1.0.0.

2017-11-28 00:07:38 ERROR Kafka010Base:46 - Exception caught in thread
c-7-aq23-000647df-ff25-48de-b92f-02f43988353e-StreamThread-6
java.lang.IllegalStateException: Record's partition does not belong to this
partition-group.
at
org.apache.kafka.streams.processor.internals.PartitionGroup.numBuffered(PartitionGroup.java:156)
at
org.apache.kafka.streams.processor.internals.StreamTask.addRecords(StreamTask.java:545)
at
org.apache.kafka.streams.processor.internals.StreamThread.addRecordsToTasks(StreamThread.java:920)
at
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:821)
at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:774)
at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:744)

Regards,
-Sameer.
2017-11-28 00:07:37 ERROR Kafka010Base:46 - Exception caught in thread 
c-7-aq23-000647df-ff25-48de-b92f-02f43988353e-StreamThread-3
org.apache.kafka.streams.errors.StreamsException: Exception caught in process. 
taskId=0_4, processor=KSTREAM-SOURCE-0000000015, 
topic=c-7-aq23-KSTREAM-BRANCHCHILD-0000000008-repartition, partition=4, 
offset=7697520
        at 
org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:232)
        at 
org.apache.kafka.streams.processor.internals.AssignedTasks.process(AssignedTasks.java:403)
        at 
org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:317)
        at 
org.apache.kafka.streams.processor.internals.StreamThread.processAndMaybeCommit(StreamThread.java:942)
        at 
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:822)
        at 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:774)
        at 
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:744)
Caused by: org.apache.kafka.common.KafkaException: Cannot perform send because 
at least one previous transactional or idempotent request has failed with 
errors.
        at 
org.apache.kafka.clients.producer.internals.TransactionManager.failIfNotReadyForSend(TransactionManager.java:278)
        at 
org.apache.kafka.clients.producer.internals.TransactionManager.maybeAddPartitionToTransaction(TransactionManager.java:263)
        at 
org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:804)
        at 
org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:760)
        at 
org.apache.kafka.streams.processor.internals.RecordCollectorImpl.send(RecordCollectorImpl.java:100)
        at 
org.apache.kafka.streams.state.internals.StoreChangeLogger.logChange(StoreChangeLogger.java:59)
        at 
org.apache.kafka.streams.state.internals.ChangeLoggingWindowBytesStore.put(ChangeLoggingWindowBytesStore.java:68)
        at 
org.apache.kafka.streams.state.internals.ChangeLoggingWindowBytesStore.put(ChangeLoggingWindowBytesStore.java:33)
        at 
org.apache.kafka.streams.state.internals.MeteredWindowStore.put(MeteredWindowStore.java:96)
        at 
org.apache.kafka.streams.state.internals.MeteredWindowStore.put(MeteredWindowStore.java:89)
        at 
org.apache.kafka.streams.kstream.internals.KStreamJoinWindow$KStreamJoinWindowProcessor.process(KStreamJoinWindow.java:64)
        at 
org.apache.kafka.streams.processor.internals.ProcessorNode$1.run(ProcessorNode.java:46)
        at 
org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:208)
        at 
org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:124)
        at 
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:85)
        at 
org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:80)
        at 
org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:216)
        ... 6 more
Caused by: org.apache.kafka.common.errors.ProducerFencedException: Producer 
attempted an operation with an old epoch. Either there is a newer producer with 
the same transactionalId, or the producer's transaction has been expired by the 
broker.
2017-11-28 00:07:38 ERROR Kafka010Base:46 - Exception caught in thread 
c-7-aq23-000647df-ff25-48de-b92f-02f43988353e-StreamThread-9
java.lang.IllegalStateException: Record's partition does not belong to this 
partition-group.
        at 
org.apache.kafka.streams.processor.internals.PartitionGroup.numBuffered(PartitionGroup.java:156)
        at 
org.apache.kafka.streams.processor.internals.StreamTask.addRecords(StreamTask.java:545)
        at 
org.apache.kafka.streams.processor.internals.StreamThread.addRecordsToTasks(StreamThread.java:920)
        at 
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:821)
        at 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:774)
        at 
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:744)
2017-11-28 00:07:38 INFO  AdvertiserCountables:84 - (-aq23)AdvCountables key 
k=LI_1644295_20171127_C,v=2950
2017-11-28 00:07:38 INFO  AdvertiserCountables:84 - (-aq23)AdvCountables key 
k=LI_2421451_20171127_I,v=177960
2017-11-28 00:07:38 INFO  AdvertiserCountables:84 - (-aq23)AdvCountables key 
k=LI_1553638_20171127_I,v=49710
2017-11-28 00:07:38 INFO  AdvertiserCountables:84 - (-aq23)AdvCountables key 
k=LI_2301832_20171128_I,v=216
2017-11-28 00:07:38 INFO  AdvertiserCountables:84 - (-aq23)AdvCountables key 
k=LI_1532271_20171127_I,v=55769
2017-11-28 00:07:38 INFO  AdvertiserCountables:84 - (-aq23)AdvCountables key 
k=LI_1532278_20171128_I,v=935
2017-11-28 00:07:38 INFO  AdvertiserCountables:84 - (-aq23)AdvCountables key 
k=LI_1516166_20171127_I,v=62855
2017-11-28 00:07:38 ERROR Kafka010Base:46 - Exception caught in thread 
c-7-aq23-000647df-ff25-48de-b92f-02f43988353e-StreamThread-11
java.lang.IllegalStateException: Record's partition does not belong to this 
partition-group.
        at 
org.apache.kafka.streams.processor.internals.PartitionGroup.numBuffered(PartitionGroup.java:156)
        at 
org.apache.kafka.streams.processor.internals.StreamTask.addRecords(StreamTask.java:545)
        at 
org.apache.kafka.streams.processor.internals.StreamThread.addRecordsToTasks(StreamThread.java:920)
        at 
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:821)
        at 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:774)
        at 
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:744)
2017-11-28 00:07:38 ERROR Kafka010Base:46 - Exception caught in thread 
c-7-aq23-000647df-ff25-48de-b92f-02f43988353e-StreamThread-6
java.lang.IllegalStateException: Record's partition does not belong to this 
partition-group.
        at 
org.apache.kafka.streams.processor.internals.PartitionGroup.numBuffered(PartitionGroup.java:156)
        at 
org.apache.kafka.streams.processor.internals.StreamTask.addRecords(StreamTask.java:545)
        at 
org.apache.kafka.streams.processor.internals.StreamThread.addRecordsToTasks(StreamThread.java:920)
        at 
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:821)
        at 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:774)
        at 
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:744)
2017-11-28 00:07:56 ERROR Kafka010Base:46 - Exception caught in thread 
c-7-aq23-000647df-ff25-48de-b92f-02f43988353e-StreamThread-10
org.apache.kafka.streams.errors.ProcessorStateException: Error opening store 
ADV_LI-aq23 at location /data/streampoc/c-7-aq23/1_36/rocksdb/ADV_LI-aq23
        at 
org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:204)
        at 
org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:174)
        at 
org.apache.kafka.streams.state.internals.RocksDBStore.init(RocksDBStore.java:185)
        at 
org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.init(ChangeLoggingKeyValueBytesStore.java:40)
        at 
org.apache.kafka.streams.state.internals.CachingKeyValueStore.init(CachingKeyValueStore.java:59)
        at 
org.apache.kafka.streams.state.internals.InnerMeteredKeyValueStore.init(InnerMeteredKeyValueStore.java:160)
        at 
org.apache.kafka.streams.state.internals.MeteredKeyValueBytesStore.init(MeteredKeyValueBytesStore.java:104)
        at 
org.apache.kafka.streams.processor.internals.AbstractTask.initializeStateStores(AbstractTask.java:224)
        at 
org.apache.kafka.streams.processor.internals.StreamTask.initialize(StreamTask.java:163)
        at 
org.apache.kafka.streams.processor.internals.AssignedTasks.initializeNewTasks(AssignedTasks.java:122)
        at 
org.apache.kafka.streams.processor.internals.TaskManager.updateNewAndRestoringTasks(TaskManager.java:260)
        at 
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:813)
        at 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:774)
        at 
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:744)
Caused by: org.rocksdb.RocksDBException: While lock file: 
/data/streampoc/c-7-aq23/1_36/rocksdb/ADV_LI-aq23/LOCK: No locks available
        at org.rocksdb.RocksDB.open(Native Method)
        at org.rocksdb.RocksDB.open(RocksDB.java:231)
        at 
org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:197)
        ... 13 more
2017-11-28 00:08:00 ERROR Kafka010Base:46 - Exception caught in thread 
c-7-aq23-000647df-ff25-48de-b92f-02f43988353e-StreamThread-1
org.apache.kafka.streams.errors.StreamsException: Exception caught in process. 
taskId=0_37, processor=KSTREAM-SOURCE-0000000012, 
topic=c-7-aq23-KSTREAM-BRANCHCHILD-0000000007-repartition, partition=37, 
offset=24786550
        at 
org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:232)
        at 
org.apache.kafka.streams.processor.internals.AssignedTasks.process(AssignedTasks.java:403)
        at 
org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:317)
        at 
org.apache.kafka.streams.processor.internals.StreamThread.processAndMaybeCommit(StreamThread.java:942)
        at 
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:822)
        at 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:774)
        at 
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:744)
Caused by: org.apache.kafka.common.KafkaException: Cannot perform send because 
at least one previous transactional or idempotent request has failed with 
errors.
        at 
org.apache.kafka.clients.producer.internals.TransactionManager.failIfNotReadyForSend(TransactionManager.java:278)
        at 
org.apache.kafka.clients.producer.internals.TransactionManager.maybeAddPartitionToTransaction(TransactionManager.java:263)
        at 
org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:804)
        at 
org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:760)
        at 
org.apache.kafka.streams.processor.internals.RecordCollectorImpl.send(RecordCollectorImpl.java:100)
        at 
org.apache.kafka.streams.state.internals.StoreChangeLogger.logChange(StoreChangeLogger.java:59)
        at 
org.apache.kafka.streams.state.internals.ChangeLoggingWindowBytesStore.put(ChangeLoggingWindowBytesStore.java:68)
        at 
org.apache.kafka.streams.state.internals.ChangeLoggingWindowBytesStore.put(ChangeLoggingWindowBytesStore.java:33)
        at 
org.apache.kafka.streams.state.internals.MeteredWindowStore.put(MeteredWindowStore.java:96)
        at 
org.apache.kafka.streams.state.internals.MeteredWindowStore.put(MeteredWindowStore.java:89)
        at 
org.apache.kafka.streams.kstream.internals.KStreamJoinWindow$KStreamJoinWindowProcessor.process(KStreamJoinWindow.java:64)
        at 
org.apache.kafka.streams.processor.internals.ProcessorNode$1.run(ProcessorNode.java:46)
        at 
org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:208)
        at 
org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:124)
        at 
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:85)
        at 
org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:80)
        at 
org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:216)
        ... 6 more
Caused by: org.apache.kafka.common.errors.ProducerFencedException: Producer 
attempted an operation with an old epoch. Either there is a newer producer with 
the same transactionalId, or the producer's transaction has been expired by the 
broker.

Reply via email to