hi all,
Faced this exception yesterday, any possible reasons for the same. At the
same time, one of the machines was restarted in my Kafka Streams cluster
and hence the job ended there.
Detailed exception trace is attached.
I am using Kafka 1.0.0.
2017-11-28 00:07:38 ERROR Kafka010Base:46 - Exception caught in thread
c-7-aq23-000647df-ff25-48de-b92f-02f43988353e-StreamThread-6
java.lang.IllegalStateException: Record's partition does not belong to this
partition-group.
at
org.apache.kafka.streams.processor.internals.PartitionGroup.numBuffered(PartitionGroup.java:156)
at
org.apache.kafka.streams.processor.internals.StreamTask.addRecords(StreamTask.java:545)
at
org.apache.kafka.streams.processor.internals.StreamThread.addRecordsToTasks(StreamThread.java:920)
at
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:821)
at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:774)
at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:744)
Regards,
-Sameer.
2017-11-28 00:07:37 ERROR Kafka010Base:46 - Exception caught in thread
c-7-aq23-000647df-ff25-48de-b92f-02f43988353e-StreamThread-3
org.apache.kafka.streams.errors.StreamsException: Exception caught in process.
taskId=0_4, processor=KSTREAM-SOURCE-0000000015,
topic=c-7-aq23-KSTREAM-BRANCHCHILD-0000000008-repartition, partition=4,
offset=7697520
at
org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:232)
at
org.apache.kafka.streams.processor.internals.AssignedTasks.process(AssignedTasks.java:403)
at
org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:317)
at
org.apache.kafka.streams.processor.internals.StreamThread.processAndMaybeCommit(StreamThread.java:942)
at
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:822)
at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:774)
at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:744)
Caused by: org.apache.kafka.common.KafkaException: Cannot perform send because
at least one previous transactional or idempotent request has failed with
errors.
at
org.apache.kafka.clients.producer.internals.TransactionManager.failIfNotReadyForSend(TransactionManager.java:278)
at
org.apache.kafka.clients.producer.internals.TransactionManager.maybeAddPartitionToTransaction(TransactionManager.java:263)
at
org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:804)
at
org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:760)
at
org.apache.kafka.streams.processor.internals.RecordCollectorImpl.send(RecordCollectorImpl.java:100)
at
org.apache.kafka.streams.state.internals.StoreChangeLogger.logChange(StoreChangeLogger.java:59)
at
org.apache.kafka.streams.state.internals.ChangeLoggingWindowBytesStore.put(ChangeLoggingWindowBytesStore.java:68)
at
org.apache.kafka.streams.state.internals.ChangeLoggingWindowBytesStore.put(ChangeLoggingWindowBytesStore.java:33)
at
org.apache.kafka.streams.state.internals.MeteredWindowStore.put(MeteredWindowStore.java:96)
at
org.apache.kafka.streams.state.internals.MeteredWindowStore.put(MeteredWindowStore.java:89)
at
org.apache.kafka.streams.kstream.internals.KStreamJoinWindow$KStreamJoinWindowProcessor.process(KStreamJoinWindow.java:64)
at
org.apache.kafka.streams.processor.internals.ProcessorNode$1.run(ProcessorNode.java:46)
at
org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:208)
at
org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:124)
at
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:85)
at
org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:80)
at
org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:216)
... 6 more
Caused by: org.apache.kafka.common.errors.ProducerFencedException: Producer
attempted an operation with an old epoch. Either there is a newer producer with
the same transactionalId, or the producer's transaction has been expired by the
broker.
2017-11-28 00:07:38 ERROR Kafka010Base:46 - Exception caught in thread
c-7-aq23-000647df-ff25-48de-b92f-02f43988353e-StreamThread-9
java.lang.IllegalStateException: Record's partition does not belong to this
partition-group.
at
org.apache.kafka.streams.processor.internals.PartitionGroup.numBuffered(PartitionGroup.java:156)
at
org.apache.kafka.streams.processor.internals.StreamTask.addRecords(StreamTask.java:545)
at
org.apache.kafka.streams.processor.internals.StreamThread.addRecordsToTasks(StreamThread.java:920)
at
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:821)
at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:774)
at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:744)
2017-11-28 00:07:38 INFO AdvertiserCountables:84 - (-aq23)AdvCountables key
k=LI_1644295_20171127_C,v=2950
2017-11-28 00:07:38 INFO AdvertiserCountables:84 - (-aq23)AdvCountables key
k=LI_2421451_20171127_I,v=177960
2017-11-28 00:07:38 INFO AdvertiserCountables:84 - (-aq23)AdvCountables key
k=LI_1553638_20171127_I,v=49710
2017-11-28 00:07:38 INFO AdvertiserCountables:84 - (-aq23)AdvCountables key
k=LI_2301832_20171128_I,v=216
2017-11-28 00:07:38 INFO AdvertiserCountables:84 - (-aq23)AdvCountables key
k=LI_1532271_20171127_I,v=55769
2017-11-28 00:07:38 INFO AdvertiserCountables:84 - (-aq23)AdvCountables key
k=LI_1532278_20171128_I,v=935
2017-11-28 00:07:38 INFO AdvertiserCountables:84 - (-aq23)AdvCountables key
k=LI_1516166_20171127_I,v=62855
2017-11-28 00:07:38 ERROR Kafka010Base:46 - Exception caught in thread
c-7-aq23-000647df-ff25-48de-b92f-02f43988353e-StreamThread-11
java.lang.IllegalStateException: Record's partition does not belong to this
partition-group.
at
org.apache.kafka.streams.processor.internals.PartitionGroup.numBuffered(PartitionGroup.java:156)
at
org.apache.kafka.streams.processor.internals.StreamTask.addRecords(StreamTask.java:545)
at
org.apache.kafka.streams.processor.internals.StreamThread.addRecordsToTasks(StreamThread.java:920)
at
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:821)
at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:774)
at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:744)
2017-11-28 00:07:38 ERROR Kafka010Base:46 - Exception caught in thread
c-7-aq23-000647df-ff25-48de-b92f-02f43988353e-StreamThread-6
java.lang.IllegalStateException: Record's partition does not belong to this
partition-group.
at
org.apache.kafka.streams.processor.internals.PartitionGroup.numBuffered(PartitionGroup.java:156)
at
org.apache.kafka.streams.processor.internals.StreamTask.addRecords(StreamTask.java:545)
at
org.apache.kafka.streams.processor.internals.StreamThread.addRecordsToTasks(StreamThread.java:920)
at
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:821)
at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:774)
at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:744)
2017-11-28 00:07:56 ERROR Kafka010Base:46 - Exception caught in thread
c-7-aq23-000647df-ff25-48de-b92f-02f43988353e-StreamThread-10
org.apache.kafka.streams.errors.ProcessorStateException: Error opening store
ADV_LI-aq23 at location /data/streampoc/c-7-aq23/1_36/rocksdb/ADV_LI-aq23
at
org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:204)
at
org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:174)
at
org.apache.kafka.streams.state.internals.RocksDBStore.init(RocksDBStore.java:185)
at
org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.init(ChangeLoggingKeyValueBytesStore.java:40)
at
org.apache.kafka.streams.state.internals.CachingKeyValueStore.init(CachingKeyValueStore.java:59)
at
org.apache.kafka.streams.state.internals.InnerMeteredKeyValueStore.init(InnerMeteredKeyValueStore.java:160)
at
org.apache.kafka.streams.state.internals.MeteredKeyValueBytesStore.init(MeteredKeyValueBytesStore.java:104)
at
org.apache.kafka.streams.processor.internals.AbstractTask.initializeStateStores(AbstractTask.java:224)
at
org.apache.kafka.streams.processor.internals.StreamTask.initialize(StreamTask.java:163)
at
org.apache.kafka.streams.processor.internals.AssignedTasks.initializeNewTasks(AssignedTasks.java:122)
at
org.apache.kafka.streams.processor.internals.TaskManager.updateNewAndRestoringTasks(TaskManager.java:260)
at
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:813)
at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:774)
at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:744)
Caused by: org.rocksdb.RocksDBException: While lock file:
/data/streampoc/c-7-aq23/1_36/rocksdb/ADV_LI-aq23/LOCK: No locks available
at org.rocksdb.RocksDB.open(Native Method)
at org.rocksdb.RocksDB.open(RocksDB.java:231)
at
org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:197)
... 13 more
2017-11-28 00:08:00 ERROR Kafka010Base:46 - Exception caught in thread
c-7-aq23-000647df-ff25-48de-b92f-02f43988353e-StreamThread-1
org.apache.kafka.streams.errors.StreamsException: Exception caught in process.
taskId=0_37, processor=KSTREAM-SOURCE-0000000012,
topic=c-7-aq23-KSTREAM-BRANCHCHILD-0000000007-repartition, partition=37,
offset=24786550
at
org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:232)
at
org.apache.kafka.streams.processor.internals.AssignedTasks.process(AssignedTasks.java:403)
at
org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:317)
at
org.apache.kafka.streams.processor.internals.StreamThread.processAndMaybeCommit(StreamThread.java:942)
at
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:822)
at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:774)
at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:744)
Caused by: org.apache.kafka.common.KafkaException: Cannot perform send because
at least one previous transactional or idempotent request has failed with
errors.
at
org.apache.kafka.clients.producer.internals.TransactionManager.failIfNotReadyForSend(TransactionManager.java:278)
at
org.apache.kafka.clients.producer.internals.TransactionManager.maybeAddPartitionToTransaction(TransactionManager.java:263)
at
org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:804)
at
org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:760)
at
org.apache.kafka.streams.processor.internals.RecordCollectorImpl.send(RecordCollectorImpl.java:100)
at
org.apache.kafka.streams.state.internals.StoreChangeLogger.logChange(StoreChangeLogger.java:59)
at
org.apache.kafka.streams.state.internals.ChangeLoggingWindowBytesStore.put(ChangeLoggingWindowBytesStore.java:68)
at
org.apache.kafka.streams.state.internals.ChangeLoggingWindowBytesStore.put(ChangeLoggingWindowBytesStore.java:33)
at
org.apache.kafka.streams.state.internals.MeteredWindowStore.put(MeteredWindowStore.java:96)
at
org.apache.kafka.streams.state.internals.MeteredWindowStore.put(MeteredWindowStore.java:89)
at
org.apache.kafka.streams.kstream.internals.KStreamJoinWindow$KStreamJoinWindowProcessor.process(KStreamJoinWindow.java:64)
at
org.apache.kafka.streams.processor.internals.ProcessorNode$1.run(ProcessorNode.java:46)
at
org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:208)
at
org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:124)
at
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:85)
at
org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:80)
at
org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:216)
... 6 more
Caused by: org.apache.kafka.common.errors.ProducerFencedException: Producer
attempted an operation with an old epoch. Either there is a newer producer with
the same transactionalId, or the producer's transaction has been expired by the
broker.
