Hi,
I again received this exception while running my streams app. I am using
Kafka 11.0.1. After restarting my app, this error got fixed.
I guess this might be due to bad network. Any pointers. Any config wherein
I can configure it for retries.
Exception trace is attached.
Regards,
-Sameer.
2017-09-26 13:27:42 INFO ClassPathXmlApplicationContext:513 - Refreshing
org.springframework.context.support.ClassPathXmlApplicationContext@6aa8ceb6:
startup date [Tue Sep 26 13:27:42 IST 2017]; root of context hierarchy
2017-09-26 13:27:42 INFO XmlBeanDefinitionReader:316 - Loading XML bean
definitions from class path resource [application-context.xml]
2017-09-26 13:27:43 INFO CountableImprClicksWriter2SpecLI:104 - State Store
Dir: /data/streampoc/
2017-09-26 13:27:43 INFO CountableImprClicksWriter2SpecLI:105 - Run Counter:
-d11359
2017-09-26 13:27:44 INFO Kafka10Base:50 - KafkaStreams processID:
83a35a1f-bc35-47dc-b87a-93eb5e1599b9
StreamsThread appId: c-7-d11359
StreamsThread clientId:
c-7-d11359-83a35a1f-bc35-47dc-b87a-93eb5e1599b9
StreamsThread threadId:
c-7-d11359-83a35a1f-bc35-47dc-b87a-93eb5e1599b9-StreamThread-1
Active tasks:
Running:
Suspended:
Restoring:
New:
Standby tasks:
Running:
Suspended:
Restoring:
New:
StreamsThread appId: c-7-d11359
StreamsThread clientId:
c-7-d11359-83a35a1f-bc35-47dc-b87a-93eb5e1599b9
StreamsThread threadId:
c-7-d11359-83a35a1f-bc35-47dc-b87a-93eb5e1599b9-StreamThread-2
Active tasks:
Running:
Suspended:
Restoring:
New:
Standby tasks:
Running:
Suspended:
Restoring:
New:
StreamsThread appId: c-7-d11359
StreamsThread clientId:
c-7-d11359-83a35a1f-bc35-47dc-b87a-93eb5e1599b9
StreamsThread threadId:
c-7-d11359-83a35a1f-bc35-47dc-b87a-93eb5e1599b9-StreamThread-3
Active tasks:
Running:
Suspended:
Restoring:
New:
Standby tasks:
Running:
Suspended:
Restoring:
New:
StreamsThread appId: c-7-d11359
StreamsThread clientId:
c-7-d11359-83a35a1f-bc35-47dc-b87a-93eb5e1599b9
StreamsThread threadId:
c-7-d11359-83a35a1f-bc35-47dc-b87a-93eb5e1599b9-StreamThread-4
Active tasks:
Running:
Suspended:
Restoring:
New:
Standby tasks:
Running:
Suspended:
Restoring:
New:
StreamsThread appId: c-7-d11359
StreamsThread clientId:
c-7-d11359-83a35a1f-bc35-47dc-b87a-93eb5e1599b9
StreamsThread threadId:
c-7-d11359-83a35a1f-bc35-47dc-b87a-93eb5e1599b9-StreamThread-5
Active tasks:
Running:
Suspended:
Restoring:
New:
Standby tasks:
Running:
Suspended:
Restoring:
New:
StreamsThread appId: c-7-d11359
StreamsThread clientId:
c-7-d11359-83a35a1f-bc35-47dc-b87a-93eb5e1599b9
StreamsThread threadId:
c-7-d11359-83a35a1f-bc35-47dc-b87a-93eb5e1599b9-StreamThread-6
Active tasks:
Running:
Suspended:
Restoring:
New:
Standby tasks:
Running:
Suspended:
Restoring:
New:
StreamsThread appId: c-7-d11359
StreamsThread clientId:
c-7-d11359-83a35a1f-bc35-47dc-b87a-93eb5e1599b9
StreamsThread threadId:
c-7-d11359-83a35a1f-bc35-47dc-b87a-93eb5e1599b9-StreamThread-7
Active tasks:
Running:
Suspended:
Restoring:
New:
Standby tasks:
Running:
Suspended:
Restoring:
New:
StreamsThread appId: c-7-d11359
StreamsThread clientId:
c-7-d11359-83a35a1f-bc35-47dc-b87a-93eb5e1599b9
StreamsThread threadId:
c-7-d11359-83a35a1f-bc35-47dc-b87a-93eb5e1599b9-StreamThread-8
Active tasks:
Running:
Suspended:
Restoring:
New:
Standby tasks:
Running:
Suspended:
Restoring:
New:
StreamsThread appId: c-7-d11359
StreamsThread clientId:
c-7-d11359-83a35a1f-bc35-47dc-b87a-93eb5e1599b9
StreamsThread threadId:
c-7-d11359-83a35a1f-bc35-47dc-b87a-93eb5e1599b9-StreamThread-9
Active tasks:
Running:
Suspended:
Restoring:
New:
Standby tasks:
Running:
Suspended:
Restoring:
New:
StreamsThread appId: c-7-d11359
StreamsThread clientId:
c-7-d11359-83a35a1f-bc35-47dc-b87a-93eb5e1599b9
StreamsThread threadId:
c-7-d11359-83a35a1f-bc35-47dc-b87a-93eb5e1599b9-StreamThread-10
Active tasks:
Running:
Suspended:
Restoring:
New:
Standby tasks:
Running:
Suspended:
Restoring:
New:
StreamsThread appId: c-7-d11359
StreamsThread clientId:
c-7-d11359-83a35a1f-bc35-47dc-b87a-93eb5e1599b9
StreamsThread threadId:
c-7-d11359-83a35a1f-bc35-47dc-b87a-93eb5e1599b9-StreamThread-11
Active tasks:
Running:
Suspended:
Restoring:
New:
Standby tasks:
Running:
Suspended:
Restoring:
New:
StreamsThread appId: c-7-d11359
StreamsThread clientId:
c-7-d11359-83a35a1f-bc35-47dc-b87a-93eb5e1599b9
StreamsThread threadId:
c-7-d11359-83a35a1f-bc35-47dc-b87a-93eb5e1599b9-StreamThread-12
Active tasks:
Running:
Suspended:
Restoring:
New:
Standby tasks:
Running:
Suspended:
Restoring:
New:
StreamsThread appId: c-7-d11359
StreamsThread clientId:
c-7-d11359-83a35a1f-bc35-47dc-b87a-93eb5e1599b9
StreamsThread threadId:
c-7-d11359-83a35a1f-bc35-47dc-b87a-93eb5e1599b9-StreamThread-13
Active tasks:
Running:
Suspended:
Restoring:
New:
Standby tasks:
Running:
Suspended:
Restoring:
New:
StreamsThread appId: c-7-d11359
StreamsThread clientId:
c-7-d11359-83a35a1f-bc35-47dc-b87a-93eb5e1599b9
StreamsThread threadId:
c-7-d11359-83a35a1f-bc35-47dc-b87a-93eb5e1599b9-StreamThread-14
Active tasks:
Running:
Suspended:
Restoring:
New:
Standby tasks:
Running:
Suspended:
Restoring:
New:
StreamsThread appId: c-7-d11359
StreamsThread clientId:
c-7-d11359-83a35a1f-bc35-47dc-b87a-93eb5e1599b9
StreamsThread threadId:
c-7-d11359-83a35a1f-bc35-47dc-b87a-93eb5e1599b9-StreamThread-15
Active tasks:
Running:
Suspended:
Restoring:
New:
Standby tasks:
Running:
Suspended:
Restoring:
New:
2017-09-26 13:27:44 INFO CountableImprClicksWriter2SpecLI:269 - Store Id:####
ADV_LI-d11359
2017-09-26 13:28:06 ERROR Kafka10Base:45 - Exception caught in thread
c-7-d11359-83a35a1f-bc35-47dc-b87a-93eb5e1599b9-StreamThread-2
org.apache.kafka.streams.errors.StreamsException: Exception caught in process.
taskId=0_3, processor=KSTREAM-SOURCE-0000000000, topic=small_50min,
partition=3, offset=2392
at
org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:202)
at
org.apache.kafka.streams.processor.internals.AssignedTasks$2.apply(AssignedTasks.java:342)
at
org.apache.kafka.streams.processor.internals.AssignedTasks.applyToRunningTasks(AssignedTasks.java:415)
at
org.apache.kafka.streams.processor.internals.AssignedTasks.process(AssignedTasks.java:334)
at
org.apache.kafka.streams.processor.internals.StreamThread.processAndPunctuate(StreamThread.java:624)
at
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:513)
at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:480)
at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:457)
Caused by: org.apache.kafka.streams.errors.StreamsException: task [0_3]
exception caught when producing
at
org.apache.kafka.streams.processor.internals.RecordCollectorImpl.checkForException(RecordCollectorImpl.java:136)
at
org.apache.kafka.streams.processor.internals.RecordCollectorImpl.send(RecordCollectorImpl.java:87)
at
org.apache.kafka.streams.processor.internals.RecordCollectorImpl.send(RecordCollectorImpl.java:76)
at
org.apache.kafka.streams.processor.internals.SinkNode.process(SinkNode.java:87)
at
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:82)
at
org.apache.kafka.streams.kstream.internals.KStreamFilter$KStreamFilterProcessor.process(KStreamFilter.java:43)
at
org.apache.kafka.streams.processor.internals.ProcessorNode$1.run(ProcessorNode.java:45)
at
org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:201)
at
org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:131)
at
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:82)
at
org.apache.kafka.streams.kstream.internals.KStreamPassThrough$KStreamPassThroughProcessor.process(KStreamPassThrough.java:33)
at
org.apache.kafka.streams.processor.internals.ProcessorNode$1.run(ProcessorNode.java:45)
at
org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:201)
at
org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:131)
at
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:96)
at
org.apache.kafka.streams.kstream.internals.KStreamBranch$KStreamBranchProcessor.process(KStreamBranch.java:45)
at
org.apache.kafka.streams.processor.internals.ProcessorNode$1.run(ProcessorNode.java:45)
at
org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:201)
at
org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:131)
at
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:82)
at
org.apache.kafka.streams.kstream.internals.KStreamMap$KStreamMapProcessor.process(KStreamMap.java:42)
at
org.apache.kafka.streams.processor.internals.ProcessorNode$1.run(ProcessorNode.java:45)
at
org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:201)
at
org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:131)
at
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:82)
at
org.apache.kafka.streams.kstream.internals.KStreamFilter$KStreamFilterProcessor.process(KStreamFilter.java:43)
at
org.apache.kafka.streams.processor.internals.ProcessorNode$1.run(ProcessorNode.java:45)
at
org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:201)
at
org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:131)
at
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:82)
at
org.apache.kafka.streams.kstream.internals.KStreamFlatMap$KStreamFlatMapProcessor.process(KStreamFlatMap.java:42)
at
org.apache.kafka.streams.processor.internals.ProcessorNode$1.run(ProcessorNode.java:45)
at
org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:201)
at
org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:131)
at
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:82)
at
org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:80)
at
org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:188)
... 7 more
Caused by: org.apache.kafka.common.errors.OutOfOrderSequenceException: The
broker received an out of order sequence number
2017-09-26 13:28:06 ERROR Kafka10Base:45 - Exception caught in thread
c-7-d11359-83a35a1f-bc35-47dc-b87a-93eb5e1599b9-StreamThread-4
org.apache.kafka.streams.errors.StreamsException: Exception caught in process.
taskId=0_18, processor=KSTREAM-SOURCE-0000000000, topic=small_50min,
partition=18, offset=828
at
org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:202)
at
org.apache.kafka.streams.processor.internals.AssignedTasks$2.apply(AssignedTasks.java:342)
at
org.apache.kafka.streams.processor.internals.AssignedTasks.applyToRunningTasks(AssignedTasks.java:415)
at
org.apache.kafka.streams.processor.internals.AssignedTasks.process(AssignedTasks.java:334)
at
org.apache.kafka.streams.processor.internals.StreamThread.processAndPunctuate(StreamThread.java:624)
at
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:513)
at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:480)
at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:457)
Caused by: org.apache.kafka.common.KafkaException: Cannot perform send because
at least one previous transactional or idempotent request has failed with
errors.
at
org.apache.kafka.clients.producer.internals.TransactionManager.failIfNotReadyForSend(TransactionManager.java:248)
at
org.apache.kafka.clients.producer.internals.TransactionManager.maybeAddPartitionToTransaction(TransactionManager.java:233)
at
org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:789)
at
org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:745)
at
org.apache.kafka.streams.processor.internals.RecordCollectorImpl.send(RecordCollectorImpl.java:98)
at
org.apache.kafka.streams.processor.internals.RecordCollectorImpl.send(RecordCollectorImpl.java:76)
at
org.apache.kafka.streams.processor.internals.SinkNode.process(SinkNode.java:87)
at
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:82)
at
org.apache.kafka.streams.kstream.internals.KStreamFilter$KStreamFilterProcessor.process(KStreamFilter.java:43)
at
org.apache.kafka.streams.processor.internals.ProcessorNode$1.run(ProcessorNode.java:45)
at
org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:201)
at
org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:131)
at
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:82)
at
org.apache.kafka.streams.kstream.internals.KStreamPassThrough$KStreamPassThroughProcessor.process(KStreamPassThrough.java:33)
at
org.apache.kafka.streams.processor.internals.ProcessorNode$1.run(ProcessorNode.java:45)
at
org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:201)
at
org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:131)
at
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:96)
at
org.apache.kafka.streams.kstream.internals.KStreamBranch$KStreamBranchProcessor.process(KStreamBranch.java:45)
at
org.apache.kafka.streams.processor.internals.ProcessorNode$1.run(ProcessorNode.java:45)
at
org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:201)
at
org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:131)
at
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:82)
at
org.apache.kafka.streams.kstream.internals.KStreamMap$KStreamMapProcessor.process(KStreamMap.java:42)
at
org.apache.kafka.streams.processor.internals.ProcessorNode$1.run(ProcessorNode.java:45)
at
org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:201)
at
org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:131)
at
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:82)
at
org.apache.kafka.streams.kstream.internals.KStreamFilter$KStreamFilterProcessor.process(KStreamFilter.java:43)
at
org.apache.kafka.streams.processor.internals.ProcessorNode$1.run(ProcessorNode.java:45)
at
org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:201)
at
org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:131)
at
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:82)
at
org.apache.kafka.streams.kstream.internals.KStreamFlatMap$KStreamFlatMapProcessor.process(KStreamFlatMap.java:42)
at
org.apache.kafka.streams.processor.internals.ProcessorNode$1.run(ProcessorNode.java:45)
at
org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:201)
at
org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:131)
at
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:82)
at
org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:80)
at
org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:188)
... 7 more
Caused by: org.apache.kafka.common.errors.OutOfOrderSequenceException: The
broker received an out of order sequence number
