[
https://issues.apache.org/jira/browse/KAFKA-5998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16826493#comment-16826493
]
Patrik Kleindl commented on KAFKA-5998:
---------------------------------------
Found a new log, again starting with the message after the state-cleaner ran.
Filtered on task 1_1, there was no rebalance or anything in the time from 19:30
to 21:03
April 25th 2019, 21:07:51.658 2019-04-25 21:07:51,658 WARN
[org.apache.kafka.streams.processor.internals.ProcessorStateManager]
(application-fde85da6-9d2f-4457-8bdb-ea1c78c8c1e2-StreamThread-1) -
[short-component-name:; transaction-id:; user-id:; creation-time:] task [1_1]
Failed to write offset checkpoint file to
/opt/app/wildfly/standalone/tmp/application-streamapp.v1/1_1/.checkpoint: {}:
java.io.FileNotFoundException:
/opt/app/wildfly/standalone/tmp/application-streamapp.v1/1_1/.checkpoint.tmp
(No such file or directory)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
April 25th 2019, 21:03:49.332 2019-04-25 21:03:49,332 INFO
[org.apache.kafka.streams.processor.internals.StateDirectory]
(application-fde85da6-9d2f-4457-8bdb-ea1c78c8c1e2-CleanupThread) -
[short-component-name:; transaction-id:; user-id:; creation-time:]
stream-thread [application-fde85da6-9d2f-4457-8bdb-ea1c78c8c1e2-CleanupThread]
Deleting obsolete state directory 1_1 for task 1_1 as 813332ms has elapsed
(cleanup delay is 600000ms).
April 25th 2019, 19:30:52.902 2019-04-25 19:30:52,902 INFO
[org.apache.kafka.streams.processor.internals.StreamThread]
(application-fde85da6-9d2f-4457-8bdb-ea1c78c8c1e2-StreamThread-1) -
[short-component-name:; transaction-id:; user-id:; creation-time:]
stream-thread [application-fde85da6-9d2f-4457-8bdb-ea1c78c8c1e2-StreamThread-1]
partition assignment took 80 ms.
current active tasks: [1_0, 0_1, 1_1, 0_3, 2_1, 1_3, 1_4, 0_5, 2_3, 1_5, 1_6,
0_7, 2_5, 0_11, 2_9, 1_11, 2_10, 2_11]
current standby tasks: []
previous active tasks: [1_0, 0_1, 1_1, 0_3, 2_1, 1_3, 1_4, 2_3, 0_5, 1_5, 1_6,
0_7, 2_5, 0_11, 2_9, 1_11, 2_10, 2_11]
April 25th 2019, 19:30:52.713 2019-04-25 19:30:52,713 INFO
[org.apache.kafka.streams.processor.internals.StreamThread]
(application-fde85da6-9d2f-4457-8bdb-ea1c78c8c1e2-StreamThread-1) -
[short-component-name:; transaction-id:; user-id:; creation-time:]
stream-thread [application-fde85da6-9d2f-4457-8bdb-ea1c78c8c1e2-StreamThread-1]
partition revocation took 764 ms.
suspended active tasks: [1_0, 0_1, 1_1, 0_3, 2_1, 1_3, 1_4, 2_3, 0_5, 1_5,
1_6, 0_7, 2_5, 0_11, 2_9, 1_11, 2_10, 2_11]
suspended standby tasks: []
April 25th 2019, 19:30:39.144 2019-04-25 19:30:39,144 INFO
[org.apache.kafka.streams.processor.internals.StreamThread]
(application-fde85da6-9d2f-4457-8bdb-ea1c78c8c1e2-StreamThread-1) -
[short-component-name:; transaction-id:; user-id:; creation-time:]
stream-thread [application-fde85da6-9d2f-4457-8bdb-ea1c78c8c1e2-StreamThread-1]
partition assignment took 29 ms.
current active tasks: [1_0, 0_1, 1_1, 0_3, 2_1, 1_3, 1_4, 2_3, 0_5, 1_5, 1_6,
0_7, 2_5, 0_11, 2_9, 1_11, 2_10, 2_11]
current standby tasks: []
previous active tasks: [1_0, 1_1, 1_3, 1_4, 2_3, 1_5, 1_6, 0_7, 2_5]
April 25th 2019, 19:30:29.619 2019-04-25 19:30:29,619 INFO
[org.apache.kafka.streams.processor.internals.StreamThread]
(application-fde85da6-9d2f-4457-8bdb-ea1c78c8c1e2-StreamThread-1) -
[short-component-name:; transaction-id:; user-id:; creation-time:]
stream-thread [application-fde85da6-9d2f-4457-8bdb-ea1c78c8c1e2-StreamThread-1]
partition revocation took 2254 ms.
suspended active tasks: [1_0, 1_1, 1_3, 1_4, 2_3, 1_5, 1_6, 0_7, 2_5]
suspended standby tasks: []
April 25th 2019, 19:30:17.158 2019-04-25 19:30:17,158 INFO
[org.apache.kafka.streams.processor.internals.StreamThread]
(application-fde85da6-9d2f-4457-8bdb-ea1c78c8c1e2-StreamThread-1) -
[short-component-name:; transaction-id:; user-id:; creation-time:]
stream-thread [application-fde85da6-9d2f-4457-8bdb-ea1c78c8c1e2-StreamThread-1]
partition assignment took 935 ms.
current active tasks: [0_0, 1_0, 0_1, 1_1, 2_0, 1_2, 0_3, 2_1, 1_3, 2_2, 1_4,
0_5, 2_3, 1_5, 0_6, 1_6, 0_7, 2_5]
current standby tasks: []
previous active tasks: [0_0, 1_0, 0_1, 1_1, 2_0, 1_2, 0_3, 2_1, 1_3, 2_2, 1_4,
0_5, 2_3, 1_5, 0_6, 1_6, 0_7, 2_5, 1_7, 0_8, 2_6, 1_8, 0_9, 2_7, 1_9, 2_8,
1_10, 0_11, 2_9, 1_11, 2_10, 2_11]
> /.checkpoint.tmp Not found exception
> ------------------------------------
>
> Key: KAFKA-5998
> URL: https://issues.apache.org/jira/browse/KAFKA-5998
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 0.11.0.0, 0.11.0.1, 2.1.1
> Reporter: Yogesh BG
> Priority: Critical
> Attachments: 5998.v1.txt, 5998.v2.txt, Topology.txt, exc.txt,
> props.txt, streams.txt
>
>
> I have one kafka broker and one kafka stream running... I am running its
> since two days under load of around 2500 msgs per second.. On third day am
> getting below exception for some of the partitions, I have 16 partitions only
> 0_0 and 0_1 gives this error
> {{09:43:25.955 [ks_0_inst-StreamThread-6] WARN
> o.a.k.s.p.i.ProcessorStateManager - Failed to write checkpoint file to
> /data/kstreams/rtp-kafkastreams/0_1/.checkpoint:
> java.io.FileNotFoundException:
> /data/kstreams/rtp-kafkastreams/0_1/.checkpoint.tmp (No such file or
> directory)
> at java.io.FileOutputStream.open(Native Method) ~[na:1.7.0_111]
> at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
> ~[na:1.7.0_111]
> at java.io.FileOutputStream.<init>(FileOutputStream.java:171)
> ~[na:1.7.0_111]
> at
> org.apache.kafka.streams.state.internals.OffsetCheckpoint.write(OffsetCheckpoint.java:73)
> ~[rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> org.apache.kafka.streams.processor.internals.ProcessorStateManager.checkpoint(ProcessorStateManager.java:324)
> ~[rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> org.apache.kafka.streams.processor.internals.StreamTask$1.run(StreamTask.java:267)
> [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:201)
> [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> org.apache.kafka.streams.processor.internals.StreamTask.commit(StreamTask.java:260)
> [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> org.apache.kafka.streams.processor.internals.StreamTask.commit(StreamTask.java:254)
> [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> org.apache.kafka.streams.processor.internals.AssignedTasks$1.apply(AssignedTasks.java:322)
> [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> org.apache.kafka.streams.processor.internals.AssignedTasks.applyToRunningTasks(AssignedTasks.java:415)
> [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> org.apache.kafka.streams.processor.internals.AssignedTasks.commit(AssignedTasks.java:314)
> [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> org.apache.kafka.streams.processor.internals.StreamThread.commitAll(StreamThread.java:700)
> [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> org.apache.kafka.streams.processor.internals.StreamThread.maybeCommit(StreamThread.java:683)
> [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:523)
> [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:480)
> [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:457)
> [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na]
> 09:43:25.974 [ks_0_inst-StreamThread-15] WARN
> o.a.k.s.p.i.ProcessorStateManager - Failed to write checkpoint file to
> /data/kstreams/rtp-kafkastreams/0_0/.checkpoint:
> java.io.FileNotFoundException:
> /data/kstreams/rtp-kafkastreams/0_0/.checkpoint.tmp (No such file or
> directory)
> at java.io.FileOutputStream.open(Native Method) ~[na:1.7.0_111]
> at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
> ~[na:1.7.0_111]
> at java.io.FileOutputStream.<init>(FileOutputStream.java:171)
> ~[na:1.7.0_111]
> at
> org.apache.kafka.streams.state.internals.OffsetCheckpoint.write(OffsetCheckpoint.java:73)
> ~[rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> org.apache.kafka.streams.processor.internals.ProcessorStateManager.checkpoint(ProcessorStateManager.java:324)
> ~[rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> org.apache.kafka.streams.processor.internals.StreamTask$1.run(StreamTask.java:267)
> [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:201)
> [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> org.apache.kafka.streams.processor.internals.StreamTask.commit(StreamTask.java:260)
> [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> org.apache.kafka.streams.processor.internals.StreamTask.commit(StreamTask.java:254)
> [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> org.apache.kafka.streams.processor.internals.AssignedTasks$1.apply(AssignedTasks.java:322)
> [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> org.apache.kafka.streams.processor.internals.AssignedTasks.applyToRunningTasks(AssignedTasks.java:415)
> [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> org.apache.kafka.streams.processor.internals.AssignedTasks.commit(AssignedTasks.java:314)
> [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> org.apache.kafka.streams.processor.internals.StreamThread.commitAll(StreamThread.java:700)
> [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> org.apache.kafka.streams.processor.internals.StreamThread.maybeCommit(StreamThread.java:683)
> [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:523)
> [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:480)
> [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na]
> at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:457)
> [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na]
> }}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
