Hi, all
Our kafka server encounter an unexpected shut down last night, I checked
the logs, it says there is a conflicted ephemeral node, but can't find what
issue it is, can anyone help check this, thanks!
[2015-04-21 03:00:00,562] INFO conflict in /brokers/ids/0 data:
{"jmx_port":-1,"timestamp":"1429556393518","host":"10.144.38.185","version":1,"port":9092}
stored data:
{"jmx_port":-1,"timestamp":"1429556241625","host":"10.144.38.185","version":1,"port":9092}
(kafka.utils.ZkUtils$)
[2015-04-21 03:00:00,564] INFO I wrote this conflicted ephemeral node
[{"jmx_port":-1,"timestamp":"1429556393518","host":"10.144.38.185","version":1,"port":9092}]
at /brokers/ids/0 a while back in a different session, hence I will backoff
for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
[2015-04-21 03:00:08,551] INFO conflict in /brokers/ids/0 data:
{"jmx_port":-1,"timestamp":"1429556393518","host":"10.144.38.185","version":1,"port":9092}
stored data:
{"jmx_port":-1,"timestamp":"1429556241625","host":"10.144.38.185","version":1,"port":9092}
(kafka.utils.ZkUtils$)
[2015-04-21 03:00:08,553] INFO I wrote this conflicted ephemeral node
[{"jmx_port":-1,"timestamp":"1429556393518","host":"10.144.38.185","version":1,"port":9092}]
at /brokers/ids/0 a while back in a different session, hence I will backoff
for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
[2015-04-21 03:00:22,539] INFO Closing socket connection to /10.144.132.152.
(kafka.network.Processor)
[2015-04-21 03:00:44,228] INFO Closing socket connection to /10.144.38.185.
(kafka.network.Processor)
[2015-04-21 03:00:45,210] INFO Registered broker 0 at path /brokers/ids/0
with address 10.144.38.185:9092. (kafka.utils.ZkUtils$)
[2015-04-21 03:00:45,210] INFO done re-registering broker
(kafka.server.KafkaHealthcheck)
[2015-04-21 03:00:45,210] INFO Subscribing to /brokers/topics path to watch
for new topics (kafka.server.KafkaHealthcheck)
[2015-04-21 03:00:45,212] ERROR Closing socket for /10.144.38.185 because
of error (kafka.network.Processor)
java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51)
at sun.nio.ch.IOUtil.write(IOUtil.java:148)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:524)
at java.nio.channels.SocketChannel.write(SocketChannel.java:493)
at
kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:56)
at kafka.network.Processor.write(SocketServer.scala:472)
at kafka.network.Processor.run(SocketServer.scala:342)
at java.lang.Thread.run(Thread.java:745)
[2015-04-21 03:00:45,215] INFO re-registering broker info in ZK for broker
0 (kafka.server.KafkaHealthcheck)
[2015-04-21 03:00:45,223] INFO conflict in /brokers/ids/0 data:
{"jmx_port":-1,"timestamp":"1429556445215","host":"10.144.38.185","version":1,"port":9092}
stored data:
{"jmx_port":-1,"timestamp":"1429556393518","host":"10.144.38.185","version":1,"port":9092}
(kafka.utils.ZkUtils$)
[2015-04-21 03:00:45,225] INFO I wrote this conflicted ephemeral node
[{"jmx_port":-1,"timestamp":"1429556445215","host":"10.144.38.185","version":1,"port":9092}]
at /brokers/ids/0 a while back in a different session, hence I will backoff
for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
[2015-04-21 03:00:53,022] INFO conflict in /brokers/ids/0 data:
{"jmx_port":-1,"timestamp":"1429556445215","host":"10.144.38.185","version":1,"port":9092}
stored data:
{"jmx_port":-1,"timestamp":"1429556393518","host":"10.144.38.185","version":1,"port":9092}
(kafka.utils.ZkUtils$)
[2015-04-21 03:00:53,888] INFO I wrote this conflicted ephemeral node
[{"jmx_port":-1,"timestamp":"1429556445215","host":"10.144.38.185","version":1,"port":9092}]
at /brokers/ids/0 a while back in a different session, hence I will backoff
for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
[2015-04-21 03:01:21,366] INFO Registered broker 0 at path /brokers/ids/0
with address 10.144.38.185:9092. (kafka.utils.ZkUtils$)
[2015-04-21 03:01:21,366] INFO done re-registering broker
(kafka.server.KafkaHealthcheck)
[2015-04-21 03:01:21,366] INFO Subscribing to /brokers/topics path to watch
for new topics (kafka.server.KafkaHealthcheck)
[2015-04-21 03:01:21,468] INFO re-registering broker info in ZK for broker
0 (kafka.server.KafkaHealthcheck)
[2015-04-21 03:01:21,471] INFO conflict in /brokers/ids/0 data:
{"jmx_port":-1,"timestamp":"1429556481468","host":"10.144.38.185","version":1,"port":9092}
stored data:
{"jmx_port":-1,"timestamp":"1429556445215","host":"10.144.38.185","version":1,"port":9092}
(kafka.utils.ZkUtils$)
[2015-04-21 03:01:21,472] INFO I wrote this conflicted ephemeral node
[{"jmx_port":-1,"timestamp":"1429556481468","host":"10.144.38.185","version":1,"port":9092}]
at /brokers/ids/0 a while back in a different session, hence I will backoff
for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
[2015-04-21 03:01:27,476] INFO conflict in /brokers/ids/0 data:
{"jmx_port":-1,"timestamp":"1429556481468","host":"10.144.38.185","version":1,"port":9092}
stored data:
{"jmx_port":-1,"timestamp":"1429556445215","host":"10.144.38.185","version":1,"port":9092}
(kafka.utils.ZkUtils$)
[2015-04-21 03:01:27,477] INFO I wrote this conflicted ephemeral node
[{"jmx_port":-1,"timestamp":"1429556481468","host":"10.144.38.185","version":1,"port":9092}]
at /brokers/ids/0 a while back in a different session, hence I will backoff
for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
[2015-04-21 03:01:33,481] INFO conflict in /brokers/ids/0 data:
{"jmx_port":-1,"timestamp":"1429556481468","host":"10.144.38.185","version":1,"port":9092}
stored data:
{"jmx_port":-1,"timestamp":"1429556445215","host":"10.144.38.185","version":1,"port":9092}
(kafka.utils.ZkUtils$)
[2015-04-21 03:01:33,483] INFO I wrote this conflicted ephemeral node
[{"jmx_port":-1,"timestamp":"1429556481468","host":"10.144.38.185","version":1,"port":9092}]
at /brokers/ids/0 a while back in a different session, hence I will backoff
for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
[2015-04-21 03:01:41,589] INFO conflict in /brokers/ids/0 data:
{"jmx_port":-1,"timestamp":"1429556481468","host":"10.144.38.185","version":1,"port":9092}
stored data:
{"jmx_port":-1,"timestamp":"1429556445215","host":"10.144.38.185","version":1,"port":9092}
(kafka.utils.ZkUtils$)
[2015-04-21 03:01:41,590] INFO I wrote this conflicted ephemeral node
[{"jmx_port":-1,"timestamp":"1429556481468","host":"10.144.38.185","version":1,"port":9092}]
at /brokers/ids/0 a while back in a different session, hence I will backoff
for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
[2015-04-21 03:01:48,398] INFO conflict in /brokers/ids/0 data:
{"jmx_port":-1,"timestamp":"1429556481468","host":"10.144.38.185","version":1,"port":9092}
stored data:
{"jmx_port":-1,"timestamp":"1429556445215","host":"10.144.38.185","version":1,"port":9092}
(kafka.utils.ZkUtils$)
[2015-04-21 03:01:48,405] INFO I wrote this conflicted ephemeral node
[{"jmx_port":-1,"timestamp":"1429556481468","host":"10.144.38.185","version":1,"port":9092}]
at /brokers/ids/0 a while back in a different session, hence I will backoff
for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
[2015-04-21 03:02:19,410] INFO Registered broker 0 at path /brokers/ids/0
with address 10.144.38.185:9092. (kafka.utils.ZkUtils$)
[2015-04-21 03:02:19,410] INFO done re-registering broker
(kafka.server.KafkaHealthcheck)
[2015-04-21 03:02:19,410] INFO Subscribing to /brokers/topics path to watch
for new topics (kafka.server.KafkaHealthcheck)
[2015-04-21 03:02:19,425] INFO re-registering broker info in ZK for broker
0 (kafka.server.KafkaHealthcheck)
[2015-04-21 03:02:19,430] INFO conflict in /brokers/ids/0 data:
{"jmx_port":-1,"timestamp":"1429556539425","host":"10.144.38.185","version":1,"port":9092}
stored data:
{"jmx_port":-1,"timestamp":"1429556481468","host":"10.144.38.185","version":1,"port":9092}
(kafka.utils.ZkUtils$)
[2015-04-21 03:02:19,432] INFO I wrote this conflicted ephemeral node
[{"jmx_port":-1,"timestamp":"1429556539425","host":"10.144.38.185","version":1,"port":9092}]
at /brokers/ids/0 a while back in a different session, hence I will backoff
for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
[2015-04-21 03:02:26,128] INFO conflict in /brokers/ids/0 data:
{"jmx_port":-1,"timestamp":"1429556539425","host":"10.144.38.185","version":1,"port":9092}
stored data:
{"jmx_port":-1,"timestamp":"1429556481468","host":"10.144.38.185","version":1,"port":9092}
(kafka.utils.ZkUtils$)
[2015-04-21 03:02:26,130] INFO I wrote this conflicted ephemeral node
[{"jmx_port":-1,"timestamp":"1429556539425","host":"10.144.38.185","version":1,"port":9092}]
at /brokers/ids/0 a while back in a different session, hence I will backoff
for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
[2015-04-21 03:02:35,785] INFO conflict in /brokers/ids/0 data:
{"jmx_port":-1,"timestamp":"1429556539425","host":"10.144.38.185","version":1,"port":9092}
stored data:
{"jmx_port":-1,"timestamp":"1429556481468","host":"10.144.38.185","version":1,"port":9092}
(kafka.utils.ZkUtils$)
[2015-04-21 03:02:35,786] INFO I wrote this conflicted ephemeral node
[{"jmx_port":-1,"timestamp":"1429556539425","host":"10.144.38.185","version":1,"port":9092}]
at /brokers/ids/0 a while back in a different session, hence I will backoff
for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
