[ https://issues.apache.org/jira/browse/KAFKA-1393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13969003#comment-13969003 ]
Guozhang Wang commented on KAFKA-1393: -------------------------------------- This might be a duplicate to KAFKA-1387 > I wrote this conflicted ephemeral node at /brokers/ids/199 a while back in a > different session, hence I will backoff for this node to be deleted by > Zookeeper and retry > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > > Key: KAFKA-1393 > URL: https://issues.apache.org/jira/browse/KAFKA-1393 > Project: Kafka > Issue Type: Bug > Affects Versions: 0.8.0 > Reporter: zuolin > Priority: Critical > > We're seeing the following log statements (over and over): > [2014-04-14 14:25:43,304] INFO re-registering broker info in ZK for broker > 199 (kafka.server.KafkaZooKeeper) > [2014-04-14 14:25:47,856] INFO Closing socket connection to /10.4.56.199. > (kafka.network.Processor) > [2014-04-14 14:25:53,691] INFO Closing socket connection to /10.4.56.199. > (kafka.network.Processor) > [2014-04-14 14:25:53,692] INFO Closing socket connection to /10.4.56.199. > (kafka.network.Processor) > [2014-04-14 14:25:54,540] INFO Closing socket connection to /10.4.56.199. > (kafka.network.Processor) > [2014-04-14 14:25:54,543] INFO Closing socket connection to /10.4.56.199. > (kafka.network.Processor) > [2014-04-14 14:26:15,997] INFO Closing socket connection to /10.4.59.170. > (kafka.network.Processor) > [2014-04-14 14:26:16,063] INFO Closing socket connection to /10.4.59.170. > (kafka.network.Processor) > [2014-04-14 14:26:16,116] INFO Closing socket connection to /10.4.59.170. > (kafka.network.Processor) > [2014-04-14 14:26:19,721] INFO Closing socket connection to /10.4.59.170. > (kafka.network.Processor) > [2014-04-14 14:26:19,722] INFO Closing socket connection to /10.4.59.170. > (kafka.network.Processor) > [2014-04-14 14:26:19,738] INFO Closing socket connection to /10.4.59.170. > (kafka.network.Processor) > [2014-04-14 14:26:19,767] INFO Closing socket connection to /10.4.59.170. > (kafka.network.Processor) > [2014-04-14 14:26:35,794] INFO Closing socket connection to /10.4.59.170. > (kafka.network.Processor) > [2014-04-14 14:26:36,009] INFO Closing socket connection to /10.4.59.170. > (kafka.network.Processor) > [2014-04-14 14:26:37,712] INFO Registered broker 199 at path /brokers/ids/199 > with address 10.4.56.199:9092. (kafka.utils.ZkUtils$) > [2014-04-14 14:26:37,789] INFO done re-registering broker > (kafka.server.KafkaZooKeeper) > [2014-04-14 14:26:37,947] INFO Subscribing to /brokers/topics path to watch > for new topics (kafka.server.KafkaZooKeeper) > [2014-04-14 14:26:38,179] INFO Closing socket connection to /10.4.56.246. > (kafka.network.Processor) > [2014-04-14 14:26:39,158] INFO Closing socket connection to /10.4.56.199. > (kafka.network.Processor) > [2014-04-14 14:26:40,398] INFO 199 successfully elected as leader > (kafka.server.ZookeeperLeaderElector) > [2014-04-14 14:26:43,332] INFO Closing socket connection to /10.4.56.246. > (kafka.network.Processor) > [2014-04-14 14:26:43,332] INFO Closing socket connection to /10.4.56.246. > (kafka.network.Processor) > [2014-04-14 14:26:43,358] INFO Closing socket connection to /10.4.56.246. > (kafka.network.Processor) > [2014-04-14 14:26:43,473] INFO Closing socket connection to /10.4.56.246. > (kafka.network.Processor) > [2014-04-14 14:26:43,701] INFO Closing socket connection to /10.4.56.246. > (kafka.network.Processor) > [2014-04-14 14:26:43,994] INFO Closing socket connection to /10.4.56.246. > (kafka.network.Processor) > [2014-04-14 14:26:43,995] INFO Closing socket connection to /10.4.56.246. > (kafka.network.Processor) > [2014-04-14 14:26:44,076] INFO Closing socket connection to /10.4.56.246. > (kafka.network.Processor) > [2014-04-14 14:26:44,377] INFO Closing socket connection to /10.4.56.246. > (kafka.network.Processor) > [2014-04-14 14:26:50,002] INFO Closing socket connection to /10.4.56.246. > (kafka.network.Processor) > [2014-04-14 14:26:50,874] INFO [Replica Manager on Broker 199]: Handling > LeaderAndIsr request > Name:LeaderAndIsrRequest;Version:0;Controller:199;ControllerEpoch:4;CorrelationId:4;ClientId:id_199-host_10.4.56.199-port_9092;PartitionState:(cacheMonitor-server,0) > -> > (LeaderAndIsrInfo:(Leader:199,ISR:199,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:199),(socket-server-4,0) > -> > (LeaderAndIsrInfo:(Leader:199,ISR:199,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:199),(push-server-3,0) > -> > (LeaderAndIsrInfo:(Leader:199,ISR:199,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:199),(socket-server-2,0) > -> > (LeaderAndIsrInfo:(Leader:199,ISR:199,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:199),(push-seat,0) > -> > (LeaderAndIsrInfo:(Leader:199,ISR:199,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:199),(socket-server-5,0) > -> > (LeaderAndIsrInfo:(Leader:199,ISR:199,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:199),(socket-seat,0) > -> > (LeaderAndIsrInfo:(Leader:199,ISR:199,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:199),(push-server-2,0) > -> > (LeaderAndIsrInfo:(Leader:199,ISR:199,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:199),(socket-server-1,0) > -> > (LeaderAndIsrInfo:(Leader:199,ISR:199,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:199),(push-server-4,0) > -> > (LeaderAndIsrInfo:(Leader:199,ISR:199,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:199),(socket-server-3,0) > -> > (LeaderAndIsrInfo:(Leader:199,ISR:199,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:199),(push-server-1,0) > -> > (LeaderAndIsrInfo:(Leader:199,ISR:199,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:199),(push-server-5,0) > -> > (LeaderAndIsrInfo:(Leader:199,ISR:199,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:199);Leaders:id:199,host:10.4.56.199,port:9092 > (kafka.server.ReplicaManager) > [2014-04-14 14:26:50,947] INFO [Replica Manager on Broker 199]: Handled > leader and isr request > Name:LeaderAndIsrRequest;Version:0;Controller:199;ControllerEpoch:4;CorrelationId:4;ClientId:id_199-host_10.4.56.199-port_9092;PartitionState:(cacheMonitor-server,0) > -> > (LeaderAndIsrInfo:(Leader:199,ISR:199,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:199),(socket-server-4,0) > -> > (LeaderAndIsrInfo:(Leader:199,ISR:199,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:199),(push-server-3,0) > -> > (LeaderAndIsrInfo:(Leader:199,ISR:199,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:199),(socket-server-2,0) > -> > (LeaderAndIsrInfo:(Leader:199,ISR:199,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:199),(push-seat,0) > -> > (LeaderAndIsrInfo:(Leader:199,ISR:199,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:199),(socket-server-5,0) > -> > (LeaderAndIsrInfo:(Leader:199,ISR:199,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:199),(socket-seat,0) > -> > (LeaderAndIsrInfo:(Leader:199,ISR:199,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:199),(push-server-2,0) > -> > (LeaderAndIsrInfo:(Leader:199,ISR:199,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:199),(socket-server-1,0) > -> > (LeaderAndIsrInfo:(Leader:199,ISR:199,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:199),(push-server-4,0) > -> > (LeaderAndIsrInfo:(Leader:199,ISR:199,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:199),(socket-server-3,0) > -> > (LeaderAndIsrInfo:(Leader:199,ISR:199,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:199),(push-server-1,0) > -> > (LeaderAndIsrInfo:(Leader:199,ISR:199,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:199),(push-server-5,0) > -> > (LeaderAndIsrInfo:(Leader:199,ISR:199,LeaderEpoch:0,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:199);Leaders:id:199,host:10.4.56.199,port:9092 > (kafka.server.ReplicaManager) > [2014-04-14 14:26:52,262] INFO Closing socket connection to /10.4.56.246. > (kafka.network.Processor) > [2014-04-14 14:26:52,547] INFO Closing socket connection to /10.4.56.246. > (kafka.network.Processor) > [2014-04-14 14:26:52,771] INFO Closing socket connection to /10.4.56.246. > (kafka.network.Processor) > [2014-04-14 14:26:52,931] INFO Closing socket connection to /10.4.56.246. > (kafka.network.Processor) > [2014-04-14 14:26:53,969] INFO Closing socket connection to /10.4.56.246. > (kafka.network.Processor) > [2014-04-14 14:26:54,272] INFO Closing socket connection to /10.4.56.246. > (kafka.network.Processor) > [2014-04-14 14:26:54,380] INFO Closing socket connection to /10.4.56.246. > (kafka.network.Processor) > [2014-04-14 14:26:54,380] INFO Closing socket connection to /10.4.56.246. > (kafka.network.Processor) > [2014-04-14 14:26:54,499] INFO New leader is 199 > (kafka.server.ZookeeperLeaderElector$LeaderChangeListener) > [2014-04-14 14:26:54,502] INFO re-registering broker info in ZK for broker > 199 (kafka.server.KafkaZooKeeper) > [2014-04-14 14:26:54,625] INFO Closing socket connection to /10.4.56.246. > (kafka.network.Processor) > [2014-04-14 14:26:54,927] INFO Closing socket connection to /10.4.56.246. > (kafka.network.Processor) > [2014-04-14 14:26:55,423] INFO Closing socket connection to /10.4.56.246. > (kafka.network.Processor) > [2014-04-14 14:26:56,680] INFO conflict in /brokers/ids/199 data: { > "host":"10.4.56.199", "jmx_port":9997, "port":9092, > "timestamp":"1397456814503", "version":1 } stored data: { > "host":"10.4.56.199", "jmx_port":9997, "port":9092, > "timestamp":"1397456743304", "version":1 } (kafka.utils.ZkUtils$) > [2014-04-14 14:26:57,101] INFO Closing socket connection to /10.4.56.246. > (kafka.network.Processor) > [2014-04-14 14:26:57,102] INFO Closing socket connection to /10.4.56.246. > (kafka.network.Processor) > [2014-04-14 14:26:57,514] INFO Closing socket connection to /10.4.56.246. > (kafka.network.Processor) > [2014-04-14 14:26:57,514] INFO Closing socket connection to /10.4.56.246. > (kafka.network.Processor) > [2014-04-14 14:26:57,523] INFO Closing socket connection to /10.4.56.199. > (kafka.network.Processor) > [2014-04-14 14:26:57,523] INFO Closing socket connection to /10.4.56.199. > (kafka.network.Processor) > [2014-04-14 14:26:57,733] INFO I wrote this conflicted ephemeral node [{ > "host":"10.4.56.199", "jmx_port":9997, "port":9092, > "timestamp":"1397456814503", "version":1 }] at /brokers/ids/199 a while back > in a different session, hence I will backoff for this node to be deleted by > Zookeeper and retry (kafka.utils.ZkUtils$) > [2014-04-14 14:27:11,616] INFO conflict in /brokers/ids/199 data: { > "host":"10.4.56.199", "jmx_port":9997, "port":9092, > "timestamp":"1397456814503", "version":1 } stored data: { > "host":"10.4.56.199", "jmx_port":9997, "port":9092, > "timestamp":"1397456743304", "version":1 } (kafka.utils.ZkUtils$) > [2014-04-14 14:27:11,695] INFO I wrote this conflicted ephemeral node [{ > "host":"10.4.56.199", "jmx_port":9997, "port":9092, > "timestamp":"1397456814503", "version":1 }] at /brokers/ids/199 a while back > in a different session, hence I will backoff for this node to be deleted by > Zookeeper and retry (kafka.utils.ZkUtils$) > [2014-04-14 14:27:12,675] INFO Closing socket connection to /10.4.56.246. > (kafka.network.Processor) > [2014-04-14 14:27:13,097] INFO Closing socket connection to /10.4.56.246. > (kafka.network.Processor) > [2014-04-14 14:27:13,252] INFO Closing socket connection to /10.4.56.246. > (kafka.network.Processor) > [2014-04-14 14:27:13,609] INFO Closing socket connection to /10.4.56.246. > (kafka.network.Processor) > [2014-04-14 14:27:23,730] INFO conflict in /brokers/ids/199 data: { > "host":"10.4.56.199", "jmx_port":9997, "port":9092, > "timestamp":"1397456814503", "version":1 } stored data: { > "host":"10.4.56.199", "jmx_port":9997, "port":9092, > "timestamp":"1397456743304", "version":1 } (kafka.utils.ZkUtils$) > [2014-04-14 14:27:23,768] INFO I wrote this conflicted ephemeral node [{ > "host":"10.4.56.199", "jmx_port":9997, "port":9092, > "timestamp":"1397456814503", "version":1 }] at /brokers/ids/199 a while back > in a different session, hence I will backoff for this node to be deleted by > Zookeeper and retry (kafka.utils.ZkUtils$) > [2014-04-14 14:27:29,933] INFO conflict in /brokers/ids/199 data: { > "host":"10.4.56.199", "jmx_port":9997, "port":9092, > "timestamp":"1397456814503", "version":1 } stored data: { > "host":"10.4.56.199", "jmx_port":9997, "port":9092, > "timestamp":"1397456743304", "version":1 } (kafka.utils.ZkUtils$) > [2014-04-14 14:27:29,940] INFO I wrote this conflicted ephemeral node [{ > "host":"10.4.56.199", "jmx_port":9997, "port":9092, > "timestamp":"1397456814503", "version":1 }] at /brokers/ids/199 a while back > in a different session, hence I will backoff for this node to be deleted by > Zookeeper and retry (kafka.utils.ZkUtils$) > [2014-04-14 14:27:36,012] INFO conflict in /brokers/ids/199 data: { > "host":"10.4.56.199", "jmx_port":9997, "port":9092, > "timestamp":"1397456814503", "version":1 } stored data: { > "host":"10.4.56.199", "jmx_port":9997, "port":9092, > "timestamp":"1397456743304", "version":1 } (kafka.utils.ZkUtils$) > [2014-04-14 14:27:36,021] INFO I wrote this conflicted ephemeral node [{ > "host":"10.4.56.199", "jmx_port":9997, "port":9092, > "timestamp":"1397456814503", "version":1 }] at /brokers/ids/199 a while back > in a different session, hence I will backoff for this node to be deleted by > Zookeeper and retry (kafka.utils.ZkUtils$) > [2014-04-14 14:27:42,064] INFO Closing socket connection to /10.4.59.170. > (kafka.network.Processor) > [2014-04-14 14:27:42,154] INFO Closing socket connection to /10.4.59.170. > (kafka.network.Processor) > [2014-04-14 14:27:42,161] INFO Closing socket connection to /10.4.59.170. > (kafka.network.Processor) > [2014-04-14 14:27:42,166] INFO Closing socket connection to /10.4.59.170. > (kafka.network.Processor) > [2014-04-14 14:27:42,403] INFO Closing socket connection to /10.4.59.170. > (kafka.network.Processor) > [2014-04-14 14:27:42,406] INFO Closing socket connection to /10.4.59.170. > (kafka.network.Processor) > [2014-04-14 14:27:42,811] INFO Closing socket connection to /10.4.59.170. > (kafka.network.Processor) > [2014-04-14 14:27:42,915] INFO conflict in /brokers/ids/199 data: { > "host":"10.4.56.199", "jmx_port":9997, "port":9092, > "timestamp":"1397456814503", "version":1 } stored data: { > "host":"10.4.56.199", "jmx_port":9997, "port":9092, > "timestamp":"1397456743304", "version":1 } (kafka.utils.ZkUtils$) > [2014-04-14 14:27:42,918] INFO I wrote this conflicted ephemeral node [{ > "host":"10.4.56.199", "jmx_port":9997, "port":9092, > "timestamp":"1397456814503", "version":1 }] at /brokers/ids/199 a while back > in a different session, hence I will backoff for this node to be deleted by > Zookeeper and retry (kafka.utils.ZkUtils$) > [2014-04-14 14:27:47,087] INFO Closing socket connection to /10.4.59.170. > (kafka.network.Processor) > [2014-04-14 14:27:47,227] INFO Closing socket connection to /10.4.59.170. > (kafka.network.Processor) > [2014-04-14 14:27:48,951] INFO conflict in /brokers/ids/199 data: { > "host":"10.4.56.199", "jmx_port":9997, "port":9092, > "timestamp":"1397456814503", "version":1 } stored data: { > "host":"10.4.56.199", "jmx_port":9997, "port":9092, > "timestamp":"1397456743304", "version":1 } (kafka.utils.ZkUtils$) > [2014-04-14 14:27:48,953] INFO I wrote this conflicted ephemeral node [{ > "host":"10.4.56.199", "jmx_port":9997, "port":9092, > "timestamp":"1397456814503", "version":1 }] at /brokers/ids/199 a while back > in a different session, hence I will backoff for this node to be deleted by > Zookeeper and retry (kafka.utils.ZkUtils$) > [2014-04-14 14:27:54,958] INFO conflict in /brokers/ids/199 data: { > "host":"10.4.56.199", "jmx_port":9997, "port":9092, > "timestamp":"1397456814503", "version":1 } stored data: { > "host":"10.4.56.199", "jmx_port":9997, "port":9092, > "timestamp":"1397456743304", "version":1 } (kafka.utils.ZkUtils$) > [2014-04-14 14:27:54,961] INFO I wrote this conflicted ephemeral node [{ > "host":"10.4.56.199", "jmx_port":9997, "port":9092, > "timestamp":"1397456814503", "version":1 }] at /brokers/ids/199 a while back > in a different session, hence I will backoff for this node to be deleted by > Zookeeper and retry (kafka.utils.ZkUtils$) > [2014-04-14 14:28:00,964] INFO conflict in /brokers/ids/199 data: { > "host":"10.4.56.199", "jmx_port":9997, "port":9092, > "timestamp":"1397456814503", "version":1 } stored data: { > "host":"10.4.56.199", "jmx_port":9997, "port":9092, > "timestamp":"1397456743304", "version":1 } (kafka.utils.ZkUtils$) > [2014-04-14 14:28:00,969] INFO I wrote this conflicted ephemeral node [{ > "host":"10.4.56.199", "jmx_port":9997, "port":9092, > "timestamp":"1397456814503", "version":1 }] at /brokers/ids/199 a while back > in a different session, hence I will backoff for this node to be deleted by > Zookeeper and retry (kafka.utils.ZkUtils$) > [2014-04-14 14:28:07,075] INFO conflict in /brokers/ids/199 data: { > "host":"10.4.56.199", "jmx_port":9997, "port":9092, > "timestamp":"1397456814503", "version":1 } stored data: { > "host":"10.4.56.199", "jmx_port":9997, "port":9092, > "timestamp":"1397456743304", "version":1 } (kafka.utils.ZkUtils$) > [2014-04-14 14:28:07,080] INFO I wrote this conflicted ephemeral node [{ > "host":"10.4.56.199", "jmx_port":9997, "port":9092, > "timestamp":"1397456814503", "version":1 }] at /brokers/ids/199 a while back > in a different session, hence I will backoff for this node to be deleted by > Zookeeper and retry (kafka.utils.ZkUtils$) > [2014-04-14 14:28:14,569] INFO conflict in /brokers/ids/199 data: { > "host":"10.4.56.199", "jmx_port":9997, "port":9092, > "timestamp":"1397456814503", "version":1 } stored data: { > "host":"10.4.56.199", "jmx_port":9997, "port":9092, > "timestamp":"1397456743304", "version":1 } (kafka.utils.ZkUtils$) > [2014-04-14 14:28:14,573] INFO I wrote this conflicted ephemeral node [{ > "host":"10.4.56.199", "jmx_port":9997, "port":9092, > "timestamp":"1397456814503", "version":1 }] at /brokers/ids/199 a while back > in a different session, hence I will backoff for this node to be deleted by > Zookeeper and retry (kafka.utils.ZkUtils$) -- This message was sent by Atlassian JIRA (v6.2#6252)