[
https://issues.apache.org/jira/browse/KAFKA-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16271906#comment-16271906
]
aarti gupta commented on KAFKA-4669:
------------------------------------
unfortunately we did not have trace enabled and did not enable it when we
encountered the problem
Here is a scrubbed larger segment of the consumer log, in case it helps
2017-11-29 21:22:37.841 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Opening socket connection to server
wdc04-dta-1-hcx-a/127.0.0.1:2181. Will not attempt to authenticate using SASL
(unknown error)
2017-11-29 21:22:37.841 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Socket connection established to
wdc04-dta-1-hcx-a/127.0.0.1:2181, initiating session
2017-11-29 21:22:37.841 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] WARN
org.apache.zookeeper.ClientCnxn- Unable to reconnect to ZooKeeper service,
session 0x15fcbdb62647c08 has expired
2017-11-29 21:22:37.841 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Unable to reconnect to ZooKeeper service,
session 0x15fcbdb62647c08 has expired, closing socket connection
2017-11-29 21:22:37.842 UTC [VrealmUpgradeService_EventListener, , ] ERROR
c.v.v.h.m.k.KafkaEventConsumerDelegate- Error fetching next record
java.lang.Exception: Error fetching next new record from kafka queue
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEventWithTimeout(KafkaEventConsumerDelegate.java:121)
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEvent(KafkaEventConsumerDelegate.java:64)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter.getNextEvent(EventListenerAdapter.java:76)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter.getNextEventAndAck(EventListenerAdapter.java:94)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter$1.run(EventListenerAdapter.java:125)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Correlation id for response
(386007) does not match request (386006), request header:
{api_key=9,api_version=3,correlation_id=386006,client_id=consumer-57}
at
org.apache.kafka.clients.NetworkClient.correlate(NetworkClient.java:752)
at
org.apache.kafka.clients.NetworkClient.parseStructMaybeUpdateThrottleTimeMetrics(NetworkClient.java:561)
at
org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:657)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:442)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:232)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:208)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:168)
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.fetchCommittedOffsets(ConsumerCoordinator.java:477)
at
org.apache.kafka.clients.consumer.KafkaConsumer.committed(KafkaConsumer.java:1346)
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEventWithTimeout(KafkaEventConsumerDelegate.java:86)
... 5 common frames omitted
2017-11-29 21:22:37.842 UTC [main-EventThread, , ] INFO
c.v.v.h.c.s.ConfigurableRunnableListener- Got a closing request- znode:
/NetworkStretchService
2017-11-29 21:22:37.842 UTC [main-EventThread, , ] INFO
org.apache.zookeeper.ZooKeeper- Initiating client connection,
connectString=localhost:2181 sessionTimeout=3000
watcher=org.apache.curator.ConnectionState@9eaa0a5
2017-11-29 21:22:37.842 UTC [main-EventThread, , ] WARN
org.apache.curator.ConnectionState- Session expired event received
2017-11-29 21:22:37.842 UTC [main-EventThread, , ] INFO
org.apache.zookeeper.ZooKeeper- Initiating client connection,
connectString=localhost:2181 sessionTimeout=3000
watcher=org.apache.curator.ConnectionState@50128862
2017-11-29 21:22:37.846 UTC [main-EventThread, , ] INFO
org.apache.zookeeper.ClientCnxn- EventThread shut down for session:
0x15fcbdb62647c05
2017-11-29 21:22:37.846 UTC [VrealmUpgradeService_EventListener, , ] ERROR
c.v.v.h.m.k.KafkaEventConsumerDelegate- Error fetching next record
java.lang.Exception: Error fetching next new record from kafka queue
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEventWithTimeout(KafkaEventConsumerDelegate.java:121)
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEvent(KafkaEventConsumerDelegate.java:64)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter.getNextEvent(EventListenerAdapter.java:76)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter.getNextEventAndAck(EventListenerAdapter.java:94)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter$1.run(EventListenerAdapter.java:125)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Correlation id for response
(386008) does not match request (386007), request header:
{api_key=9,api_version=3,correlation_id=386007,client_id=consumer-57}
at
org.apache.kafka.clients.NetworkClient.correlate(NetworkClient.java:752)
at
org.apache.kafka.clients.NetworkClient.parseStructMaybeUpdateThrottleTimeMetrics(NetworkClient.java:561)
at
org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:657)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:442)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:232)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:208)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:168)
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.fetchCommittedOffsets(ConsumerCoordinator.java:477)
at
org.apache.kafka.clients.consumer.KafkaConsumer.committed(KafkaConsumer.java:1346)
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEventWithTimeout(KafkaEventConsumerDelegate.java:86)
... 5 common frames omitted
2017-11-29 21:22:40.700 UTC [main-EventThread, , ] INFO
org.apache.zookeeper.ClientCnxn- EventThread shut down for session:
0x15fcbdb62647c08
2017-11-29 21:22:40.700 UTC [RemotingService_EventListener, , ] ERROR
c.v.v.h.m.k.KafkaEventConsumerDelegate- Error fetching next record
java.lang.Exception: Error fetching next new record from kafka queue
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEventWithTimeout(KafkaEventConsumerDelegate.java:121)
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEvent(KafkaEventConsumerDelegate.java:64)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter.getNextEvent(EventListenerAdapter.java:76)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter.getNextEventAndAck(EventListenerAdapter.java:94)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter$1.run(EventListenerAdapter.java:125)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Correlation id for response
(384729) does not match request (384728), request header:
{api_key=9,api_version=3,correlation_id=384728,client_id=consumer-36}
at
org.apache.kafka.clients.NetworkClient.correlate(NetworkClient.java:752)
at
org.apache.kafka.clients.NetworkClient.parseStructMaybeUpdateThrottleTimeMetrics(NetworkClient.java:561)
at
org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:657)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:442)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:232)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:208)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:168)
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.fetchCommittedOffsets(ConsumerCoordinator.java:477)
at
org.apache.kafka.clients.consumer.KafkaConsumer.committed(KafkaConsumer.java:1346)
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEventWithTimeout(KafkaEventConsumerDelegate.java:86)
... 5 common frames omitted
2017-11-29 21:22:40.701 UTC [VrealmUpgradeService_EventListener, , ] ERROR
c.v.v.h.m.k.KafkaEventConsumerDelegate- Error fetching next record
java.lang.Exception: Error fetching next new record from kafka queue
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEventWithTimeout(KafkaEventConsumerDelegate.java:121)
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEvent(KafkaEventConsumerDelegate.java:64)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter.getNextEvent(EventListenerAdapter.java:76)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter.getNextEventAndAck(EventListenerAdapter.java:94)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter$1.run(EventListenerAdapter.java:125)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Correlation id for response
(386009) does not match request (386008), request header:
{api_key=9,api_version=3,correlation_id=386008,client_id=consumer-57}
at
org.apache.kafka.clients.NetworkClient.correlate(NetworkClient.java:752)
at
org.apache.kafka.clients.NetworkClient.parseStructMaybeUpdateThrottleTimeMetrics(NetworkClient.java:561)
at
org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:657)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:442)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:232)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:208)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:168)
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.fetchCommittedOffsets(ConsumerCoordinator.java:477)
at
org.apache.kafka.clients.consumer.KafkaConsumer.committed(KafkaConsumer.java:1346)
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEventWithTimeout(KafkaEventConsumerDelegate.java:86)
... 5 common frames omitted
2017-11-29 21:22:40.702 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Opening socket connection to server
wdc04-dta-1-hcx-a/127.0.0.1:2181. Will not attempt to authenticate using SASL
(unknown error)
2017-11-29 21:22:40.702 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Socket connection established to
wdc04-dta-1-hcx-a/127.0.0.1:2181, initiating session
2017-11-29 21:22:40.704 UTC [VrealmUpgradeService_EventListener, , ] ERROR
c.v.v.h.m.k.KafkaEventConsumerDelegate- Error fetching next record
java.lang.Exception: Error fetching next new record from kafka queue
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEventWithTimeout(KafkaEventConsumerDelegate.java:121)
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEvent(KafkaEventConsumerDelegate.java:64)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter.getNextEvent(EventListenerAdapter.java:76)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter.getNextEventAndAck(EventListenerAdapter.java:94)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter$1.run(EventListenerAdapter.java:125)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Correlation id for response
(386010) does not match request (386009), request header:
{api_key=9,api_version=3,correlation_id=386009,client_id=consumer-57}
at
org.apache.kafka.clients.NetworkClient.correlate(NetworkClient.java:752)
at
org.apache.kafka.clients.NetworkClient.parseStructMaybeUpdateThrottleTimeMetrics(NetworkClient.java:561)
at
org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:657)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:442)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:232)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:208)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:168)
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.fetchCommittedOffsets(ConsumerCoordinator.java:477)
at
org.apache.kafka.clients.consumer.KafkaConsumer.committed(KafkaConsumer.java:1346)
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEventWithTimeout(KafkaEventConsumerDelegate.java:86)
... 5 common frames omitted
2017-11-29 21:22:40.704 UTC [RemotingService_EventListener, , ] ERROR
c.v.v.h.m.k.KafkaEventConsumerDelegate- Error fetching next record
java.lang.Exception: Error fetching next new record from kafka queue
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEventWithTimeout(KafkaEventConsumerDelegate.java:121)
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEvent(KafkaEventConsumerDelegate.java:64)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter.getNextEvent(EventListenerAdapter.java:76)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter.getNextEventAndAck(EventListenerAdapter.java:94)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter$1.run(EventListenerAdapter.java:125)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Correlation id for response
(384730) does not match request (384729), request header:
{api_key=9,api_version=3,correlation_id=384729,client_id=consumer-36}
at
org.apache.kafka.clients.NetworkClient.correlate(NetworkClient.java:752)
at
org.apache.kafka.clients.NetworkClient.parseStructMaybeUpdateThrottleTimeMetrics(NetworkClient.java:561)
at
org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:657)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:442)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:232)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:208)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:168)
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.fetchCommittedOffsets(ConsumerCoordinator.java:477)
at
org.apache.kafka.clients.consumer.KafkaConsumer.committed(KafkaConsumer.java:1346)
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEventWithTimeout(KafkaEventConsumerDelegate.java:86)
... 5 common frames omitted
2017-11-29 21:22:40.705 UTC [VrealmUpgradeService_EventListener, , ] ERROR
c.v.v.h.m.k.KafkaEventConsumerDelegate- Error fetching next record
java.lang.Exception: Error fetching next new record from kafka queue
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEventWithTimeout(KafkaEventConsumerDelegate.java:121)
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEvent(KafkaEventConsumerDelegate.java:64)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter.getNextEvent(EventListenerAdapter.java:76)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter.getNextEventAndAck(EventListenerAdapter.java:94)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter$1.run(EventListenerAdapter.java:125)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Correlation id for response
(386011) does not match request (386010), request header:
{api_key=9,api_version=3,correlation_id=386010,client_id=consumer-57}
at
org.apache.kafka.clients.NetworkClient.correlate(NetworkClient.java:752)
at
org.apache.kafka.clients.NetworkClient.parseStructMaybeUpdateThrottleTimeMetrics(NetworkClient.java:561)
at
org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:657)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:442)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:232)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:208)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:168)
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.fetchCommittedOffsets(ConsumerCoordinator.java:477)
at
org.apache.kafka.clients.consumer.KafkaConsumer.committed(KafkaConsumer.java:1346)
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEventWithTimeout(KafkaEventConsumerDelegate.java:86)
... 5 common frames omitted
2017-11-29 21:22:40.705 UTC [RemotingService_EventListener, , ] ERROR
c.v.v.h.m.k.KafkaEventConsumerDelegate- Error fetching next record
java.lang.Exception: Error fetching next new record from kafka queue
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEventWithTimeout(KafkaEventConsumerDelegate.java:121)
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEvent(KafkaEventConsumerDelegate.java:64)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter.getNextEvent(EventListenerAdapter.java:76)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter.getNextEventAndAck(EventListenerAdapter.java:94)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter$1.run(EventListenerAdapter.java:125)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Correlation id for response
(384731) does not match request (384730), request header:
{api_key=9,api_version=3,correlation_id=384730,client_id=consumer-36}
at
org.apache.kafka.clients.NetworkClient.correlate(NetworkClient.java:752)
at
org.apache.kafka.clients.NetworkClient.parseStructMaybeUpdateThrottleTimeMetrics(NetworkClient.java:561)
at
org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:657)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:442)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:232)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:208)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:168)
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.fetchCommittedOffsets(ConsumerCoordinator.java:477)
at
org.apache.kafka.clients.consumer.KafkaConsumer.committed(KafkaConsumer.java:1346)
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEventWithTimeout(KafkaEventConsumerDelegate.java:86)
... 5 common frames omitted
2017-11-29 21:22:40.705 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Session establishment complete on server
wdc04-dta-1-hcx-a/127.0.0.1:2181, sessionid = 0x15fcbdb62647c1e, negotiated
timeout = 4000
2017-11-29 21:22:40.705 UTC [VrealmUpgradeService_EventListener, , ] ERROR
c.v.v.h.m.k.KafkaEventConsumerDelegate- Error fetching next record
java.lang.Exception: Error fetching next new record from kafka queue
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEventWithTimeout(KafkaEventConsumerDelegate.java:121)
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEvent(KafkaEventConsumerDelegate.java:64)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter.getNextEvent(EventListenerAdapter.java:76)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter.getNextEventAndAck(EventListenerAdapter.java:94)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter$1.run(EventListenerAdapter.java:125)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Correlation id for response
(386012) does not match request (386011), request header:
{api_key=9,api_version=3,correlation_id=386011,client_id=consumer-57}
at
org.apache.kafka.clients.NetworkClient.correlate(NetworkClient.java:752)
at
org.apache.kafka.clients.NetworkClient.parseStructMaybeUpdateThrottleTimeMetrics(NetworkClient.java:561)
at
org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:657)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:442)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:232)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:208)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:168)
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.fetchCommittedOffsets(ConsumerCoordinator.java:477)
at
org.apache.kafka.clients.consumer.KafkaConsumer.committed(KafkaConsumer.java:1346)
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEventWithTimeout(KafkaEventConsumerDelegate.java:86)
... 5 common frames omitted
2017-11-29 21:22:40.706 UTC [VrealmUpgradeService_EventListener, , ] ERROR
c.v.v.h.m.k.KafkaEventConsumerDelegate- Error fetching next record
java.lang.Exception: Error fetching next new record from kafka queue
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEventWithTimeout(KafkaEventConsumerDelegate.java:121)
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEvent(KafkaEventConsumerDelegate.java:64)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter.getNextEvent(EventListenerAdapter.java:76)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter.getNextEventAndAck(EventListenerAdapter.java:94)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter$1.run(EventListenerAdapter.java:125)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Correlation id for response
(386013) does not match request (386012), request header:
{api_key=9,api_version=3,correlation_id=386012,client_id=consumer-57}
at
org.apache.kafka.clients.NetworkClient.correlate(NetworkClient.java:752)
at
org.apache.kafka.clients.NetworkClient.parseStructMaybeUpdateThrottleTimeMetrics(NetworkClient.java:561)
at
org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:657)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:442)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:232)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:208)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:168)
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.fetchCommittedOffsets(ConsumerCoordinator.java:477)
at
org.apache.kafka.clients.consumer.KafkaConsumer.committed(KafkaConsumer.java:1346)
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEventWithTimeout(KafkaEventConsumerDelegate.java:86)
... 5 common frames omitted
2017-11-29 21:22:40.707 UTC [VrealmUpgradeService_EventListener, , ] ERROR
c.v.v.h.m.k.KafkaEventConsumerDelegate- Error fetching next record
java.lang.Exception: Error fetching next new record from kafka queue
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEventWithTimeout(KafkaEventConsumerDelegate.java:121)
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEvent(KafkaEventConsumerDelegate.java:64)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter.getNextEvent(EventListenerAdapter.java:76)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter.getNextEventAndAck(EventListenerAdapter.java:94)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter$1.run(EventListenerAdapter.java:125)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Correlation id for response
(386014) does not match request (386013), request header:
{api_key=9,api_version=3,correlation_id=386013,client_id=consumer-57}
at
org.apache.kafka.clients.NetworkClient.correlate(NetworkClient.java:752)
at
org.apache.kafka.clients.NetworkClient.parseStructMaybeUpdateThrottleTimeMetrics(NetworkClient.java:561)
at
org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:657)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:442)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:232)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:208)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:168)
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.fetchCommittedOffsets(ConsumerCoordinator.java:477)
at
org.apache.kafka.clients.consumer.KafkaConsumer.committed(KafkaConsumer.java:1346)
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEventWithTimeout(KafkaEventConsumerDelegate.java:86)
... 5 common frames omitted
2017-11-29 21:22:40.708 UTC [main-EventThread, , ] INFO
org.apache.zookeeper.ClientCnxn- EventThread shut down for session:
0x15fcbdb62647c09
2017-11-29 21:22:34.471 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] WARN
org.apache.zookeeper.ClientCnxn- Client session timed out, have not heard from
server in 11660ms for sessionid 0x0
2017-11-29 21:22:44.226 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Unable to read additional data from server
sessionid 0x15fcbdb62647c0c, likely server has closed socket, closing socket
connection and attempting reconnect
2017-11-29 21:22:44.226 UTC [main-EventThread, , ] WARN
org.apache.curator.ConnectionState- Session expired event received
2017-11-29 21:22:44.226 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Unable to read additional data from server
sessionid 0x15fcbdb62647c0f, likely server has closed socket, closing socket
connection and attempting reconnect
2017-11-29 21:22:44.226 UTC [main-EventThread, , ] INFO
org.apache.zookeeper.ZooKeeper- Initiating client connection,
connectString=localhost:2181 sessionTimeout=3000
watcher=org.apache.curator.ConnectionState@268a8a24
2017-11-29 21:22:44.227 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Unable to read additional data from server
sessionid 0x15fcbdb62647c10, likely server has closed socket, closing socket
connection and attempting reconnect
2017-11-29 21:22:44.227 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Unable to read additional data from server
sessionid 0x15fcbdb62647c13, likely server has closed socket, closing socket
connection and attempting reconnect
2017-11-29 21:22:44.227 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Unable to read additional data from server
sessionid 0x15fcbdb62647c17, likely server has closed socket, closing socket
connection and attempting reconnect
2017-11-29 21:22:44.227 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Unable to read additional data from server
sessionid 0x15fcbdb62647c15, likely server has closed socket, closing socket
connection and attempting reconnect
2017-11-29 21:22:44.227 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Unable to read additional data from server
sessionid 0x15fcbdb62647c14, likely server has closed socket, closing socket
connection and attempting reconnect
2017-11-29 21:22:44.228 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Unable to read additional data from server
sessionid 0x15fcbdb62647c16, likely server has closed socket, closing socket
connection and attempting reconnect
2017-11-29 21:22:44.228 UTC [main-EventThread, , ] INFO
org.apache.zookeeper.ClientCnxn- EventThread shut down for session:
0x15fcbdb62647bfe
2017-11-29 21:22:44.228 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Unable to read additional data from server
sessionid 0x15fcbdb62647c19, likely server has closed socket, closing socket
connection and attempting reconnect
2017-11-29 21:22:44.228 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Unable to read additional data from server
sessionid 0x15fcbdb62647c1a, likely server has closed socket, closing socket
connection and attempting reconnect
2017-11-29 21:22:44.228 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Unable to read additional data from server
sessionid 0x15fcbdb62647c1b, likely server has closed socket, closing socket
connection and attempting reconnect
2017-11-29 21:22:44.229 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Opening socket connection to server
wdc04-dta-1-hcx-a/127.0.0.1:2181. Will not attempt to authenticate using SASL
(unknown error)
2017-11-29 21:22:44.229 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Socket connection established to
wdc04-dta-1-hcx-a/127.0.0.1:2181, initiating session
2017-11-29 21:22:44.232 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] WARN
org.apache.zookeeper.ClientCnxn- Client session timed out, have not heard from
server in 6391ms for sessionid 0x0
2017-11-29 21:22:44.232 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Client session timed out, have not heard from
server in 6391ms for sessionid 0x0, closing socket connection and attempting
reconnect
2017-11-29 21:22:44.233 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] WARN
org.apache.zookeeper.ClientCnxn- Client session timed out, have not heard from
server in 6398ms for sessionid 0x0
2017-11-29 21:22:44.233 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Client session timed out, have not heard from
server in 6398ms for sessionid 0x0, closing socket connection and attempting
reconnect
2017-11-29 21:22:44.234 UTC [main-EventThread, , ] INFO
c.v.v.h.orchestration.Application-
com.vmware.vchs.hybridity.orchestration.Application,
com.vmware.vchs.hybridity.orchestration.Application , inside
process(WatchedEvent)
2017-11-29 21:22:44.235 UTC [main-EventThread, , ] WARN
c.v.v.h.container.DataMonitor- Got an event that is not handled:Disconnected
2017-11-29 21:22:44.235 UTC [main-EventThread, , ] INFO
org.apache.zookeeper.ClientCnxn- EventThread shut down for session:
0x15fcbdb62647be4
2017-11-29 21:22:44.235 UTC [main-EventThread, , ] INFO
org.apache.zookeeper.ClientCnxn- EventThread shut down for session:
0x15fcbdb62647bdc
2017-11-29 21:22:34.472 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Socket connection established to
wdc04-dta-1-hcx-a/127.0.0.1:2181, initiating session
2017-11-29 21:22:34.477 UTC [main-EventThread, , ] INFO
org.apache.zookeeper.ZooKeeper- Initiating client connection,
connectString=localhost:2181 sessionTimeout=3000
watcher=org.apache.curator.ConnectionState@4f2aa529
2017-11-29 21:22:34.478 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Socket connection established to
wdc04-dta-1-hcx-a/127.0.0.1:2181, initiating session
2017-11-29 21:22:44.236 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Session establishment complete on server
wdc04-dta-1-hcx-a/127.0.0.1:2181, sessionid = 0x15fcbdb62647c1f, negotiated
timeout = 4000
2017-11-29 21:22:34.478 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Socket connection established to
wdc04-dta-1-hcx-a/127.0.0.1:2181, initiating session
2017-11-29 21:22:37.837 UTC [main-EventThread, , ] INFO
c.v.v.h.c.s.ConfigurableRunnableListener- Got a closing request- znode:
/VrealmUpgradeService
2017-11-29 21:22:37.839 UTC [main-EventThread, , ] WARN
org.apache.curator.ConnectionState- Session expired event received
2017-11-29 21:22:47.640 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] WARN
org.apache.zookeeper.ClientCnxn- Client session timed out, have not heard from
server in 6935ms for sessionid 0x15fcbdb62647c1e
2017-11-29 21:22:47.641 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Opening socket connection to server
wdc04-dta-1-hcx-a/127.0.0.1:2181. Will not attempt to authenticate using SASL
(unknown error)
2017-11-29 21:22:47.643 UTC [RemotingService_EventListener, , ] ERROR
c.v.v.h.m.k.KafkaEventConsumerDelegate- Error fetching next record
java.lang.Exception: Error fetching next new record from kafka queue
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEventWithTimeout(KafkaEventConsumerDelegate.java:121)
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEvent(KafkaEventConsumerDelegate.java:64)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter.getNextEvent(EventListenerAdapter.java:76)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter.getNextEventAndAck(EventListenerAdapter.java:94)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter$1.run(EventListenerAdapter.java:125)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Correlation id for response
(384732) does not match request (384731), request header:
{api_key=9,api_version=3,correlation_id=384731,client_id=consumer-36}
at
org.apache.kafka.clients.NetworkClient.correlate(NetworkClient.java:752)
at
org.apache.kafka.clients.NetworkClient.parseStructMaybeUpdateThrottleTimeMetrics(NetworkClient.java:561)
at
org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:657)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:442)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:232)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:208)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:168)
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.fetchCommittedOffsets(ConsumerCoordinator.java:477)
at
org.apache.kafka.clients.consumer.KafkaConsumer.committed(KafkaConsumer.java:1346)
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEventWithTimeout(KafkaEventConsumerDelegate.java:86)
... 5 common frames omitted
2017-11-29 21:22:47.644 UTC [VrealmUpgradeService_EventListener, , ] ERROR
c.v.v.h.m.k.KafkaEventConsumerDelegate- Error fetching next record
java.lang.Exception: Error fetching next new record from kafka queue
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEventWithTimeout(KafkaEventConsumerDelegate.java:121)
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEvent(KafkaEventConsumerDelegate.java:64)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter.getNextEvent(EventListenerAdapter.java:76)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter.getNextEventAndAck(EventListenerAdapter.java:94)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter$1.run(EventListenerAdapter.java:125)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Correlation id for response
(386015) does not match request (386014), request header:
{api_key=9,api_version=3,correlation_id=386014,client_id=consumer-57}
at
org.apache.kafka.clients.NetworkClient.correlate(NetworkClient.java:752)
at
org.apache.kafka.clients.NetworkClient.parseStructMaybeUpdateThrottleTimeMetrics(NetworkClient.java:561)
at
org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:657)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:442)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:232)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:208)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:168)
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.fetchCommittedOffsets(ConsumerCoordinator.java:477)
at
org.apache.kafka.clients.consumer.KafkaConsumer.committed(KafkaConsumer.java:1346)
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEventWithTimeout(KafkaEventConsumerDelegate.java:86)
... 5 common frames omitted
2017-11-29 21:22:47.645 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Client session timed out, have not heard from
server in 11660ms for sessionid 0x0, closing socket connection and attempting
reconnect
2017-11-29 21:22:47.646 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Socket connection established to
wdc04-dta-1-hcx-a/127.0.0.1:2181, initiating session
2017-11-29 21:22:47.647 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Client session timed out, have not heard from
server in 6935ms for sessionid 0x15fcbdb62647c1e, closing socket connection and
attempting reconnect
2017-11-29 21:22:47.647 UTC [main-EventThread, , ] INFO
org.apache.zookeeper.ZooKeeper- Initiating client connection,
connectString=localhost:2181 sessionTimeout=3000
watcher=org.apache.curator.ConnectionState@7245a579
2017-11-29 21:22:47.647 UTC [main-EventThread, , ] WARN
org.apache.curator.ConnectionState- Session expired event received
2017-11-29 21:22:47.647 UTC [main-EventThread, , ] INFO
org.apache.zookeeper.ZooKeeper- Initiating client connection,
connectString=localhost:2181 sessionTimeout=3000
watcher=org.apache.curator.ConnectionState@257de07c
2017-11-29 21:22:47.647 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] WARN
org.apache.zookeeper.ClientCnxn- Client session timed out, have not heard from
server in 13169ms for sessionid 0x15fcbdb62647bdd
2017-11-29 21:22:47.647 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Client session timed out, have not heard from
server in 13169ms for sessionid 0x15fcbdb62647bdd, closing socket connection
and attempting reconnect
2017-11-29 21:22:47.647 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Opening socket connection to server
wdc04-dta-1-hcx-a/127.0.0.1:2181. Will not attempt to authenticate using SASL
(unknown error)
2017-11-29 21:22:47.647 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Socket connection established to
wdc04-dta-1-hcx-a/127.0.0.1:2181, initiating session
2017-11-29 21:22:47.647 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] WARN
org.apache.zookeeper.ClientCnxn- Client session timed out, have not heard from
server in 13170ms for sessionid 0x15fcbdb62647bde
2017-11-29 21:22:47.647 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Client session timed out, have not heard from
server in 13170ms for sessionid 0x15fcbdb62647bde, closing socket connection
and attempting reconnect
2017-11-29 21:22:47.648 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] WARN
org.apache.zookeeper.ClientCnxn- Client session timed out, have not heard from
server in 13176ms for sessionid 0x0
2017-11-29 21:22:47.648 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Client session timed out, have not heard from
server in 13176ms for sessionid 0x0, closing socket connection and attempting
reconnect
2017-11-29 21:22:47.649 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Opening socket connection to server
wdc04-dta-1-hcx-a/127.0.0.1:2181. Will not attempt to authenticate using SASL
(unknown error)
2017-11-29 21:22:47.649 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Socket connection established to
wdc04-dta-1-hcx-a/127.0.0.1:2181, initiating session
2017-11-29 21:22:47.651 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Session establishment complete on server
wdc04-dta-1-hcx-a/127.0.0.1:2181, sessionid = 0x15fcbdb62647c21, negotiated
timeout = 4000
2017-11-29 21:22:47.652 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Session establishment complete on server
wdc04-dta-1-hcx-a/127.0.0.1:2181, sessionid = 0x15fcbdb62647c20, negotiated
timeout = 4000
2017-11-29 21:22:47.652 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Session establishment complete on server
wdc04-dta-1-hcx-a/127.0.0.1:2181, sessionid = 0x15fcbdb62647c23, negotiated
timeout = 4000
2017-11-29 21:22:47.655 UTC [RemotingService_EventListener, , ] ERROR
c.v.v.h.m.k.KafkaEventConsumerDelegate- Error fetching next record
java.lang.Exception: Error fetching next new record from kafka queue
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEventWithTimeout(KafkaEventConsumerDelegate.java:121)
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEvent(KafkaEventConsumerDelegate.java:64)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter.getNextEvent(EventListenerAdapter.java:76)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter.getNextEventAndAck(EventListenerAdapter.java:94)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter$1.run(EventListenerAdapter.java:125)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Correlation id for response
(384733) does not match request (384732), request header:
{api_key=9,api_version=3,correlation_id=384732,client_id=consumer-36}
at
org.apache.kafka.clients.NetworkClient.correlate(NetworkClient.java:752)
at
org.apache.kafka.clients.NetworkClient.parseStructMaybeUpdateThrottleTimeMetrics(NetworkClient.java:561)
at
org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:657)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:442)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:232)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:208)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:168)
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.fetchCommittedOffsets(ConsumerCoordinator.java:477)
at
org.apache.kafka.clients.consumer.KafkaConsumer.committed(KafkaConsumer.java:1346)
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEventWithTimeout(KafkaEventConsumerDelegate.java:86)
... 5 common frames omitted
2017-11-29 21:22:47.655 UTC [VrealmUpgradeService_EventListener, , ] ERROR
c.v.v.h.m.k.KafkaEventConsumerDelegate- Error fetching next record
java.lang.Exception: Error fetching next new record from kafka queue
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEventWithTimeout(KafkaEventConsumerDelegate.java:121)
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEvent(KafkaEventConsumerDelegate.java:64)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter.getNextEvent(EventListenerAdapter.java:76)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter.getNextEventAndAck(EventListenerAdapter.java:94)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter$1.run(EventListenerAdapter.java:125)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Correlation id for response
(386016) does not match request (386015), request header:
{api_key=9,api_version=3,correlation_id=386015,client_id=consumer-57}
at
org.apache.kafka.clients.NetworkClient.correlate(NetworkClient.java:752)
at
org.apache.kafka.clients.NetworkClient.parseStructMaybeUpdateThrottleTimeMetrics(NetworkClient.java:561)
at
org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:657)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:442)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:232)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:208)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:168)
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.fetchCommittedOffsets(ConsumerCoordinator.java:477)
at
org.apache.kafka.clients.consumer.KafkaConsumer.committed(KafkaConsumer.java:1346)
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEventWithTimeout(KafkaEventConsumerDelegate.java:86)
... 5 common frames omitted
2017-11-29 21:22:47.658 UTC [RemotingService_EventListener, , ] ERROR
c.v.v.h.m.k.KafkaEventConsumerDelegate- Error fetching next record
java.lang.Exception: Error fetching next new record from kafka queue
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEventWithTimeout(KafkaEventConsumerDelegate.java:121)
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEvent(KafkaEventConsumerDelegate.java:64)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter.getNextEvent(EventListenerAdapter.java:76)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter.getNextEventAndAck(EventListenerAdapter.java:94)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter$1.run(EventListenerAdapter.java:125)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Correlation id for response
(384734) does not match request (384733), request header:
{api_key=9,api_version=3,correlation_id=384733,client_id=consumer-36}
at
org.apache.kafka.clients.NetworkClient.correlate(NetworkClient.java:752)
at
org.apache.kafka.clients.NetworkClient.parseStructMaybeUpdateThrottleTimeMetrics(NetworkClient.java:561)
at
org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:657)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:442)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:232)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:208)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:168)
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.fetchCommittedOffsets(ConsumerCoordinator.java:477)
at
org.apache.kafka.clients.consumer.KafkaConsumer.committed(KafkaConsumer.java:1346)
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEventWithTimeout(KafkaEventConsumerDelegate.java:86)
... 5 common frames omitted
2017-11-29 21:22:47.659 UTC [VrealmUpgradeService_EventListener, , ] ERROR
c.v.v.h.m.k.KafkaEventConsumerDelegate- Error fetching next record
java.lang.Exception: Error fetching next new record from kafka queue
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEventWithTimeout(KafkaEventConsumerDelegate.java:121)
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEvent(KafkaEventConsumerDelegate.java:64)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter.getNextEvent(EventListenerAdapter.java:76)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter.getNextEventAndAck(EventListenerAdapter.java:94)
at
com.vmware.vchs.hybridity.messaging.adapter.EventListenerAdapter$1.run(EventListenerAdapter.java:125)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Correlation id for response
(386017) does not match request (386016), request header:
{api_key=9,api_version=3,correlation_id=386016,client_id=consumer-57}
at
org.apache.kafka.clients.NetworkClient.correlate(NetworkClient.java:752)
at
org.apache.kafka.clients.NetworkClient.parseStructMaybeUpdateThrottleTimeMetrics(NetworkClient.java:561)
at
org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:657)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:442)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:232)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:208)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:168)
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.fetchCommittedOffsets(ConsumerCoordinator.java:477)
at
org.apache.kafka.clients.consumer.KafkaConsumer.committed(KafkaConsumer.java:1346)
at
com.vmware.vchs.hybridity.messaging.kafka.KafkaEventConsumerDelegate.getNextEventWithTimeout(KafkaEventConsumerDelegate.java:86)
... 5 common frames omitted
2017-11-29 21:22:47.659 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Opening socket connection to server
wdc04-dta-1-hcx-a/127.0.0.1:2181. Will not attempt to authenticate using SASL
(unknown error)
2017-11-29 21:22:47.660 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Socket connection established to
wdc04-dta-1-hcx-a/127.0.0.1:2181, initiating session
2017-11-29 21:22:47.661 UTC [main-EventThread, , ] INFO
org.apache.zookeeper.ClientCnxn- EventThread shut down for session:
0x15fcbdb62647c07
2017-11-29 21:22:47.666 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Session establishment complete on server
wdc04-dta-1-hcx-a/127.0.0.1:2181, sessionid = 0x15fcbdb62647c24, negotiated
timeout = 4000
2017-11-29 21:22:47.680 UTC [main-EventThread, , ] INFO
org.apache.zookeeper.ClientCnxn- EventThread shut down for session:
0x15fcbdb62647be2
2017-11-29 21:22:50.504 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] WARN
org.apache.zookeeper.ClientCnxn- Client session timed out, have not heard from
server in 6268ms for sessionid 0x15fcbdb62647c1f
2017-11-29 21:22:50.504 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] WARN
org.apache.zookeeper.ClientCnxn- Client session timed out, have not heard from
server in 2852ms for sessionid 0x15fcbdb62647c23
2017-11-29 21:22:50.505 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Client session timed out, have not heard from
server in 6268ms for sessionid 0x15fcbdb62647c1f, closing socket connection and
attempting reconnect
2017-11-29 21:22:50.505 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Client session timed out, have not heard from
server in 2852ms for sessionid 0x15fcbdb62647c23, closing socket connection and
attempting reconnect
2017-11-29 21:22:50.505 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] WARN
org.apache.zookeeper.ClientCnxn- Client session timed out, have not heard from
server in 2854ms for sessionid 0x15fcbdb62647c21
2017-11-29 21:22:50.505 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] WARN
org.apache.zookeeper.ClientCnxn- Client session timed out, have not heard from
server in 2853ms for sessionid 0x15fcbdb62647c20
2017-11-29 21:22:50.505 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Client session timed out, have not heard from
server in 2854ms for sessionid 0x15fcbdb62647c21, closing socket connection and
attempting reconnect
2017-11-29 21:22:50.505 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Client session timed out, have not heard from
server in 2853ms for sessionid 0x15fcbdb62647c20, closing socket connection and
attempting reconnect
2017-11-29 21:22:50.505 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] WARN
org.apache.zookeeper.ClientCnxn- Client session timed out, have not heard from
server in 2839ms for sessionid 0x15fcbdb62647c24
2017-11-29 21:22:50.505 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Opening socket connection to server
wdc04-dta-1-hcx-a/127.0.0.1:2181. Will not attempt to authenticate using SASL
(unknown error)
2017-11-29 21:22:50.506 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Opening socket connection to server
wdc04-dta-1-hcx-a/127.0.0.1:2181. Will not attempt to authenticate using SASL
(unknown error)
2017-11-29 21:22:50.506 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Socket connection established to
wdc04-dta-1-hcx-a/127.0.0.1:2181, initiating session
2017-11-29 21:22:50.505 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Client session timed out, have not heard from
server in 2839ms for sessionid 0x15fcbdb62647c24, closing socket connection and
attempting reconnect
2017-11-29 21:22:50.506 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Socket connection established to
wdc04-dta-1-hcx-a/127.0.0.1:2181, initiating session
2017-11-29 21:22:50.506 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Opening socket connection to server
wdc04-dta-1-hcx-a/127.0.0.1:2181. Will not attempt to authenticate using SASL
(unknown error)
2017-11-29 21:22:50.506 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Socket connection established to
wdc04-dta-1-hcx-a/127.0.0.1:2181, initiating session
2017-11-29 21:22:50.506 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Opening socket connection to server
wdc04-dta-1-hcx-a/127.0.0.1:2181. Will not attempt to authenticate using SASL
(unknown error)
2017-11-29 21:22:50.506 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Socket connection established to
wdc04-dta-1-hcx-a/127.0.0.1:2181, initiating session
2017-11-29 21:22:50.506 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Opening socket connection to server
wdc04-dta-1-hcx-a/127.0.0.1:2181. Will not attempt to authenticate using SASL
(unknown error)
2017-11-29 21:22:50.506 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Socket connection established to
wdc04-dta-1-hcx-a/127.0.0.1:2181, initiating session
2017-11-29 21:22:50.506 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Opening socket connection to server
wdc04-dta-1-hcx-a/127.0.0.1:2181. Will not attempt to authenticate using SASL
(unknown error)
2017-11-29 21:22:50.506 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Socket connection established to
wdc04-dta-1-hcx-a/127.0.0.1:2181, initiating session
2017-11-29 21:22:50.506 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Opening socket connection to server
wdc04-dta-1-hcx-a/127.0.0.1:2181. Will not attempt to authenticate using SASL
(unknown error)
2017-11-29 21:22:50.506 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Opening socket connection to server
wdc04-dta-1-hcx-a/127.0.0.1:2181. Will not attempt to authenticate using SASL
(unknown error)
2017-11-29 21:22:50.506 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Opening socket connection to server
wdc04-dta-1-hcx-a/127.0.0.1:2181. Will not attempt to authenticate using SASL
(unknown error)
2017-11-29 21:22:50.506 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Socket connection established to
wdc04-dta-1-hcx-a/127.0.0.1:2181, initiating session
2017-11-29 21:22:50.506 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Socket connection established to
wdc04-dta-1-hcx-a/127.0.0.1:2181, initiating session
2017-11-29 21:22:50.507 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Opening socket connection to server
wdc04-dta-1-hcx-a/127.0.0.1:2181. Will not attempt to authenticate using SASL
(unknown error)
2017-11-29 21:22:50.507 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Socket connection established to
wdc04-dta-1-hcx-a/127.0.0.1:2181, initiating session
2017-11-29 21:22:50.507 UTC [main-EventThread, , ] INFO
org.apache.zookeeper.ClientCnxn- EventThread shut down for session:
0x15fcbdb62647c01
2017-11-29 21:22:50.507 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Socket connection established to
wdc04-dta-1-hcx-a/127.0.0.1:2181, initiating session
2017-11-29 21:22:50.507 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] WARN
org.apache.zookeeper.ClientCnxn- Unable to reconnect to ZooKeeper service,
session 0x15fcbdb62647c0b has expired
2017-11-29 21:22:50.507 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Opening socket connection to server
wdc04-dta-1-hcx-a/127.0.0.1:2181. Will not attempt to authenticate using SASL
(unknown error)
2017-11-29 21:22:50.507 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Socket connection established to
wdc04-dta-1-hcx-a/127.0.0.1:2181, initiating session
2017-11-29 21:22:50.507 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] WARN
org.apache.zookeeper.ClientCnxn- Unable to reconnect to ZooKeeper service,
session 0x15fcbdb62647c03 has expired
2017-11-29 21:22:50.508 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Unable to reconnect to ZooKeeper service,
session 0x15fcbdb62647c03 has expired, closing socket connection
2017-11-29 21:22:50.508 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] WARN
org.apache.zookeeper.ClientCnxn- Unable to reconnect to ZooKeeper service,
session 0x15fcbdb62647c04 has expired
2017-11-29 21:22:50.508 UTC [main-SendThread(wdc04-dta-1-hcx-a:2181), , ] INFO
org.apache.zookeeper.ClientCnxn- Unable to reconnect to ZooKeeper service,
session 0x15fcbdb62647c04 has expired, closing socket connection
> KafkaProducer.flush hangs when NetworkClient.handleCompletedReceives throws
> exception
> -------------------------------------------------------------------------------------
>
> Key: KAFKA-4669
> URL: https://issues.apache.org/jira/browse/KAFKA-4669
> Project: Kafka
> Issue Type: Bug
> Components: clients
> Affects Versions: 0.9.0.1
> Reporter: Cheng Ju
> Assignee: Rajini Sivaram
> Priority: Critical
> Labels: reliability
> Fix For: 0.11.0.1, 1.0.0
>
>
> There is no try catch in NetworkClient.handleCompletedReceives. If an
> exception is thrown after inFlightRequests.completeNext(source), then the
> corresponding RecordBatch's done will never get called, and
> KafkaProducer.flush will hang on this RecordBatch.
> I've checked 0.10 code and think this bug does exist in 0.10 versions.
> A real case. First a correlateId not match exception happens:
> 13 Jan 2017 17:08:24,059 ERROR [kafka-producer-network-thread | producer-21]
> (org.apache.kafka.clients.producer.internals.Sender.run:130) - Uncaught
> error in kafka producer I/O thread:
> java.lang.IllegalStateException: Correlation id for response (703766) does
> not match request (703764)
> at
> org.apache.kafka.clients.NetworkClient.correlate(NetworkClient.java:477)
> at
> org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:440)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:265)
> at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216)
> at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:128)
> at java.lang.Thread.run(Thread.java:745)
> Then jstack shows the thread is hanging on:
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> at
> org.apache.kafka.clients.producer.internals.ProduceRequestResult.await(ProduceRequestResult.java:57)
> at
> org.apache.kafka.clients.producer.internals.RecordAccumulator.awaitFlushCompletion(RecordAccumulator.java:425)
> at
> org.apache.kafka.clients.producer.KafkaProducer.flush(KafkaProducer.java:544)
> at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:224)
> at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
> at java.lang.Thread.run(Thread.java:745)
> client code
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
