Hi Kafka users!
I have a legacy kafka cluster that hosts a topic of click stream events. Topic
is consumed by 10 instances of an app with the same consumer group. Sometimes I
see exception is logs of those consumers like this:
org.apache.kafka.common.protocol.types.SchemaException: Error reading field
'responses': Error reading field 'partition_responses': Error reading field
'record_set': java.lang.IllegalArgumentException
at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:73)
at
org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:464)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:279)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:303)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:197)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:187)
at
org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:877)
at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:829)
at
org.apache.flink.streaming.connectors.kafka.internal.KafkaConsumerThread.run(KafkaConsumerThread.java:257)
After that app instance is restarted from the previously commited offset and
continues without any errors. Broker logs are full of exceptions like this:
ERROR Processor got uncaught exception. (kafka.network.Processor)
java.lang.ArrayIndexOutOfBoundsException
Broker exceptions occur much more often then exceptions on consumer(regularly
each minute on broker VS sporadically less then 10 times per day on consumer)
and it seems that doesn't cause consumer to experience any problems (but never
the less these problems may have the same reason).
By this time I havent found niether reasons nor regularity of consumer
exception. I've found an issue[1] that seems to be the same, but it's not
resolved by now.
May be someone has already met this type of error and knows it reason?
kafka broker version: kafka_2.11-0.9.0.0
kafka client version: kafka-clients-0.9.0.0
[1] https://issues.apache.org/jira/browse/KAFKA-4961
P.S. I wish I could just upgrade kafka broker version, but currently it is not
the option.
