Hi Kafka Team, I am trying to find out Kafka Internal and how a message can be corrupted or lost at brokers side.
I have refer to following documentations for monitoring: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Internals http://kafka.apache.org/documentation.html#monitoring I am looking at following beans: "kafka.server":type="BrokerTopicMetrics",name="test-FailedProduceRequestsPerSec" "kafka.server":type="BrokerTopicMetrics",name="test-BytesRejectedPerSec" I see following exception on Brokers side rejecting due to large request. This is great but it does not show the source ip of prodcuer that caused this issue ? Is there any way to log and capture this ? *[2014-10-14 22:09:53,262] ERROR [KafkaApi-2] Error processing ProducerRequest with correlation id 28795280 from client XXXXXon partition [XXX,17] (kafka.server.KafkaApis)kafka.common.MessageSizeTooLargeException: Message size is 2924038 bytes which exceeds the maximum configured message size of 2097152. * Can you this be reported as separate metric "MessageSizeTooLargeException per topic" ? Also, what is best way to find the CRC check error from the consumer side ? How do you debug this ? e.g log line: *11 Dec 2014 07:22:33,387 ERROR [pool-15-thread-4] * *kafka.message.InvalidMessageException: Message is corrupt (stored crc = 1834644195, computed crc = 2374999037)* Also, is there any jira open to update with list all latest metrics and its format and what it means ? http://kafka.apache.org/documentation.html#monitoring. Please see attached image for list of all metrics. Version of Broker is 0.8.1.1. Thanks, Bhavesh
