Hi, I'm trying to stream large message with Kafka into Spark. Generally this has been working nicely, but I found one message (5.1MB in size) which is clogging my pipeline up. I have these settings in server.properties: fetch.message.max.bytes=10485760 replica.fetch.max.bytes=10485760 message.max.bytes=10485760 fetch.size=10485760
I'm not getting any obvious errors in the logs and I can retrieve the large message with this command: kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic mytopic --fetch-size=10485760 I noticed recently after digging into this problem that the kafkaServer.out log is complaining that the fetch.message.max.bytes parameter is not valid: [2014-06-25 11:33:36,547] WARN Property fetch.message.max.bytes is not valid (kafka.utils.VerifiableProperties) [2014-06-25 11:33:36,547] WARN Property fetch.size is not valid (kafka.utils.VerifiableProperties) That seems like the most critical parameter for my needs. It is apparently not recognizing that it is a parameter despite it being listed on the configuration website (https://kafka.apache.org/08/configuration.html). I'm using 0.8.1.1. Any ideas? many thanks for reading this!
