but I found one message (5.1MB in size) which is clogging my pipeline up Have you ensured that the "fetch.message.max.bytes" on the consumer config is set to > 5.1 MB?
On Thu, Jun 26, 2014 at 6:14 PM, Louis Clark <sfgypsy...@gmail.com> wrote: > in the consumer.properties file, I've got (default?): > > zookeeper.connect=127.0.0.1:2181 > > zookeeper.connection.timeout.ms=1000000 > > group.id=test-consumer-group > > thanks, > > -Louis > > > On Thu, Jun 26, 2014 at 6:04 PM, Guozhang Wang <wangg...@gmail.com> wrote: > > > Hi Louis, > > > > What are your consumer's config properties? > > > > Guozhang > > > > > > On Thu, Jun 26, 2014 at 5:54 PM, Louis Clark <sfgypsy...@gmail.com> > wrote: > > > >> Hi, I'm trying to stream large message with Kafka into Spark. Generally > >> this has been working nicely, but I found one message (5.1MB in size) > >> which > >> is clogging my pipeline up. I have these settings in server.properties: > >> fetch.message.max.bytes=10485760 > >> replica.fetch.max.bytes=10485760 > >> message.max.bytes=10485760 > >> fetch.size=10485760 > >> > >> I'm not getting any obvious errors in the logs and I can retrieve the > >> large > >> message with this command: > >> kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning > >> --topic mytopic --fetch-size=10485760 > >> > >> I noticed recently after digging into this problem that the > >> kafkaServer.out > >> log is complaining that the fetch.message.max.bytes parameter is not > >> valid: > >> [2014-06-25 11:33:36,547] WARN Property fetch.message.max.bytes is not > >> valid (kafka.utils.VerifiableProperties) > >> [2014-06-25 11:33:36,547] WARN Property fetch.size is not valid > >> (kafka.utils.VerifiableProperties) > >> That seems like the most critical parameter for my needs. It is > >> apparently > >> not recognizing that it is a parameter despite it being listed on the > >> configuration website (https://kafka.apache.org/08/configuration.html). > >> I'm using 0.8.1.1. Any ideas? > >> > >> many thanks for reading this! > >> > > > > > > > > -- > > -- Guozhang > > >
