thanks for the help. For others who happen upon this thread, the problem
was indeed on the consumer side. Spark (0.9.1) needs a bit of help setting
the Kafka properties for big messages.
// setup Kafka with manual parameters to allow big messaging
//see
spark/external/kafka/src/main/scala
I believe so. I have set
fetch.message.max.bytes=10485760
in both the consumer.properties and the server.properties config files,
then restarted kafka -> same problem. I'm following up on some of
Guozhang's other suggestions now.
One thing I'm confused about (I should read the docs again) is w
but I found one message (5.1MB in size) which
is clogging my pipeline up
Have you ensured that the "fetch.message.max.bytes" on the consumer config
is set to > 5.1 MB?
On Thu, Jun 26, 2014 at 6:14 PM, Louis Clark wrote:
> in the consumer.properties file, I've got (default?):
>
> zookeeper.conn
in the consumer.properties file, I've got (default?):
zookeeper.connect=127.0.0.1:2181
zookeeper.connection.timeout.ms=100
group.id=test-consumer-group
thanks,
-Louis
On Thu, Jun 26, 2014 at 6:04 PM, Guozhang Wang wrote:
> Hi Louis,
>
> What are your consumer's config properties?
>
> G
Hi Louis,
What are your consumer's config properties?
Guozhang
On Thu, Jun 26, 2014 at 5:54 PM, Louis Clark wrote:
> Hi, I'm trying to stream large message with Kafka into Spark. Generally
> this has been working nicely, but I found one message (5.1MB in size) which
> is clogging my pipeline
Hi, I'm trying to stream large message with Kafka into Spark. Generally
this has been working nicely, but I found one message (5.1MB in size) which
is clogging my pipeline up. I have these settings in server.properties:
fetch.message.max.bytes=10485760
replica.fetch.max.bytes=10485760
message.max
