Re: Consumer behavior when message exceeds fetch.message.max.bytes

2013-08-01 Thread Jun Rao
Yes. It's good to enforce that. Could you file a jira and attach your patch there? Thanks, Jun On Thu, Aug 1, 2013 at 7:39 AM, Sam Meder wrote: > Seems like a good idea to enforce this? Maybe something like this: > > diff --git a/core/src/main/scala/kafka/server/KafkaConfig.scala > b/core/src/

Re: Consumer behavior when message exceeds fetch.message.max.bytes

2013-08-01 Thread Sam Meder
Seems like a good idea to enforce this? Maybe something like this: diff --git a/core/src/main/scala/kafka/server/KafkaConfig.scala b/core/src/main/scala/kafka/server/KafkaConfig.scala index a64b210..1c3bfdd 100644 --- a/core/src/main/scala/kafka/server/KafkaConfig.scala +++ b/core/src/main/scala/

Re: Consumer behavior when message exceeds fetch.message.max.bytes

2013-08-01 Thread Sam Meder
We put together a test case that showws the exception is never thrown. I looked into it a bit more and filed https://issues.apache.org/jira/browse/KAFKA-994 which has the details. /Sam On Aug 1, 2013, at 7:03 AM, Jun Rao wrote: > If you look at ConsumerIterator, we will throw MessageSizeTooLa

Re: Consumer behavior when message exceeds fetch.message.max.bytes

2013-07-31 Thread Jun Rao
server: replica.fetch.max.bytes should be >= message.max.bytes. Otherwise, the follower will get stuck when replicating data from the leader. Thanks, Jun On Wed, Jul 31, 2013 at 10:10 AM, Sam Meder wrote: > I also noticed that there are two properties related to messages size on > the server:

Re: Consumer behavior when message exceeds fetch.message.max.bytes

2013-07-31 Thread Jun Rao
If you look at ConsumerIterator, we will throw MessageSizeTooLargeException to the caller if the fetch size is too small. Thanks, Jun On Wed, Jul 31, 2013 at 8:25 AM, Sam Meder wrote: > We're expecting to occasionally have to deal with pretty large messages > being sent to Kafka. We will of co

Re: Consumer behavior when message exceeds fetch.message.max.bytes

2013-07-31 Thread Sam Meder
I also noticed that there are two properties related to messages size on the server: replica.fetch.max.bytes and message.max.bytes. What happens when replica.fetch.max.bytes is lower than message.max.bytes? Should there even be two properties? /Sam On Jul 31, 2013, at 5:25 PM, Sam Meder wrote