Re: Consumer behavior when message exceeds fetch.message.max.bytes

2013-08-01 Thread Jun Rao
Yes. It's good to enforce that. Could you file a jira and attach your patch there? Thanks, Jun On Thu, Aug 1, 2013 at 7:39 AM, Sam Meder wrote: > Seems like a good idea to enforce this? Maybe something like this: > > diff --git a/core/src/main/scala/kafka/server/KafkaConfig.scala > b/core/src/

Re: Consumer behavior when message exceeds fetch.message.max.bytes

2013-08-01 Thread Sam Meder
Seems like a good idea to enforce this? Maybe something like this: diff --git a/core/src/main/scala/kafka/server/KafkaConfig.scala b/core/src/main/scala/kafka/server/KafkaConfig.scala index a64b210..1c3bfdd 100644 --- a/core/src/main/scala/kafka/server/KafkaConfig.scala +++ b/core/src/main/scala/

Re: Consumer behavior when message exceeds fetch.message.max.bytes

2013-08-01 Thread Sam Meder
We put together a test case that showws the exception is never thrown. I looked into it a bit more and filed https://issues.apache.org/jira/browse/KAFKA-994 which has the details. /Sam On Aug 1, 2013, at 7:03 AM, Jun Rao wrote: > If you look at ConsumerIterator, we will throw MessageSizeTooLa

Re: Consumer behavior when message exceeds fetch.message.max.bytes

2013-07-31 Thread Jun Rao
server: replica.fetch.max.bytes should be >= message.max.bytes. Otherwise, the follower will get stuck when replicating data from the leader. Thanks, Jun On Wed, Jul 31, 2013 at 10:10 AM, Sam Meder wrote: > I also noticed that there are two properties related to messages size on > the server:

Re: Consumer behavior when message exceeds fetch.message.max.bytes

2013-07-31 Thread Jun Rao
If you look at ConsumerIterator, we will throw MessageSizeTooLargeException to the caller if the fetch size is too small. Thanks, Jun On Wed, Jul 31, 2013 at 8:25 AM, Sam Meder wrote: > We're expecting to occasionally have to deal with pretty large messages > being sent to Kafka. We will of co

Re: Consumer behavior when message exceeds fetch.message.max.bytes

2013-07-31 Thread Sam Meder
I also noticed that there are two properties related to messages size on the server: replica.fetch.max.bytes and message.max.bytes. What happens when replica.fetch.max.bytes is lower than message.max.bytes? Should there even be two properties? /Sam On Jul 31, 2013, at 5:25 PM, Sam Meder wrote

Consumer behavior when message exceeds fetch.message.max.bytes

2013-07-31 Thread Sam Meder
We're expecting to occasionally have to deal with pretty large messages being sent to Kafka. We will of course set the fetch size appropriately high, but are concerned about the behavior when the message exceeds the fetch size. As far as I can tell the current behavior when a message that is too