Seems like a good idea to enforce this? Maybe something like this: diff --git a/core/src/main/scala/kafka/server/KafkaConfig.scala b/core/src/main/scala/kafka/server/KafkaConfig.scala index a64b210..1c3bfdd 100644 --- a/core/src/main/scala/kafka/server/KafkaConfig.scala +++ b/core/src/main/scala/kafka/server/KafkaConfig.scala @@ -198,7 +198,7 @@ class KafkaConfig private (val props: VerifiableProperties) extends ZKConfig(pro val replicaSocketReceiveBufferBytes = props.getInt(ReplicaSocketReceiveBufferBytesProp, ConsumerConfig.SocketBufferSize) /* the number of byes of messages to attempt to fetch */ - val replicaFetchMaxBytes = props.getInt(ReplicaFetchMaxBytesProp, ConsumerConfig.FetchSize) + val replicaFetchMaxBytes = props.getIntInRange(ReplicaFetchMaxBytesProp, ConsumerConfig.FetchSize, (messageMaxBytes, Int.MaxValue)) /* max wait time for each fetcher request issued by follower replicas*/ val replicaFetchWaitMaxMs = props.getInt(ReplicaFetchWaitMaxMsProp, 500)
Not sure is message.max.bytes only counts payload or whole message + any headers, so it may be that it should be a bit larger even. /Sam On Aug 1, 2013, at 7:04 AM, Jun Rao <jun...@gmail.com> wrote: > server: replica.fetch.max.bytes should be >= message.max.bytes. Otherwise, > the follower will get stuck when replicating data from the leader. > > Thanks, > > Jun > > > On Wed, Jul 31, 2013 at 10:10 AM, Sam Meder <sam.me...@jivesoftware.com>wrote: > >> I also noticed that there are two properties related to messages size on >> the server: replica.fetch.max.bytes and message.max.bytes. What happens >> when replica.fetch.max.bytes is lower than message.max.bytes? Should there >> even be two properties? >> >> /Sam >> >> On Jul 31, 2013, at 5:25 PM, Sam Meder <sam.me...@jivesoftware.com> wrote: >> >>> We're expecting to occasionally have to deal with pretty large messages >> being sent to Kafka. We will of course set the fetch size appropriately >> high, but are concerned about the behavior when the message exceeds the >> fetch size. As far as I can tell the current behavior when a message that >> is too large is encountered is to pretend it is not there and not notify >> the consumer in any way. IMO it would be better to throw an exception than >> silently ignoring the issue (with the current code one can't really >> distinguish a large message from no data at all). >>> >>> Thoughts? >>> >>> /Sam >> >>
