That's right. If one message in a batch exceeds the size limit, the whole
batch is rejected.

When determining message.max.bytes, the most important thing to consider is
probably memory since currently we need to allocate memory for a full
message in the broker and the producer and the consumer client.

Thanks,

Jun


On Wed, Aug 27, 2014 at 9:52 PM, Alexis Midon <
alexis.mi...@airbedandbreakfast.com> wrote:

> am I miss reading this loop:
>
> https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/log/Log.scala#L265-L269
>
> it seems like all messages from `validMessages` (which is
> ByteBufferMessageSet) are NOT appended if one of the message size exceeds
> the limit.
>
> I hope I'm missing something.
>
>
>
> On Wed, Aug 27, 2014 at 9:38 PM, Alexis Midon <
> alexis.mi...@airbedandbreakfast.com> wrote:
>
> > Hi Jun,
> >
> > thanks for you answer.
> > Unfortunately the size won't help much, I'd like to see the actual
> message
> > data.
> >
> > By the way what are the things to consider when deciding on
> > `message.max.bytes` value?
> >
> >
> >
> >
> >
> >
> > On Wed, Aug 27, 2014 at 9:06 PM, Jun Rao <jun...@gmail.com> wrote:
> >
> >> The message size check is currently only done on the broker. If you
> enable
> >> trace level logging in RequestChannel, you will see the produce request,
> >> which includes the size of each partition.
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >>
> >> On Wed, Aug 27, 2014 at 4:40 PM, Alexis Midon <
> >> alexis.mi...@airbedandbreakfast.com> wrote:
> >>
> >> > Hello,
> >> >
> >> > my brokers are reporting that some received messages exceed the
> >> > `message.max.bytes` value.
> >> > I'd like to know what producers are at fault but It is pretty much
> >> > impossible:
> >> > - the brokers don't log the content of the rejected messages
> >> > - the log messages do not contain the IP of the producers
> >> > - on the consumer side, no exception is thrown (afaik it is because
> >> Ack-0
> >> > is used). The only kind of notification is to closed the connection.
> >> >
> >> > [1] Do you have any suggestions to track down the guilty producers or
> >> find
> >> > out the message content?
> >> >
> >> > Even though it makes total sense to have the limit defined and applied
> >> on
> >> > the brokers, I was thinking that this check could also be applied by
> the
> >> > producers. Some google results suggest that `message.max.bytes` might
> be
> >> > used by the producers but I can't find any trace of that behavior in
> the
> >> > code.
> >> >
> >> > The closest thing I have is
> >> >
> >> >
> >>
> https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/producer/SyncProducer.scala#L67
> >> > but it simply logs the message size and content and the log level is
> >> trace.
> >> >
> >> > [2] could you please confirm if such a producer-side check exists?
> >> >
> >> >
> >> > thanks!
> >> >
> >> > Alexis
> >> >
> >>
> >
> >
>

Reply via email to