[
https://issues.apache.org/jira/browse/KAFKA-1756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14199656#comment-14199656
]
Jun Rao commented on KAFKA-1756:
--------------------------------
I am also wondering if we need to make max.message.bytes a topic level config.
This makes replica fetchers as well as tools like MirrorMaker harder to
configure since they have to be aware of all per topic level values. I am not
sure if there is a strong use case to customize a different max message size
per topic.
> never allow the replica fetch size to be less than the max message size
> -----------------------------------------------------------------------
>
> Key: KAFKA-1756
> URL: https://issues.apache.org/jira/browse/KAFKA-1756
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8.1.1, 0.8.2
> Reporter: Joe Stein
> Priority: Blocker
> Fix For: 0.8.2
>
>
> There exists a very hazardous scenario where if the max.message.bytes is
> greather than the replica.fetch.max.bytes the message will never replicate.
> This will bring the ISR down to 1 (eventually/quickly once
> replica.lag.max.messages is reached). If during this window the leader itself
> goes out of the ISR then the new leader will commit the last offset it
> replicated. This is also bad for sync producers with -1 ack because they will
> all block (heard affect caused upstream) in this scenario too.
> The fix here is two fold
> 1) when setting max.message.bytes using kafka-topics we must check first each
> and every broker (which will need some thought about how todo this because of
> the topiccommand zk notification) that max.message.bytes <=
> replica.fetch.max.bytes and if it is NOT then DO NOT create the topic
> 2) if you change this in server.properties then the broker should not start
> if max.message.bytes > replica.fetch.max.bytes
> This does beg the question/issue some about centralizing certain/some/all
> configurations so that inconsistencies do not occur (where broker 1 has
> max.message.bytes > replica.fetch.max.bytes but broker 2 max.message.bytes <=
> replica.fetch.max.bytes because of error in properties). I do not want to
> conflate this ticket but I think it is worth mentioning/bringing up here as
> it is a good example where it could make sense.
> I set this as BLOCKER for 0.8.2-beta because we did so much work to enable
> consistency vs availability and 0 data loss this corner case should be part
> of 0.8.2-final
> Also, I could go one step further (though I would not consider this part as a
> blocker for 0.8.2 but interested to what other folks think) about a consumer
> replica fetch size so that if the message max is increased messages will no
> longer be consumed (since the consumer fetch max would be < max.message.bytes
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
