[
https://issues.apache.org/jira/browse/KAFKA-1409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-1409:
---------------------------------
Attachment: KAFKA-1409_2014-04-24_16:26:06.patch
> oversized messages can slow down the brokers
> --------------------------------------------
>
> Key: KAFKA-1409
> URL: https://issues.apache.org/jira/browse/KAFKA-1409
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8.2
> Reporter: Jun Rao
> Assignee: Guozhang Wang
> Priority: Critical
> Fix For: 0.8.2
>
> Attachments: KAFKA-1409.patch, KAFKA-1409_2014-04-23_14:10:03.patch,
> KAFKA-1409_2014-04-23_14:25:48.patch, KAFKA-1409_2014-04-23_15:20:43.patch,
> KAFKA-1409_2014-04-24_16:26:06.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> There are 2 main issues.
> 1. The broker first decompresses and then recompresses each message (to
> assign new offsets) before validating the message size (we have to do the
> validation after recompression since the message size could change). So, it
> can spend many secs to decompress/recompress an oversized message, only to be
> dropped later. While this was happening, a request thread was tied up, which
> reduced the capacity on the broker.
> 2. Both the fetch and producer requests need to hold a leader lock (per
> partition). So, if the producer is slow in appending the log, it will block
> other producer/fetch requests on the same partition.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
