[
https://issues.apache.org/jira/browse/KAFKA-3718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Onur Karaman updated KAFKA-3718:
--------------------------------
Fix Version/s: 0.10.0.0
> propagate all KafkaConfig __consumer_offsets configs to OffsetConfig
> instantiation
> ----------------------------------------------------------------------------------
>
> Key: KAFKA-3718
> URL: https://issues.apache.org/jira/browse/KAFKA-3718
> Project: Kafka
> Issue Type: Bug
> Reporter: Onur Karaman
> Assignee: Onur Karaman
> Fix For: 0.10.0.0
>
>
> Kafka has two configurable compression codecs: the one used by the client
> (source codec) and the one finally used when storing into the log (target
> codec). The target codec defaults to KafkaConfig.compressionType and can be
> dynamically configured through zookeeper.
> The GroupCoordinator appends group membership information into the
> __consumer_offsets topic by:
> 1. making a message with group membership information
> 2. making a MessageSet with the single message compressed with the source
> codec
> 3. doing a log.append on the MessageSet
> Without this patch, KafkaConfig.offsetsTopicCompressionCodec doesn't get
> propagated to OffsetConfig instantiation, so GroupMetadataManager uses a
> source codec of NoCompressionCodec when making the MessageSet. Let's say we
> have enough group information such that the message formed exceeds
> KafkaConfig.messageMaxBytes before compression but would fall below the
> threshold after compression using our source codec. Even if we had
> dynamically configured __consumer_offsets with our favorite compression
> codec, the log.append will throw RecordTooLargeException during
> analyzeAndValidateMessageSet since the message was unexpectedly uncompressed
> instead of having been compressed with the source codec defined by
> KafkaConfig.offsetsTopicCompressionCodec.
> NOTE: even after this issue is resolved, preliminary tests show that LinkedIn
> will still hit RecordTooLargeException with large groups that consume many
> topics (like MirrorMakers with wildcard consumption of .*) since fully
> expanded subscription and assignment state for each member is put into a
> single record. But this is a first step in the right direction.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
