[ https://issues.apache.org/jira/browse/KAFKA-2213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14558017#comment-14558017 ]
Manikumar Reddy commented on KAFKA-2213: ---------------------------------------- CASE A: In normal scenarios, log are already written with broker/topic compression type. During compaction, we are just compacting with same compression type. CASE B: In some scenarios, we may change the compression type of existing topics using per-topic compression config. In this case, logs may have messages with different compression types. So we want to handle this during compaction? Currently during compaction, we are decompressing a message, and writing back compacted message(less size). So we are maintaining producer side batching (with less size). In CASE B, we have to change the compression type(non-compressed messages -> compressed, compressed -> non-compressed, compressed -> compressed). AFAIK, batching on producer side controlled by batch.size( in bytes) config. Do we need to introduce similar server side parameter in bytes/no. of messages. > Log cleaner should write compacted messages using configured compression type > ----------------------------------------------------------------------------- > > Key: KAFKA-2213 > URL: https://issues.apache.org/jira/browse/KAFKA-2213 > Project: Kafka > Issue Type: Bug > Reporter: Joel Koshy > > In KAFKA-1374 the log cleaner was improved to handle compressed messages. > There were a couple of follow-ups from that: > * We write compacted messages using the original compression type in the > compressed message-set. We should instead append all retained messages with > the configured broker compression type of the topic. > * While compressing messages we should ideally do some batching before > compression. > * Investigate the use of the client compressor. (See the discussion in the > RBs for KAFKA-1374) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
