[
https://issues.apache.org/jira/browse/KAFKA-2213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14564676#comment-14564676
]
Manikumar Reddy edited comment on KAFKA-2213 at 5/29/15 6:58 PM:
-----------------------------------------------------------------
Uploaded a patch which writes the compacted messages using the configured
broker compression type.
a) If the log contains messages with multiple compression types and configured
broker compression type is "producer", then will write the messages with
latest message compression type.
b) No special batching is introduced. Currently on each iteration , we will try
to compact a maximum of maxMessageSize bytes. So the compacted message will be
less than maxMessageSize byes
c) Updated the LogIntegrationTest to include broker compression. This may not
be required, as it increases the test run time.
d) Used client MemoryRecords/Compressor classes
was (Author: omkreddy):
Uploaded a patch which writes the compacted messages using the configured
broker compression type.
a) If the log contains messages with multiple compression types and configured
broker compression type is "producer", then will write the messages with
latest message compression type.
b) No special batching is introduced. Currently on each iteration , we will try
to compact a maximum of maxMessageSize bytes. So the compacted message will be
less than maxMessageSize byes
c) Updated the LogIntegrationTest to include broker compression. This may not
be required, as it increases the test run time.
I will investigate Compressor usage in next patch.
> Log cleaner should write compacted messages using configured compression type
> -----------------------------------------------------------------------------
>
> Key: KAFKA-2213
> URL: https://issues.apache.org/jira/browse/KAFKA-2213
> Project: Kafka
> Issue Type: Bug
> Reporter: Joel Koshy
> Assignee: Manikumar Reddy
> Attachments: KAFKA-2213.patch, KAFKA-2213_2015-05-30_00:23:01.patch
>
>
> In KAFKA-1374 the log cleaner was improved to handle compressed messages.
> There were a couple of follow-ups from that:
> * We write compacted messages using the original compression type in the
> compressed message-set. We should instead append all retained messages with
> the configured broker compression type of the topic.
> * While compressing messages we should ideally do some batching before
> compression.
> * Investigate the use of the client compressor. (See the discussion in the
> RBs for KAFKA-1374)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
