[
https://issues.apache.org/jira/browse/KAFKA-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14165366#comment-14165366
]
Jay Kreps commented on KAFKA-1499:
----------------------------------
Specifically the nuance I thought made this better was to change the
configuration slightly from the proposal in this patch. The problem with
broker.compression.enable/compression.type was that the way compression worked
was dependent on whether the enable/disable flag was set to true, which is
confusing. Joel had an idea that I thought fixed this, let's see if other
people agree. We instead have compression.type which is a broker-level config
as well as a topic override. However this config takes the following options:
NONE, GZIP, SNAPPY, PRODUCER. GZIP and SNAPPY are self-explanatory. NONE means
that regardless of what the producer sets, the broker writes the message
decompressed. PRODUCER means the broker attempts to retain whatever is used by
the producer (this will just be approximate in the case of a log compaction and
a mixture of codecs). We will default the behavior to PRODUCER so things will
work as they do today for most people.
Thoughts?
Personally I think although there isn't an incredibly pressing use case this
would be a good generalization of how things work today that isn't too
confusing.
Let's get consensus from the rest of the people who chimed in on this ticket on
how things should work and whether this is worth doing before we proceed,
though, so we don't cause to much churn for [~omkreddy] who is actually doing
the work here. [~nehanarkhede], [~junrao], [~guozhang], [~gwenshap] thoughts?
> Broker-side compression configuration
> -------------------------------------
>
> Key: KAFKA-1499
> URL: https://issues.apache.org/jira/browse/KAFKA-1499
> Project: Kafka
> Issue Type: New Feature
> Reporter: Joel Koshy
> Assignee: Manikumar Reddy
> Labels: newbie++
> Fix For: 0.8.2
>
> Attachments: KAFKA-1499.patch, KAFKA-1499.patch,
> KAFKA-1499_2014-08-15_14:20:27.patch, KAFKA-1499_2014-08-21_21:44:27.patch,
> KAFKA-1499_2014-09-21_15:57:23.patch, KAFKA-1499_2014-09-23_14:45:38.patch,
> KAFKA-1499_2014-09-24_14:20:33.patch, KAFKA-1499_2014-09-24_14:24:54.patch,
> KAFKA-1499_2014-09-25_11:05:57.patch
>
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> A given topic can have messages in mixed compression codecs. i.e., it can
> also have a mix of uncompressed/compressed messages.
> It will be useful to support a broker-side configuration to recompress
> messages to a specific compression codec. i.e., all messages (for all
> topics) on the broker will be compressed to this codec. We could have
> per-topic overrides as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
