Hi Dan, Decompression generally only happens in the broker if the topic has a particular compression algorithm set, and the producer is using a different one - then the broker will decompress records from the producer, then recompress it using the topic's configured algorithm. (The LogCleaner will also decompress then recompress records when compacting compressed topics).
The consumer decompresses compressed record batches it receives. In my opinion, using topic compression instead of producer compression would only make sense if the overhead of a few more CPU cycles compression uses was not tolerable for the producing app. In all of my use cases, network throughput becomes a bottleneck long before producer compression CPU cost does. For your "if X, do Y" formulation I'd say - if your producer is sending tiny batches, do some analysis of compressed vs. uncompressed size for your given compression algorithm - you may find that compression overhead increases batch size for tiny batches. If you're sending a large amount of data, do tune your batching and use compression to reduce data being sent over the wire. If you can tell us more about what your problem domain, there might be more advice that's applicable :) Cheers, Liam Clarke-Hutchinson On Tue, 15 Mar 2022 at 10:05, Dan Hill <quietgol...@gmail.com> wrote: > Hi. I looked around for advice about Kafka compression. I've seen mixed > and conflicting advice. > > Is there any sorta "if X, do Y" type of documentation around Kafka > compression? > > Any advice? Any good posts to read that talk about this trade off? > > *Detailed comments* > I tried looking for producer vs topic compression. I didn't find much. > Some of the information I see is back from 2011 (which I'm guessing is > pretty stale). > > I can guess some potential benefits but I don't know if they are actually > real. I've also seen some sites claim certain trade offs but it's unclear > if they're true. > > It looks like I can modify an existing topic's compression. I don't know > if that actually works. I'd assume it'd just impact data going forward. > > I've seen multiple sites say that decompression happens in the broker and > multiple that say it happens in the consumer. >
