I just want to bring up that idea of no server side de/recompression again. Features like KAFKA-1499 <https://issues.apache.org/jira/browse/KAFKA-1499> seem to steer the project into a different direction and the fact that tickets like KAFKA-845 <https://issues.apache.org/jira/browse/KAFKA-845> are not getting much attention gives the same impression. This is something my head keeps spinning around almost 24/7 recently.

The problem I see is that CPU's are not the cheapest part of a new server and if you can spare a gigahertz or some cores by just making sure your configs are the same across all producers I would always opt for the operational overhead instead of the bigger servers. I think this will usually decrease the tco's of kafka installations.

I am currently not familiar enough with the codebase to judge if server side decompression happens before acknowledge. If so, these would be some additional milliseconds to respond faster if we could spare de/recompression.

Those are my thoughts about server side de/recompression. It would be great if I could get some responses and thoughts back.

Jan



On 07.11.2014 00:23, Jay Kreps wrote:
I suspect it is possible to save and reuse the CRCs though it might be a
bit of an invasive change. I suspect the first usage is when we are
checking the validity of the messages and the second is from when we
rebuild the compressed message set (I'm assuming you guys are using
compression because I think we optimize this out otherwise). Technically I
think the CRCs stay the same.

An alternative approach, though, would be working to remove the need for
recompression entirely on the broker side by making the offsets in the
compressed message relative to the base offset of the message set. This is
a much more invasive change but potentially better as it would also remove
the recompression done on the broker which is also CPU heavy.

-Jay

On Thu, Nov 6, 2014 at 2:36 PM, Allen Wang <aw...@netflix.com.invalid>
wrote:

Sure. Here is the link to the screen shot of jmc with the JTR file loaded:

http://picpaste.com/fligh-recorder-crc.png



On Thu, Nov 6, 2014 at 2:12 PM, Neha Narkhede <neha.narkh...@gmail.com>
wrote:

Allen,

Apache mailing lists don't allow attachments. Could you please link to a
pastebin or something?

Thanks,
Neha

On Thu, Nov 6, 2014 at 12:02 PM, Allen Wang <aw...@netflix.com.invalid>
wrote:

After digging more into the stack trace got from flight recorder (which
is
attached), it seems that Kafka (0.8.1.1) can optimize the usage of
Crc32.
The stack trace shows that Crc32 is invoked twice from Log.append().
First
is from the line number 231:

val appendInfo = analyzeAndValidateMessageSet(messages)

The second time is from line 252 in the same function:

validMessages = validMessages.assignOffsets(offset, appendInfo.codec)

If one of the Crc32 invocation can be eliminated, we are looking at
saving
at least 7% of CPU usage.

Thanks,
Allen




On Wed, Nov 5, 2014 at 6:32 PM, Allen Wang <aw...@netflix.com> wrote:

Hi,

Using flight recorder, we have observed high CPU usage of CRC32
(kafka.utils.Crc32.update()) on Kafka broker. It uses as much as 25%
of
CPU
on an instance. Tracking down stack trace, this method is invoked by
ReplicaFetcherThread.

Is there any tuning we can do to reduce this?

Also on the topic of CPU utilization, we observed that overall CPU
utilization is proportional to AllTopicsBytesInPerSec metric. Does
this
metric include incoming replication traffic?

Thanks,
Allen



Reply via email to