[
https://issues.apache.org/jira/browse/KAFKA-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15128038#comment-15128038
]
Ismael Juma commented on KAFKA-3174:
------------------------------------
Becket, I think the best way to validate the reasoning is to benchmark the
producer with varying message sizes (and potentially with compression
enabled/disabled).
Also, I'll say it again just to make this clear for others reading this: the
JDK does not use the CRC32 instruction in SSE 4.2 because the CPU instruction
uses a different polynomial than the one in ZIP. The Intel instruction is for
CRC32-C. Relevant quote:
"CRC (Cyclic Redundancy Check) is a remainder from dividing your message by a
polynomial. Most popular file formats and protocols (Ethernet, MPEG-2, ZIP,
RAR, 7-Zip, GZip, and PNG) use the polynomial 0x04C11DB7, while Intel's
hardware implementation is based on another polynomial, 0x1EDC6F41 (used in
iSCSI and Btrfs). Newly designed protocols and formats can choose the second
polynomial to benefit from hardware acceleration, but CRC-32 with polynomial
0x04C11DB7 has to be calculated in software. The CRC32 instruction is not
supported by AMD processors."
http://www.strchr.com/crc32_popcnt
The JDK simply tries to optimise the software implementation via SIMD
instructions.
> Re-evaluate the CRC32 class performance.
> ----------------------------------------
>
> Key: KAFKA-3174
> URL: https://issues.apache.org/jira/browse/KAFKA-3174
> Project: Kafka
> Issue Type: Improvement
> Affects Versions: 0.9.0.0
> Reporter: Jiangjie Qin
> Assignee: Jiangjie Qin
> Fix For: 0.9.0.1
>
>
> We used org.apache.kafka.common.utils.CRC32 in clients because it has better
> performance than java.util.zip.CRC32 in Java 1.6.
> In a recent test I ran it looks in Java 1.8 the CRC32 class is 2x as fast as
> the Crc32 class we are using now. We may want to re-evaluate the performance
> of Crc32 class and see it makes sense to simply use java CRC32 instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
