[
https://issues.apache.org/jira/browse/KAFKA-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15126959#comment-15126959
]
Jiangjie Qin commented on KAFKA-3174:
-------------------------------------
[~ijuma] My test code was as below. Initially I only tested 1MB size. I added
testing on different size after saw your comments above.
{code}
public static void main(String[] args) {
int[] sizes = {8, 16, 32, 128, 1024, 65536, 1048576};
for (int size : sizes) {
byte[] bytes = new byte[size];
Random random = new Random();
random.nextBytes(bytes);
int loop = 5000 * (1048576 / size);
long start = System.currentTimeMillis();
for (int i = 0; i < loop; i++) {
Crc32 crc32 = new Crc32();
crc32.update(bytes, 0, bytes.length);
}
System.out.println(String.format("KCrc32: Size = %d\t, time = %d",
size, (System.currentTimeMillis() - start)));
start = System.currentTimeMillis();
for (int i = 0; i < loop; i++) {
CRC32 crc32 = new CRC32();
crc32.update(bytes, 0, bytes.length);
}
System.out.println(String.format("JCrc32: Size = %d\t, time =
%d\n", size, (System.currentTimeMillis() - start)));
}
}
{code}
And here is the output:
{code}
KCrc32: Size = 8 , time = 10400
JCrc32: Size = 8 , time = 9907
KCrc32: Size = 16 , time = 6959
JCrc32: Size = 16 , time = 8419
KCrc32: Size = 32 , time = 5596
JCrc32: Size = 32 , time = 5587
KCrc32: Size = 128 , time = 4397
JCrc32: Size = 128 , time = 3305
KCrc32: Size = 1024 , time = 4115
JCrc32: Size = 1024 , time = 2392
KCrc32: Size = 65536 , time = 4087
JCrc32: Size = 65536 , time = 2296
KCrc32: Size = 1048576 , time = 4078
JCrc32: Size = 1048576 , time = 2298
{code}
>From the output above, it looks for size < 32 bytes KCrc32 and JCrc32 is
>comparable (except 16 bytes). After size >= 64, JCrc32 is faster. My ~2x
>result came from 1MB size.
In this Intel paper, they mentioned that the CRC32 instruction is actually in
SSE4.2, which was introduced in Nov 2008.
http://www.intel.com/content/dam/www/public/us/en/documents/white-papers/crc-iscsi-polynomial-crc32-instruction-paper.pdf
Wikipedia says the same thing.
https://en.wikipedia.org/wiki/SSE4#SSE4.2
AMD started to support SSE4.2 in Oct 2011.
I ran the above test on both my desktop (Intel(R) Xeon(R) CPU E5-2620 v2 @
2.10GHz) and macbook (Intel(R) Core(TM) i7-4558U CPU @ 2.80GHz), both of them
have SSE4.2 support.
> Re-evaluate the CRC32 class performance.
> ----------------------------------------
>
> Key: KAFKA-3174
> URL: https://issues.apache.org/jira/browse/KAFKA-3174
> Project: Kafka
> Issue Type: Improvement
> Affects Versions: 0.9.0.0
> Reporter: Jiangjie Qin
> Assignee: Jiangjie Qin
> Fix For: 0.9.0.1
>
>
> We used org.apache.kafka.common.utils.CRC32 in clients because it has better
> performance than java.util.zip.CRC32 in Java 1.6.
> In a recent test I ran it looks in Java 1.8 the CRC32 class is 2x as fast as
> the Crc32 class we are using now. We may want to re-evaluate the performance
> of Crc32 class and see it makes sense to simply use java CRC32 instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
