[
https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13939839#comment-13939839
]
Guozhang Wang commented on KAFKA-1253:
--------------------------------------
So it seems except the baseline heuristics #4, all others are doing pretty well
in reducing possibility of reallocation.
In terms of memory efficiency, I re-run the second experiment and the memory
usage (= #.bytes written / #.bytes as buffer capacity) is the following:
The first experiments are done with 10K messages half of which are random bits
and the other half all 1's, batch size = 16K, recorded #.reallocations, linger
time 1 second.
GZIP:
|| message size || 1K || 10K || 100K ||
|heuristic0|0.16393931370995588|0.6360426060498783|0.5027185390566682|
|heuristic1|0.08041873550981989|0.6361295505149862|0.5026977360002329|
|heuristic2|0.037097550186846885|0.6360425366609646|0.5027069612634141|
|heuristic3|0.07121086649677909|0.321025252352334|0.5027315390995597|
|heuristic4|0.07140996394426881|0.634673472157968|0.5027017302210279|
SNAPPY:
|| message size || 1K || 10K || 100K ||
|heuristic0|0.6140103928071738|0.6618065853371036|0.5244664440669362|
|heuristic1|0.3794715113367395|0.6618042001500875|0.524466444066936|
|heuristic2|0.44042297446570144|0.661811600542026|0.524466444066936|
|heuristic3|0.5112405714891525|0.33291628770530224|0.5244664440669362|
|heuristic4|0.4887840474881868|0.6618042070775481|0.5244664440669362|
> Implement compression in new producer
> -------------------------------------
>
> Key: KAFKA-1253
> URL: https://issues.apache.org/jira/browse/KAFKA-1253
> Project: Kafka
> Issue Type: Sub-task
> Components: producer
> Reporter: Jay Kreps
> Assignee: Guozhang Wang
> Attachments: KAFKA-1253.patch, KAFKA-1253_2014-02-21_16:15:21.patch,
> KAFKA-1253_2014-02-21_17:55:52.patch, KAFKA-1253_2014-02-24_13:31:50.patch,
> KAFKA-1253_2014-02-26_17:31:30.patch, KAFKA-1253_2014-03-06_17:48:11.patch,
> KAFKA-1253_2014-03-07_16:34:33.patch, KAFKA-1253_2014-03-10_14:35:56.patch,
> KAFKA-1253_2014-03-10_14:39:58.patch, KAFKA-1253_2014-03-10_15:27:47.patch,
> KAFKA-1253_2014-03-14_13:46:40.patch, KAFKA-1253_2014-03-14_17:39:53.patch,
> KAFKA-1253_2014-03-17_15:56:04.patch, compression-fix.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)
