[
https://issues.apache.org/jira/browse/KAFKA-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxim Ivanov updated KAFKA-1933:
--------------------------------
Description:
This patch adds finer locking when appending to log. It breaks
global append lock into 2 sequential and 1 parallel phase.
Basic idea is to allow every thread to "reserve" offsets in non
overlapping ranges, then do compression in parallel and then
"commit" write to log in the same order offsets where reserved.
On my Core i3 M370 @2.4Ghz (2 cores + HT) it resulted in following performance
boost:
LZ4: 7.2 sec -> 4.2 sec
Gzip: 62.3 sec -> 26.9 sec
Kafka was configured to run 4 IO threads, data was pushed using 5 netcat
instances pushing in parallel batches of 200 msg 6.2 kb each (510 MB in total,
82180 messages in total)
was:
This patch adds finer locking when appending to log. It breaks
global append lock into 2 sequential and 1 parallel phase.
Basic idea is to allow every thread to "reserve" offsets in non
overlapping ranges, then do compression in parallel and then
"commit" write to log in the same order offsets where reserved.
On my Core i3 M370 @2.4Ghz (2 cores + HT) it resulted in following performance
boost:
LZ4: 7.2 sec -> 3.9 sec
Gzip: 62.3 sec -> 24.8 sec
Kafka was configured to run 4 IO threads, data was pushed using 5 netcat
instances pushing in parallel batches of 200 msg 6.2 kb each (510 MB in total,
82180 messages in total)
> Fine-grained locking in log append
> ----------------------------------
>
> Key: KAFKA-1933
> URL: https://issues.apache.org/jira/browse/KAFKA-1933
> Project: Kafka
> Issue Type: Improvement
> Components: log
> Reporter: Maxim Ivanov
> Assignee: Jay Kreps
> Priority: Minor
> Fix For: 0.8.2
>
> Attachments: KAFKA-1933.patch, KAFKA-1933_2015-02-09_12:15:06.patch
>
>
> This patch adds finer locking when appending to log. It breaks
> global append lock into 2 sequential and 1 parallel phase.
> Basic idea is to allow every thread to "reserve" offsets in non
> overlapping ranges, then do compression in parallel and then
> "commit" write to log in the same order offsets where reserved.
> On my Core i3 M370 @2.4Ghz (2 cores + HT) it resulted in following
> performance boost:
> LZ4: 7.2 sec -> 4.2 sec
> Gzip: 62.3 sec -> 26.9 sec
> Kafka was configured to run 4 IO threads, data was pushed using 5 netcat
> instances pushing in parallel batches of 200 msg 6.2 kb each (510 MB in
> total, 82180 messages in total)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
