[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-03-26 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13948225#comment-13948225 ] Guozhang Wang commented on KAFKA-1253: -- Updated reviewboard https://reviews.apache.or

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-03-25 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13947293#comment-13947293 ] Guozhang Wang commented on KAFKA-1253: -- Just realized the old producer performance ha

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-03-25 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13947129#comment-13947129 ] Guozhang Wang commented on KAFKA-1253: -- Updated reviewboard https://reviews.apache.or

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-03-25 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13947117#comment-13947117 ] Guozhang Wang commented on KAFKA-1253: -- The comparison of the latest patch with trunk

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-03-22 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13944287#comment-13944287 ] Guozhang Wang commented on KAFKA-1253: -- Updated reviewboard https://reviews.apache.or

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-03-19 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13941177#comment-13941177 ] Guozhang Wang commented on KAFKA-1253: -- Updated reviewboard https://reviews.apache.or

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-03-18 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13939998#comment-13939998 ] Guozhang Wang commented on KAFKA-1253: -- Updated reviewboard https://reviews.apache.or

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-03-18 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13939953#comment-13939953 ] Guozhang Wang commented on KAFKA-1253: -- According to these numbers, I think using jus

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-03-18 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13939839#comment-13939839 ] Guozhang Wang commented on KAFKA-1253: -- So it seems except the baseline heuristics #4

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-03-18 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13939579#comment-13939579 ] Guozhang Wang commented on KAFKA-1253: -- The first experiments are done with 10K messa

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-03-18 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13939556#comment-13939556 ] Jay Kreps commented on KAFKA-1253: -- What is the memory usage efficiency for each approach

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-03-18 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13939543#comment-13939543 ] Guozhang Wang commented on KAFKA-1253: -- Here is the experimental results for various

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-03-17 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13938520#comment-13938520 ] Guozhang Wang commented on KAFKA-1253: -- Updated reviewboard https://reviews.apache.or

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-03-15 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13936359#comment-13936359 ] Guozhang Wang commented on KAFKA-1253: -- Thanks a lot Jay, I think these numbers tell

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-03-15 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13936300#comment-13936300 ] Jay Kreps commented on KAFKA-1253: -- Also, in case helpful, here was my java code to test

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-03-15 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13936298#comment-13936298 ] Jay Kreps commented on KAFKA-1253: -- Found this formatting helpful: {noformat} Linger = 50

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-03-14 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13935994#comment-13935994 ] Jay Kreps commented on KAFKA-1253: -- As a reference the unix gzip command on random data s

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-03-14 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13935889#comment-13935889 ] Guozhang Wang commented on KAFKA-1253: -- Updated reviewboard https://reviews.apache.or

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-03-14 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13935730#comment-13935730 ] Guozhang Wang commented on KAFKA-1253: -- message/sec from trunk: 68390.0971 > Impleme

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-03-14 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13935622#comment-13935622 ] Jay Kreps commented on KAFKA-1253: -- What is the performance difference with and without t

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-03-14 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13935617#comment-13935617 ] Guozhang Wang commented on KAFKA-1253: -- Updated reviewboard https://reviews.apache.or

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-03-14 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13935548#comment-13935548 ] Guozhang Wang commented on KAFKA-1253: -- Yes. And I dig a little bit deeper for the re

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-03-14 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13935287#comment-13935287 ] Neha Narkhede commented on KAFKA-1253: -- [~guozhang] What metric does the above test m

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-03-14 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13935241#comment-13935241 ] Guozhang Wang commented on KAFKA-1253: -- With the new code, new performance numbers (1

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-03-10 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13926348#comment-13926348 ] Guozhang Wang commented on KAFKA-1253: -- Updated reviewboard https://reviews.apache.or

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-03-10 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13926296#comment-13926296 ] Guozhang Wang commented on KAFKA-1253: -- Some interesting performance numbers (1000 me

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-03-10 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13926267#comment-13926267 ] Guozhang Wang commented on KAFKA-1253: -- Updated reviewboard https://reviews.apache.or

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-03-10 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13926257#comment-13926257 ] Guozhang Wang commented on KAFKA-1253: -- Updated reviewboard https://reviews.apache.or

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-03-07 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13924565#comment-13924565 ] Guozhang Wang commented on KAFKA-1253: -- Updated reviewboard https://reviews.apache.or

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-03-06 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13923400#comment-13923400 ] Guozhang Wang commented on KAFKA-1253: -- Updated reviewboard https://reviews.apache.or

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-02-26 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13913881#comment-13913881 ] Guozhang Wang commented on KAFKA-1253: -- Updated reviewboard https://reviews.apache.or

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-02-24 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13910844#comment-13910844 ] Guozhang Wang commented on KAFKA-1253: -- Updated reviewboard https://reviews.apache.or

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-02-21 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13909138#comment-13909138 ] Guozhang Wang commented on KAFKA-1253: -- Updated reviewboard https://reviews.apache.or

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-02-21 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13909027#comment-13909027 ] Guozhang Wang commented on KAFKA-1253: -- Updated reviewboard https://reviews.apache.or

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-02-21 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13908874#comment-13908874 ] Guozhang Wang commented on KAFKA-1253: -- Thanks Jay, will incorporate into the patch.

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-02-20 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13907746#comment-13907746 ] Guozhang Wang commented on KAFKA-1253: -- Thanks for the comments, I think I should ela

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-02-19 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13906620#comment-13906620 ] Jay Kreps commented on KAFKA-1253: -- I think this approach may work but we have to work ou

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-02-19 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13906544#comment-13906544 ] Guozhang Wang commented on KAFKA-1253: -- One note is that in this preliminary patch I

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-02-19 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13906532#comment-13906532 ] Guozhang Wang commented on KAFKA-1253: -- Created reviewboard https://reviews.apache.or

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-02-19 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13906533#comment-13906533 ] Guozhang Wang commented on KAFKA-1253: -- Uploaded a first-step patch which illustrate

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-02-19 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13906422#comment-13906422 ] Jay Kreps commented on KAFKA-1253: -- Also the high-level idea I had for implementation was

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-02-19 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13906407#comment-13906407 ] Jay Kreps commented on KAFKA-1253: -- This will be tricky but is possible. Here are a coup

[jira] [Commented] (KAFKA-1253) Implement compression in new producer

2014-02-19 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13906339#comment-13906339 ] Guozhang Wang commented on KAFKA-1253: -- It seems incremental compression would not be