C++ Client Library -- libkafka-asio
Hello there, I'm currently working on a C++ client library, implementing the Kafka protocol using Boost ASIO. You can find the source code and some examples on github: https://github.com/danieljoos/libkafka-asio I tried to add it to the "Clients" section of the Kafka wiki, but either I'm to blind to see the "Edit" button, or I just don't have enough permissions to edit the page ;-) In case you like the library, it would be very nice, if someone with sufficient permissions for the wiki could add it there. Thanks. Best regards, Daniel
Re: C++ Client Library -- libkafka-asio
Cool, Looks nice. I was looking for something similar a year ago. We also ended up rolling our own. https://github.com/bitbouncer/csi-kafka Have you got any performance figures? /svante 2015-03-22 14:29 GMT+01:00 Daniel Joos : > Hello there, > > I'm currently working on a C++ client library, implementing the Kafka > protocol using Boost ASIO. > You can find the source code and some examples on github: > https://github.com/danieljoos/libkafka-asio > > I tried to add it to the "Clients" section of the Kafka wiki, but either > I'm to blind to see the "Edit" button, or I just don't have enough > permissions to edit the page ;-) > In case you like the library, it would be very nice, if someone with > sufficient permissions for the wiki could add it there. > > Thanks. > Best regards, > > Daniel > >
[jira] [Updated] (KAFKA-2021) Consolidate test classes for KafkaConfig
[ https://issues.apache.org/jira/browse/KAFKA-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrii Biletskyi updated KAFKA-2021: Attachment: KAFKA-2021.patch > Consolidate test classes for KafkaConfig > > > Key: KAFKA-2021 > URL: https://issues.apache.org/jira/browse/KAFKA-2021 > Project: Kafka > Issue Type: Task >Reporter: Gwen Shapira >Assignee: Andrii Biletskyi >Priority: Minor > Attachments: KAFKA-2021.patch > > > We have kafka.server.KafkaConfigTest, KafkaConfigConfigDefTest and > kafka.unit.KafkaTest (in a file called KafkaConfigTest.scala) > I think consolidating them into one test class (or at list renaming so it > will be clear how they are different) will make a lot of sense. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Review Request 32376: Patch for KAFKA-2021
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32376/ --- Review request for kafka. Bugs: KAFKA-2021 https://issues.apache.org/jira/browse/KAFKA-2021 Repository: kafka Description --- KAFKA-2021- consolidate test classes for KafkaConfig. Renamed related to Kafka.scala, merged KafkaConfigTest and KafkaConfigConfigDefTest Diffs - core/src/test/scala/unit/kafka/KafkaConfigTest.scala core/src/test/scala/unit/kafka/server/KafkaConfigConfigDefTest.scala 191251d1340b5e5b2d649b37af3c6c1896d07e6e core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala 7f47e6f9a74314ed9e9f19d0c97931f3f2e49259 Diff: https://reviews.apache.org/r/32376/diff/ Testing --- Thanks, Andrii Biletskyi
[jira] [Commented] (KAFKA-2021) Consolidate test classes for KafkaConfig
[ https://issues.apache.org/jira/browse/KAFKA-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14375085#comment-14375085 ] Andrii Biletskyi commented on KAFKA-2021: - Created reviewboard https://reviews.apache.org/r/32376/diff/ against branch origin/trunk > Consolidate test classes for KafkaConfig > > > Key: KAFKA-2021 > URL: https://issues.apache.org/jira/browse/KAFKA-2021 > Project: Kafka > Issue Type: Task >Reporter: Gwen Shapira >Assignee: Andrii Biletskyi >Priority: Minor > Attachments: KAFKA-2021.patch > > > We have kafka.server.KafkaConfigTest, KafkaConfigConfigDefTest and > kafka.unit.KafkaTest (in a file called KafkaConfigTest.scala) > I think consolidating them into one test class (or at list renaming so it > will be clear how they are different) will make a lot of sense. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2021) Consolidate test classes for KafkaConfig
[ https://issues.apache.org/jira/browse/KAFKA-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrii Biletskyi updated KAFKA-2021: Status: Patch Available (was: Open) > Consolidate test classes for KafkaConfig > > > Key: KAFKA-2021 > URL: https://issues.apache.org/jira/browse/KAFKA-2021 > Project: Kafka > Issue Type: Task >Reporter: Gwen Shapira >Assignee: Andrii Biletskyi >Priority: Minor > Attachments: KAFKA-2021.patch > > > We have kafka.server.KafkaConfigTest, KafkaConfigConfigDefTest and > kafka.unit.KafkaTest (in a file called KafkaConfigTest.scala) > I think consolidating them into one test class (or at list renaming so it > will be clear how they are different) will make a lot of sense. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 32376: Patch for KAFKA-2021
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32376/#review77356 --- It looks like some tests are missing? like the uncleanLeaderElection ones? - Gwen Shapira On March 22, 2015, 5:46 p.m., Andrii Biletskyi wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/32376/ > --- > > (Updated March 22, 2015, 5:46 p.m.) > > > Review request for kafka. > > > Bugs: KAFKA-2021 > https://issues.apache.org/jira/browse/KAFKA-2021 > > > Repository: kafka > > > Description > --- > > KAFKA-2021- consolidate test classes for KafkaConfig. Renamed related to > Kafka.scala, merged KafkaConfigTest and KafkaConfigConfigDefTest > > > Diffs > - > > core/src/test/scala/unit/kafka/KafkaConfigTest.scala > core/src/test/scala/unit/kafka/server/KafkaConfigConfigDefTest.scala > 191251d1340b5e5b2d649b37af3c6c1896d07e6e > core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala > 7f47e6f9a74314ed9e9f19d0c97931f3f2e49259 > > Diff: https://reviews.apache.org/r/32376/diff/ > > > Testing > --- > > > Thanks, > > Andrii Biletskyi > >
Re: Review Request 32376: Patch for KAFKA-2021
> On March 22, 2015, 6:57 p.m., Gwen Shapira wrote: > > It looks like some tests are missing? like the uncleanLeaderElection ones? Those had been covered in the main test - testFromPropsToProps, already in the first version of the patch. - Andrii --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32376/#review77356 --- On March 22, 2015, 5:46 p.m., Andrii Biletskyi wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/32376/ > --- > > (Updated March 22, 2015, 5:46 p.m.) > > > Review request for kafka. > > > Bugs: KAFKA-2021 > https://issues.apache.org/jira/browse/KAFKA-2021 > > > Repository: kafka > > > Description > --- > > KAFKA-2021- consolidate test classes for KafkaConfig. Renamed related to > Kafka.scala, merged KafkaConfigTest and KafkaConfigConfigDefTest > > > Diffs > - > > core/src/test/scala/unit/kafka/KafkaConfigTest.scala > core/src/test/scala/unit/kafka/server/KafkaConfigConfigDefTest.scala > 191251d1340b5e5b2d649b37af3c6c1896d07e6e > core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala > 7f47e6f9a74314ed9e9f19d0c97931f3f2e49259 > > Diff: https://reviews.apache.org/r/32376/diff/ > > > Testing > --- > > > Thanks, > > Andrii Biletskyi > >
Review Request 32378: Patch for KAFKA-1293
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32378/ --- Review request for kafka. Bugs: KAFKA-1293 https://issues.apache.org/jira/browse/KAFKA-1293 Repository: kafka Description --- KAFKA-1293 # Housekeeping/standardization for multiple command arguments. Diffs - core/src/main/scala/kafka/admin/PreferredReplicaLeaderElectionCommand.scala 79b5e0aeb4af77d356f80107685cd9a7d508ef16 core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala 979992b68af3723cd229845faff81c641123bb88 core/src/main/scala/kafka/admin/TopicCommand.scala f400b71f8444fffd3fc1d8398a283682390eba4e core/src/main/scala/kafka/tools/ConsoleConsumer.scala 910691e88ccc66a1542d0ea85bb2f732861d805e core/src/main/scala/kafka/tools/ConsoleProducer.scala 00265f9f4a4b6c6a9aa023e5be5faf297f77bf31 core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala d1e7c434e77859d746b8dc68dd5d5a3740425e79 core/src/main/scala/kafka/tools/ConsumerPerformance.scala c39c0672ff8d04361df99a7fcd98448eea8717ed core/src/main/scala/kafka/tools/DumpLogSegments.scala fe2cc11b75f370beb9cb87ebc9ed01b63fd65f87 core/src/main/scala/kafka/tools/ExportZkOffsets.scala 4d051bc2db12f0e15aa6a3289abeb9dd25d373d2 core/src/main/scala/kafka/tools/GetOffsetShell.scala 3d9293e4abbe3f4a4a2bc5833385747c604d5a95 core/src/main/scala/kafka/tools/ImportZkOffsets.scala abe09721b13f71320510fd1a01c1917470450c6e core/src/main/scala/kafka/tools/JmxTool.scala 1d1a120c45ff70fbd60df5b147ca230eb1ef50de core/src/main/scala/kafka/tools/KafkaMigrationTool.java 026d8198678d3e3e41feddcdaa53b67dad5c0e64 core/src/main/scala/kafka/tools/MirrorMaker.scala 4f3c4c872e144195bb4b742b802fa3b931edb534 core/src/main/scala/kafka/tools/PerfConfig.scala d073acf9adf4b7fd07e5dd9db5331a7952d0ecf4 core/src/main/scala/kafka/tools/ProducerPerformance.scala bc25cd2f371c1b18aba4e7d760357e448acfd5c0 core/src/main/scala/kafka/tools/ReplayLogProducer.scala 2b8537b89fafa7b29fd9d2262c8cdf88cf00e527 core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala ba6ddd7a909df79a0f7d45e8b4a2af94ea0fceb6 core/src/main/scala/kafka/tools/SimpleConsumerPerformance.scala 900f7df9f5ce0deb19e6a6739a776a7d5b12bfaa core/src/main/scala/kafka/tools/SimpleConsumerShell.scala b4f903b6c7c3bb725cac7c05eb1f885906413c4d core/src/main/scala/kafka/tools/StateChangeLogMerger.scala b34b8c7d7d4caaab230719a67048310e0fbb3280 core/src/main/scala/kafka/tools/TestLogCleaning.scala af496f7c547a5ac7a4096a6af325dad0d8feec6f core/src/main/scala/kafka/tools/VerifyConsumerRebalance.scala aef8361b73a0934641fc4f5cee942b5b50f3e7d7 core/src/test/scala/other/kafka/TestLinearWriteSpeed.scala 7211c2529c1db76100432737da7a1d1d221dfba0 core/src/test/scala/other/kafka/TestOffsetManager.scala a106379d96ab3cbdf54a9f162db44a6df177e733 Diff: https://reviews.apache.org/r/32378/diff/ Testing --- Thanks, Matt Warhaftig
[jira] [Updated] (KAFKA-1293) Mirror maker housecleaning
[ https://issues.apache.org/jira/browse/KAFKA-1293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Warhaftig updated KAFKA-1293: -- Attachment: KAFKA-1293.patch > Mirror maker housecleaning > -- > > Key: KAFKA-1293 > URL: https://issues.apache.org/jira/browse/KAFKA-1293 > Project: Kafka > Issue Type: Bug > Components: tools >Affects Versions: 0.8.1 >Reporter: Jay Kreps >Priority: Minor > Labels: usability > Attachments: KAFKA-1293.patch > > > Mirror maker uses it's own convention for command-line arguments, e.g. > --num.producers, where everywhere else follows the unix convention like > --num-producers. This is annoying because when running different tools you > have to constantly remember whatever quirks of the person who wrote that tool. > Mirror maker should also have a top-level wrapper script in bin/ to make tab > completion work and so you don't have to remember the fully qualified class > name. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1293) Mirror maker housecleaning
[ https://issues.apache.org/jira/browse/KAFKA-1293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14375146#comment-14375146 ] Matt Warhaftig commented on KAFKA-1293: --- Created reviewboard https://reviews.apache.org/r/32378/diff/ against branch origin/trunk > Mirror maker housecleaning > -- > > Key: KAFKA-1293 > URL: https://issues.apache.org/jira/browse/KAFKA-1293 > Project: Kafka > Issue Type: Bug > Components: tools >Affects Versions: 0.8.1 >Reporter: Jay Kreps >Priority: Minor > Labels: usability > Attachments: KAFKA-1293.patch > > > Mirror maker uses it's own convention for command-line arguments, e.g. > --num.producers, where everywhere else follows the unix convention like > --num-producers. This is annoying because when running different tools you > have to constantly remember whatever quirks of the person who wrote that tool. > Mirror maker should also have a top-level wrapper script in bin/ to make tab > completion work and so you don't have to remember the fully qualified class > name. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1293) Mirror maker housecleaning
[ https://issues.apache.org/jira/browse/KAFKA-1293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14375154#comment-14375154 ] Matt Warhaftig commented on KAFKA-1293: --- In addition to Mirror Maker other tools don't follow argument naming conventions. So, going a step further than the original request I have submitted a patch with updates to: * Standardize tool argument names. * Properly format argument descriptions (into sentences) and add any missing "REQUIRED" notes. * Add 'help' argument to any top-level tool scripts that were missing it. Since argument name changes would have the most impact here is the list: |*Command*|*Old Argument Name*|*New Argument Name*| |kafka-console-consumer.sh|consumer.config|consumer-config| |kafka-consumer-offset-checker.sh|socket.timeout.ms|socket-timeout-ms| |kafka-consumer-offset-checker.sh|retry.backoff.ms|retry-backoff-ms | |ExportZkOffsets.scala|zkconnect|zookeeper| |ImportZkOffsets.scala|zkconnect|zookeeper| |KafkaMigrationTool.java|consumer.config|consumer-config| |KafkaMigrationTool.java|producer.config|producer-config| |KafkaMigrationTool.java|num.producers|num-producers| |KafkaMigrationTool.java|num.streams|num-streams| |KafkaMigrationTool.java|queue.size|queue-size| |kafka-mirror-maker.sh|consumer.config|consumer-config| |kafka-mirror-maker.sh|producer.config|producer-config| |kafka-mirror-maker.sh|num.streams|num-streams| |kafka-producer-perf-test.sh|request-num-acks|request-required-acks| |kafka-replay-log-producer.sh|inputtopic|input-topic| |kafka-replay-log-producer.sh|outputtopic|output-topic| |kafka-replica-verification.sh|topic-white-list|whitelist| |kafka-replica-verification.sh|report-interval-ms|reporting-interval| |kafka-simple-consumer-shell.sh|fetchsize|fetch-size| |TestLogCleaning.scala|zk|zookeeper| |VerifyConsumerRebalance.scala|zookeeper.connect|zookeeper| |TestLinearWriteSpeed.scala|compression|compression-codec| |TestOffsetManager.scala|thread-count|threads| |TestOffsetManager.scala|reporting-interval-ms|reporting-interval| No worries if some of these changes are too dramatic to make it to the codebase. > Mirror maker housecleaning > -- > > Key: KAFKA-1293 > URL: https://issues.apache.org/jira/browse/KAFKA-1293 > Project: Kafka > Issue Type: Bug > Components: tools >Affects Versions: 0.8.1 >Reporter: Jay Kreps >Priority: Minor > Labels: usability > Attachments: KAFKA-1293.patch > > > Mirror maker uses it's own convention for command-line arguments, e.g. > --num.producers, where everywhere else follows the unix convention like > --num-producers. This is annoying because when running different tools you > have to constantly remember whatever quirks of the person who wrote that tool. > Mirror maker should also have a top-level wrapper script in bin/ to make tab > completion work and so you don't have to remember the fully qualified class > name. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 32378: Patch for KAFKA-1293
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32378/ --- (Updated March 22, 2015, 8:33 p.m.) Review request for kafka. Bugs: KAFKA-1293 https://issues.apache.org/jira/browse/KAFKA-1293 Repository: kafka Description --- KAFKA-1293 # Housekeeping/standardization for multiple command arguments. Diffs - core/src/main/scala/kafka/admin/PreferredReplicaLeaderElectionCommand.scala 79b5e0aeb4af77d356f80107685cd9a7d508ef16 core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala 979992b68af3723cd229845faff81c641123bb88 core/src/main/scala/kafka/admin/TopicCommand.scala f400b71f8444fffd3fc1d8398a283682390eba4e core/src/main/scala/kafka/tools/ConsoleConsumer.scala 910691e88ccc66a1542d0ea85bb2f732861d805e core/src/main/scala/kafka/tools/ConsoleProducer.scala 00265f9f4a4b6c6a9aa023e5be5faf297f77bf31 core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala d1e7c434e77859d746b8dc68dd5d5a3740425e79 core/src/main/scala/kafka/tools/ConsumerPerformance.scala c39c0672ff8d04361df99a7fcd98448eea8717ed core/src/main/scala/kafka/tools/DumpLogSegments.scala fe2cc11b75f370beb9cb87ebc9ed01b63fd65f87 core/src/main/scala/kafka/tools/ExportZkOffsets.scala 4d051bc2db12f0e15aa6a3289abeb9dd25d373d2 core/src/main/scala/kafka/tools/GetOffsetShell.scala 3d9293e4abbe3f4a4a2bc5833385747c604d5a95 core/src/main/scala/kafka/tools/ImportZkOffsets.scala abe09721b13f71320510fd1a01c1917470450c6e core/src/main/scala/kafka/tools/JmxTool.scala 1d1a120c45ff70fbd60df5b147ca230eb1ef50de core/src/main/scala/kafka/tools/KafkaMigrationTool.java 026d8198678d3e3e41feddcdaa53b67dad5c0e64 core/src/main/scala/kafka/tools/MirrorMaker.scala 4f3c4c872e144195bb4b742b802fa3b931edb534 core/src/main/scala/kafka/tools/PerfConfig.scala d073acf9adf4b7fd07e5dd9db5331a7952d0ecf4 core/src/main/scala/kafka/tools/ProducerPerformance.scala bc25cd2f371c1b18aba4e7d760357e448acfd5c0 core/src/main/scala/kafka/tools/ReplayLogProducer.scala 2b8537b89fafa7b29fd9d2262c8cdf88cf00e527 core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala ba6ddd7a909df79a0f7d45e8b4a2af94ea0fceb6 core/src/main/scala/kafka/tools/SimpleConsumerPerformance.scala 900f7df9f5ce0deb19e6a6739a776a7d5b12bfaa core/src/main/scala/kafka/tools/SimpleConsumerShell.scala b4f903b6c7c3bb725cac7c05eb1f885906413c4d core/src/main/scala/kafka/tools/StateChangeLogMerger.scala b34b8c7d7d4caaab230719a67048310e0fbb3280 core/src/main/scala/kafka/tools/TestLogCleaning.scala af496f7c547a5ac7a4096a6af325dad0d8feec6f core/src/main/scala/kafka/tools/VerifyConsumerRebalance.scala aef8361b73a0934641fc4f5cee942b5b50f3e7d7 core/src/test/scala/other/kafka/TestLinearWriteSpeed.scala 7211c2529c1db76100432737da7a1d1d221dfba0 core/src/test/scala/other/kafka/TestOffsetManager.scala a106379d96ab3cbdf54a9f162db44a6df177e733 Diff: https://reviews.apache.org/r/32378/diff/ Testing (updated) --- Verified that all updated tool commands continued to function after new argument name and argument descriptions were added. Also, verified that unit tests (./gradlew testAll) were not negatively affected (all passed locally). Note that http://kafka.apache.org/documentation.html (for Mirror Maker) and multiple wiki pages (ex. https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330) still need to be updated to the new argument names. Thanks, Matt Warhaftig
[jira] [Created] (KAFKA-2039) Update Scala to 2.10.5 and 2.11.6
Ismael Juma created KAFKA-2039: -- Summary: Update Scala to 2.10.5 and 2.11.6 Key: KAFKA-2039 URL: https://issues.apache.org/jira/browse/KAFKA-2039 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.2.1 Reporter: Ismael Juma Priority: Minor Scala 2.10.5 (the last release of the 2.10.x series) is binary compatible with 2.10.4 and Scala 2.11.6 is binary compatible with 2.11.5. For details of the changes in each release, see: * http://www.scala-lang.org/news/2.10.5 * http://www.scala-lang.org/news/2.11.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2039) Update Scala to 2.10.5 and 2.11.6
[ https://issues.apache.org/jira/browse/KAFKA-2039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-2039: --- Attachment: kafka-2039.patch Update Scala to 2.10.5 and 2.11.6. > Update Scala to 2.10.5 and 2.11.6 > - > > Key: KAFKA-2039 > URL: https://issues.apache.org/jira/browse/KAFKA-2039 > Project: Kafka > Issue Type: Improvement >Affects Versions: 0.8.2.1 >Reporter: Ismael Juma >Priority: Minor > Attachments: kafka-2039.patch > > > Scala 2.10.5 (the last release of the 2.10.x series) is binary compatible > with 2.10.4 and Scala 2.11.6 is binary compatible with 2.11.5. For details > of the changes in each release, see: > * http://www.scala-lang.org/news/2.10.5 > * http://www.scala-lang.org/news/2.11.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (KAFKA-2039) Update Scala to 2.10.5 and 2.11.6
[ https://issues.apache.org/jira/browse/KAFKA-2039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-2039: --- Comment: was deleted (was: Update Scala to 2.10.5 and 2.11.6.) > Update Scala to 2.10.5 and 2.11.6 > - > > Key: KAFKA-2039 > URL: https://issues.apache.org/jira/browse/KAFKA-2039 > Project: Kafka > Issue Type: Improvement >Affects Versions: 0.8.2.1 >Reporter: Ismael Juma >Priority: Minor > Attachments: kafka-2039.patch > > > Scala 2.10.5 (the last release of the 2.10.x series) is binary compatible > with 2.10.4 and Scala 2.11.6 is binary compatible with 2.11.5. For details > of the changes in each release, see: > * http://www.scala-lang.org/news/2.10.5 > * http://www.scala-lang.org/news/2.11.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2039) Update Scala to 2.10.5 and 2.11.6
[ https://issues.apache.org/jira/browse/KAFKA-2039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-2039: --- Status: Patch Available (was: Open) I've attached a patch that updates Scala to 2.10.5 and 2.11.6. > Update Scala to 2.10.5 and 2.11.6 > - > > Key: KAFKA-2039 > URL: https://issues.apache.org/jira/browse/KAFKA-2039 > Project: Kafka > Issue Type: Improvement >Affects Versions: 0.8.2.1 >Reporter: Ismael Juma >Priority: Minor > Attachments: kafka-2039.patch > > > Scala 2.10.5 (the last release of the 2.10.x series) is binary compatible > with 2.10.4 and Scala 2.11.6 is binary compatible with 2.11.5. For details > of the changes in each release, see: > * http://www.scala-lang.org/news/2.10.5 > * http://www.scala-lang.org/news/2.11.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [Discussion] Using Client Requests and Responses in Server
In case anyone is still following this thread, I need a bit of help :) The old FetchResponse.PartitionData included a MessageSet object. The new FetchResponse.PartitionData includes a ByteBuffer. However, when we read from logs, we return a MessageSet, and as far as I can see, these can't be converted to ByteBuffers (at least not without copying their data). Did anyone consider how to reconcile the MessageSets with the new FetchResponse objects? Gwen On Sat, Mar 21, 2015 at 6:54 PM, Gwen Shapira wrote: > Note: I'm also treating ZkUtils as if it was a public API (i.e. converting > objects that are returned into o.a.k.common equivalents but not changing > ZkUtils itself). > I know its not public, but I suspect I'm not the only developer here who > has tons of external code that uses it. > > Gwen > > On Wed, Mar 18, 2015 at 5:48 PM, Gwen Shapira > wrote: > >> We can't rip them out completely, unfortunately - the SimpleConsumer uses >> them. >> >> So we'll need conversion at some point. I'll try to make the >> conversion point "just before hitting a public API that we can't >> modify", and hopefully it won't look too arbitrary. >> >> >> >> On Wed, Mar 18, 2015 at 5:24 PM, Jay Kreps wrote: >> > I think either approach is okay in the short term. However our goal >> should >> > be to eventually get rid of that duplicate code, so if you are up for >> just >> > ripping and cutting that may get us there sooner. >> > >> > -Jay >> > >> > On Wed, Mar 18, 2015 at 5:19 PM, Gwen Shapira >> wrote: >> > >> >> Thanks! >> >> >> >> Another clarification: >> >> The Common request/responses use slightly different infrastructure >> >> objects: Node instead of Broker, TopicPartition instead of >> >> TopicAndPartition and few more. >> >> >> >> I can write utilities to convert Node to Broker to minimize the scope >> >> of the change. >> >> Or I can start replacing Brokers with Nodes across the board. >> >> >> >> I'm currently taking the second approach - i.e, if MetadataRequest is >> >> now returning Node, I'm changing the entire line of dependencies to >> >> use Nodes instead of broker. >> >> >> >> Is this acceptable, or do we want to take a more minimal approach for >> >> this patch and do a larger replacement as a follow up? >> >> >> >> Gwen >> >> >> >> >> >> >> >> >> >> On Wed, Mar 18, 2015 at 3:32 PM, Jay Kreps >> wrote: >> >> > Great. >> >> > >> >> > For (3) yeah I think we should just think through the end-to-end >> pattern >> >> > for these versioned requests since it seems like we will have a >> number of >> >> > them. The serialization code used as you described gets us to the >> right >> >> > Struct which the user would then wrap in something like >> ProduceRequest. >> >> > Presumably there would just be one ProduceRequest that would >> internally >> >> > fill in things like null or otherwise adapt the struct to a usable >> >> object. >> >> > On the response side we would have the version from the request to >> use >> >> for >> >> > correct versioning. On question is whether this is enough or whether >> we >> >> > need to have switches in KafkaApis to do things like >> >> >if(produceRequest.version == 3) >> >> >// do something >> >> >else >> >> > // do something else >> >> > >> >> > Basically it would be good to be able to write a quick wiki that was >> like >> >> > "how to add or modify a kafka api" that explained the right way to >> do all >> >> > this. >> >> > >> >> > I don't think any of this necessarily blocks this ticket since at the >> >> > moment we don't have tons of versions of requests out there. >> >> > >> >> > -Jay >> >> > >> >> > On Wed, Mar 18, 2015 at 2:50 PM, Gwen Shapira > > >> >> wrote: >> >> > >> >> >> See inline responses: >> >> >> >> >> >> On Wed, Mar 18, 2015 at 2:26 PM, Jay Kreps >> wrote: >> >> >> > Hey Gwen, >> >> >> > >> >> >> > This makes sense to me. >> >> >> > >> >> >> > A couple of thoughts, mostly confirming what you said I think: >> >> >> > >> >> >> >1. Ideally we would move completely over to the new style of >> >> request >> >> >> >definition for server-side processing, even for the internal >> >> >> requests. This >> >> >> >way all requests would have the same header/body struct stuff. >> As >> >> you >> >> >> say >> >> >> >for the internal requests we can just delete the scala code. >> For >> >> the >> >> >> old >> >> >> >clients they will continue to use their old request definitions >> >> until >> >> >> we >> >> >> >eol them. I would propose that new changes will go only into >> the >> >> new >> >> >> >request/response objects and the old scala ones will be >> permanently >> >> >> stuck >> >> >> >on their current version until discontinued. So after this >> change >> >> >> that old >> >> >> >scala code could be considered frozen. >> >> >> >> >> >> SimpleConsumer is obviously stuck with the old request/response. >> >> >> >> >> >> The Producers can be converted to the common request/response >> without >> >> >> brea
[jira] [Commented] (KAFKA-873) Consider replacing zkclient with curator (with zkclient-bridge)
[ https://issues.apache.org/jira/browse/KAFKA-873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14375366#comment-14375366 ] Grant Henke commented on KAFKA-873: --- Is changing over to Curator something that still should/needs to be done? It seams to have some longer term benefits and now may be a good time to make the change. The zkclient-bridge is no longer part of the Apache Curator project and I think a straight conversion is likely what would be best. I can start working on this if there is interest. It may need a KIP too. > Consider replacing zkclient with curator (with zkclient-bridge) > --- > > Key: KAFKA-873 > URL: https://issues.apache.org/jira/browse/KAFKA-873 > Project: Kafka > Issue Type: Improvement >Affects Versions: 0.8.0 >Reporter: Scott Clasen > > If zkclient was replaced with curator and curator-x-zkclient-bridge it would > be initially a drop-in replacement > https://github.com/Netflix/curator/wiki/ZKClient-Bridge > With the addition of a few more props to ZkConfig, and a bit of code this > would open up the possibility of using ACLs in zookeeper (which arent > supported directly by zkclient), as well as integrating with netflix > exhibitor for those of us using that. > Looks like KafkaZookeeperClient needs some love anyhow... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (KAFKA-873) Consider replacing zkclient with curator (with zkclient-bridge)
[ https://issues.apache.org/jira/browse/KAFKA-873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke reassigned KAFKA-873: - Assignee: Grant Henke > Consider replacing zkclient with curator (with zkclient-bridge) > --- > > Key: KAFKA-873 > URL: https://issues.apache.org/jira/browse/KAFKA-873 > Project: Kafka > Issue Type: Improvement >Affects Versions: 0.8.0 >Reporter: Scott Clasen >Assignee: Grant Henke > > If zkclient was replaced with curator and curator-x-zkclient-bridge it would > be initially a drop-in replacement > https://github.com/Netflix/curator/wiki/ZKClient-Bridge > With the addition of a few more props to ZkConfig, and a bit of code this > would open up the possibility of using ACLs in zookeeper (which arent > supported directly by zkclient), as well as integrating with netflix > exhibitor for those of us using that. > Looks like KafkaZookeeperClient needs some love anyhow... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [Discussion] Using Client Requests and Responses in Server
Ack, yeah, forgot about that. It's not just a difference of wrappers. The server side actually sends the bytes lazily using FileChannel.transferTo. We need to make it possible to carry over that optimization. In some sense what we want to be able to do is set a value to a Send instead of a ByteBuffer. Let me try to add that support to the protocol definition stuff, will probably take me a few days to free up time. -Jay On Sun, Mar 22, 2015 at 7:44 PM, Gwen Shapira wrote: > In case anyone is still following this thread, I need a bit of help :) > > The old FetchResponse.PartitionData included a MessageSet object. > The new FetchResponse.PartitionData includes a ByteBuffer. > > However, when we read from logs, we return a MessageSet, and as far as I > can see, these can't be converted to ByteBuffers (at least not without > copying their data). > > Did anyone consider how to reconcile the MessageSets with the new > FetchResponse objects? > > Gwen > > > On Sat, Mar 21, 2015 at 6:54 PM, Gwen Shapira > wrote: > > > Note: I'm also treating ZkUtils as if it was a public API (i.e. > converting > > objects that are returned into o.a.k.common equivalents but not changing > > ZkUtils itself). > > I know its not public, but I suspect I'm not the only developer here who > > has tons of external code that uses it. > > > > Gwen > > > > On Wed, Mar 18, 2015 at 5:48 PM, Gwen Shapira > > wrote: > > > >> We can't rip them out completely, unfortunately - the SimpleConsumer > uses > >> them. > >> > >> So we'll need conversion at some point. I'll try to make the > >> conversion point "just before hitting a public API that we can't > >> modify", and hopefully it won't look too arbitrary. > >> > >> > >> > >> On Wed, Mar 18, 2015 at 5:24 PM, Jay Kreps wrote: > >> > I think either approach is okay in the short term. However our goal > >> should > >> > be to eventually get rid of that duplicate code, so if you are up for > >> just > >> > ripping and cutting that may get us there sooner. > >> > > >> > -Jay > >> > > >> > On Wed, Mar 18, 2015 at 5:19 PM, Gwen Shapira > >> wrote: > >> > > >> >> Thanks! > >> >> > >> >> Another clarification: > >> >> The Common request/responses use slightly different infrastructure > >> >> objects: Node instead of Broker, TopicPartition instead of > >> >> TopicAndPartition and few more. > >> >> > >> >> I can write utilities to convert Node to Broker to minimize the scope > >> >> of the change. > >> >> Or I can start replacing Brokers with Nodes across the board. > >> >> > >> >> I'm currently taking the second approach - i.e, if MetadataRequest is > >> >> now returning Node, I'm changing the entire line of dependencies to > >> >> use Nodes instead of broker. > >> >> > >> >> Is this acceptable, or do we want to take a more minimal approach for > >> >> this patch and do a larger replacement as a follow up? > >> >> > >> >> Gwen > >> >> > >> >> > >> >> > >> >> > >> >> On Wed, Mar 18, 2015 at 3:32 PM, Jay Kreps > >> wrote: > >> >> > Great. > >> >> > > >> >> > For (3) yeah I think we should just think through the end-to-end > >> pattern > >> >> > for these versioned requests since it seems like we will have a > >> number of > >> >> > them. The serialization code used as you described gets us to the > >> right > >> >> > Struct which the user would then wrap in something like > >> ProduceRequest. > >> >> > Presumably there would just be one ProduceRequest that would > >> internally > >> >> > fill in things like null or otherwise adapt the struct to a usable > >> >> object. > >> >> > On the response side we would have the version from the request to > >> use > >> >> for > >> >> > correct versioning. On question is whether this is enough or > whether > >> we > >> >> > need to have switches in KafkaApis to do things like > >> >> >if(produceRequest.version == 3) > >> >> >// do something > >> >> >else > >> >> > // do something else > >> >> > > >> >> > Basically it would be good to be able to write a quick wiki that > was > >> like > >> >> > "how to add or modify a kafka api" that explained the right way to > >> do all > >> >> > this. > >> >> > > >> >> > I don't think any of this necessarily blocks this ticket since at > the > >> >> > moment we don't have tons of versions of requests out there. > >> >> > > >> >> > -Jay > >> >> > > >> >> > On Wed, Mar 18, 2015 at 2:50 PM, Gwen Shapira < > gshap...@cloudera.com > >> > > >> >> wrote: > >> >> > > >> >> >> See inline responses: > >> >> >> > >> >> >> On Wed, Mar 18, 2015 at 2:26 PM, Jay Kreps > >> wrote: > >> >> >> > Hey Gwen, > >> >> >> > > >> >> >> > This makes sense to me. > >> >> >> > > >> >> >> > A couple of thoughts, mostly confirming what you said I think: > >> >> >> > > >> >> >> >1. Ideally we would move completely over to the new style of > >> >> request > >> >> >> >definition for server-side processing, even for the internal > >> >> >> requests. This > >> >> >> >way all requests would have the same header/body struct > st
