[GitHub] kafka pull request #2142: Merge pull request #1 from apache/trunk

GitHub user husthang opened a pull request: https://github.com/apache/kafka/pull/2142 Merge pull request #1 from apache/trunk update test You can merge this pull request into a Git repository by running: $ git pull https://github.com/husthang/kafka trunk Alternatively you can

[jira] [Commented] (KAFKA-4406) Add support for custom Java Security Providers in configuration

[ https://issues.apache.org/jira/browse/KAFKA-4406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670080#comment-15670080 ] Magnus Reftel commented on KAFKA-4406: -- The PR adds it for both broker and clients, s

[jira] [Commented] (KAFKA-4406) Add support for custom Java Security Providers in configuration

[ https://issues.apache.org/jira/browse/KAFKA-4406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670416#comment-15670416 ] Rajini Sivaram commented on KAFKA-4406: --- Java security providers are used not just f

[jira] [Commented] (KAFKA-4406) Add support for custom Java Security Providers in configuration

[ https://issues.apache.org/jira/browse/KAFKA-4406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670468#comment-15670468 ] Magnus Reftel commented on KAFKA-4406: -- Right, so `ssl.` is out. How about `security.

[jira] [Commented] (KAFKA-4406) Add support for custom Java Security Providers in configuration

[ https://issues.apache.org/jira/browse/KAFKA-4406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670575#comment-15670575 ] Rajini Sivaram commented on KAFKA-4406: --- Perhaps it is worth writing a small KIP exp

[jira] [Commented] (KAFKA-4414) Unexpected "Halting because log truncation is not allowed"

[ https://issues.apache.org/jira/browse/KAFKA-4414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670636#comment-15670636 ] Meyer Kizner commented on KAFKA-4414: - What value would you suggest? We're already usi

Re: [VOTE] KIP-72 - Allow putting a bound on memory consumed by Incoming requests

Hi, Radai, Thanks for the updated proposal. +1 overall. A couple of comments below. 1. Our current convention is to avoid using getters. Could you change getSize and getAvailableMemory accordingly? Also, size is bit ambiguous, could we use sth like capacity? 2. This is more on the implementation

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

Sorry for sending this late. Please see the replies/comments inline : - In practice, the up-conversion may not be used until all the clients are upgraded. - I am not sure what you meant by this. Up conversion is actually done for old producer clients producing to the new broker

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

One comment below. On Wed, Nov 16, 2016 at 5:08 PM, Mayuresh Gharat wrote: >- If we don't bump up the magic byte, on the broker side, the broker >will always have to look at both tombstone bit and the value when do the >compaction. Assuming we do not bump up the magic byte, >imag

[jira] [Commented] (KAFKA-4391) On Windows, Kafka server stops with uncaught exception after coming back from sleep

[ https://issues.apache.org/jira/browse/KAFKA-4391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15671078#comment-15671078 ] Guozhang Wang commented on KAFKA-4391: -- Currently we do not have plans to release new

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

Hi Ismael, That's a very good point which I might have not considered earlier. Here is a plan that I can think of: Stage 1) The broker from now on, up converts the message to have the tombstone marker. The log compaction thread does log compaction based on both null and tombstone marker. This is

[jira] [Commented] (KAFKA-4322) StateRestoreCallback begin and end indication

[ https://issues.apache.org/jira/browse/KAFKA-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15671093#comment-15671093 ] Guozhang Wang commented on KAFKA-4322: -- Thanks [~markshelton], the reason I mentioned

[GitHub] kafka pull request #2136: MINOR: Remove unused `ByteBoundedBlockingQueue` cl...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/2136 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[jira] [Commented] (KAFKA-4414) Unexpected "Halting because log truncation is not allowed"

[ https://issues.apache.org/jira/browse/KAFKA-4414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15671200#comment-15671200 ] James Cheng commented on KAFKA-4414: Possibly related to https://issues.apache.org/jir

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

David, I think it would be better implementing such synchronization (i.e. making sure all consumers has done fetching to that point, and no one will ever want to go back and re-consume) on the admin side, not on the broker side, since 1) we want to keep the broker system to be simple enough, and r

Build failed in Jenkins: kafka-trunk-jdk8 #1043

See Changes: [jason] MINOR: Remove unused `ByteBoundedBlockingQueue` class and -- [...truncated 6405 lines...] kafka.api.AdminClientTest > testDescribeConsumerGroup STARTED kafka.api.AdminClie

[GitHub] kafka pull request #1791: KAFKA-4089: KafkaProducer expires batch when metad...

Github user sutambe closed the pull request at: https://github.com/apache/kafka/pull/1791 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[jira] [Commented] (KAFKA-4089) KafkaProducer raises Batch Expired exception

[ https://issues.apache.org/jira/browse/KAFKA-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15671351#comment-15671351 ] ASF GitHub Bot commented on KAFKA-4089: --- Github user sutambe closed the pull request

[GitHub] kafka pull request #2132: add a space to separate two words

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/2132 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] kafka pull request #2143: KAFKA-4400: Enabling configurable prefix for sink ...

GitHub user skyahead opened a pull request: https://github.com/apache/kafka/pull/2143 KAFKA-4400: Enabling configurable prefix for sink connectors' consume… …r group names. Author: Tianji Li Reviewers: Ewen Cheslack-Postava You can merge this pull request int

[jira] [Commented] (KAFKA-4400) Prefix for sink task consumer groups should be configurable

[ https://issues.apache.org/jira/browse/KAFKA-4400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15671433#comment-15671433 ] ASF GitHub Bot commented on KAFKA-4400: --- GitHub user skyahead opened a pull request:

[jira] [Commented] (KAFKA-4400) Prefix for sink task consumer groups should be configurable

[ https://issues.apache.org/jira/browse/KAFKA-4400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15671457#comment-15671457 ] Tianji Li commented on KAFKA-4400: -- [~ewencp] We got bugged by this very issue a few time

Build failed in Jenkins: kafka-trunk-jdk7 #1693

See Changes: [wangguoz] MINOR: add a space to separate two words -- [...truncated 3898 lines...] kafka.server.DynamicConfigChangeTest > shouldParseWildcardReplicationQuotaProperties STARTED ka

[jira] [Commented] (KAFKA-4322) StateRestoreCallback begin and end indication

[ https://issues.apache.org/jira/browse/KAFKA-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15671564#comment-15671564 ] Mark Shelton commented on KAFKA-4322: - I have a custom store and processor and if no i

[jira] [Resolved] (KAFKA-4359) Streams integration tests should not use commit interval of 1

[ https://issues.apache.org/jira/browse/KAFKA-4359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-4359. -- Resolution: Fixed Fix Version/s: (was: 0.10.1.1) 0.10.2.0 Issue re

[jira] [Commented] (KAFKA-4359) Streams integration tests should not use commit interval of 1

[ https://issues.apache.org/jira/browse/KAFKA-4359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15671584#comment-15671584 ] ASF GitHub Bot commented on KAFKA-4359: --- Github user asfgit closed the pull request

[GitHub] kafka pull request #2124: KAFKA-4359: Removed commit interval

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/2124 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

Hi Ismael, This is something I can think of for migration plan: So the migration plan can look something like this, with up conversion : 1) Currently lets say we have Broker at version x. 2) Currently we have clients at version x. 3) a) We move the version to Broker(x+1) : supports both tombstone

Build failed in Jenkins: kafka-trunk-jdk8 #1044

See Changes: [wangguoz] MINOR: add a space to separate two words -- [...truncated 14368 lines...] org.apache.kafka.streams.KafkaStreamsTest > shouldNotGetTaskWithKeyAndSerializerWhenNotRunning

[jira] [Created] (KAFKA-4416) Add a '--group' option to the console consumer

Vahid Hashemian created KAFKA-4416: -- Summary: Add a '--group' option to the console consumer Key: KAFKA-4416 URL: https://issues.apache.org/jira/browse/KAFKA-4416 Project: Kafka Issue Type:

Jenkins build is back to normal : kafka-trunk-jdk7 #1694

See

Jenkins build is back to normal : kafka-trunk-jdk8 #1045

See

[GitHub] kafka pull request #2060: KAFKA-4211: Update system tests to use the new con...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/2060 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[jira] [Commented] (KAFKA-4211) Change system tests to use the new consumer by default

[ https://issues.apache.org/jira/browse/KAFKA-4211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15672206#comment-15672206 ] ASF GitHub Bot commented on KAFKA-4211: --- Github user asfgit closed the pull request

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

Hi Mayuresh, Thanks for describing your plan in detail. I understand the concern about having to support both old and new way for a long time. In my opinion, we don't have a choice in that matter. We can't change semantics of the message format without having a long transition period. And we can't

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

Renu, Mayuresh and I had an offline discussion, and following is a brief summary. 1. We agreed that not bumping up magic value may result in losing zero copy during migration. 2. Given that bumping up magic value is almost free and has benefit of avoiding potential performance issue. It is probabl

[GitHub] kafka pull request #2130: MINOR: Extract SCALA_BINARY_VERSION from SCALA_VER...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/2130 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] kafka pull request #2088: Cross compile to Scala 2.12.0-RC2

Github user leachbj closed the pull request at: https://github.com/apache/kafka/pull/2088 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] kafka pull request #2113: KAFKA-4376: Cross compile to Scala 2.12.0

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/2113 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[jira] [Commented] (KAFKA-4376) Add scala 2.12 support

[ https://issues.apache.org/jira/browse/KAFKA-4376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15672605#comment-15672605 ] ASF GitHub Bot commented on KAFKA-4376: --- Github user asfgit closed the pull request

[jira] [Resolved] (KAFKA-4376) Add scala 2.12 support

[ https://issues.apache.org/jira/browse/KAFKA-4376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-4376. Resolution: Fixed > Add scala 2.12 support > -- > > Key: KAFKA-4

[jira] [Updated] (KAFKA-4376) Add scala 2.12 support

[ https://issues.apache.org/jira/browse/KAFKA-4376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-4376: --- Fix Version/s: 0.10.2.0 > Add scala 2.12 support > -- > > Key: KAF

[jira] [Assigned] (KAFKA-4415) Reduce time to create and send MetadataUpdateRequest

[ https://issues.apache.org/jira/browse/KAFKA-4415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin reassigned KAFKA-4415: --- Assignee: Dong Lin > Reduce time to create and send MetadataUpdateRequest > -

Build failed in Jenkins: kafka-trunk-jdk8 #1047

See Changes: [jason] KAFKA-4211; Update system test services to use the new consumer by [ismael] MINOR: Extract SCALA_BINARY_VERSION from SCALA_VERSION -- [...truncated 7947 lines...] kafka.in

Build failed in Jenkins: kafka-trunk-jdk8 #1048

See Changes: [ismael] KAFKA-4376; Cross compile to Scala 2.12.0 -- [...truncated 3898 lines...] kafka.integration.SaslPlaintextTopicMetadataTest > testIsrAfterBrokerShutDownAndJoinsBack STARTE

Build failed in Jenkins: kafka-trunk-jdk7 #1696

See Changes: [ismael] MINOR: Extract SCALA_BINARY_VERSION from SCALA_VERSION -- [...truncated 14205 lines...] org.apache.kafka.streams.kstream.KStreamBuilderTest > shouldThrowExceptionWhenTopic

[GitHub] kafka pull request #2142: Merge pull request #1 from apache/trunk

Github user husthang closed the pull request at: https://github.com/apache/kafka/pull/2142 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is en

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

Thanks guys, for discussing this offline and getting some consensus. So its clear for myself and others what is proposed now (i think i understand, but want to make sure) Could i ask either directly update the kip to detail the migration strategy, or (re-)state your offline discussed and agreed

Kafka high cpu usage

Hi, Our production env uses Kafka 0.9.0.1 cluster of 12 m3.large nodes. Partitions count per broker is ~450, percent of leaders per broker is 30-40%. The average messages load is ~3K/s, bytes flow in is ~10MB/s and bytes flow out is ~60 MB/s. We observed strange behaviour while putting one instan