[jira] [Updated] (KAFKA-3103) Transient Failure in kafka.integration.PlaintextTopicMetadataTest.testIsrAfterBrokerShutDownAndJoinsBack

2016-01-28 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-3103: --- Reviewer: Guozhang Wang Status: Patch Available (was: Open) > Transient Failure in > kafka.int

[jira] [Resolved] (KAFKA-3155) Transient Failure in kafka.integration.PlaintextTopicMetadataTest.testIsrAfterBrokerShutDownAndJoinsBack

2016-01-28 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-3155. Resolution: Duplicate Duplicate of KAFKA-3103. > Transient Failure in > kafka.integration.Plaintex

[jira] [Resolved] (KAFKA-2836) FAILING TEST: SaslSslTopicMetadataTest

2016-01-28 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-2836. Resolution: Duplicate Marking as a duplicate of KAFKA-3103, which has a PR. > FAILING TEST: SaslSsl

Re: [DISCUSS] KIP-43: Kafka SASL enhancements

2016-01-28 Thread Rajini Sivaram
Tao, We currently add the security provider in a static initializer in our login module. This ensures that the security provider is always installed before Kafka creates SaslServer/SaslClient. As you say, it is also possible to insert code into your application to add security provider before Kafk

JIRA access

2016-01-28 Thread Magnus Edenhill
Hey, could I have access to assign myself to issues in the Apache JIRA? Username is edenhill. Looking to provide some patches for KAFKA-3160. Regards, Magnus

Re: [DISCUSS] KIP-43: Kafka SASL enhancements

2016-01-28 Thread tao xiao
Sounds like a good approach to add provider in login module. Would love to see updates in the PR to reflect the changes in Login and AuthCallbackHandler. On Thu, 28 Jan 2016 at 19:31 Rajini Sivaram wrote: > Tao, > > We currently add the security provider in a static initializer in our login > mo

[GitHub] kafka pull request: KAFKA-3068: Keep track of bootstrap nodes inst...

2016-01-28 Thread enothereska
Github user enothereska closed the pull request at: https://github.com/apache/kafka/pull/804 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Commented] (KAFKA-3068) NetworkClient may connect to a different Kafka cluster than originally configured

2016-01-28 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15121671#comment-15121671 ] ASF GitHub Bot commented on KAFKA-3068: --- Github user enothereska closed the pull req

[jira] [Commented] (KAFKA-1464) Add a throttling option to the Kafka replication tool

2016-01-28 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15121691#comment-15121691 ] Jay Kreps commented on KAFKA-1464: -- I agree that the key difference is in-sync vs out-of-

[jira] [Commented] (KAFKA-1464) Add a throttling option to the Kafka replication tool

2016-01-28 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15121706#comment-15121706 ] Jay Kreps commented on KAFKA-1464: -- Another issue this raises is that a partition might h

Apache Kafka Contributor list

2016-01-28 Thread Denise Fernandez
Hi, I have just made my first Kafka PR and would like to be added to the contributor list so that I can assign myself to Jira issues and make changes to documentation. How do I go about getting this access? Thanks, Denise

Kafka Consumer stops receiving messages.

2016-01-28 Thread Nikhil Bansal
Hi All, My kafka consumer suddenly stops receiving messages and lags keep on increasing. Stack trace shows me the following for all the consumer threads. *"pool-43-thread-20" prio=10 tid=0x7f9a1d9d5000 nid=0x17888 waiting on condition [0x7f95d8e0d000]* * java.lang.Thread.State: WAITIN

[GitHub] kafka pull request: KAFKA-3068: Remove retry with nodesEverSeen

2016-01-28 Thread enothereska
GitHub user enothereska opened a pull request: https://github.com/apache/kafka/pull/823 KAFKA-3068: Remove retry with nodesEverSeen @ewencp @ijuma if this looks good please merge when you can. Thanks. You can merge this pull request into a Git repository by running: $ git pull

[jira] [Commented] (KAFKA-3068) NetworkClient may connect to a different Kafka cluster than originally configured

2016-01-28 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15121772#comment-15121772 ] ASF GitHub Bot commented on KAFKA-3068: --- GitHub user enothereska opened a pull reque

[GitHub] kafka pull request: KAFKA-3161: Fixed ProducerConfig/ConsumerConfi...

2016-01-28 Thread crhyne
GitHub user crhyne opened a pull request: https://github.com/apache/kafka/pull/824 KAFKA-3161: Fixed ProducerConfig/ConsumerConfig so that defaults are used in java.util.Properties This impacts the ProducerConfig and ConsumerConfig. I added a unit test to reflect the new case. Wh

[jira] [Commented] (KAFKA-3161) Refactor Java client's use of the Properties class

2016-01-28 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15121798#comment-15121798 ] ASF GitHub Bot commented on KAFKA-3161: --- GitHub user crhyne opened a pull request:

[jira] [Created] (KAFKA-3164) Document client and mirrormaker upgrade procedure/requirements

2016-01-28 Thread Grant Henke (JIRA)
Grant Henke created KAFKA-3164: -- Summary: Document client and mirrormaker upgrade procedure/requirements Key: KAFKA-3164 URL: https://issues.apache.org/jira/browse/KAFKA-3164 Project: Kafka Iss

Re: Apache Kafka Contributor list

2016-01-28 Thread Guozhang Wang
Hi Denise, What is your apache id? Guozhang On Thu, Jan 28, 2016 at 12:07 AM, Denise Fernandez wrote: > Hi, > > I have just made my first Kafka PR and would like to be added to the > contributor list so that I can assign myself to Jira issues and make > changes to documentation. How do I go ab

Re: JIRA access

2016-01-28 Thread Guozhang Wang
Done. Guozhang On Thu, Jan 28, 2016 at 4:16 AM, Magnus Edenhill wrote: > Hey, > > could I have access to assign myself to issues in the Apache JIRA? > Username is edenhill. > Looking to provide some patches for KAFKA-3160. > > Regards, > Magnus > -- -- Guozhang

[jira] [Updated] (KAFKA-3064) Improve resync method to waste less time and data transfer

2016-01-28 Thread Michael Graff (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Graff updated KAFKA-3064: - Affects Version/s: 0.8.2.1 0.9.0.0 > Improve resync method to waste less ti

[jira] [Commented] (KAFKA-3155) Transient Failure in kafka.integration.PlaintextTopicMetadataTest.testIsrAfterBrokerShutDownAndJoinsBack

2016-01-28 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15121853#comment-15121853 ] Guozhang Wang commented on KAFKA-3155: -- [~ijuma] Not sure if it the same error since

[jira] [Reopened] (KAFKA-3155) Transient Failure in kafka.integration.PlaintextTopicMetadataTest.testIsrAfterBrokerShutDownAndJoinsBack

2016-01-28 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma reopened KAFKA-3155: > Transient Failure in > kafka.integration.PlaintextTopicMetadataTest.testIsrAfterBrokerShutDownAndJoin

[jira] [Commented] (KAFKA-3155) Transient Failure in kafka.integration.PlaintextTopicMetadataTest.testIsrAfterBrokerShutDownAndJoinsBack

2016-01-28 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15121877#comment-15121877 ] Ismael Juma commented on KAFKA-3155: Hmm, that's true. It was stated in the PR for KAF

[jira] [Commented] (KAFKA-3064) Improve resync method to waste less time and data transfer

2016-01-28 Thread Michael Graff (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15121886#comment-15121886 ] Michael Graff commented on KAFKA-3064: -- Any thought on this issue? If all we had to

Re: [DISCUSS] KIP-42: Add Producer and Consumer Interceptors

2016-01-28 Thread Becket Qin
I am +1 on #1.2 and #3. #2: Regarding CRC, I am not sure if users care about CRC. is there any specific use case? Currently we validate messages by calling ensureValid() to verify the checksum and throw exception if it does not match. Message size would be useful. We can add that to ConsumerRecor

[jira] [Created] (KAFKA-3165) Fix ignored parameters in the gradle "All" tasks

2016-01-28 Thread Grant Henke (JIRA)
Grant Henke created KAFKA-3165: -- Summary: Fix ignored parameters in the gradle "All" tasks Key: KAFKA-3165 URL: https://issues.apache.org/jira/browse/KAFKA-3165 Project: Kafka Issue Type: Sub-ta

[GitHub] kafka pull request: KAFKA-3147 : Memory records is not writable in...

2016-01-28 Thread MayureshGharat
GitHub user MayureshGharat opened a pull request: https://github.com/apache/kafka/pull/825 KAFKA-3147 : Memory records is not writable in MirrorMaker Remove the batch from the RecordAccumulator once its closed while aborting batches. Make sure we don't accept new batch appends to Re

[jira] [Commented] (KAFKA-3147) Memory records is not writable in MirrorMaker

2016-01-28 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15122084#comment-15122084 ] ASF GitHub Bot commented on KAFKA-3147: --- GitHub user MayureshGharat opened a pull re

Re: [DISCUSS] KIP-42: Add Producer and Consumer Interceptors

2016-01-28 Thread Anna Povzner
Hi Becket, The use-case for CRC is end-to-end audit, rather than checking whether a single message is corrupt or not. Regarding record size, I was thinking to extract record size from Record. That will include header overhead as well. I think total record size will tell users how much bandwidth t

[jira] [Commented] (KAFKA-1791) Corrupt index after safe shutdown and restart

2016-01-28 Thread sandeep reddy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15122134#comment-15122134 ] sandeep reddy commented on KAFKA-1791: -- There was an error in one of the threads duri

[jira] [Commented] (KAFKA-3060) Refactor MeteredXXStore

2016-01-28 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15122147#comment-15122147 ] ASF GitHub Bot commented on KAFKA-3060: --- GitHub user guozhangwang opened a pull requ

[GitHub] kafka pull request: KAFKA-3060: Refactor MeteredStore and RockDBSt...

2016-01-28 Thread guozhangwang
GitHub user guozhangwang opened a pull request: https://github.com/apache/kafka/pull/826 KAFKA-3060: Refactor MeteredStore and RockDBStore Impl Changes include: 1) Move logging logic from MeteredXXXStore to internal stores, and leave WindowedStore API clean by removed all i

Re: [DISCUSS] KIP-42: Add Producer and Consumer Interceptors

2016-01-28 Thread Becket Qin
Anna, Using CRC to do end2end auditing might be very costly because you will need to collect all the CRC from both producer and consumer. And it is based on the assumption that broker does not modify the record. Can you shed some idea on how end to end auditing will be using the CRC before we deci

Jenkins build is back to normal : kafka_0.9.0_jdk7 #97

2016-01-28 Thread Apache Jenkins Server
See

Re: kafka-consumer-groups.sh doesn't work when consumers are off

2016-01-28 Thread Cliff Rhyne
Just following up on this concern. Is there a constraint that prevents ConsumerGroupCommand from reporting offsets on a group if no members are connected, or is this just the current implementation? Thanks, Cliff On Mon, Jan 25, 2016 at 3:50 PM, Cliff Rhyne wrote: > I'm running into a few chal

[jira] [Created] (KAFKA-3166) Disable SSL client authentication for SASL_SSL security protocol

2016-01-28 Thread Ismael Juma (JIRA)
Ismael Juma created KAFKA-3166: -- Summary: Disable SSL client authentication for SASL_SSL security protocol Key: KAFKA-3166 URL: https://issues.apache.org/jira/browse/KAFKA-3166 Project: Kafka I

[jira] [Assigned] (KAFKA-3166) Disable SSL client authentication for SASL_SSL security protocol

2016-01-28 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma reassigned KAFKA-3166: -- Assignee: Ismael Juma > Disable SSL client authentication for SASL_SSL security protocol >

[GitHub] kafka pull request: KAFKA-3166; Disable SSL client authentication ...

2016-01-28 Thread ijuma
GitHub user ijuma opened a pull request: https://github.com/apache/kafka/pull/827 KAFKA-3166; Disable SSL client authentication for SASL_SSL security protocol Also: * Fixed a bug in `createSslConfig` where we were always generating a keystore even if `useClientCert` was

[jira] [Commented] (KAFKA-3166) Disable SSL client authentication for SASL_SSL security protocol

2016-01-28 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15122259#comment-15122259 ] ASF GitHub Bot commented on KAFKA-3166: --- GitHub user ijuma opened a pull request:

[jira] [Updated] (KAFKA-3166) Disable SSL client authentication for SASL_SSL security protocol

2016-01-28 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-3166: --- Reviewer: Jun Rao Status: Patch Available (was: Open) > Disable SSL client authentication for S

[jira] [Commented] (KAFKA-3166) Disable SSL client authentication for SASL_SSL security protocol

2016-01-28 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15122266#comment-15122266 ] Ismael Juma commented on KAFKA-3166: Thoughts [~rsivaram] [~harsha_ch]? > Disable SSL

[GitHub] kafka pull request: KAFKA-3103; Transient Failure in testIsrAfterB...

2016-01-28 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/822 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabl

[jira] [Commented] (KAFKA-3103) Transient Failure in kafka.integration.PlaintextTopicMetadataTest.testIsrAfterBrokerShutDownAndJoinsBack

2016-01-28 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15122270#comment-15122270 ] ASF GitHub Bot commented on KAFKA-3103: --- Github user asfgit closed the pull request

[jira] [Updated] (KAFKA-3103) Transient Failure in kafka.integration.PlaintextTopicMetadataTest.testIsrAfterBrokerShutDownAndJoinsBack

2016-01-28 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-3103: - Resolution: Fixed Fix Version/s: 0.9.0.1 Status: Resolved (was: Patch Available)

[jira] [Updated] (KAFKA-3103) Transient Failure in kafka.integration.PlaintextTopicMetadataTest.testIsrAfterBrokerShutDownAndJoinsBack

2016-01-28 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-3103: - Assignee: Denise Fernandez (was: Guozhang Wang) > Transient Failure in > kafka.integration.Plain

[jira] [Assigned] (KAFKA-3103) Transient Failure in kafka.integration.PlaintextTopicMetadataTest.testIsrAfterBrokerShutDownAndJoinsBack

2016-01-28 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang reassigned KAFKA-3103: Assignee: Guozhang Wang > Transient Failure in > kafka.integration.PlaintextTopicMetadataT

[jira] [Commented] (KAFKA-3103) Transient Failure in kafka.integration.PlaintextTopicMetadataTest.testIsrAfterBrokerShutDownAndJoinsBack

2016-01-28 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15122276#comment-15122276 ] Guozhang Wang commented on KAFKA-3103: -- Thanks [~rowdyrabbit]. Just added you to the

Build failed in Jenkins: kafka_0.9.0_jdk7 #98

2016-01-28 Thread Apache Jenkins Server
See Changes: [wangguoz] KAFKA-3103; Fix transient Failure in -- [...truncated 585 lines...] kafka.server.ServerShutdownTest > testCleanShutdown PASSED kafka.server.ServerShutdownTest > testCleanS

Build failed in Jenkins: kafka-trunk-jdk8 #324

2016-01-28 Thread Apache Jenkins Server
See Changes: [wangguoz] KAFKA-3103; Fix transient Failure in -- Started by an SCM change [EnvInject] - Loading node environment variables. Building remotely on ubuntu-2 (docker Ubuntu ubuntu yaho

[jira] [Created] (KAFKA-3167) Use local to the workspace cache and recreate it on every build

2016-01-28 Thread Ismael Juma (JIRA)
Ismael Juma created KAFKA-3167: -- Summary: Use local to the workspace cache and recreate it on every build Key: KAFKA-3167 URL: https://issues.apache.org/jira/browse/KAFKA-3167 Project: Kafka Is

[jira] [Commented] (KAFKA-2676) AclCommandTest has wrong package name

2016-01-28 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15122331#comment-15122331 ] ASF GitHub Bot commented on KAFKA-2676: --- GitHub user kichristensen opened a pull req

[GitHub] kafka pull request: KAFKA-2676: AclCommandTest has wrong package n...

2016-01-28 Thread kichristensen
GitHub user kichristensen opened a pull request: https://github.com/apache/kafka/pull/828 KAFKA-2676: AclCommandTest has wrong package name Also fixed a couple of other tests with the same issue. This is my original work and I license the work to the project under the proje

Re: kafka-consumer-groups.sh doesn't work when consumers are off

2016-01-28 Thread Guozhang Wang
Hi Cliff, Short answer to your question is it is just the current implementation. The coordinator stores the offsets as messages in an internal topic and also keeps the latest offset values in in-memory. It answers ConsumerGroupRequest using its cached offset, and upon the consumer group being re

[jira] [Commented] (KAFKA-3060) Refactor MeteredXXStore

2016-01-28 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15122348#comment-15122348 ] ASF GitHub Bot commented on KAFKA-3060: --- GitHub user guozhangwang reopened a pull re

[GitHub] kafka pull request: KAFKA-3060: Refactor MeteredStore and RockDBSt...

2016-01-28 Thread guozhangwang
Github user guozhangwang closed the pull request at: https://github.com/apache/kafka/pull/826 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] kafka pull request: KAFKA-3060: Refactor MeteredStore and RockDBSt...

2016-01-28 Thread guozhangwang
GitHub user guozhangwang reopened a pull request: https://github.com/apache/kafka/pull/826 KAFKA-3060: Refactor MeteredStore and RockDBStore Impl Changes include: 1) Move logging logic from MeteredXXXStore to internal stores, and leave WindowedStore API clean by removed all

[jira] [Commented] (KAFKA-3060) Refactor MeteredXXStore

2016-01-28 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15122347#comment-15122347 ] ASF GitHub Bot commented on KAFKA-3060: --- Github user guozhangwang closed the pull re

[jira] [Created] (KAFKA-3168) Failure in kafka.integration.PrimitiveApiTest.testPipelinedProduceRequests

2016-01-28 Thread Guozhang Wang (JIRA)
Guozhang Wang created KAFKA-3168: Summary: Failure in kafka.integration.PrimitiveApiTest.testPipelinedProduceRequests Key: KAFKA-3168 URL: https://issues.apache.org/jira/browse/KAFKA-3168 Project: Kaf

Re: kafka-consumer-groups.sh doesn't work when consumers are off

2016-01-28 Thread Cliff Rhyne
Hi Guozhang, That looks like it might help but feels like there might be some gaps. Would it be able to survive restarts of the kafka broker? How long would it stay in the cache (and is that configurable)? If it expires from the cache, what's the cache-miss operation look like? (yes, a lot of t

Re: [DISCUSS] KIP-42: Add Producer and Consumer Interceptors

2016-01-28 Thread Anna Povzner
Hi Becket, It will be up to the interceptor to implement their audit or monitoring strategy. I would also imagine there is more than one good way to do audit. So, I agree that some of the interceptors may not use CRC, and we will not require it. The question is now whether intercepting CRCs is nee

Build failed in Jenkins: kafka-trunk-jdk7 #997

2016-01-28 Thread Apache Jenkins Server
See Changes: [wangguoz] KAFKA-3103; Fix transient Failure in -- [...truncated 4741 lines...] org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > testCreateConnectorExists PA

Re: [DISCUSS] KIP-42: Add Producer and Consumer Interceptors

2016-01-28 Thread Todd Palino
It may be difficult (or nearly impossible) to get actual compressed bytes for a message from a compressed batch, but I do think it’s useful information to have available for the very reason noted, bandwidth consumed. Does it make sense to have an interceptor at the batch level that can provide this

Re: [DISCUSS] KIP-42: Add Producer and Consumer Interceptors

2016-01-28 Thread Anna Povzner
Regarding record size as bytes sent over the wire, my concern is that it is almost impossible to calculate per-record. We could do as: 1) compressed bytes / number of records in a compressed message, as Todd mentioned; or 2) or same as #1 but take it proportional to uncompressed record size vs. tot

Re: kafka-consumer-groups.sh doesn't work when consumers are off

2016-01-28 Thread Guozhang Wang
Once the offset is written to the log it is persistent and hence should survive broker failures. And its retention policy is configurable. It may be a bit misleading in saying "in-memory cache" in my previous email: the brokers just keep the in-memory map of [group, partition] -> latest_offset, wh

[jira] [Commented] (KAFKA-3159) Kafka consumer 0.9.0.0 client poll is very CPU intensive under certain conditions

2016-01-28 Thread Jason Gustafson (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15122467#comment-15122467 ] Jason Gustafson commented on KAFKA-3159: [~ra...@signalfx.com] Looks like these EO

[jira] [Commented] (KAFKA-3159) Kafka consumer 0.9.0.0 client poll is very CPU intensive under certain conditions

2016-01-28 Thread Jason Gustafson (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15122507#comment-15122507 ] Jason Gustafson commented on KAFKA-3159: However, I do see significantly more exce

[jira] [Commented] (KAFKA-3159) Kafka consumer 0.9.0.0 client poll is very CPU intensive under certain conditions

2016-01-28 Thread Rajiv Kurian (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15122594#comment-15122594 ] Rajiv Kurian commented on KAFKA-3159: - I don't enable compression on the topic. Howeve

[jira] [Commented] (KAFKA-3159) Kafka consumer 0.9.0.0 client poll is very CPU intensive under certain conditions

2016-01-28 Thread Jason Gustafson (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15122607#comment-15122607 ] Jason Gustafson commented on KAFKA-3159: You can use the DumpLogSegmentsTool. The

[jira] [Commented] (KAFKA-3159) Kafka consumer 0.9.0.0 client poll is very CPU intensive under certain conditions

2016-01-28 Thread Rajiv Kurian (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15122614#comment-15122614 ] Rajiv Kurian commented on KAFKA-3159: - Actually I managed to dig through the logs and

[jira] [Commented] (KAFKA-3159) Kafka consumer 0.9.0.0 client poll is very CPU intensive under certain conditions

2016-01-28 Thread Jason Gustafson (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15122640#comment-15122640 ] Jason Gustafson commented on KAFKA-3159: [~ra...@signalfx.com] Here's a patch I've

[jira] [Updated] (KAFKA-3152) kafka-acl doesn't allow space in principal name

2016-01-28 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-3152: --- Resolution: Fixed Fix Version/s: (was: 0.9.1.0) Status: Resolved (was: Patch Available)

[jira] [Commented] (KAFKA-3159) Kafka consumer 0.9.0.0 client poll is very CPU intensive under certain conditions

2016-01-28 Thread Rajiv Kurian (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15122659#comment-15122659 ] Rajiv Kurian commented on KAFKA-3159: - Thanks Jason. I can try to do that early next w

[GitHub] kafka pull request: Fix mis-spell in ConsumerConfig.scala

2016-01-28 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/4 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled

Build failed in Jenkins: kafka_0.9.0_jdk7 #99

2016-01-28 Thread Apache Jenkins Server
See Changes: [junrao] KAFKA-3152; kafka-acl doesn't allow space in principal name (backport [wangguoz] MINOR: Fix mis-spell in ConsumerConfig.scala -- Started by an SCM change [EnvInject] - Loadi

Build failed in Jenkins: kafka-trunk-jdk7 #998

2016-01-28 Thread Apache Jenkins Server
See Changes: [wangguoz] MINOR: Fix mis-spell in ConsumerConfig.scala -- Started by an SCM change [EnvInject] - Loading node environment variables. Building remotely on H11 (docker Ubuntu ubuntu y

[jira] [Commented] (KAFKA-1545) java.net.InetAddress.getLocalHost in KafkaHealthcheck.register may fail on some irregular hostnames

2016-01-28 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15122719#comment-15122719 ] Guozhang Wang commented on KAFKA-1545: -- Left some comments in the PR, closing this ti

[jira] [Resolved] (KAFKA-1545) java.net.InetAddress.getLocalHost in KafkaHealthcheck.register may fail on some irregular hostnames

2016-01-28 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-1545. -- Resolution: Not A Problem > java.net.InetAddress.getLocalHost in KafkaHealthcheck.register may f

[GitHub] kafka pull request: KAFKA-948 : Update ReplicaStateMachine.scala

2016-01-28 Thread dibbhatt
Github user dibbhatt closed the pull request at: https://github.com/apache/kafka/pull/5 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabl

[jira] [Commented] (KAFKA-948) ISR list in LeaderAndISR path not updated for partitions when Broker (which is not leader) is down

2016-01-28 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15122731#comment-15122731 ] ASF GitHub Bot commented on KAFKA-948: -- Github user dibbhatt closed the pull request a

Re: [DISCUSS] KIP-42: Add Producer and Consumer Interceptors

2016-01-28 Thread Anna Povzner
On a second thought, yes, I think we should expose record size that represents application bytes. This is Becket's option #1. I updated the KIP wiki with new fields in RecordMetadata and ConsumerRecord. I would like to start a voting thread tomorrow if there are no objections or more things to di

Re: kafka-consumer-groups.sh doesn't work when consumers are off

2016-01-28 Thread tao xiao
Guozhang, The old ConsumerOffsetChecker works for new consumer too with offset stored in Kafka. I tested it with mirror maker with new consumer enabled. it is able to show offset during mirror maker running and after its shutdown. On Fri, 29 Jan 2016 at 06:34 Guozhang Wang wrote: > Once the off

Build failed in Jenkins: kafka-trunk-jdk8 #325

2016-01-28 Thread Apache Jenkins Server
See Changes: [wangguoz] MINOR: Fix mis-spell in ConsumerConfig.scala -- [...truncated 82 lines...] /x1/jenkins/jenkins-slave/workspace/kafka-trunk-jdk8/core/src/main/scala/kafka/server/KafkaServe

[GitHub] kafka pull request: KAFKA-2143: fix replica offset truncate to beg...

2016-01-28 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/129 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabl

[jira] [Updated] (KAFKA-2143) Replicas get ahead of leader and fail

2016-01-28 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-2143: - Resolution: Fixed Fix Version/s: 0.9.0.1 Status: Resolved (was: Patch Available)

[jira] [Commented] (KAFKA-2143) Replicas get ahead of leader and fail

2016-01-28 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15122919#comment-15122919 ] ASF GitHub Bot commented on KAFKA-2143: --- Github user asfgit closed the pull request

[jira] [Updated] (KAFKA-2221) Log the entire cause which caused a reconnect in the SimpleConsumer

2016-01-28 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-2221: - Resolution: Fixed Fix Version/s: 0.9.0.1 Status: Resolved (was: Patch Available)

[GitHub] kafka pull request: Log the real exception which triggered a recon...

2016-01-28 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/138 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabl

[jira] [Commented] (KAFKA-2221) Log the entire cause which caused a reconnect in the SimpleConsumer

2016-01-28 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15122923#comment-15122923 ] ASF GitHub Bot commented on KAFKA-2221: --- Github user asfgit closed the pull request

[jira] [Updated] (KAFKA-2435) More optimally balanced partition assignment strategy

2016-01-28 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-2435: - Reviewer: Jason Gustafson Assigning to [~hachikuji] to review. > More optimally balanced partitio

[jira] [Updated] (KAFKA-2522) ConsumerGroupCommand sends all output to STDOUT

2016-01-28 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-2522: - Reviewer: Ismael Juma > ConsumerGroupCommand sends all output to STDOUT >

[GitHub] kafka pull request: Add Rolling Upgrade Notes to Security Docs

2016-01-28 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/625 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabl

[GitHub] kafka pull request: KAFKA-2653: Kafka Streams Stateful API Design

2016-01-28 Thread guozhangwang
Github user guozhangwang closed the pull request at: https://github.com/apache/kafka/pull/691 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Commented] (KAFKA-2653) Stateful operations in the KStream DSL layer

2016-01-28 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15122951#comment-15122951 ] ASF GitHub Bot commented on KAFKA-2653: --- Github user guozhangwang closed the pull re

Re: kafka-consumer-groups.sh doesn't work when consumers are off

2016-01-28 Thread Guozhang Wang
Tao, Hmm that is a bit wired since ConsumerOffsetChecker itself does not talk to brokers at all, but only through ZK. Guozhang On Thu, Jan 28, 2016 at 6:07 PM, tao xiao wrote: > Guozhang, > > The old ConsumerOffsetChecker works for new consumer too with offset stored > in Kafka. I tested it wi

Re: kafka-consumer-groups.sh doesn't work when consumers are off

2016-01-28 Thread tao xiao
it first issues an offsetrequest to broker and check if offset is stored in broker if not it will queries zk https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala#L171 On Fri, 29 Jan 2016 at 13:11 Guozhang Wang wrote: > Tao, > > Hmm that is a bit

Jenkins build is back to normal : kafka_0.9.0_jdk7 #100

2016-01-28 Thread Apache Jenkins Server
See

Build failed in Jenkins: kafka-trunk-jdk7 #999

2016-01-28 Thread Apache Jenkins Server
See Changes: [wangguoz] KAFKA-2143: fix replica offset truncate to beginning during leader [wangguoz] KAFKA-2221: Log the real exception which triggered a reconnect -- [...truncated 7056 lines..

Build failed in Jenkins: kafka-trunk-jdk7 #1000

2016-01-28 Thread Apache Jenkins Server
See Changes: [wangguoz] MINOR: Add Rolling Upgrade Notes to Security Docs -- Started by an SCM change [EnvInject] - Loading node environment variables. Building remotely on ubuntu3 (Ubuntu ubunt

Build failed in Jenkins: kafka-trunk-jdk8 #326

2016-01-28 Thread Apache Jenkins Server
See Changes: [wangguoz] KAFKA-2143: fix replica offset truncate to beginning during leader [wangguoz] KAFKA-2221: Log the real exception which triggered a reconnect [wangguoz] MINOR: Add Rolling Upgrade Notes to Security Docs ---

Re: JIRA access

2016-01-28 Thread Magnus Edenhill
Thanks! 2016-01-28 17:36 GMT+01:00 Guozhang Wang : > Done. > > Guozhang > > On Thu, Jan 28, 2016 at 4:16 AM, Magnus Edenhill > wrote: > > > Hey, > > > > could I have access to assign myself to issues in the Apache JIRA? > > Username is edenhill. > > Looking to provide some patches for KAFKA-3160

  1   2   >