[jira] [Created] (KAFKA-3156) Got EOFException When use 0.9.0.0 Kafka clients with broker 0.8.1.2.2

Manyu Gou created KAFKA-3156: Summary: Got EOFException When use 0.9.0.0 Kafka clients with broker 0.8.1.2.2 Key: KAFKA-3156 URL: https://issues.apache.org/jira/browse/KAFKA-3156 Project: Kafka

Build failed in Jenkins: kafka-trunk-jdk8 #321

See Changes: [wangguoz] MINOR: join test for windowed keys -- [...truncated 2817 lines...] kafka.log.BrokerCompressionTest > testBrokerSideCompression[0] PASSED kafka.log.BrokerCompressionTest

[jira] [Created] (KAFKA-3157) Mirror maker doesn't commit offset with new consumer when there is no more messages

TAO XIAO created KAFKA-3157: --- Summary: Mirror maker doesn't commit offset with new consumer when there is no more messages Key: KAFKA-3157 URL: https://issues.apache.org/jira/browse/KAFKA-3157 Project: Kafk

[jira] [Updated] (KAFKA-3157) Mirror maker doesn't commit offset with new consumer when there is no more messages

[ https://issues.apache.org/jira/browse/KAFKA-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] TAO XIAO updated KAFKA-3157: Description: Mirror maker doesn't commit messages with new consumer enabled when messages are sent to sourc

Re: [DISCUSS] KIP-43: Kafka SASL enhancements

Tao, Thank you for the explanation. I couldn't find a standard Java interface that would be suitable, so will define one based on your requirement and update the KIP. Regards, Rajini On Wed, Jan 27, 2016 at 2:12 AM, tao xiao wrote: > Hi Rajini, > > One requirement I have is to refresh the log

[GitHub] kafka pull request: KAFKA-3152; kafka-acl doesn't allow space in p...

GitHub user ijuma opened a pull request: https://github.com/apache/kafka/pull/818 KAFKA-3152; kafka-acl doesn't allow space in principal name * Add quotes to `$@` in shell scripts This is necessary for correct processing of quotes in the user command. * Minor improve

[jira] [Commented] (KAFKA-3152) kafka-acl doesn't allow space in principal name

[ https://issues.apache.org/jira/browse/KAFKA-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15119066#comment-15119066 ] ASF GitHub Bot commented on KAFKA-3152: --- GitHub user ijuma opened a pull request:

[jira] [Updated] (KAFKA-3152) kafka-acl doesn't allow space in principal name

[ https://issues.apache.org/jira/browse/KAFKA-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-3152: --- Reviewer: Jun Rao Status: Patch Available (was: In Progress) > kafka-acl doesn't allow space in

[GitHub] kafka pull request: MINOR: Fix syntax used for comment in JAAS con...

GitHub user ijuma opened a pull request: https://github.com/apache/kafka/pull/819 MINOR: Fix syntax used for comment in JAAS config file Simple fix, but important because the incorrect syntax causes the server not to start. You can merge this pull request into a Git repository by r

Re: [DISCUSS] KIP-43: Kafka SASL enhancements

Thanks Rajini. The other thing in my mind is that we should find a way to expose subject to SASL so that other mechanisms are able to use the principal and credentials stored in subject to do authentication. I am thinking to have below interface that can be extended by users to build the SASL clie

[GitHub] kafka pull request: KAFKA-3152; kafka-acl doesn't allow space in p...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/818 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabl

[jira] [Commented] (KAFKA-3152) kafka-acl doesn't allow space in principal name

[ https://issues.apache.org/jira/browse/KAFKA-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15119698#comment-15119698 ] ASF GitHub Bot commented on KAFKA-3152: --- Github user asfgit closed the pull request

[jira] [Commented] (KAFKA-1493) Use a well-documented LZ4 compression format and remove redundant LZ4HC option

[ https://issues.apache.org/jira/browse/KAFKA-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15119703#comment-15119703 ] Dana Powers commented on KAFKA-1493: Hi all - it appears that the header checksum (HC)

[jira] [Commented] (KAFKA-3006) Make collection default container type for sequences in the consumer API

[ https://issues.apache.org/jira/browse/KAFKA-3006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15119739#comment-15119739 ] Pierre-Yves Ritschard commented on KAFKA-3006: -- [~ijuma] [~gwenshap] Can I re

Build failed in Jenkins: kafka-trunk-jdk7 #995

See Changes: [junrao] KAFKA-3152; kafka-acl doesn't allow space in principal name -- Started by an SCM change [EnvInject] - Loading node environment variables. Building remotely on ubuntu-6 (dock

Build failed in Jenkins: kafka-trunk-jdk8 #322

See Changes: [junrao] KAFKA-3152; kafka-acl doesn't allow space in principal name -- Started by an SCM change [EnvInject] - Loading node environment variables. Building remotely on H11 (docker Ub

[jira] [Created] (KAFKA-3158) ConsumerGroupCommand should tell whether group is actually dead

Jason Gustafson created KAFKA-3158: -- Summary: ConsumerGroupCommand should tell whether group is actually dead Key: KAFKA-3158 URL: https://issues.apache.org/jira/browse/KAFKA-3158 Project: Kafka

[GitHub] kafka pull request: MINOR: Fix syntax used for comment in JAAS con...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/819 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabl

Re: [DISCUSS] KIP-32 - Add CreateTime and LogAppendTime to Kafka message

Hi Guozhang, That makes sense. I will update the KIP wiki and bump up the voting thread to let people know about this change. Thanks, Jiangjie (Becket) Qin On Tue, Jan 26, 2016 at 10:55 PM, Guozhang Wang wrote: > One motivation of my proposal is actually to avoid any clients trying to > read

[jira] [Created] (KAFKA-3159) Kafka consumer 0.9.0.0 client poll is very CPU intensive under certain conditions

Rajiv Kurian created KAFKA-3159: --- Summary: Kafka consumer 0.9.0.0 client poll is very CPU intensive under certain conditions Key: KAFKA-3159 URL: https://issues.apache.org/jira/browse/KAFKA-3159 Projec

[jira] [Commented] (KAFKA-1493) Use a well-documented LZ4 compression format and remove redundant LZ4HC option

[ https://issues.apache.org/jira/browse/KAFKA-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15119920#comment-15119920 ] Magnus Edenhill commented on KAFKA-1493: [~dana.powers] I can confirm this is the

[jira] [Updated] (KAFKA-3159) Kafka consumer 0.9.0.0 client poll is very CPU intensive under certain conditions

[ https://issues.apache.org/jira/browse/KAFKA-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajiv Kurian updated KAFKA-3159: Description: We are using the new kafka consumer with the following config (as logged by kafka) me

[jira] [Updated] (KAFKA-3159) Kafka consumer 0.9.0.0 client poll is very CPU intensive under certain conditions

[ https://issues.apache.org/jira/browse/KAFKA-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajiv Kurian updated KAFKA-3159: Description: We are using the new kafka consumer with the following config (as logged by kafka) me

[jira] [Assigned] (KAFKA-3159) Kafka consumer 0.9.0.0 client poll is very CPU intensive under certain conditions

[ https://issues.apache.org/jira/browse/KAFKA-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson reassigned KAFKA-3159: -- Assignee: Jason Gustafson > Kafka consumer 0.9.0.0 client poll is very CPU intensive u

[jira] [Updated] (KAFKA-3159) Kafka consumer 0.9.0.0 client poll is very CPU intensive under certain conditions

[ https://issues.apache.org/jira/browse/KAFKA-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajiv Kurian updated KAFKA-3159: Description: We are using the new kafka consumer with the following config (as logged by kafka) me

[jira] [Updated] (KAFKA-3159) Kafka consumer 0.9.0.0 client poll is very CPU intensive under certain conditions

[ https://issues.apache.org/jira/browse/KAFKA-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajiv Kurian updated KAFKA-3159: Description: We are using the new kafka consumer with the following config (as logged by kafka) me

Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.

Hi folks, During the implementation of the KIP, Guozhang raised another proposal on how to indicate the message timestamp type used by messages. The difference between current and the new proposal only differs on messages that are a) compressed, and b) using LogAppendTime For compressed messages

Re: [DISCUSS] KIP-42: Add Producer and Consumer Interceptors

Mayuresh, Regarding your use case about mirror maker. Is it good enough as long as we know there is no message for the topic in the producer anymore? If that is the case, call producer.flush() is sufficient. Thanks, Jiangjie (Becket) Qin On Tue, Jan 26, 2016 at 6:18 PM, Mayuresh Gharat wrote:

Re: [DISCUSS] KIP-42: Add Producer and Consumer Interceptors

Calling producer.flush(), flushes all the data. So this is OK. But when you are running Mirror maker, I am not sure there is a way to flush() from outside. Thanks, Mayuresh On Wed, Jan 27, 2016 at 11:08 AM, Becket Qin wrote: > Mayuresh, > > Regarding your use case about mirror maker. Is it go

[jira] [Commented] (KAFKA-3147) Memory records is not writable in MirrorMaker

[ https://issues.apache.org/jira/browse/KAFKA-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15120004#comment-15120004 ] Jiangjie Qin commented on KAFKA-3147: - [~junrao] [~mgharat] I think the issue is what

[jira] [Commented] (KAFKA-3152) kafka-acl doesn't allow space in principal name

[ https://issues.apache.org/jira/browse/KAFKA-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15120007#comment-15120007 ] ASF GitHub Bot commented on KAFKA-3152: --- GitHub user ijuma opened a pull request:

[jira] [Updated] (KAFKA-3152) kafka-acl doesn't allow space in principal name

[ https://issues.apache.org/jira/browse/KAFKA-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-3152: --- Fix Version/s: 0.9.1.0 0.9.0.1 > kafka-acl doesn't allow space in principal name >

Build failed in Jenkins: kafka_0.9.0_jdk7 #96

See Changes: [wangguoz] MINOR: Fix syntax used for comment in JAAS config file -- [...truncated 3797 lines...] kafka.log.LogTest > testParseTopicPartitionNameForMissingTopic PASSED kafka.log.LogT

[GitHub] kafka pull request: KAFKA-3152; kafka-acl doesn't allow space in p...

GitHub user ijuma opened a pull request: https://github.com/apache/kafka/pull/820 KAFKA-3152; kafka-acl doesn't allow space in principal name (backport to 0.9.0) This is a backport of the trunk PR and it excludes the test changes due to conflicts and the fact that the changes

[jira] [Assigned] (KAFKA-3147) Memory records is not writable in MirrorMaker

[ https://issues.apache.org/jira/browse/KAFKA-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat reassigned KAFKA-3147: -- Assignee: Mayuresh Gharat > Memory records is not writable in MirrorMaker > ---

[jira] [Commented] (KAFKA-3147) Memory records is not writable in MirrorMaker

[ https://issues.apache.org/jira/browse/KAFKA-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15120014#comment-15120014 ] Mayuresh Gharat commented on KAFKA-3147: [~becket_qin] sure. I will take a look an

Build failed in Jenkins: kafka-trunk-jdk7 #996

See Changes: [wangguoz] MINOR: Fix syntax used for comment in JAAS config file -- [...truncated 3689 lines...] kafka.log.BrokerCompressionTest > testBrokerSideCompression[0] PASSED kafka.log.Br

Re: [DISCUSS] KIP-43: Kafka SASL enhancements

Hi Tao, *javax.security.auth.callback.**CallbackHandler *is the standard way in which SASL clients and server obtain additional mechanism specific input. *AuthCallbackHandler *simply extends this interface to propagate configuration properties. I was going to provide SASL mechanism and Subject to

[jira] [Created] (KAFKA-3160) Kafka LZ4 framing code miscalculates header checksum

Dana Powers created KAFKA-3160: -- Summary: Kafka LZ4 framing code miscalculates header checksum Key: KAFKA-3160 URL: https://issues.apache.org/jira/browse/KAFKA-3160 Project: Kafka Issue Type: Bu

[jira] [Commented] (KAFKA-1493) Use a well-documented LZ4 compression format and remove redundant LZ4HC option

[ https://issues.apache.org/jira/browse/KAFKA-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15120072#comment-15120072 ] Dana Powers commented on KAFKA-1493: filed KAFKA-3160 > Use a well-documented LZ4 com

Build failed in Jenkins: kafka-trunk-jdk8 #323

See Changes: [wangguoz] MINOR: Fix syntax used for comment in JAAS config file -- [...truncated 6712 lines...] org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > testGetCon

Re: [DISCUSS] KIP-45 Standardize all client sequence interaction on j.u.Collection.

Hi Pierre, Thanks for your persistence on this issue. I've gone back and forth on this a few times. The current API can definitely be annoying in some cases, but breaking compatibility still sucks. We do have the @Unstable annotation on the API, but it's unclear what exactly it means and I'm guess

[jira] [Created] (KAFKA-3161) Refactor Java client's use of the Properties class

Cliff Rhyne created KAFKA-3161: -- Summary: Refactor Java client's use of the Properties class Key: KAFKA-3161 URL: https://issues.apache.org/jira/browse/KAFKA-3161 Project: Kafka Issue Type: Impr

[jira] [Updated] (KAFKA-3161) Refactor Java client's use of the Properties class

[ https://issues.apache.org/jira/browse/KAFKA-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cliff Rhyne updated KAFKA-3161: --- Description: The KafkaConsumer takes a Properties class for the config, but then instead of using it'

Re: [DISCUSS] KIP-45 Standardize all client sequence interaction on j.u.Collection.

Hi Jason, Thanks for weighing in on this. Here's my take: - I initially opted for overloading, but this met resistance (most vocally from Jay Kreps). I don't have strong feelings either way (I tend to prefer the current proposal without overloading but would understand the need to add it b

[jira] [Resolved] (KAFKA-3156) Got EOFException When use 0.9.0.0 Kafka clients with broker 0.8.1.2.2

[ https://issues.apache.org/jira/browse/KAFKA-3156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-3156. - Resolution: Won't Fix This is the expected behavior. You must upgrade the brokers before using ne

[jira] [Commented] (KAFKA-3006) Make collection default container type for sequences in the consumer API

[ https://issues.apache.org/jira/browse/KAFKA-3006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15120288#comment-15120288 ] Gwen Shapira commented on KAFKA-3006: - Sure. Generally wait few days for consensus to

Re: [DISCUSS] KIP-45 Standardize all client sequence interaction on j.u.Collection.

I have a minor preference toward modifying the API. Because it is source-compatible and protocol-compatible, the only case that will break is if you use client code from one version but run with a JAR from a different version, which sounds like a pretty weird setup in general. Its not a strong pre

Re: [DISCUSS] KIP-45 Standardize all client sequence interaction on j.u.Collection.

Hi Pierre and Jason, A comment below. On Wed, Jan 27, 2016 at 9:01 PM, Jason Gustafson wrote: > Hi Pierre, > > Thanks for your persistence on this issue. I've gone back and forth on this > a few times. The current API can definitely be annoying in some cases, but > breaking compatibility still

Re: [DISCUSS] KIP-45 Standardize all client sequence interaction on j.u.Collection.

>From the user code I see that uses the new consumer API so far, here are my two cents: 1. For subscribe() and assign(), because we are doing an replacement, taking either a List() or Collection() makes sense. 2. For seek(), pause(), resume(), it depends on how easily user can use them. If we

Re: [DISCUSS] KIP-42: Add Producer and Consumer Interceptors

Hi Mayuresh, I see why you would want to check for messages left in the RecordAccumulator. However, I don't think this will completely solve the problem. Messages could be in-flight somewhere else, like in the socket, or there maybe in-flight messages on the consumer side of the MirrorMaker. So, i

[jira] [Commented] (KAFKA-3160) Kafka LZ4 framing code miscalculates header checksum

[ https://issues.apache.org/jira/browse/KAFKA-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15120439#comment-15120439 ] Ismael Juma commented on KAFKA-3160: [~edenhill], you're working on this, right? You s

Re: [DISCUSS] KIP-42: Add Producer and Consumer Interceptors

Hi All, The KIP wiki page is now up-to-date with the scope we have agreed on: Producer and Consumer Interceptors with a minimal set of mutable API that are not dependent on producer and consumer internal implementation. I have few more API details that I would like to bring attention to or/and di

[jira] [Created] (KAFKA-3162) KIP-42: Add Producer and Consumer Interceptors

Anna Povzner created KAFKA-3162: --- Summary: KIP-42: Add Producer and Consumer Interceptors Key: KAFKA-3162 URL: https://issues.apache.org/jira/browse/KAFKA-3162 Project: Kafka Issue Type: Bug

[jira] [Updated] (KAFKA-3162) KIP-42: Add Producer and Consumer Interceptors

[ https://issues.apache.org/jira/browse/KAFKA-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-3162: --- Description: This JIRA is for KIP-42 implementation, which includes: 1. Add ProducerInterceptor inter

[jira] [Work started] (KAFKA-3162) KIP-42: Add Producer and Consumer Interceptors

[ https://issues.apache.org/jira/browse/KAFKA-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-3162 started by Anna Povzner. --- > KIP-42: Add Producer and Consumer Interceptors >

[jira] [Commented] (KAFKA-2076) Add an API to new consumer to allow user get high watermark of partitions.

[ https://issues.apache.org/jira/browse/KAFKA-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15120501#comment-15120501 ] Jiangjie Qin commented on KAFKA-2076: - [~jkreps] Good point. From consumer point of vi

[jira] [Commented] (KAFKA-3147) Memory records is not writable in MirrorMaker

[ https://issues.apache.org/jira/browse/KAFKA-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15120591#comment-15120591 ] Jun Rao commented on KAFKA-3147: It also seems that in RecordAccumulator.abortBatches(), i

[jira] [Commented] (KAFKA-3147) Memory records is not writable in MirrorMaker

[ https://issues.apache.org/jira/browse/KAFKA-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15120613#comment-15120613 ] Jiangjie Qin commented on KAFKA-3147: - Yes, we should do that as well. > Memory recor

Re: [DISCUSS] KIP-43: Kafka SASL enhancements

The callback works for me as long as it has access to Subject and mechs. The other thing is how we can inject the customized security provider via Security.addProvider()? If I want to implement my own SASL mech I need to call the addProvider() before SASL.create so that my own implementation of SAS

[jira] [Created] (KAFKA-3163) KIP-33 - Add a time based log index to Kafka

Jiangjie Qin created KAFKA-3163: --- Summary: KIP-33 - Add a time based log index to Kafka Key: KAFKA-3163 URL: https://issues.apache.org/jira/browse/KAFKA-3163 Project: Kafka Issue Type: Task

[DISCUSS] KIP-33 - Add a time based log index to Kafka

Hi, Since KIP-31 and KIP-32 has passed. I would like to kick off the discussion on KIP-33, which propose to add a time based log index to Kafka. Here is the KIP wiki link: https://cwiki.apache.org/confluence/display/KAFKA/KIP-33+-+Add+a+time+based+log+index A good amount of the KIP has been touc

Re: [DISCUSS] KIP-42: Add Producer and Consumer Interceptors

Anna, Thanks for being diligent. +1 on #1.2 and sounds good on #3. I recommend adding checksum and size fields to RecordMetadata and ConsumerRecord instead of exposing metadata piecemeal in the interceptor APIs. Thanks, Neha On Wed, Jan 27, 2016 at 4:10 PM, Anna Povzner wrote: > Hi All, > > T

[jira] [Updated] (KAFKA-3157) Mirror maker doesn't commit offset with new consumer when there is no more messages

[ https://issues.apache.org/jira/browse/KAFKA-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] TAO XIAO updated KAFKA-3157: Status: Patch Available (was: Open) > Mirror maker doesn't commit offset with new consumer when there is no

[GitHub] kafka pull request: KAFKA-3157 - Mirror maker doesn't commit offse...

GitHub user xiaotao183 opened a pull request: https://github.com/apache/kafka/pull/821 KAFKA-3157 - Mirror maker doesn't commit offset Mirror maker doesn't commit offset with new consumer enabled when data volume is low. This is caused by infinite loop in ```receive()``` which would

[jira] [Commented] (KAFKA-3157) Mirror maker doesn't commit offset with new consumer when there is no more messages

[ https://issues.apache.org/jira/browse/KAFKA-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15120954#comment-15120954 ] ASF GitHub Bot commented on KAFKA-3157: --- GitHub user xiaotao183 opened a pull reques

[GitHub] kafka pull request: KAFKA-3103 fix - also fixes KAFKA-3155 and KAF...

GitHub user rowdyrabbit opened a pull request: https://github.com/apache/kafka/pull/822 KAFKA-3103 fix - also fixes KAFKA-3155 and KAFKA-2836 All three defects have the same root cause. Root cause is ClientUtils.fetchTopicMetadata returns the BrokerEndPoints in a non-dete

[jira] [Commented] (KAFKA-3103) Transient Failure in kafka.integration.PlaintextTopicMetadataTest.testIsrAfterBrokerShutDownAndJoinsBack

[ https://issues.apache.org/jira/browse/KAFKA-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15120960#comment-15120960 ] ASF GitHub Bot commented on KAFKA-3103: --- GitHub user rowdyrabbit opened a pull reque