Re: [VOTE] KIP-128: Add ByteArrayConverter for Kafka Connect

Me too. +1. -- Dongjin Lee Software developer in Line+. So interested in massive-scale machine learning. facebook: www.facebook.com/dongjin.lee.kr (http://www.facebook.com/dongjin.lee.kr) linkedin: kr.linkedin.com/in/dongjinleekr (http://kr.linkedin.c

Re: [DISCUSS] KIP-124: Request rate quotas

Hi Todd, Thank you for the review. For SSL, the case that is not covered is Scenario 6 in the KIP that Ismael pointed out. For clusters with only SSL or PLAINTEXT, byte rate quotas work well, but for clusters with both SSL and PLAINTEXT, network thread utilization also needs to be taken into acco

[jira] [Commented] (KAFKA-4669) KafkaProducer.flush hangs when NetworkClient.handleCompletedReceives throws exception

[ https://issues.apache.org/jira/browse/KAFKA-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15901075#comment-15901075 ] Edoardo Comar commented on KAFKA-4669: -- The NPE moved from broker1 to another broker

[jira] [Commented] (KAFKA-4864) Kafka Secure Migrator tool doesn't secure all the nodes

[ https://issues.apache.org/jira/browse/KAFKA-4864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15901088#comment-15901088 ] Ismael Juma commented on KAFKA-4864: Thanks for the report. I think the expectation wa

[jira] [Commented] (KAFKA-4669) KafkaProducer.flush hangs when NetworkClient.handleCompletedReceives throws exception

[ https://issues.apache.org/jira/browse/KAFKA-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15901095#comment-15901095 ] Ismael Juma commented on KAFKA-4669: By the way, the code where the NPE was happening

[jira] [Created] (KAFKA-4866) Kafka console consumer property is ignored

Frank Lyaruu created KAFKA-4866: --- Summary: Kafka console consumer property is ignored Key: KAFKA-4866 URL: https://issues.apache.org/jira/browse/KAFKA-4866 Project: Kafka Issue Type: Bug

[jira] [Commented] (KAFKA-4669) KafkaProducer.flush hangs when NetworkClient.handleCompletedReceives throws exception

[ https://issues.apache.org/jira/browse/KAFKA-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15901349#comment-15901349 ] Edoardo Comar commented on KAFKA-4669: -- thanks [~ijuma] yes we're planning to move to

[jira] [Commented] (KAFKA-4569) Transient failure in org.apache.kafka.clients.consumer.KafkaConsumerTest.testWakeupWithFetchDataAvailable

[ https://issues.apache.org/jira/browse/KAFKA-4569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15901556#comment-15901556 ] Jason Gustafson commented on KAFKA-4569: Sorry for the late response. If I underst

Re: [DISCUSS] KIP-124: Request rate quotas

Hi, Todd, Thanks for the feedback. I just want to clarify your second point. If the limit percentage is per thread and the thread counts are changed, the absolute processing limit for existing users haven't changed and there is no need to adjust them. On the other hand, if the limit percentage is

[jira] [Created] (KAFKA-4867) zookeeper-security-migration.sh does not clear ACLs from all nodes

Stevo Slavic created KAFKA-4867: --- Summary: zookeeper-security-migration.sh does not clear ACLs from all nodes Key: KAFKA-4867 URL: https://issues.apache.org/jira/browse/KAFKA-4867 Project: Kafka

[jira] [Assigned] (KAFKA-4863) Querying window store may return unwanted keys

[ https://issues.apache.org/jira/browse/KAFKA-4863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damian Guy reassigned KAFKA-4863: - Assignee: Damian Guy (was: Eno Thereska) > Querying window store may return unwanted keys >

[GitHub] kafka pull request #1562: KAFKA-3908; Set ReceiveBufferSize for socket used ...

Github user lindong28 closed the pull request at: https://github.com/apache/kafka/pull/1562 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is e

[jira] [Commented] (KAFKA-3908) Set SendBufferSize for socket used by Processor

[ https://issues.apache.org/jira/browse/KAFKA-3908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15901702#comment-15901702 ] ASF GitHub Bot commented on KAFKA-3908: --- Github user lindong28 closed the pull reque

[GitHub] kafka pull request #2656: MINOR: Use ConcurrentMap for ConsumerNetworkClient...

GitHub user hachikuji opened a pull request: https://github.com/apache/kafka/pull/2656 MINOR: Use ConcurrentMap for ConsumerNetworkClient UnsentRequests You can merge this pull request into a Git repository by running: $ git pull https://github.com/hachikuji/kafka minor-cleanu

[jira] [Created] (KAFKA-4868) Optimize RocksDb config for fast recovery/bulk load

Eno Thereska created KAFKA-4868: --- Summary: Optimize RocksDb config for fast recovery/bulk load Key: KAFKA-4868 URL: https://issues.apache.org/jira/browse/KAFKA-4868 Project: Kafka Issue Type: I

[GitHub] kafka pull request #2617: KAFKA-4722: Add application.id to StreamThread nam...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/2617 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[jira] [Updated] (KAFKA-4722) Add application.id to StreamThread name

[ https://issues.apache.org/jira/browse/KAFKA-4722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-4722: - Resolution: Fixed Fix Version/s: 0.11.0.0 Status: Resolved (was: Patch Available

[jira] [Commented] (KAFKA-4722) Add application.id to StreamThread name

[ https://issues.apache.org/jira/browse/KAFKA-4722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15901865#comment-15901865 ] ASF GitHub Bot commented on KAFKA-4722: --- Github user asfgit closed the pull request

[jira] [Created] (KAFKA-4869) 0.10.2.0 release notes incorrectly include KIP-115

Yeva Byzek created KAFKA-4869: - Summary: 0.10.2.0 release notes incorrectly include KIP-115 Key: KAFKA-4869 URL: https://issues.apache.org/jira/browse/KAFKA-4869 Project: Kafka Issue Type: Bug

[jira] [Updated] (KAFKA-4869) 0.10.2.0 release notes incorrectly include KIP-115

[ https://issues.apache.org/jira/browse/KAFKA-4869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yeva Byzek updated KAFKA-4869: -- Description: >From http://kafka.apache.org/documentation.html : bq. The offsets.topic.replication.facto

[jira] [Updated] (KAFKA-4869) 0.10.2.0 release notes incorrectly include KIP-115

[ https://issues.apache.org/jira/browse/KAFKA-4869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yeva Byzek updated KAFKA-4869: -- Description: >From http://kafka.apache.org/documentation.html : bq. The offsets.topic.replication.facto

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi, Dong, Thanks for the updated KIP. A few more comments below. 1.1 and 1.2: I am still not sure there is enough benefit of reusing ReplicaFetchThread to move data across disks. (a) A big part of ReplicaFetchThread is to deal with issuing and tracking fetch requests. So, it doesn't feel that we

[jira] [Commented] (KAFKA-4222) Transient failure in QueryableStateIntegrationTest.queryOnRebalance

[ https://issues.apache.org/jira/browse/KAFKA-4222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15902014#comment-15902014 ] Matthias J. Sax commented on KAFKA-4222: This did not occur anymore. Can we close

[jira] [Commented] (KAFKA-4222) Transient failure in QueryableStateIntegrationTest.queryOnRebalance

[ https://issues.apache.org/jira/browse/KAFKA-4222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15902017#comment-15902017 ] Eno Thereska commented on KAFKA-4222: - Sounds good. > Transient failure in QueryableS

[jira] [Commented] (KAFKA-4474) Poor kafka-streams throughput

[ https://issues.apache.org/jira/browse/KAFKA-4474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15902026#comment-15902026 ] Eno Thereska commented on KAFKA-4474: - [~agomez] I haven't forgotten about this and we

Re: [DISCUSS] KIP-124: Request rate quotas

Rajini - I understand what you’re saying, but the point I’m making is that I don’t believe we need to take it into account directly. The CPU utilization of the network threads is directly proportional to the number of bytes being sent. The more bytes, the more CPU that is required for SSL (or othe

[GitHub] kafka pull request #2657: KAFKA-4861; GroupMetadataManager record is rejecte...

GitHub user ijuma opened a pull request: https://github.com/apache/kafka/pull/2657 KAFKA-4861; GroupMetadataManager record is rejected if broker configured with LogAppendTime The record should be created with CreateTime (like in the producer). The conversion to LogAppendTime ha

[jira] [Commented] (KAFKA-4861) log.message.timestamp.type=LogAppendTime breaks Kafka based consumers

[ https://issues.apache.org/jira/browse/KAFKA-4861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15902079#comment-15902079 ] ASF GitHub Bot commented on KAFKA-4861: --- GitHub user ijuma opened a pull request:

[GitHub] kafka pull request #2647: MINOR: Add varint serde utilities for new message ...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/2647 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[jira] [Assigned] (KAFKA-4861) log.message.timestamp.type=LogAppendTime breaks Kafka based consumers

[ https://issues.apache.org/jira/browse/KAFKA-4861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma reassigned KAFKA-4861: -- Assignee: Ismael Juma (was: Jason Gustafson) > log.message.timestamp.type=LogAppendTime breaks

[jira] [Updated] (KAFKA-4861) log.message.timestamp.type=LogAppendTime breaks Kafka based consumers

[ https://issues.apache.org/jira/browse/KAFKA-4861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-4861: --- Fix Version/s: 0.11.0.0 > log.message.timestamp.type=LogAppendTime breaks Kafka based consumers >

[jira] [Updated] (KAFKA-4861) log.message.timestamp.type=LogAppendTime breaks Kafka based consumers

[ https://issues.apache.org/jira/browse/KAFKA-4861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-4861: --- Status: Patch Available (was: Open) > log.message.timestamp.type=LogAppendTime breaks Kafka based con

[GitHub] kafka pull request #2658: MINOR: Introduce NetworkClient.hasInFlightRequests

GitHub user ijuma opened a pull request: https://github.com/apache/kafka/pull/2658 MINOR: Introduce NetworkClient.hasInFlightRequests It’s a minor optimisation, but simple enough. You can merge this pull request into a Git repository by running: $ git pull https://github.com/

Re: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API

Hi, sorry for not replying earlier and thanks for all your feedback. After some more discussions I updated the KIP. The new proposal puts some other design considerations into account, that I want to highlight shortly. Those considerations, automatically resolve the concerns raised. First some an

[jira] [Resolved] (KAFKA-4222) Transient failure in QueryableStateIntegrationTest.queryOnRebalance

[ https://issues.apache.org/jira/browse/KAFKA-4222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-4222. Resolution: Resolved > Transient failure in QueryableStateIntegrationTest.queryOnRebalance >

[GitHub] kafka pull request #2659: KAFKA-4840 : BufferPool errors can cause buffer po...

GitHub user smccauliff opened a pull request: https://github.com/apache/kafka/pull/2659 KAFKA-4840 : BufferPool errors can cause buffer pool to go into a bad state You can merge this pull request into a Git repository by running: $ git pull https://github.com/smccauliff/kafka

[jira] [Commented] (KAFKA-4840) There are are still cases where producer buffer pool will not remove waiters.

[ https://issues.apache.org/jira/browse/KAFKA-4840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15902125#comment-15902125 ] ASF GitHub Bot commented on KAFKA-4840: --- GitHub user smccauliff opened a pull reques

[jira] [Updated] (KAFKA-4840) There are are still cases where producer buffer pool will not remove waiters.

[ https://issues.apache.org/jira/browse/KAFKA-4840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean McCauliff updated KAFKA-4840: -- Description: In BufferPool.allocate(int size, long maxTimeToBlockMs): If a Throwable other than

[jira] [Updated] (KAFKA-4840) There are are still cases where producer buffer pool will not remove waiters.

[ https://issues.apache.org/jira/browse/KAFKA-4840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean McCauliff updated KAFKA-4840: -- Status: Patch Available (was: Open) > There are are still cases where producer buffer pool will

Build failed in Jenkins: kafka-trunk-jdk7 #1999

See Changes: [wangguoz] KAFKA-4722: Add application.id to StreamThread name -- [...truncated 848.58 KB...] org.apache.kafka.streams.processor.internals.StateRestorerTest >

[GitHub] kafka pull request #2656: MINOR: Use ConcurrentMap for ConsumerNetworkClient...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/2656 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

Re: [DISCUSS] KIP 130: Expose states of active tasks to KafkaStreams public API

Hi Guozhang Thank you for your feedback. I've started to look more deeply into the code. As you mention, it would be more clever to use the current StreamMetadata API to expose these information. I think exposing metrics through JMX is great for building monitoring dashboards using some tools lik

Features ideas - need feedback: message deletion and consumer improvements

Hi, Two ideas that I would like to get feedback on before putting KIPs together. 1) Ability to have the kafka client consumer “skip” data that can’t be de-serialized. it would be a consumer config such as “ignore.deserialization.errors” (got better naming?) that defaults to false to it’s backw

Jenkins build is back to normal : kafka-trunk-jdk7 #2000

See

[GitHub] kafka pull request #2660: MINOR: Make ConfigDef safer by not using empty str...

GitHub user ewencp opened a pull request: https://github.com/apache/kafka/pull/2660 MINOR: Make ConfigDef safer by not using empty string for NO_DEFAULT_VALUE. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ewencp/kafka minor-ma

Features ideas - need feedback

Hi, Two ideas that I would like to get feedback on before putting KIPs together. 1) Ability to have the kafka client consumer “skip” data that can’t be de-serialized. it would be a consumer config such as “ignore.deserialization.errors” (got better naming?) that defaults to false to it’s backw

[GitHub] kafka pull request #2528: KAFKA-4745 -Optimize close to remove unnecessary f...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/2528 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[jira] [Commented] (KAFKA-4745) KafkaLZ4BlockOutputStream.java incorrectly finishes the last frame

[ https://issues.apache.org/jira/browse/KAFKA-4745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15902249#comment-15902249 ] ASF GitHub Bot commented on KAFKA-4745: --- Github user asfgit closed the pull request

Re: [DISCUSS] KIP-117: Add a public AdministrativeClient API for Kafka admin operations

On Mon, Mar 6, 2017, at 11:55, Ismael Juma wrote: > Thanks Colin. > > I am familiar with the protocol semantics, but we need to document the > API > for users who don't know the protocol. I still think it would be valuable > to have some examples of how the API would be used for common use cases.

[jira] [Assigned] (KAFKA-4866) Kafka console consumer property is ignored

[ https://issues.apache.org/jira/browse/KAFKA-4866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huxi reassigned KAFKA-4866: --- Assignee: huxi > Kafka console consumer property is ignored > -- > >

[GitHub] kafka pull request #2661: kafka-4866: Kafka console consumer property is ign...

GitHub user amethystic opened a pull request: https://github.com/apache/kafka/pull/2661 kafka-4866: Kafka console consumer property is ignored Added `print.value` config in ConsoleConsumer to match what the quickstart document says. You can merge this pull request into a Git reposi

Re: [jira] [Commented] (KAFKA-2729) Cached zkVersion not equal to that in zookeeper, broker not recovering.

Jun, I see that line elsewhere in the cluster. I don't see it happening on that particular broker that ran into the problem. On Mon, Mar 6, 2017 at 5:02 PM, Jun Rao (JIRA) wrote: > > [ https://issues.apache.org/jira/browse/KAFKA-2729?page= > com.atlassian.jira.plugin.system.issuetabpanels:

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hey Jun, Thanks for the detailed explanation. I will use the separate thread pool to move replica between log directories. I will let you know when the KIP has been updated to use a separate thread pool. Here is my response to your other questions: 1.3 My idea is that the ReplicaMoveThread that

Build failed in Jenkins: kafka-trunk-jdk8 #1338

See Changes: [ismael] MINOR: Use ConcurrentMap for ConsumerNetworkClient UnsentRequests [ismael] KAFKA-4745; Remove unnecessary flush in -- [...truncated 95.53 KB...] kaf

[jira] [Created] (KAFKA-4870) A question about broker down , the server is doing partition master election,the client producer may send msg fail . How the producer deal with the situation ??

zhaoziyan created KAFKA-4870: Summary: A question about broker down , the server is doing partition master election,the client producer may send msg fail . How the producer deal with the situation ?? Key: KAFKA-4870

[jira] [Resolved] (KAFKA-4864) Kafka Secure Migrator tool doesn't secure all the nodes

[ https://issues.apache.org/jira/browse/KAFKA-4864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-4864. Resolution: Fixed Fix Version/s: 0.10.2.1 0.11.0.0 Issue resolved by pull request

[GitHub] kafka pull request #2655: KAFKA-4864 added correct zookeeper nodes for secur...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/2655 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[jira] [Commented] (KAFKA-4864) Kafka Secure Migrator tool doesn't secure all the nodes

[ https://issues.apache.org/jira/browse/KAFKA-4864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15902532#comment-15902532 ] ASF GitHub Bot commented on KAFKA-4864: --- Github user asfgit closed the pull request

[jira] [Created] (KAFKA-4871) Kafka doesn't respect TTL on Zookeeper hostname - crash if zookeeper IP changes

Stephane Maarek created KAFKA-4871: -- Summary: Kafka doesn't respect TTL on Zookeeper hostname - crash if zookeeper IP changes Key: KAFKA-4871 URL: https://issues.apache.org/jira/browse/KAFKA-4871 Pro

[jira] [Created] (KAFKA-4872) Getting java.io.IOException in kafka 0.10.1.0 & 0.10.1.1

Khushboo Sangal created KAFKA-4872: -- Summary: Getting java.io.IOException in kafka 0.10.1.0 & 0.10.1.1 Key: KAFKA-4872 URL: https://issues.apache.org/jira/browse/KAFKA-4872 Project: Kafka Is

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi, Dong, Thanks for the reply. 1.3 So the thread gets the lock, checks if caught up and releases the lock if not? Then, in the case when there is continuous incoming data, the thread may never get a chance to swap. One way to address this is when the thread is getting really close in catching up

SASL security issue

Hi I've found a security issue in the kafka SASL implementation. It seems that ticket refreshments are not necessary to keep client-broker connection up. Test scenario: Client sucessfully connects to the broker using SASL_SSL security protocol. Kerberos server is provided by Windows Server 2012 an