[DISCUSS] KIP-101: Alter Replication Protocol to use Leader Generation rather than High Watermark for Truncation

Hi All: We are using Kafka 0.8.2.2 in our sit enviornment, and meeting an data lose case when all brokers (2 brokers) going down and restart again. I am tring to understand the log management and recovery mechanism in kafka and i found a useful description document: KIP-101 - Alter Replicati

[jira] [Created] (KAFKA-5985) Mention the need to close store iterators

Stanislav Chizhov created KAFKA-5985: Summary: Mention the need to close store iterators Key: KAFKA-5985 URL: https://issues.apache.org/jira/browse/KAFKA-5985 Project: Kafka Issue Type: I

[jira] [Resolved] (KAFKA-5961) NullPointerException when consumer restore read messages with null key.

[ https://issues.apache.org/jira/browse/KAFKA-5961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andres Gomez Ferrer resolved KAFKA-5961. Resolution: Fixed > NullPointerException when consumer restore read messages with nu

[GitHub] kafka pull request #3973: KAFKA-5958: Global stores access state restore lis...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3973 ---

[GitHub] kafka pull request #3976: MINOR: Follow-up cleanup of KAFKA-5960

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3976 ---

[GitHub] kafka pull request #3939: KAFKA-5949: User Callback Exceptions need to be ha...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3939 ---

[GitHub] kafka pull request #3979: KAFKA-5979: Use single AtomicCounter to generate i...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3979 ---

[GitHub] kafka pull request #3961: KAFKA-5976: RequestChannel.sendResponse should rec...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3961 ---

[GitHub] kafka pull request #3981: HOTFIX: fix build compilation error

GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/3981 HOTFIX: fix build compilation error You can merge this pull request into a Git repository by running: $ git pull https://github.com/dguy/kafka fix-build Alternatively you can review and apply th

[jira] [Resolved] (KAFKA-5976) RequestChannel.sendReponse records incorrect size for NetworkSend with TRACE logging

[ https://issues.apache.org/jira/browse/KAFKA-5976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-5976. Resolution: Fixed Fix Version/s: 1.0.0 > RequestChannel.sendReponse records incorrect size fo

[GitHub] kafka pull request #3875: MINOR: Use full package name when classes referenc...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3875 ---

Re: [DISCUSS] KIP-204 : adding records deletion operation to the new Admin Client API

Hi, maybe we want to start without the delete records policy for now waiting for a real needs. So I'm removing it from the KIP. I hope for more comments on this KIP-204 so that we can start a vote on Monday. Thanks. Paolo Patierno Senior Software Engineer (IoT) @ Red Hat Microsoft MVP on Az

[jira] [Resolved] (KAFKA-3986) completedReceives can contain closed channels

[ https://issues.apache.org/jira/browse/KAFKA-3986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-3986. Resolution: Fixed Fix Version/s: (was: 1.1.0) 1.0.0 > completedReceive

[jira] [Created] (KAFKA-5986) Streams State Restoration never completes when logging is disabled

Damian Guy created KAFKA-5986: - Summary: Streams State Restoration never completes when logging is disabled Key: KAFKA-5986 URL: https://issues.apache.org/jira/browse/KAFKA-5986 Project: Kafka I

Build failed in Jenkins: kafka-trunk-jdk9 #67

See Changes: [damian.guy] KAFKA-5958; Global stores access state restore listener [ismael] KAFKA-5960; Follow-up cleanup [damian.guy] KAFKA-5949; User Callback Exceptions need to be handled properly [damian.guy]

[GitHub] kafka pull request #3982: TRIVIAL: fix connect Flatten error message

GitHub user sv3nd opened a pull request: https://github.com/apache/kafka/pull/3982 TRIVIAL: fix connect Flatten error message In case of an error while flattening a record with schema, the Flatten transform was reporting an error about a record without schema, as follows:

[GitHub] kafka pull request #3981: HOTFIX: fix build compilation error

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3981 ---

[GitHub] kafka pull request #3983: KAFKA-5986: Streams State Restoration never comple...

GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/3983 KAFKA-5986: Streams State Restoration never completes when logging is disabled When logging is disabled and there are state stores the task never transitions from restoring to running. This is because

Jenkins build is back to normal : kafka-trunk-jdk7 #2825

See

Jenkins build is back to normal : kafka-trunk-jdk9 #68

See

[GitHub] kafka pull request #3984: MINOR: update streams quickstart for KIP-182

GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/3984 MINOR: update streams quickstart for KIP-182 You can merge this pull request into a Git repository by running: $ git pull https://github.com/dguy/kafka quickstart-update Alternatively you can re

Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

I'm starting to think about KIP-179 again. In order to have more manageably-scoped KIPs and PRs I think it might be worth factoring-out the throttling part into a separate KIP. Wdyt? Keeping the throttling discussion in this thread for the moment... The throttling behaviour is currently spread ac

[GitHub] kafka pull request #3959: KAFKA-5901: Added Connect metrics specific to sour...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3959 ---

[jira] [Resolved] (KAFKA-5901) Create Connect metrics for source tasks

[ https://issues.apache.org/jira/browse/KAFKA-5901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava resolved KAFKA-5901. -- Resolution: Fixed Issue resolved by pull request 3959 [https://github.com/apache

Build failed in Jenkins: kafka-trunk-jdk8 #2081

See Changes: [me] KAFKA-5901: Added Connect metrics specific to source tasks (KIP-196) -- [...truncated 362.46 KB...] kafka.security.auth.SimpleAclAuthorizerTest > testAc

KIP-167 Updates

All, I have updated KIP-167 to include the bootstrapping status of any GlobalKTables defined in the application. The KIP has been updated: https://cwiki.apache.org/confluence/display/KAFKA/KIP-167%3A+Add+interface+for+the+state+store+restoration+process Thanks, Bill

KafkaSpout not consuming the first uncommitted offset data from kafka

Hi Kafka, I have a trident topology in storm which consumes data from kafka. Now i am seeing an issue in KafkaSpout. This is not consuming the very first tthe first uncommitted offset data from kafka. My storm version is 1.1.1 and kafka version is 0.11.0.0. I have a topic say X and partition of t

Re: [VOTE] KIP-175: Additional '--describe' views for ConsumerGroupCommand

I'm bumping this up as it's awaiting one more binding +1, but I'd like to also mention a recent change to the KIP. Since the current DescribeGroup response protocol does not include member-specific information such as preferred assignment strategies, or topic subscriptions, I've removed the cor

Re: KIP-167 Updates

Looks good. Please update the discussion thread link. On Thu, Sep 28, 2017 at 12:01 PM, Bill Bejeck wrote: > All, > > I have updated KIP-167 to include the bootstrapping status of any > GlobalKTables defined in the application. > > The KIP has been updated: > https://cwiki.apache.org/confluence

[jira] [Created] (KAFKA-5987) Kafka metrics templates used in document generation should maintain order of tags

Randall Hauch created KAFKA-5987: Summary: Kafka metrics templates used in document generation should maintain order of tags Key: KAFKA-5987 URL: https://issues.apache.org/jira/browse/KAFKA-5987 Proje

[GitHub] kafka pull request #3969: KAFKA-5888: System test to check ordering of messa...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3969 ---

[GitHub] kafka pull request #3985: KAFKA-5987: Maintain order of metric tags in gener...

GitHub user rhauch opened a pull request: https://github.com/apache/kafka/pull/3985 KAFKA-5987: Maintain order of metric tags in generated documentation The `MetricNameTemplate` is changed to used a `LinkedHashSet` to maintain the same order of the tags that are passed in. This tag

Build failed in Jenkins: kafka-trunk-jdk8 #2082

See Changes: [jason] KAFKA-5888; System test to check ordering of messages with transactions -- [...truncated 1.37 MB...] org.apache.kafka.common.security.authenticator.C

[GitHub] kafka pull request #3986: KAFKA-5949: Follow-up after latest KIP-161 changes

GitHub user mjsax opened a pull request: https://github.com/apache/kafka/pull/3986 KAFKA-5949: Follow-up after latest KIP-161 changes - compare KAFKA-5958 You can merge this pull request into a Git repository by running: $ git pull https://github.com/mjsax/kafka kafka-5949-ex

[jira] [Resolved] (KAFKA-5552) testTransactionalProducerTopicAuthorizationExceptionInCommit fails

[ https://issues.apache.org/jira/browse/KAFKA-5552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apurva Mehta resolved KAFKA-5552. - Resolution: Cannot Reproduce Fix Version/s: (was: 1.1.0) > testTransactionalProducerTop

[GitHub] kafka pull request #3705: KAFKA-5746: Add new metrics to support health chec...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3705 ---

[jira] [Created] (KAFKA-5988) Consider removing StreamThread#STREAM_THREAD_ID_SEQUENCE

Ted Yu created KAFKA-5988: - Summary: Consider removing StreamThread#STREAM_THREAD_ID_SEQUENCE Key: KAFKA-5988 URL: https://issues.apache.org/jira/browse/KAFKA-5988 Project: Kafka Issue Type: Improvem

Build failed in Jenkins: kafka-trunk-jdk8 #2083

See Changes: [ismael] KAFKA-5746; Add new metrics to support health checks (KIP-188) -- [...truncated 564.38 KB...] kafka.security.auth.SimpleAclAuthorizerTest > testDis

Build failed in Jenkins: kafka-trunk-jdk7 #2828

See Changes: [ismael] KAFKA-5746; Add new metrics to support health checks (KIP-188) -- [...truncated 385.58 KB...] kafka.log.BrokerCompressionTest > testBrokerSideCompre

[jira] [Created] (KAFKA-5989) disableLogging() causes an initialization loop

Tuan Nguyen created KAFKA-5989: -- Summary: disableLogging() causes an initialization loop Key: KAFKA-5989 URL: https://issues.apache.org/jira/browse/KAFKA-5989 Project: Kafka Issue Type: Bug

Build failed in Jenkins: kafka-trunk-jdk9 #71

See Changes: [ismael] KAFKA-5746; Add new metrics to support health checks (KIP-188) -- [...truncated 1.64 MB...] kafka.message.MessageTest > testFieldValues STARTED kafka.

[GitHub] kafka-site pull request #87: Update main bullet verbiage on the homepage

Github user asfgit closed the pull request at: https://github.com/apache/kafka-site/pull/87 ---

[jira] [Created] (KAFKA-5990) Add generated documentation for Connect metrics

Randall Hauch created KAFKA-5990: Summary: Add generated documentation for Connect metrics Key: KAFKA-5990 URL: https://issues.apache.org/jira/browse/KAFKA-5990 Project: Kafka Issue Type: Sub

[jira] [Created] (KAFKA-5991) Change Consumer per partition lag metrics to put topic-partition-id in tags instead of metric name

Kevin Lu created KAFKA-5991: --- Summary: Change Consumer per partition lag metrics to put topic-partition-id in tags instead of metric name Key: KAFKA-5991 URL: https://issues.apache.org/jira/browse/KAFKA-5991

[GitHub] kafka pull request #3987: KAFKA-5990: Enable generation of metrics docs for ...

GitHub user rhauch opened a pull request: https://github.com/apache/kafka/pull/3987 KAFKA-5990: Enable generation of metrics docs for Connect A new mechanism was added recently to the Metrics framework to make it easier to generate the documentation. It uses a registry with a Metri

[GitHub] kafka pull request #3988: KAFKA-5967 Ineffective check of negative value in ...

GitHub user shivsantham opened a pull request: https://github.com/apache/kafka/pull/3988 KAFKA-5967 Ineffective check of negative value in CompositeReadOnlyKe… package name: org.apache.kafka.streams.state.internals Minor change to approximateNumEntries() method in CompositeRea

[jira] [Resolved] (KAFKA-5989) disableLogging() causes an initialization loop

[ https://issues.apache.org/jira/browse/KAFKA-5989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-5989. Resolution: Duplicate \cc [~damianguy] It this is not a duplicate, please reopen. > disable

[GitHub] kafka pull request #3989: KAFKA-5746; Fix conversion count computed in `down...

GitHub user ijuma opened a pull request: https://github.com/apache/kafka/pull/3989 KAFKA-5746; Fix conversion count computed in `downConvert` It should be the number of records instead of the number of batches. A few additional clean-ups: minor renames, removal of un

[GitHub] kafka pull request #3942: KAFKA-5957: Prevent second deallocate if response ...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3942 ---

Build failed in Jenkins: kafka-trunk-jdk8 #2084

See Changes: [ismael] KAFKA-5957; Prevent second deallocate if response for aborted batch -- [...truncated 553.28 KB...] kafka.security.auth.SimpleAclAuthorizerTest > te

Build failed in Jenkins: kafka-trunk-jdk7 #2829

See Changes: [ismael] KAFKA-5957; Prevent second deallocate if response for aborted batch -- [...truncated 426.55 KB...] kafka.log.ProducerStateManagerTest > testNonTran

Build failed in Jenkins: kafka-trunk-jdk9 #72

See Changes: [ismael] KAFKA-5957; Prevent second deallocate if response for aborted batch -- [...truncated 1.47 MB...] kafka.javaapi.message.ByteBufferMessageSetTest > testE

[jira] [Created] (KAFKA-5992) Better Java Documentation for AdminClient Exceptions

Stephane Maarek created KAFKA-5992: -- Summary: Better Java Documentation for AdminClient Exceptions Key: KAFKA-5992 URL: https://issues.apache.org/jira/browse/KAFKA-5992 Project: Kafka Issue

[jira] [Created] (KAFKA-5993) Kafka AdminClient does not support standard security settings

Stephane Maarek created KAFKA-5993: -- Summary: Kafka AdminClient does not support standard security settings Key: KAFKA-5993 URL: https://issues.apache.org/jira/browse/KAFKA-5993 Project: Kafka

Re: [VOTE] KIP-175: Additional '--describe' views for ConsumerGroupCommand

+1 (non-binding) On Sep 28, 2017 12:13 PM, "Vahid S Hashemian" wrote: > I'm bumping this up as it's awaiting one more binding +1, but I'd like to > also mention a recent change to the KIP. > > Since the current DescribeGroup response protocol does not include > member-specific information such a

Issue using Kafka ZkUtils

Hello Team, I have the below issue with Kafka ZkUtils API. When I am trying to debug intelliJ points me to the ZkUtils class but when I tried to use it, the methods listed under ZkUtils object only pop up in the auto-suggestions. import kafka.utils.{TestUtils, ZKStringSerializer, ZkUtils}

Request to add to Contributors list

Hi There, I'm a beginner to the Apache Kafka project. Could you please add me to the contributors list. My Account credentials: Username: sssanthalingam Email: siva.santhalin...@gmail.com Full Name: siva santhalingam Thanks, Siva

Re: Request to add to Contributors list

Hi, Siva, Thanks for your interest. Added you to the contributor list. Jun On Thu, Sep 28, 2017 at 5:52 PM, Siva Santhalingam < siva.santhalin...@gmail.com> wrote: > Hi There, > > I'm a beginner to the Apache Kafka project. Could you please add me to the > contributors list. > > My Account cred

[GitHub] kafka pull request #3990: simple change to KafkaHealth

GitHub user prasincs opened a pull request: https://github.com/apache/kafka/pull/3990 simple change to KafkaHealth You can merge this pull request into a Git repository by running: $ git pull https://github.com/prasincs/kafka KAFKA-5473 Alternatively you can review and apply