[jira] [Commented] (KAFKA-2072) Add StopReplica request/response to o.a.k.common.requests and replace the usage in core module

[ https://issues.apache.org/jira/browse/KAFKA-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716229#comment-14716229 ] David Jacot commented on KAFKA-2072: [~ijuma] Do you also need rest of this one for KA

[jira] [Commented] (KAFKA-2074) Add UpdateMetadata request/response to o.a.k.common.requests and replace its use in core module

[ https://issues.apache.org/jira/browse/KAFKA-2074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716259#comment-14716259 ] Ismael Juma commented on KAFKA-2074: Doing this as part of KAFKA-2411, so assigning it

[jira] [Work started] (KAFKA-2074) Add UpdateMetadata request/response to o.a.k.common.requests and replace its use in core module

[ https://issues.apache.org/jira/browse/KAFKA-2074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-2074 started by Ismael Juma. -- > Add UpdateMetadata request/response to o.a.k.common.requests and replace its > use

[jira] [Assigned] (KAFKA-2074) Add UpdateMetadata request/response to o.a.k.common.requests and replace its use in core module

[ https://issues.apache.org/jira/browse/KAFKA-2074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma reassigned KAFKA-2074: -- Assignee: Ismael Juma (was: Sriharsha Chintalapani) > Add UpdateMetadata request/response to o

[jira] [Work started] (KAFKA-2065) Add ControlledShutdown to org.apache.kafka.common.requests and replace current use in core module

[ https://issues.apache.org/jira/browse/KAFKA-2065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-2065 started by Ismael Juma. -- > Add ControlledShutdown to org.apache.kafka.common.requests and replace > current

[jira] [Work started] (KAFKA-2067) Add LeaderAndISR request/response to org.apache.kafka.common.requests and replace usage in core module

[ https://issues.apache.org/jira/browse/KAFKA-2067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-2067 started by Ismael Juma. -- > Add LeaderAndISR request/response to org.apache.kafka.common.requests and > replac

[jira] [Commented] (KAFKA-2072) Add StopReplica request/response to o.a.k.common.requests and replace the usage in core module

[ https://issues.apache.org/jira/browse/KAFKA-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716261#comment-14716261 ] Ismael Juma commented on KAFKA-2072: Migration of `KafkaAPIs` is not needed for KAFKA-

[jira] [Commented] (KAFKA-2478) KafkaConsumer javadoc example seems wrong

[ https://issues.apache.org/jira/browse/KAFKA-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716378#comment-14716378 ] Dmitry Stratiychuk commented on KAFKA-2478: --- Can certainly do for the javadoc ex

[jira] [Commented] (KAFKA-2478) KafkaConsumer javadoc example seems wrong

[ https://issues.apache.org/jira/browse/KAFKA-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716428#comment-14716428 ] Ismael Juma commented on KAFKA-2478: [~devstr], yes, you can submit a PR. Instructions

[GitHub] kafka pull request: KAFKA-2425; Copy latest docs to kafka repo doc...

GitHub user omkreddy opened a pull request: https://github.com/apache/kafka/pull/171 KAFKA-2425; Copy latest docs to kafka repo docs/ directory This PR copies the latest kafka docs to kafka repo docs directory. Here I have copied 0.8.3/ directory contents from svn website repo to ka

[jira] [Commented] (KAFKA-2425) Migrate website from SVN to Git

[ https://issues.apache.org/jira/browse/KAFKA-2425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716436#comment-14716436 ] ASF GitHub Bot commented on KAFKA-2425: --- GitHub user omkreddy opened a pull request:

[jira] [Commented] (KAFKA-2425) Migrate website from SVN to Git

[ https://issues.apache.org/jira/browse/KAFKA-2425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716460#comment-14716460 ] ASF GitHub Bot commented on KAFKA-2425: --- GitHub user omkreddy reopened a pull reques

[GitHub] kafka pull request: KAFKA-2425; Copy latest docs to kafka repo doc...

GitHub user omkreddy reopened a pull request: https://github.com/apache/kafka/pull/171 KAFKA-2425; Copy latest docs to kafka repo docs/ directory This PR copies the latest kafka docs to kafka repo docs directory. Here I have copied 0.8.3/ directory contents from svn website repo to

[GitHub] kafka pull request: KAFKA-2425; Copy latest docs to kafka repo doc...

Github user omkreddy closed the pull request at: https://github.com/apache/kafka/pull/171 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[jira] [Commented] (KAFKA-2425) Migrate website from SVN to Git

[ https://issues.apache.org/jira/browse/KAFKA-2425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716459#comment-14716459 ] ASF GitHub Bot commented on KAFKA-2425: --- Github user omkreddy closed the pull reques

[jira] [Created] (KAFKA-2479) Add CopycatExceptions to indicate transient and permanent errors in a connector/task

Ewen Cheslack-Postava created KAFKA-2479: Summary: Add CopycatExceptions to indicate transient and permanent errors in a connector/task Key: KAFKA-2479 URL: https://issues.apache.org/jira/browse/KAFKA-2479

[jira] [Created] (KAFKA-2481) Allow copycat sinks to request periodic invocation of put even if no new data is available

Ewen Cheslack-Postava created KAFKA-2481: Summary: Allow copycat sinks to request periodic invocation of put even if no new data is available Key: KAFKA-2481 URL: https://issues.apache.org/jira/browse/KAFK

[jira] [Created] (KAFKA-2480) Handle non-CopycatExceptions from

Ewen Cheslack-Postava created KAFKA-2480: Summary: Handle non-CopycatExceptions from Key: KAFKA-2480 URL: https://issues.apache.org/jira/browse/KAFKA-2480 Project: Kafka Issue Type: S

[jira] [Updated] (KAFKA-2480) Handle non-CopycatExceptions from SinkTasks

[ https://issues.apache.org/jira/browse/KAFKA-2480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-2480: - Summary: Handle non-CopycatExceptions from SinkTasks (was: Handle non-CopycatExce

[jira] [Created] (KAFKA-2482) Allow copycat sink tasks to pause/resume consumption of specific topic partitions

Ewen Cheslack-Postava created KAFKA-2482: Summary: Allow copycat sink tasks to pause/resume consumption of specific topic partitions Key: KAFKA-2482 URL: https://issues.apache.org/jira/browse/KAFKA-2482

[jira] [Created] (KAFKA-2483) Add support for batch/scheduled Copycat connectors

Ewen Cheslack-Postava created KAFKA-2483: Summary: Add support for batch/scheduled Copycat connectors Key: KAFKA-2483 URL: https://issues.apache.org/jira/browse/KAFKA-2483 Project: Kafka

[jira] [Created] (KAFKA-2484) Add schema projection utilities

Ewen Cheslack-Postava created KAFKA-2484: Summary: Add schema projection utilities Key: KAFKA-2484 URL: https://issues.apache.org/jira/browse/KAFKA-2484 Project: Kafka Issue Type: Sub

Kafka New Consumer Performance Test ThroughPut Degradation

Hi, We have built the latest kafka from https://github.com/apache/kafka based on this commit id 436b7ddc386eb688ba0f12836710f5e4bcaa06c8 . We ran the performance test on a 3 node kafka cluster. There is a huge throughput degradation using the new-consumer compared to the regular consumer. Below ar

[GitHub] kafka pull request: KAFKA-2367: Add Copycat runtime data API.

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/163 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabl

[jira] [Updated] (KAFKA-2367) Add Copycat runtime data API

[ https://issues.apache.org/jira/browse/KAFKA-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2367: Resolution: Fixed Status: Resolved (was: Patch Available) Issue resolved by pull request 16

[jira] [Commented] (KAFKA-2367) Add Copycat runtime data API

[ https://issues.apache.org/jira/browse/KAFKA-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14717315#comment-14717315 ] ASF GitHub Bot commented on KAFKA-2367: --- Github user asfgit closed the pull request

Jenkins build is back to normal : Kafka-trunk #604

See

[GitHub] kafka pull request: KAFKA-2377: Add basic system test for copycat ...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/150 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabl

[jira] [Commented] (KAFKA-2377) Add copycat system tests

[ https://issues.apache.org/jira/browse/KAFKA-2377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14717506#comment-14717506 ] ASF GitHub Bot commented on KAFKA-2377: --- Github user asfgit closed the pull request

[jira] [Updated] (KAFKA-2377) Add copycat system tests

[ https://issues.apache.org/jira/browse/KAFKA-2377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2377: Resolution: Fixed Status: Resolved (was: Patch Available) Issue resolved by pull request 15

Re: [DISCUSS] Client-side Assignment for New Consumer

I actually feel these set of tests (whatever they may be) are somewhat irrelevant here. My main concern with the current client-side proposal (i.e., without Becket's follow-up suggestions) is that it makes a significant compromise to the original charter of the new consumer - i.e., reduce/eliminate

[GitHub] kafka pull request: KAFKA-2475: Make Copycat only have a Converter...

GitHub user ewencp opened a pull request: https://github.com/apache/kafka/pull/172 KAFKA-2475: Make Copycat only have a Converter class instead of Serializer, Deserializer, and Converter. The Converter class now translates directly between byte[] and Copycat's data API instead

[jira] [Commented] (KAFKA-2475) Reduce copycat configs to only specify a converter or serializer, not both

[ https://issues.apache.org/jira/browse/KAFKA-2475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14717551#comment-14717551 ] ASF GitHub Bot commented on KAFKA-2475: --- GitHub user ewencp opened a pull request:

[jira] [Updated] (KAFKA-2475) Reduce copycat configs to only specify a converter or serializer, not both

[ https://issues.apache.org/jira/browse/KAFKA-2475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-2475: - Reviewer: Gwen Shapira Status: Patch Available (was: Open) > Reduce copycat

Re: [DISCUSS] Client-side Assignment for New Consumer

Hey Joel, I really don't think we should do both. There are pros and cons but we should make a decision and work on operationalizing one approach. Much of really making something like this work is getting all the bugs out, getting monitoring in place, getting rigorous system tests in place. Trying

Re: Kafka New Consumer Performance Test ThroughPut Degradation

I don't think the commands are really equivalent despite just adding the --new-consumer flag. ConsumerPerformance uses a single thread when using the new consumer (it literally just allocates the consumer, loops until it's consumed enough, then exits), whereas the old consumer uses a bunch of addit

Re: Kafka New Consumer Performance Test ThroughPut Degradation

I think this is likely a regression. The two clients had more or less equivalent performance when we checked in the code (see my post on this earlier in the year). Looks like maybe we broke something up in the interim? On my laptop the new consumer perf seems to have dropped from about ~200MB/sec

[GitHub] kafka pull request: MINOR: kafkatest add manifest

GitHub user granders opened a pull request: https://github.com/apache/kafka/pull/173 MINOR: kafkatest add manifest This patch makes it possible to publish kafkatest (system test package) to pypi and use it as a library in other projects by: - including necessary static reso

[jira] [Created] (KAFKA-2485) Allow producer performance to take properties from a file via --consumer.config command line parameter

Dong Lin created KAFKA-2485: --- Summary: Allow producer performance to take properties from a file via --consumer.config command line parameter Key: KAFKA-2485 URL: https://issues.apache.org/jira/browse/KAFKA-2485

[GitHub] kafka pull request: KAFKA-2485; Allow producer performance to take...

GitHub user lindong28 opened a pull request: https://github.com/apache/kafka/pull/174 KAFKA-2485; Allow producer performance to take properties from a file… … via --consumer.config command line parameter You can merge this pull request into a Git repository by running: $ gi

[jira] [Work started] (KAFKA-2485) Allow producer performance to take properties from a file via --consumer.config command line parameter

[ https://issues.apache.org/jira/browse/KAFKA-2485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-2485 started by Dong Lin. --- > Allow producer performance to take properties from a file via > --consumer.config comman

[jira] [Updated] (KAFKA-2485) Allow producer performance to take properties from a file via --consumer.config command line parameter

[ https://issues.apache.org/jira/browse/KAFKA-2485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated KAFKA-2485: Status: Patch Available (was: In Progress) > Allow producer performance to take properties from a file via

[jira] [Commented] (KAFKA-2485) Allow producer performance to take properties from a file via --consumer.config command line parameter

[ https://issues.apache.org/jira/browse/KAFKA-2485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14717815#comment-14717815 ] ASF GitHub Bot commented on KAFKA-2485: --- GitHub user lindong28 opened a pull request

[jira] [Commented] (KAFKA-2485) Allow producer performance to take properties from a file via --consumer.config command line parameter

[ https://issues.apache.org/jira/browse/KAFKA-2485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14717952#comment-14717952 ] Ewen Cheslack-Postava commented on KAFKA-2485: -- [~lindong] This looks good an

[jira] [Updated] (KAFKA-2485) Allow producer performance to take properties from a file via --producer.config command line parameter

[ https://issues.apache.org/jira/browse/KAFKA-2485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-2485: - Summary: Allow producer performance to take properties from a file via --producer.

[jira] [Commented] (KAFKA-2485) Allow producer performance to take properties from a file via --producer.config command line parameter

[ https://issues.apache.org/jira/browse/KAFKA-2485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14717963#comment-14717963 ] Dong Lin commented on KAFKA-2485: - [~ewencp] I didn't add this to the Java producer perfor

Re: Kafka New Consumer Performance Test ThroughPut Degradation

436b7ddc386eb688ba0f12836710f5e4bcaa06c8 is pretty recent and there could be some current consumer improvement patches that introduces some regression. I would suggest doing a binary search in the log from 3f8480ccfb011eb43da774737597c597f703e11b (maybe even earlier?) to do a quick check. Guozhang

[GitHub] kafka pull request: KAFKA-2468: SIGINT during Kafka server startup...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/167 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabl

[jira] [Resolved] (KAFKA-2468) SIGINT during Kafka server startup can leave server deadlocked

[ https://issues.apache.org/jira/browse/KAFKA-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-2468. -- Resolution: Fixed Fix Version/s: 0.8.3 Issue resolved by pull request 167 [https://github

[jira] [Commented] (KAFKA-2468) SIGINT during Kafka server startup can leave server deadlocked

[ https://issues.apache.org/jira/browse/KAFKA-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14718017#comment-14718017 ] ASF GitHub Bot commented on KAFKA-2468: --- Github user asfgit closed the pull request

[jira] [Updated] (KAFKA-2461) request logger no longer logs extra information in debug mode

[ https://issues.apache.org/jira/browse/KAFKA-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish K Singh updated KAFKA-2461: -- Status: Patch Available (was: Open) > request logger no longer logs extra information in debug

[jira] [Updated] (KAFKA-2307) Drop ConsumerOffsetChecker completely

[ https://issues.apache.org/jira/browse/KAFKA-2307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish K Singh updated KAFKA-2307: -- Fix Version/s: (was: 0.8.3) 0.9.0 > Drop ConsumerOffsetChecker completely

[jira] [Commented] (KAFKA-2471) Replicas Order and Leader out of sync

[ https://issues.apache.org/jira/browse/KAFKA-2471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14718087#comment-14718087 ] Ashish K Singh commented on KAFKA-2471: --- Could you attach server.log > Replicas Ord

[jira] [Commented] (KAFKA-1912) Create a simple request re-routing facility

[ https://issues.apache.org/jira/browse/KAFKA-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14718092#comment-14718092 ] Ashish K Singh commented on KAFKA-1912: --- [~jkreps], [~guozhang] are we still plannin

[jira] [Created] (KAFKA-2486) New consumer performance

Ewen Cheslack-Postava created KAFKA-2486: Summary: New consumer performance Key: KAFKA-2486 URL: https://issues.apache.org/jira/browse/KAFKA-2486 Project: Kafka Issue Type: Sub-task

Re: Kafka New Consumer Performance Test ThroughPut Degradation

Tried bisecting, but turns out things were broken for some time. We really need some system tests in place to avoid letting even new code break for so long. At 49026f11781181c38e9d5edb634be9d27245c961 (May 14th), we went from good performance -> an error due to broker apparently not accepting the