[jira] [Updated] (KAFKA-2431) Test SSL/TLS impact on performance

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-2431:
---
Reviewer: Jun Rao

> Test SSL/TLS impact on performance
> --
>
> Key: KAFKA-2431
> URL: https://issues.apache.org/jira/browse/KAFKA-2431
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Ismael Juma
>Assignee: Ben Stopford
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> Test new Producer and new Consumer performance with and without SSL/TLS once 
> the SSL/TLS branch is integrated.
> The ideal scenario is that SSL/TLS would not have an impact if disabled. When 
> enabled, there will be some overhead (encryption and the inability to use 
> `SendFile`) and it will be good to quantify it. The encryption overhead is 
> reduced if recent JDKs are used with CPUs that support AES-specific 
> instructions (https://en.wikipedia.org/wiki/AES_instruction_set).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (KAFKA-2587) Transient test failure: `SimpleAclAuthorizerTest`

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma reopened KAFKA-2587:


As Jason said, this is still happening.

> Transient test failure: `SimpleAclAuthorizerTest`
> -
>
> Key: KAFKA-2587
> URL: https://issues.apache.org/jira/browse/KAFKA-2587
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Ismael Juma
>Assignee: Parth Brahmbhatt
> Fix For: 0.9.0.0
>
>
> I've seen `SimpleAclAuthorizerTest ` fail a couple of times since its recent 
> introduction. Here's one such build:
> https://builds.apache.org/job/kafka-trunk-git-pr/576/console
> [~parth.brahmbhatt], can you please take a look and see if it's an easy fix?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1326) New consumer checklist

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-1326:
---
Affects Version/s: (was: 0.10.0.0)
   0.8.2.1

> New consumer checklist
> --
>
> Key: KAFKA-1326
> URL: https://issues.apache.org/jira/browse/KAFKA-1326
> Project: Kafka
>  Issue Type: New Feature
>  Components: consumer
>Affects Versions: 0.8.2.1
>Reporter: Neha Narkhede
>Assignee: Neha Narkhede
>  Labels: feature
> Fix For: 0.9.0.0
>
>
> We will use this JIRA to track the list of issues to resolve to get a working 
> new consumer client. The consumer client can work in phases -
> 1. Add new consumer APIs and configs
> 2. Refactor Sender. We will need to use some common APIs from Sender.java 
> (https://issues.apache.org/jira/browse/KAFKA-1316)
> 3. Add metadata fetch and refresh functionality to the consumer (This will 
> require https://issues.apache.org/jira/browse/KAFKA-1316)
> 4. Add functionality to support subscribe(TopicPartition...partitions). This 
> will add SimpleConsumer functionality to the new consumer. This does not 
> include any group management related work.
> 5. Add ability to commit offsets to Kafka. This will include adding 
> functionality to the commit()/commitAsync()/committed() APIs. This still does 
> not include any group management related work.
> 6. Add functionality to the offsetsBeforeTime() API.
> 7. Add consumer co-ordinator election to the server. This will only add a new 
> module for the consumer co-ordinator, but not necessarily all the logic to do 
> group management. 
> At this point, we will have a fully functional standalone consumer and a 
> server side co-ordinator module. This will be a good time to start adding 
> group management functionality to the server and consumer.
> 8. Add failure detection capability to the consumer when group management is 
> used. This will not include any rebalancing logic, just the ability to detect 
> failures using session.timeout.ms.
> 9. Add rebalancing logic to the server and consumer. This will be a tricky 
> and potentially large change since it will involve implementing the group 
> management protocol.
> 10. Add system tests for the new consumer
> 11. Add metrics 
> 12. Convert mirror maker to use the new consumer.
> 13. Convert perf test to use the new consumer
> 14. Performance testing and analysis.
> 15. Review and fine tune log4j logging



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2604) Remove `completeAll` and improve timeout passed to `Selector.poll` from `NetworkClient.poll`

[Ismael Juma (JIRA)]

Ismael Juma created KAFKA-2604:
--

 Summary: Remove `completeAll` and improve timeout passed to 
`Selector.poll` from `NetworkClient.poll`
 Key: KAFKA-2604
 URL: https://issues.apache.org/jira/browse/KAFKA-2604
 Project: Kafka
  Issue Type: Sub-task
Reporter: Ismael Juma
Assignee: Ismael Juma
 Fix For: 0.9.0.0


Now that KAFKA-2120 has been merged, we can improve a few things:

1. Remove `NetworkClient.completeAll`
2. Call `Selector.poll` with a timeout that is the minimum of timeout, metadata 
timeout and request timeout
3. Disallow `Selector.poll(-1)` as it's not used and not something that we want 
generally (and one can always use `Selector.poll(Integer.MAX_VALUE)` if one 
really wants it



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work started] (KAFKA-2604) Remove `completeAll` and improve timeout passed to `Selector.poll` from `NetworkClient.poll`

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-2604 started by Ismael Juma.
--
> Remove `completeAll` and improve timeout passed to `Selector.poll` from 
> `NetworkClient.poll`
> 
>
> Key: KAFKA-2604
> URL: https://issues.apache.org/jira/browse/KAFKA-2604
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Ismael Juma
>Assignee: Ismael Juma
> Fix For: 0.9.0.0
>
>
> Now that KAFKA-2120 has been merged, we can improve a few things:
> 1. Remove `NetworkClient.completeAll`
> 2. Call `Selector.poll` with a timeout that is the minimum of timeout, 
> metadata timeout and request timeout
> 3. Disallow `Selector.poll(-1)` as it's not used and not something that we 
> want generally (and one can always use `Selector.poll(Integer.MAX_VALUE)` if 
> one really wants it



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2640; Remove `completeAll` and improve t...

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/272

KAFKA-2640; Remove `completeAll` and improve timeout passed to 
`Selector.poll` from `NetworkClient.poll`



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
kafka-2640-remove-complete-all-poll-timeout

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/272.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #272


commit 76b6d12c9c60954b58ed1cb0bc12dfbca1c6c0d8
Author: Ismael Juma 
Date:   2015-10-02T08:51:26Z

Remove unused `NetworkClient.completeAll` methods

They block indefinitely which is an anti-pattern.

commit 78bbfaafe83158c913a1828ac9a940dfb3d44383
Author: Ismael Juma 
Date:   2015-10-02T08:56:42Z

`NetworkClient.poll` should not call `Selector.poll` with a value higher 
than `requestTimeoutMs`

commit 1c2e6ed2bf324538dc5ada9893cb3d43cda97ee6
Author: Ismael Juma 
Date:   2015-10-02T08:57:19Z

Disallow `Selector.poll(-1)` as it's not something we do or want to do




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-2604) Remove `completeAll` and improve timeout passed to `Selector.poll` from `NetworkClient.poll`

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-2604:
---
Reviewer: Jason Gustafson
  Status: Patch Available  (was: In Progress)

Jason, can you please review this?

> Remove `completeAll` and improve timeout passed to `Selector.poll` from 
> `NetworkClient.poll`
> 
>
> Key: KAFKA-2604
> URL: https://issues.apache.org/jira/browse/KAFKA-2604
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Ismael Juma
>Assignee: Ismael Juma
> Fix For: 0.9.0.0
>
>
> Now that KAFKA-2120 has been merged, we can improve a few things:
> 1. Remove `NetworkClient.completeAll`
> 2. Call `Selector.poll` with a timeout that is the minimum of timeout, 
> metadata timeout and request timeout
> 3. Disallow `Selector.poll(-1)` as it's not used and not something that we 
> want generally (and one can always use `Selector.poll(Integer.MAX_VALUE)` if 
> one really wants it



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2604) Remove `completeAll` and improve timeout passed to `Selector.poll` from `NetworkClient.poll`

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14940960#comment-14940960
 ] 

Ismael Juma commented on KAFKA-2604:


PR link: https://github.com/apache/kafka/pull/272

> Remove `completeAll` and improve timeout passed to `Selector.poll` from 
> `NetworkClient.poll`
> 
>
> Key: KAFKA-2604
> URL: https://issues.apache.org/jira/browse/KAFKA-2604
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Ismael Juma
>Assignee: Ismael Juma
> Fix For: 0.9.0.0
>
>
> Now that KAFKA-2120 has been merged, we can improve a few things:
> 1. Remove `NetworkClient.completeAll`
> 2. Call `Selector.poll` with a timeout that is the minimum of timeout, 
> metadata timeout and request timeout
> 3. Disallow `Selector.poll(-1)` as it's not used and not something that we 
> want generally (and one can always use `Selector.poll(Integer.MAX_VALUE)` if 
> one really wants it



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-2517) Performance Regression post SSL implementation

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma reassigned KAFKA-2517:
--

Assignee: Ismael Juma  (was: Ben Stopford)

> Performance Regression post SSL implementation
> --
>
> Key: KAFKA-2517
> URL: https://issues.apache.org/jira/browse/KAFKA-2517
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ben Stopford
>Assignee: Ismael Juma
> Fix For: 0.9.0.0
>
>
> It would appear that we incurred a performance regression on submission of 
> the SSL work affecting the performance of the new Kafka Consumer. 
> Running with 1KB messages. Macbook 2.3 GHz Intel Core i7, 8GB, APPLE SSD 
> SM256E. Single server instance. All local. 
> kafka-consumer-perf-test.sh ... --messages 300  --new-consumer
> Pre-SSL changes (commit 503bd36647695e8cc91893ffb80346dd03eb0bc5)
> Steady state throughputs = 234.8 MB/s
> (2861.5913, 234.8261, 3000596, 246233.0543)
> Post-SSL changes (commit 13c432f7952de27e9bf8cb4adb33a91ae3a4b738) 
> Steady state throughput =  178.1 MB/s  
> (2861.5913, 178.1480, 3000596, 186801.7182)
> Implication is a 25% reduction in consumer throughput for these test 
> conditions. 
> This appears to be caused by the use of PlaintextTransportLayer rather than 
> SocketChannel in FileMessageSet.writeTo() meaning a zero copy transfer is not 
> invoked.
> Switching to the use of a SocketChannel directly in FileMessageSet.writeTo()  
> yields the following result:
> Steady state throughput =  281.8 MB/s
> (2861.5913, 281.8191, 3000596, 295508.7650)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2517) Performance Regression post SSL implementation

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14940991#comment-14940991
 ] 

Ismael Juma commented on KAFKA-2517:


Talked to Ben about this and he said it was OK for me to pick this up as he's 
busy at the moment.

> Performance Regression post SSL implementation
> --
>
> Key: KAFKA-2517
> URL: https://issues.apache.org/jira/browse/KAFKA-2517
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ben Stopford
>Assignee: Ismael Juma
> Fix For: 0.9.0.0
>
>
> It would appear that we incurred a performance regression on submission of 
> the SSL work affecting the performance of the new Kafka Consumer. 
> Running with 1KB messages. Macbook 2.3 GHz Intel Core i7, 8GB, APPLE SSD 
> SM256E. Single server instance. All local. 
> kafka-consumer-perf-test.sh ... --messages 300  --new-consumer
> Pre-SSL changes (commit 503bd36647695e8cc91893ffb80346dd03eb0bc5)
> Steady state throughputs = 234.8 MB/s
> (2861.5913, 234.8261, 3000596, 246233.0543)
> Post-SSL changes (commit 13c432f7952de27e9bf8cb4adb33a91ae3a4b738) 
> Steady state throughput =  178.1 MB/s  
> (2861.5913, 178.1480, 3000596, 186801.7182)
> Implication is a 25% reduction in consumer throughput for these test 
> conditions. 
> This appears to be caused by the use of PlaintextTransportLayer rather than 
> SocketChannel in FileMessageSet.writeTo() meaning a zero copy transfer is not 
> invoked.
> Switching to the use of a SocketChannel directly in FileMessageSet.writeTo()  
> yields the following result:
> Steady state throughput =  281.8 MB/s
> (2861.5913, 281.8191, 3000596, 295508.7650)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2517; Performance Regression post SSL im...

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/273

KAFKA-2517; Performance Regression post SSL implementation (zero copy)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
kafka-2517-ssl-zero-copy-regression

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/273.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #273


commit fb88692a9e4f62677b91069db017c12ba5b42edd
Author: Ismael Juma 
Date:   2015-10-02T10:10:12Z

Introduce `TransportLayer.transferFrom` and use it from `FileMessageSet`




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2517) Performance Regression post SSL implementation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14940993#comment-14940993
 ] 

ASF GitHub Bot commented on KAFKA-2517:
---

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/273

KAFKA-2517; Performance Regression post SSL implementation (zero copy)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
kafka-2517-ssl-zero-copy-regression

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/273.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #273


commit fb88692a9e4f62677b91069db017c12ba5b42edd
Author: Ismael Juma 
Date:   2015-10-02T10:10:12Z

Introduce `TransportLayer.transferFrom` and use it from `FileMessageSet`




> Performance Regression post SSL implementation
> --
>
> Key: KAFKA-2517
> URL: https://issues.apache.org/jira/browse/KAFKA-2517
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ben Stopford
>Assignee: Ismael Juma
> Fix For: 0.9.0.0
>
>
> It would appear that we incurred a performance regression on submission of 
> the SSL work affecting the performance of the new Kafka Consumer. 
> Running with 1KB messages. Macbook 2.3 GHz Intel Core i7, 8GB, APPLE SSD 
> SM256E. Single server instance. All local. 
> kafka-consumer-perf-test.sh ... --messages 300  --new-consumer
> Pre-SSL changes (commit 503bd36647695e8cc91893ffb80346dd03eb0bc5)
> Steady state throughputs = 234.8 MB/s
> (2861.5913, 234.8261, 3000596, 246233.0543)
> Post-SSL changes (commit 13c432f7952de27e9bf8cb4adb33a91ae3a4b738) 
> Steady state throughput =  178.1 MB/s  
> (2861.5913, 178.1480, 3000596, 186801.7182)
> Implication is a 25% reduction in consumer throughput for these test 
> conditions. 
> This appears to be caused by the use of PlaintextTransportLayer rather than 
> SocketChannel in FileMessageSet.writeTo() meaning a zero copy transfer is not 
> invoked.
> Switching to the use of a SocketChannel directly in FileMessageSet.writeTo()  
> yields the following result:
> Steady state throughput =  281.8 MB/s
> (2861.5913, 281.8191, 3000596, 295508.7650)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2517) Performance Regression post SSL implementation

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-2517:
---
Status: Patch Available  (was: Open)

[~junrao], can you please review?

> Performance Regression post SSL implementation
> --
>
> Key: KAFKA-2517
> URL: https://issues.apache.org/jira/browse/KAFKA-2517
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ben Stopford
>Assignee: Ismael Juma
> Fix For: 0.9.0.0
>
>
> It would appear that we incurred a performance regression on submission of 
> the SSL work affecting the performance of the new Kafka Consumer. 
> Running with 1KB messages. Macbook 2.3 GHz Intel Core i7, 8GB, APPLE SSD 
> SM256E. Single server instance. All local. 
> kafka-consumer-perf-test.sh ... --messages 300  --new-consumer
> Pre-SSL changes (commit 503bd36647695e8cc91893ffb80346dd03eb0bc5)
> Steady state throughputs = 234.8 MB/s
> (2861.5913, 234.8261, 3000596, 246233.0543)
> Post-SSL changes (commit 13c432f7952de27e9bf8cb4adb33a91ae3a4b738) 
> Steady state throughput =  178.1 MB/s  
> (2861.5913, 178.1480, 3000596, 186801.7182)
> Implication is a 25% reduction in consumer throughput for these test 
> conditions. 
> This appears to be caused by the use of PlaintextTransportLayer rather than 
> SocketChannel in FileMessageSet.writeTo() meaning a zero copy transfer is not 
> invoked.
> Switching to the use of a SocketChannel directly in FileMessageSet.writeTo()  
> yields the following result:
> Steady state throughput =  281.8 MB/s
> (2861.5913, 281.8191, 3000596, 295508.7650)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2566) Improve Jenkins set-up

[Flavio Junqueira (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14941044#comment-14941044
 ] 

Flavio Junqueira commented on KAFKA-2566:
-

I'm a member of the hudson-jobadmin group and I can make this change, but I'm 
not a committer of this project. I'm willing to make this change if this isn't 
an issue with the PMC.

> Improve Jenkins set-up
> --
>
> Key: KAFKA-2566
> URL: https://issues.apache.org/jira/browse/KAFKA-2566
> Project: Kafka
>  Issue Type: Task
>Reporter: Ismael Juma
>
> There are currently two Jenkins jobs:
> https://builds.apache.org/job/Kafka-trunk
> https://builds.apache.org/job/kafka-trunk-git-pr
> They both run with Java 7 and execute the following gradle command:
> ./gradlew -PscalaVersion=2.10.1 test
> There are a few issues with this:
> * We don't test Java 8 even though that's the only stable release of the JDK 
> that still receives security fixes
> * We are testing with Scala 2.10.1 even though we should be testing with 
> Scala 2.10.5
> * We are not testing with Scala 2.11.x
> * We are not doing clean builds
> I suggest the following:
> 1. Change the `kafka-trunk-git-pr` job to use the `./gradlew clean test` 
> command.
> 2. Change the `Kafka-trunk` job to use the `./gradlew clean jarAll docsJarAll 
> testAll` command.
> 3. Introduce a kafka-trunk-jdk8 job with the command `./gradlew clean jarAll 
> docsJarAll testAll`
> This is a compromise that doesn't slow down the PR job (which is executed 
> much more often) while still testing trunk in all of our supported JDK and 
> Scala versions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2605) Replace `catch: Throwable` clauses with `NonFatal` or `NonControl`

[Ismael Juma (JIRA)]

Ismael Juma created KAFKA-2605:
--

 Summary: Replace `catch: Throwable` clauses with `NonFatal` or 
`NonControl`
 Key: KAFKA-2605
 URL: https://issues.apache.org/jira/browse/KAFKA-2605
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2.2
Reporter: Ismael Juma


The Kafka codebase includes a number of instances where we do `catch t: 
Throwable` where we should really be doing `catch NonFatal(t)` or `catch 
NonControl(t)` where `NonFatal` is part of the standard library and 
`NonControl` is something like:

{code}
object NonControl {
   def apply(t: Throwable): Boolean = t match {
 case _: ControlThrowable => false
 case _ => true
   }
  def unapply(t: Throwable): Option[Throwable] = if (apply(t)) Some(t) else None
}
{code}

We can also use `NonControl` to replace cases like (it's more concise and has 
the same behaviour):

{code}
  case e: ControlThrowable => throw e
  case e: Throwable => ...
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2605) Replace `catch: Throwable` clauses with `NonFatal` or `NonControl`

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-2605:
---
Labels: newbie  (was: )

> Replace `catch: Throwable` clauses with `NonFatal` or `NonControl`
> --
>
> Key: KAFKA-2605
> URL: https://issues.apache.org/jira/browse/KAFKA-2605
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.2
>Reporter: Ismael Juma
>  Labels: newbie
>
> The Kafka codebase includes a number of instances where we do `catch t: 
> Throwable` where we should really be doing `catch NonFatal(t)` or `catch 
> NonControl(t)` where `NonFatal` is part of the standard library and 
> `NonControl` is something like:
> {code}
> object NonControl {
>def apply(t: Throwable): Boolean = t match {
>  case _: ControlThrowable => false
>  case _ => true
>}
>   def unapply(t: Throwable): Option[Throwable] = if (apply(t)) Some(t) else 
> None
> }
> {code}
> We can also use `NonControl` to replace cases like (it's more concise and has 
> the same behaviour):
> {code}
>   case e: ControlThrowable => throw e
>   case e: Throwable => ...
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2605) Replace `catch: Throwable` clauses with `NonFatal` or `NonControl`

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14941054#comment-14941054
 ] 

Ismael Juma commented on KAFKA-2605:


cc [~jjkoshy]

> Replace `catch: Throwable` clauses with `NonFatal` or `NonControl`
> --
>
> Key: KAFKA-2605
> URL: https://issues.apache.org/jira/browse/KAFKA-2605
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.2
>Reporter: Ismael Juma
>  Labels: newbie
>
> The Kafka codebase includes a number of instances where we do `catch t: 
> Throwable` where we should really be doing `catch NonFatal(t)` or `catch 
> NonControl(t)` where `NonFatal` is part of the standard library and 
> `NonControl` is something like:
> {code}
> object NonControl {
>def apply(t: Throwable): Boolean = t match {
>  case _: ControlThrowable => false
>  case _ => true
>}
>   def unapply(t: Throwable): Option[Throwable] = if (apply(t)) Some(t) else 
> None
> }
> {code}
> We can also use `NonControl` to replace cases like (it's more concise and has 
> the same behaviour):
> {code}
>   case e: ControlThrowable => throw e
>   case e: Throwable => ...
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2605) Replace `catch: Throwable` clauses with `NonFatal` or `NonControl`

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-2605:
---
Issue Type: Improvement  (was: Bug)

> Replace `catch: Throwable` clauses with `NonFatal` or `NonControl`
> --
>
> Key: KAFKA-2605
> URL: https://issues.apache.org/jira/browse/KAFKA-2605
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.2
>Reporter: Ismael Juma
>  Labels: newbie
>
> The Kafka codebase includes a number of instances where we do `catch t: 
> Throwable` where we should really be doing `catch NonFatal(t)` or `catch 
> NonControl(t)` where `NonFatal` is part of the standard library and 
> `NonControl` is something like:
> {code}
> object NonControl {
>def apply(t: Throwable): Boolean = t match {
>  case _: ControlThrowable => false
>  case _ => true
>}
>   def unapply(t: Throwable): Option[Throwable] = if (apply(t)) Some(t) else 
> None
> }
> {code}
> We can also use `NonControl` to replace cases like (it's more concise and has 
> the same behaviour):
> {code}
>   case e: ControlThrowable => throw e
>   case e: Throwable => ...
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2606) Remove kafka.utils.Time in favour of o.a.kafka.common.utils.Time

[Ismael Juma (JIRA)]

Ismael Juma created KAFKA-2606:
--

 Summary: Remove kafka.utils.Time in favour of 
o.a.kafka.common.utils.Time
 Key: KAFKA-2606
 URL: https://issues.apache.org/jira/browse/KAFKA-2606
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2.2
Reporter: Ismael Juma
Priority: Minor


They duplicate each other at the moment and some server classes actually need 
an instance of both types, which is annoying.

It's worth noting that `kafka.utils.MockTime` includes a `scheduler` that is 
used by some tests while `o.a.kafka.common.utils.Time` does not. We either need 
to add this functionality or change the tests not to need it anymore.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2606) Remove kafka.utils.Time in favour of o.a.kafka.common.utils.Time

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-2606:
---
Issue Type: Improvement  (was: Bug)

> Remove kafka.utils.Time in favour of o.a.kafka.common.utils.Time
> 
>
> Key: KAFKA-2606
> URL: https://issues.apache.org/jira/browse/KAFKA-2606
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.2
>Reporter: Ismael Juma
>Priority: Minor
>  Labels: newbie
>
> They duplicate each other at the moment and some server classes actually need 
> an instance of both types, which is annoying.
> It's worth noting that `kafka.utils.MockTime` includes a `scheduler` that is 
> used by some tests while `o.a.kafka.common.utils.Time` does not. We either 
> need to add this functionality or change the tests not to need it anymore.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2607) Review `Time` interface and its usage

[Ismael Juma (JIRA)]

Ismael Juma created KAFKA-2607:
--

 Summary: Review `Time` interface and its usage
 Key: KAFKA-2607
 URL: https://issues.apache.org/jira/browse/KAFKA-2607
 Project: Kafka
  Issue Type: Improvement
Affects Versions: 0.8.2.2
Reporter: Ismael Juma


Two of `Time` interface's methods are `milliseconds` and `nanoseconds` which 
are implemented in `SystemTime` as follows:

{code}
@Override
public long milliseconds() {
return System.currentTimeMillis();
}

@Override
public long nanoseconds() {
return System.nanoTime();
}
{code}


The issue with this interface is that it makes it seem that the difference is 
about the unit (`ms` versus `ns`) whereas it's much more than that:

https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks

We should probably change the names of the methods and review our usage to see 
if we're using the right one in the various places.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2607) Review `Time` interface and its usage

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14941066#comment-14941066
 ] 

Ismael Juma commented on KAFKA-2607:


We should probably do this after KAFKA-2606 to make the change simpler.

> Review `Time` interface and its usage
> -
>
> Key: KAFKA-2607
> URL: https://issues.apache.org/jira/browse/KAFKA-2607
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.2
>Reporter: Ismael Juma
>  Labels: newbie
>
> Two of `Time` interface's methods are `milliseconds` and `nanoseconds` which 
> are implemented in `SystemTime` as follows:
> {code}
> @Override
> public long milliseconds() {
> return System.currentTimeMillis();
> }
> @Override
> public long nanoseconds() {
> return System.nanoTime();
> }
> {code}
> The issue with this interface is that it makes it seem that the difference is 
> about the unit (`ms` versus `ns`) whereas it's much more than that:
> https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks
> We should probably change the names of the methods and review our usage to 
> see if we're using the right one in the various places.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2600) Make KStream interfaces compatible with Java 8 java.util.function

[Randall Hauch (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Randall Hauch updated KAFKA-2600:
-
Description: 
As suggested by [~rhauch], if we make the interface method names as the same to 
java.util.function.[Functions]:

https://docs.oracle.com/javase/8/docs/api/java/util/function/package-summary.html

Our goal is to simply align names and concepts with those in the Java 8 API for 
ease of learning and use.

  was:
As suggested by [~rhauch], if we make the interface method names as the same to 
java.util.function.[Functions]:

https://docs.oracle.com/javase/8/docs/api/java/util/function/package-summary.html

Then we can migrate to Java8 later by simply extending them and not requiring a 
re-compile on the user side.


> Make KStream interfaces compatible with Java 8 java.util.function
> -
>
> Key: KAFKA-2600
> URL: https://issues.apache.org/jira/browse/KAFKA-2600
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
>Assignee: Randall Hauch
> Fix For: 0.9.0.0
>
>
> As suggested by [~rhauch], if we make the interface method names as the same 
> to java.util.function.[Functions]:
> https://docs.oracle.com/javase/8/docs/api/java/util/function/package-summary.html
> Our goal is to simply align names and concepts with those in the Java 8 API 
> for ease of learning and use.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2600) Make KStream interfaces compatible with Java 8 java.util.function

[Randall Hauch (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14941220#comment-14941220
 ] 

Randall Hauch commented on KAFKA-2600:
--

My original suggestion was to make these changes so we can eventually _evolve_ 
from KStream's custom functional interfaces to Java8 standard functional 
interfaces. These changes would make it possible for our functional interfaces 
to extend Java 8's and remain binary compatible, but changing `KStream` methods 
to *use* the Java 8 functional interfaces is not binary compatible. Overloading 
is also not an option, since that wreaks havoc with lambdas and makes it 
ambiguous as to which method is being used. Thus, the goal of these changes is 
to simply align names and concepts with those in the Java 8 API for ease of 
learning and use. The attached PR still does this.

> Make KStream interfaces compatible with Java 8 java.util.function
> -
>
> Key: KAFKA-2600
> URL: https://issues.apache.org/jira/browse/KAFKA-2600
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
>Assignee: Randall Hauch
> Fix For: 0.9.0.0
>
>
> As suggested by [~rhauch], if we make the interface method names as the same 
> to java.util.function.[Functions]:
> https://docs.oracle.com/javase/8/docs/api/java/util/function/package-summary.html
> Our goal is to simply align names and concepts with those in the Java 8 API 
> for ease of learning and use.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Java KafkaClient

[Laubenthal, Horst (NEUSTA CONSULTING)]

Hi,

We are just trying to use one of your Kafka Clients: KafkaConsumer  
(org.apache.kafka.clients.consumer) and was wondering why no Messages are 
returned by the poll method described in your examples...

Then, I saw your implementation of the poll method and was a bit confused:

/**
 * Fetches data for the topics or partitions specified using one of the 
subscribe APIs. It is an error to not have subscribed to
 * any topics or partitions before polling for data.
 *  
 * The offset used for fetching the data is governed by whether or not 
{@link #seek(Map) seek(offsets)}
 * is used. If {@link #seek(Map) seek(offsets)} is used, it will use the 
specified offsets on startup and
 * on every rebalance, to consume data from that offset sequentially on 
every poll. If not, it will use the last checkpointed offset 
 * using {@link #commit(Map, boolean) commit(offsets, sync)} 
 * for the subscribed list of partitions.
 * @param timeout  The time, in milliseconds, spent waiting in poll if data 
is not available. If 0, waits indefinitely. Must not be negative
 * @return map of topic to records since the last fetch for the subscribed 
list of topics and partitions
 */
@Override
public Map> poll(long timeout) {
// TODO Auto-generated method stub
return null;
}

Didn't you want to implement this?

Thanks
Horst

[jira] [Updated] (KAFKA-2598) Add Test with authorizer for producer and consumer

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-2598:
---
Priority: Blocker  (was: Major)

> Add Test with authorizer for producer and consumer
> --
>
> Key: KAFKA-2598
> URL: https://issues.apache.org/jira/browse/KAFKA-2598
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security, unit tests
>Affects Versions: 0.8.2.2
>Reporter: Parth Brahmbhatt
>Assignee: Parth Brahmbhatt
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> Now that we have all the authorizer code merged into trunk we should add a 
> test that enables authorizer and tests that only authorized users can 
> produce/consume from topics or issue cluster actions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Java KafkaClient

[Ashish Singh]

Hello Horst,

I think you are using Kafka version <= 0.8.2. The new consumer,
org.apache.kafka.clients.consumer, is being worked on and will be available
in 0.9.0, however it will be marked unstable for 0.9.0. You can follow Apache
Kafka docs 
to get started with consuming from your Kafka cluster.
​

On Fri, Oct 2, 2015 at 1:15 AM, Laubenthal, Horst (NEUSTA CONSULTING) <
horst.laubent...@otto.de> wrote:

> Hi,
>
> We are just trying to use one of your Kafka Clients: KafkaConsumer
> (org.apache.kafka.clients.consumer) and was wondering why no Messages are
> returned by the poll method described in your examples...
>
> Then, I saw your implementation of the poll method and was a bit confused:
>
> /**
>  * Fetches data for the topics or partitions specified using one of
> the subscribe APIs. It is an error to not have subscribed to
>  * any topics or partitions before polling for data.
>  * 
>  * The offset used for fetching the data is governed by whether or not
> {@link #seek(Map) seek(offsets)}
>  * is used. If {@link #seek(Map) seek(offsets)} is used, it will use
> the specified offsets on startup and
>  * on every rebalance, to consume data from that offset sequentially
> on every poll. If not, it will use the last checkpointed offset
>  * using {@link #commit(Map, boolean) commit(offsets, sync)}
>  * for the subscribed list of partitions.
>  * @param timeout  The time, in milliseconds, spent waiting in poll if
> data is not available. If 0, waits indefinitely. Must not be negative
>  * @return map of topic to records since the last fetch for the
> subscribed list of topics and partitions
>  */
> @Override
> public Map> poll(long timeout) {
> // TODO Auto-generated method stub
> return null;
> }
>
> Didn't you want to implement this?
>
> Thanks
> Horst




-- 

Regards,
Ashish

[jira] [Commented] (KAFKA-2566) Improve Jenkins set-up

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14941390#comment-14941390
 ] 

Guozhang Wang commented on KAFKA-2566:
--

[~fpj] Sure go ahead.

> Improve Jenkins set-up
> --
>
> Key: KAFKA-2566
> URL: https://issues.apache.org/jira/browse/KAFKA-2566
> Project: Kafka
>  Issue Type: Task
>Reporter: Ismael Juma
>
> There are currently two Jenkins jobs:
> https://builds.apache.org/job/Kafka-trunk
> https://builds.apache.org/job/kafka-trunk-git-pr
> They both run with Java 7 and execute the following gradle command:
> ./gradlew -PscalaVersion=2.10.1 test
> There are a few issues with this:
> * We don't test Java 8 even though that's the only stable release of the JDK 
> that still receives security fixes
> * We are testing with Scala 2.10.1 even though we should be testing with 
> Scala 2.10.5
> * We are not testing with Scala 2.11.x
> * We are not doing clean builds
> I suggest the following:
> 1. Change the `kafka-trunk-git-pr` job to use the `./gradlew clean test` 
> command.
> 2. Change the `Kafka-trunk` job to use the `./gradlew clean jarAll docsJarAll 
> testAll` command.
> 3. Introduce a kafka-trunk-jdk8 job with the command `./gradlew clean jarAll 
> docsJarAll testAll`
> This is a compromise that doesn't slow down the PR job (which is executed 
> much more often) while still testing trunk in all of our supported JDK and 
> Scala versions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2587) Transient test failure: `SimpleAclAuthorizerTest`

[Parth Brahmbhatt (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14941474#comment-14941474
 ] 

Parth Brahmbhatt commented on KAFKA-2587:
-

I looked at the code to reason around why this can happen. The state reported 
is indeed one of the valid states during our test 
https://github.com/apache/kafka/blob/5764e54de147af81aac85acc00687c23e9646a5c/core/src/test/scala/unit/kafka/security/auth/SimpleAclAuthorizerTest.scala#L217

After that line we actually remove all acls for that resource, add one acl back 
to it and remove that one acl. All those steps pass verification. 
https://github.com/apache/kafka/blob/5764e54de147af81aac85acc00687c23e9646a5c/core/src/test/scala/unit/kafka/security/auth/SimpleAclAuthorizerTest.scala#L225
 and 
https://github.com/apache/kafka/blob/5764e54de147af81aac85acc00687c23e9646a5c/core/src/test/scala/unit/kafka/security/auth/SimpleAclAuthorizerTest.scala#L226

Given we are using the same instance of the authorizer the cache of that 
instance is immediately updated for both add and remove. 
https://github.com/apache/kafka/blob/5764e54de147af81aac85acc00687c23e9646a5c/core/src/main/scala/kafka/security/auth/SimpleAclAuthorizer.scala#L171
https://github.com/apache/kafka/blob/5764e54de147af81aac85acc00687c23e9646a5c/core/src/main/scala/kafka/security/auth/SimpleAclAuthorizer.scala#L189

The only other place that can update the cache is notification handler as part 
of handling acl-changed notification. 
https://github.com/apache/kafka/blob/5764e54de147af81aac85acc00687c23e9646a5c/core/src/main/scala/kafka/security/auth/SimpleAclAuthorizer.scala#L269

However in that case we read the data from zookeeper and then update the cache. 
If the notifications processing was delayed for some reason, it should still 
read the acls from zk and then update the cache. 
There are pathological cases that can lead to this failure , for example:
1) Notification handler starts, reads acls from zk and a thread switch happens 
before it can update the cache
2) All the other cache updates go through (remove resource, add the acl, remove 
the acl). 
3) Before verification finishes for the last "remove one acl" a thread switch 
happens and notification handler update the cache with stale acls that it read 
before. 

Even with this case there should be follow up notifications about adding an acl 
and removing an acl which should again cause the notification process to read 
state from zookeeper and update the cache to correct state. Plus this seems 
unlikely enough that it would not happen every other day.

I will continue to look into this. In the meantime if this is a continuous dev 
pain, we can remove the last 3 lines of test that removes the last acl and 
tries to verify that the zookeeper path is deleted. 

> Transient test failure: `SimpleAclAuthorizerTest`
> -
>
> Key: KAFKA-2587
> URL: https://issues.apache.org/jira/browse/KAFKA-2587
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Ismael Juma
>Assignee: Parth Brahmbhatt
> Fix For: 0.9.0.0
>
>
> I've seen `SimpleAclAuthorizerTest ` fail a couple of times since its recent 
> introduction. Here's one such build:
> https://builds.apache.org/job/kafka-trunk-git-pr/576/console
> [~parth.brahmbhatt], can you please take a look and see if it's an easy fix?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2587) Transient test failure: `SimpleAclAuthorizerTest`

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14941502#comment-14941502
 ] 

Ismael Juma commented on KAFKA-2587:


Hey Parth, thanks for looking into this. If it's hard to track this down, we 
could leave it open for now and see if it happens again and with what 
frequency. Based on your current understanding, is this a test problem or 
something that could also happen in production?

> Transient test failure: `SimpleAclAuthorizerTest`
> -
>
> Key: KAFKA-2587
> URL: https://issues.apache.org/jira/browse/KAFKA-2587
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Ismael Juma
>Assignee: Parth Brahmbhatt
> Fix For: 0.9.0.0
>
>
> I've seen `SimpleAclAuthorizerTest ` fail a couple of times since its recent 
> introduction. Here's one such build:
> https://builds.apache.org/job/kafka-trunk-git-pr/576/console
> [~parth.brahmbhatt], can you please take a look and see if it's an easy fix?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2587) Transient test failure: `SimpleAclAuthorizerTest`

[Parth Brahmbhatt (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14941533#comment-14941533
 ] 

Parth Brahmbhatt commented on KAFKA-2587:
-

Hey Ismael,

As mentioned above, in production given all notifications will be processed 
sequentially and as part of each notification we read the acls from zookeeper 
(source of truth) this issue should not happen in production as long as 
zookeeper guarantees read after write consistency. We may run into this issue 
in the case where we observe inconsistency due to zookeeper not supporting 
"Simultaneously Consistent Cross-Client Views". Please see Notes section in 
http://zookeeper.apache.org/doc/trunk/zookeeperProgrammers.html#ch_zkGuarantees.
 

I looked at the zkClient code to see if we can call sync before read to avoid 
any consistency concerns but zkClient does not support sync.

> Transient test failure: `SimpleAclAuthorizerTest`
> -
>
> Key: KAFKA-2587
> URL: https://issues.apache.org/jira/browse/KAFKA-2587
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Ismael Juma
>Assignee: Parth Brahmbhatt
> Fix For: 0.9.0.0
>
>
> I've seen `SimpleAclAuthorizerTest ` fail a couple of times since its recent 
> introduction. Here's one such build:
> https://builds.apache.org/job/kafka-trunk-git-pr/576/console
> [~parth.brahmbhatt], can you please take a look and see if it's an easy fix?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2566) Improve Jenkins set-up

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14941567#comment-14941567
 ] 

Guozhang Wang commented on KAFKA-2566:
--

[~junrao] [~ijuma] [~fpj] I browse through the configs and here is a complete 
proposal:

1. kafka-trunk-jdk7: "Poll SCM" every hour, GRADLE_2_1,  jdk-1.7u51, "./gradlew 
clean jarAll docsJarAll testAll"

2. kafka-trunk-jdk8: "Poll SCM" every hour, GRADLE_2_1,  jdk1.8.0_45, 
"./gradlew clean jarAll docsJarAll testAll"

3. kafka-trunk-git-pr-jdk8: "Pull requests", GRADLE_2_1,  jdk1.8.0_45, 
"./gradlew -PscalaVersion=2.11.7 clean test"

> Improve Jenkins set-up
> --
>
> Key: KAFKA-2566
> URL: https://issues.apache.org/jira/browse/KAFKA-2566
> Project: Kafka
>  Issue Type: Task
>Reporter: Ismael Juma
>
> There are currently two Jenkins jobs:
> https://builds.apache.org/job/Kafka-trunk
> https://builds.apache.org/job/kafka-trunk-git-pr
> They both run with Java 7 and execute the following gradle command:
> ./gradlew -PscalaVersion=2.10.1 test
> There are a few issues with this:
> * We don't test Java 8 even though that's the only stable release of the JDK 
> that still receives security fixes
> * We are testing with Scala 2.10.1 even though we should be testing with 
> Scala 2.10.5
> * We are not testing with Scala 2.11.x
> * We are not doing clean builds
> I suggest the following:
> 1. Change the `kafka-trunk-git-pr` job to use the `./gradlew clean test` 
> command.
> 2. Change the `Kafka-trunk` job to use the `./gradlew clean jarAll docsJarAll 
> testAll` command.
> 3. Introduce a kafka-trunk-jdk8 job with the command `./gradlew clean jarAll 
> docsJarAll testAll`
> This is a compromise that doesn't slow down the PR job (which is executed 
> much more often) while still testing trunk in all of our supported JDK and 
> Scala versions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2452) enable new consumer in mirror maker

[Samuel Zhou (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14941609#comment-14941609
 ] 

Samuel Zhou commented on KAFKA-2452:


[~ijuma] thank you for the information.!

Another issue I found is that when I was trying to get lags for the mirror 
maker by using 
bin/kafka-consumer-groups.sh --zookeeper zk.host/kafka_ssl --group 
sslMirrorMaker --describe

I got no information of consumer group sslMirrorMaker from zookeeper. I noticed 
that the patch only use zookeeper connector for old consumer. So how could I 
get the lags for MirrorMaker for new consumer? Thanks!

> enable new consumer in mirror maker
> ---
>
> Key: KAFKA-2452
> URL: https://issues.apache.org/jira/browse/KAFKA-2452
> Project: Kafka
>  Issue Type: Sub-task
>  Components: core
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Jiangjie Qin
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> We need to an an option to enable the new consumer in mirror maker.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Corrupted Message Issue

[Siddhartha Singh Sandhu]

Hi,

I am receiving this in the server logs.
[2015-10-02 17:58:31,726] ERROR [KafkaApi-0] Error processing
ProducerRequest with correlation id 103815 from client
prod-dcs-stats-producer-1034 on partition [prod-dcs-stats,0]
(kafka.server.KafkaApis)
kafka.message.InvalidMessageException: Message is corrupt (stored crc =
423648070, computed crc = 1402134007)
at kafka.message.Message.ensureValid(Message.scala:166)
at
kafka.log.Log$$anonfun$analyzeAndValidateMessageSet$1.apply(Log.scala:327)
at
kafka.log.Log$$anonfun$analyzeAndValidateMessageSet$1.apply(Log.scala:315)
at scala.collection.Iterator$class.foreach(Iterator.scala:772)
at kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:32)
at kafka.log.Log.analyzeAndValidateMessageSet(Log.scala:315)
at kafka.log.Log.append(Log.scala:228)
at kafka.cluster.Partition.appendMessagesToLeader(Partition.scala:354)
at
kafka.server.KafkaApis$$anonfun$appendToLocalLog$2.apply(KafkaApis.scala:285)
at
kafka.server.KafkaApis$$anonfun$appendToLocalLog$2.apply(KafkaApis.scala:275)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95)
at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95)
at scala.collection.Iterator$class.foreach(Iterator.scala:772)
at scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:157)
at
scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:190)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:45)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:95)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:233)
at scala.collection.mutable.HashMap.map(HashMap.scala:45)
at kafka.server.KafkaApis.appendToLocalLog(KafkaApis.scala:275)
at kafka.server.KafkaApis.handleProducerRequest(KafkaApis.scala:201)
at kafka.server.KafkaApis.handle(KafkaApis.scala:68)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:42)
at java.lang.Thread.run(Thread.java:744)

What are potential ways in which a message can be corrupted?

Thank you for your time.
-Khris

Re: KAFKA-2364 migrate docs from SVN to git

[Gwen Shapira]

Hi,

I created asf-git under https://git-wip-us.apache.org/repos/asf/kafka-site.
git and pushed our existing docs in there.
What do we need to do to get infra to show this in our website?

Next steps:
1) Minor fix to PR 171
2) Merge PR 171
3) Get Apache to show our git site
4) Update wiki with "contributing to docs" process

Gwen



On Tue, Sep 15, 2015 at 8:40 AM, Manikumar Reddy 
wrote:

> Hi Gwen,
>
> We need to create new branch named "asf-site"  in new git repository[1].
> This is requirement from Apache Infra
> for git based websites [2].  After creating new branch, we will the copy
> the existing to svn repo contents to
> new branch.
>
>
> 1. https://git-wip-us.apache.org/repos/asf/kafka-site.git
> 2. https://issues.apache.org/jira/browse/INFRA-10143
>
>
> Kumar
>
> On Tue, Sep 15, 2015 at 2:19 AM, Gwen Shapira  wrote:
>
> > Hi Manikumar,
> >
> > Sorry for huge delay!
> >
> > 1) This looks good, I'll get it in
> >
> > 2) I'm confused - do we need a new branch or a new repository? it looks
> > like you already got a new repository, so why do we need a branch as
> well?
> >
> >
> >
> > On Wed, Sep 2, 2015 at 8:11 AM, Manikumar Reddy 
> > wrote:
> >
> > > Jun/Gwen/Guozhang,
> > >Need your help to complete this.
> > >
> > >   (1) Copy latest docs to kafka repo:
> > > https://github.com/apache/kafka/pull/171
> > >
> > >   (2) svn site repo -> git site repo migration : need committer help to
> > > create a branch "asf-site".
> > >
> > >new git site repo :
> > > https://git-wip-us.apache.org/repos/asf/kafka-site.git
> > >
> > > Kumar
> > >
> > > On Wed, Aug 26, 2015 at 7:43 PM, Manikumar Reddy  >
> > > wrote:
> > >
> > > > Hi Guozhang,
> > > >
> > > >   Our plan is to follow Gwen's suggested approach and migrate the
> > > existing
> > > > svn site repo to new git repo.
> > > >
> > > >   (1) Gwen's suggestion will help to us maintain latest docs in Kafka
> > > repo
> > > > itself.  We periodically need to copy these latest docs to site
> repo. I
> > > > will submit patch for this.
> > > >
> > > >   (2)  svn repo -> git repo  migration will help us to integrate site
> > > repo
> > > > to git tooling/github. It will be easy to maintain the site repo and
> > > > changes.  So we have created new git repo for docs and need committer
> > > help
> > > > to create a branch "asf-site".
> > > >
> > > >new git repo:
> > https://git-wip-us.apache.org/repos/asf/kafka-site.git
> > > >
> > > >   Hope this clears the confusion.
> > > >
> > > > Kumar
> > > > I thought Gwen's suggestion was to us a separate folder in the same
> > repo
> > > > for docs instead of a separate branch, Gwen can correct me if I was
> > > wrong?
> > > >
> > > > Guozhang
> > > >
> > > > On Mon, Aug 24, 2015 at 10:31 AM, Manikumar Reddy <
> > ku...@nmsworks.co.in>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > >Infra team created git repo for kafka site docs.
> > > > >
> > > > >Gwen/Guozhang,
> > > > >Need your help to create a branch "asf-site" and copy the
> > > exiting
> > > > > svn contents to that branch.
> > > > >
> > > > > git repo:
> https://git-wip-us.apache.org/repos/asf/kafka-site.git
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/INFRA-10143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14709630#comment-14709630
> > > > >
> > > > > Kumar
> > > > >
> > > > > On Fri, Aug 21, 2015 at 6:16 PM, Ismael Juma 
> > > wrote:
> > > > >
> > > > > > My preference would be to do `2` because it reduces the number of
> > > tools
> > > > > we
> > > > > > need to know. If we want to clone the repo for the generated
> site,
> > we
> > > > can
> > > > > > use the same tools as we do for the code repo and we can watch
> for
> > > > > changes
> > > > > > on GitHub, etc.
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Fri, Aug 21, 2015 at 1:34 PM, Manikumar Reddy <
> > > ku...@nmsworks.co.in
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi All,
> > > > > > >
> > > > > > > Can we finalize the  approach? So that we can proceed further.
> > > > > > >
> > > > > > > 1. Gwen's suggestion + existing svn repo
> > > > > > > 2. Gwen's suggestion + new git repo for docs
> > > > > > >
> > > > > > > kumar
> > > > > > >
> > > > > > > On Thu, Aug 20, 2015 at 11:48 PM, Manikumar Reddy <
> > > > > ku...@nmsworks.co.in>
> > > > > > > wrote:
> > > > > > >
> > > > > > > >   Also can we migrate svn repo to git repo? This will help us
> > to
> > > > fix
> > > > > > > > occasional  doc changes/bug fixes through github PR.
> > > > > > > >
> > > > > > > > On Thu, Aug 20, 2015 at 4:04 AM, Guozhang Wang <
> > > wangg...@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Gwen: I remembered it wrong. We would not need another round
> > of
> > > > > > voting.
> > > > > > > >>
> > > > > > > >> On Wed, Aug 19, 2015 at 3:06 PM, Gwen Shapira <
> > > g...@confluent.io>
> > > > > > > wrote:
> > > > > > > >>
> > > > > > > >> > Looki

Re: KAFKA-2364 migrate docs from SVN to git

[Ashish]

On Fri, Oct 2, 2015 at 12:38 PM, Gwen Shapira  wrote:
> Hi,
>
> I created asf-git under https://git-wip-us.apache.org/repos/asf/kafka-site.
> git and pushed our existing docs in there.
> What do we need to do to get infra to show this in our website?
>
> Next steps:
> 1) Minor fix to PR 171
> 2) Merge PR 171
> 3) Get Apache to show our git site
> 4) Update wiki with "contributing to docs" process
>
> Gwen

Does this mean doc patches would be welcome only for a production
release? else you would need a diff branch for accepting docs for
trunk features/patches which don't get reflected on the website.

Does Infra supports git based sites?

[jira] [Updated] (KAFKA-2517) Performance Regression post SSL implementation

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-2517:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Issue resolved by pull request 273
[https://github.com/apache/kafka/pull/273]

> Performance Regression post SSL implementation
> --
>
> Key: KAFKA-2517
> URL: https://issues.apache.org/jira/browse/KAFKA-2517
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ben Stopford
>Assignee: Ismael Juma
> Fix For: 0.9.0.0
>
>
> It would appear that we incurred a performance regression on submission of 
> the SSL work affecting the performance of the new Kafka Consumer. 
> Running with 1KB messages. Macbook 2.3 GHz Intel Core i7, 8GB, APPLE SSD 
> SM256E. Single server instance. All local. 
> kafka-consumer-perf-test.sh ... --messages 300  --new-consumer
> Pre-SSL changes (commit 503bd36647695e8cc91893ffb80346dd03eb0bc5)
> Steady state throughputs = 234.8 MB/s
> (2861.5913, 234.8261, 3000596, 246233.0543)
> Post-SSL changes (commit 13c432f7952de27e9bf8cb4adb33a91ae3a4b738) 
> Steady state throughput =  178.1 MB/s  
> (2861.5913, 178.1480, 3000596, 186801.7182)
> Implication is a 25% reduction in consumer throughput for these test 
> conditions. 
> This appears to be caused by the use of PlaintextTransportLayer rather than 
> SocketChannel in FileMessageSet.writeTo() meaning a zero copy transfer is not 
> invoked.
> Switching to the use of a SocketChannel directly in FileMessageSet.writeTo()  
> yields the following result:
> Steady state throughput =  281.8 MB/s
> (2861.5913, 281.8191, 3000596, 295508.7650)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2517) Performance Regression post SSL implementation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14941820#comment-14941820
 ] 

ASF GitHub Bot commented on KAFKA-2517:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/273


> Performance Regression post SSL implementation
> --
>
> Key: KAFKA-2517
> URL: https://issues.apache.org/jira/browse/KAFKA-2517
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ben Stopford
>Assignee: Ismael Juma
> Fix For: 0.9.0.0
>
>
> It would appear that we incurred a performance regression on submission of 
> the SSL work affecting the performance of the new Kafka Consumer. 
> Running with 1KB messages. Macbook 2.3 GHz Intel Core i7, 8GB, APPLE SSD 
> SM256E. Single server instance. All local. 
> kafka-consumer-perf-test.sh ... --messages 300  --new-consumer
> Pre-SSL changes (commit 503bd36647695e8cc91893ffb80346dd03eb0bc5)
> Steady state throughputs = 234.8 MB/s
> (2861.5913, 234.8261, 3000596, 246233.0543)
> Post-SSL changes (commit 13c432f7952de27e9bf8cb4adb33a91ae3a4b738) 
> Steady state throughput =  178.1 MB/s  
> (2861.5913, 178.1480, 3000596, 186801.7182)
> Implication is a 25% reduction in consumer throughput for these test 
> conditions. 
> This appears to be caused by the use of PlaintextTransportLayer rather than 
> SocketChannel in FileMessageSet.writeTo() meaning a zero copy transfer is not 
> invoked.
> Switching to the use of a SocketChannel directly in FileMessageSet.writeTo()  
> yields the following result:
> Steady state throughput =  281.8 MB/s
> (2861.5913, 281.8191, 3000596, 295508.7650)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2517; Performance Regression post SSL im...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/273


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: MINOR: Fix Vagrant setup script for use on Fed...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/246


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: MINOR: remove no longer needed CommitType

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/258


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: Kafka-trunk #654

[Apache Jenkins Server]

See 

Changes:

[junrao] KAFKA-2517; Performance Regression post SSL implementation (zero copy)

[wangguoz] MINOR: Fix Vagrant setup script for use on Fedora

[wangguoz] MINOR: remove no longer needed CommitType

--
[...truncated 1728 lines...]
kafka.common.ConfigTest > testInvalidClientIds PASSED

kafka.admin.AddPartitionsTest > testReplicaPlacement PASSED

kafka.controller.ControllerFailoverTest > testMetadataUpdate PASSED

kafka.network.SocketServerTest > testMaxConnectionsPerIp PASSED

kafka.network.SocketServerTest > simpleRequest PASSED

kafka.network.SocketServerTest > testSessionPrincipal PASSED

kafka.network.SocketServerTest > testSocketsCloseOnShutdown PASSED

kafka.network.SocketServerTest > testMaxConnectionsPerIPOverrides PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[0] PASSED

kafka.admin.DeleteTopicTest > testPartitionReassignmentDuringDeleteTopic PASSED

kafka.api.ConsumerTest > testListTopics PASSED

kafka.network.SocketServerTest > testSSLSocketServer PASSED

kafka.integration.FetcherTest > testFetcher PASSED

kafka.zk.ZKEphemeralTest > testZkWatchedEphemeralRecursive PASSED

kafka.zk.ZKEphemeralTest > testOverlappingSessions PASSED

kafka.network.SocketServerTest > tooBigRequestIsRejected PASSED

kafka.zk.ZKEphemeralTest > testEphemeralNodeCleanup PASSED

kafka.zk.ZKEphemeralTest > testZkWatchedEphemeral PASSED

kafka.zk.ZKEphemeralTest > testSameSession PASSED

kafka.api.QuotasTest > testThrottledProducerConsumer PASSED

kafka.utils.ReplicationUtilsTest > testUpdateLeaderAndIsr PASSED

kafka.utils.ReplicationUtilsTest > testGetLeaderIsrAndEpochForPartition PASSED

kafka.admin.DeleteTopicTest > testDeleteNonExistingTopic PASSED

kafka.integration.UncleanLeaderElectionTest > testUncleanLeaderElectionEnabled 
PASSED

kafka.api.ConsumerTest > testExpandingTopicSubscriptions PASSED

kafka.log.LogTest > testCorruptLog PASSED

kafka.log.LogTest > testLogRecoversToCorrectOffset PASSED

kafka.log.LogTest > testReopenThenTruncate PASSED

kafka.log.LogTest > testParseTopicPartitionNameForMissingPartition PASSED

kafka.log.LogTest > testParseTopicPartitionNameForEmptyName PASSED

kafka.log.LogTest > testOpenDeletesObsoleteFiles PASSED

kafka.log.LogTest > testSizeBasedLogRoll PASSED

kafka.log.LogTest > testTimeBasedLogRollJitter PASSED

kafka.log.LogTest > testParseTopicPartitionName PASSED

kafka.admin.DeleteTopicTest > testRecreateTopicAfterDeletion PASSED

kafka.log.LogTest > testTruncateTo PASSED

kafka.log.LogTest > testCleanShutdownFile PASSED

kafka.api.ProducerFailureHandlingTest > testCannotSendToInternalTopic PASSED

kafka.admin.DeleteTopicTest > testAddPartitionDuringDeleteTopic PASSED

kafka.api.ConsumerBounceTest > testConsumptionWithBrokerFailures PASSED

kafka.utils.ByteBoundedBlockingQueueTest > testByteBoundedBlockingQueue PASSED

kafka.api.ConsumerTest > testPatternUnsubscription PASSED

kafka.api.ProducerFailureHandlingTest > testTooLargeRecordWithAckOne PASSED

kafka.admin.DeleteTopicTest > testDeleteTopicWithAllAliveReplicas PASSED

kafka.api.ConsumerTest > testGroupConsumption PASSED

kafka.api.ProducerFailureHandlingTest > testWrongBrokerList PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[1] PASSED

kafka.api.ProducerFailureHandlingTest > testNotEnoughReplicas PASSED

kafka.admin.DeleteTopicTest > testDeleteTopicDuringAddPartition PASSED

kafka.integration.UncleanLeaderElectionTest > 
testCleanLeaderElectionDisabledByTopicOverride PASSED

kafka.api.ConsumerTest > testPartitionsFor PASSED

kafka.api.ProducerFailureHandlingTest > testNonExistentTopic PASSED

kafka.api.ConsumerTest > testAutoCommitOnRebalance PASSED

kafka.api.ProducerFailureHandlingTest > testInvalidPartition PASSED

kafka.api.ProducerFailureHandlingTest > testSendAfterClosed PASSED

kafka.integration.RollingBounceTest > testRollingBounce PASSED

kafka.common.TopicTest > testInvalidTopicNames PASSED

kafka.common.TopicTest > testTopicHasCollision PASSED

kafka.common.TopicTest > testTopicHasCollisionChars PASSED

kafka.api.ConsumerTest > testSimpleConsumption PASSED

kafka.api.ProducerFailureHandlingTest > testTooLargeRecordWithAckZero PASSED

kafka.api.ProducerFailureHandlingTest > 
testNotEnoughReplicasAfterBrokerShutdown PASSED

kafka.utils.UtilsTest > testAbs PASSED

kafka.utils.UtilsTest > testReplaceSuffix PASSED

kafka.api.ConsumerTest > testPartitionPauseAndResume PASSED

kafka.utils.UtilsTest > testDoublyLinkedList PASSED

kafka.utils.UtilsTest > testCircularIterator PASSED

kafka.utils.UtilsTest > testReadBytes PASSED

kafka.utils.UtilsTest > testCsvList PASSED

kafka.utils.UtilsTest > testReadInt PASSED

kafka.utils.UtilsTest > testCsvMap PASSED

kafka.utils.UtilsTest > testInLock PASSED

kafka.utils.UtilsTest > testSwallow PASSED

kafka.integration.UncleanLeaderElectionTest > testUncleanLeaderElectionDisabled 
PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[2] PASSED

kaf

Re: KAFKA-2364 migrate docs from SVN to git

[Ismael Juma]

Thanks Gwen. Comments below.

On Fri, Oct 2, 2015 at 8:38 PM, Gwen Shapira  wrote:

> I created asf-git under https://git-wip-us.apache.org/repos/asf/kafka-site
> .
> git and pushed our existing docs in there.
> What do we need to do to get infra to show this in our website?
>

You need to reopen the ticket that was closed due to inactivity and tell
them that the branch was created:

https://issues.apache.org/jira/browse/INFRA-10143

Next steps:
> 1) Minor fix to PR 171
> 2) Merge PR 171
> 3) Get Apache to show our git site
> 4) Update wiki with "contributing to docs" process
>

The contributing to website instructions are on the website at the moment,
so we need to update that too:

http://kafka.apache.org/contributing.html

And we also need to update the release instructions.

Ismael

Re: KAFKA-2364 migrate docs from SVN to git

[Ismael Juma]

On Fri, Oct 2, 2015 at 9:36 PM, Ashish  wrote:
>
> Does this mean doc patches would be welcome only for a production
> release? else you would need a diff branch for accepting docs for
> trunk features/patches which don't get reflected on the website.
>

Yes, you will contribute changes to the relevant branch of the code
repository and this will be pushed to the website Git repo as needed.

Does Infra supports git based sites?
>

Yes.

Ismael

[jira] [Commented] (KAFKA-2566) Improve Jenkins set-up

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14941903#comment-14941903
 ] 

Ismael Juma commented on KAFKA-2566:


Guozhang, it looks good to me apart from the PR job. We should use the lowest 
version of Java and Scala to ensure people don't use features from the newer 
version (as per my original suggestion).

> Improve Jenkins set-up
> --
>
> Key: KAFKA-2566
> URL: https://issues.apache.org/jira/browse/KAFKA-2566
> Project: Kafka
>  Issue Type: Task
>Reporter: Ismael Juma
>
> There are currently two Jenkins jobs:
> https://builds.apache.org/job/Kafka-trunk
> https://builds.apache.org/job/kafka-trunk-git-pr
> They both run with Java 7 and execute the following gradle command:
> ./gradlew -PscalaVersion=2.10.1 test
> There are a few issues with this:
> * We don't test Java 8 even though that's the only stable release of the JDK 
> that still receives security fixes
> * We are testing with Scala 2.10.1 even though we should be testing with 
> Scala 2.10.5
> * We are not testing with Scala 2.11.x
> * We are not doing clean builds
> I suggest the following:
> 1. Change the `kafka-trunk-git-pr` job to use the `./gradlew clean test` 
> command.
> 2. Change the `Kafka-trunk` job to use the `./gradlew clean jarAll docsJarAll 
> testAll` command.
> 3. Introduce a kafka-trunk-jdk8 job with the command `./gradlew clean jarAll 
> docsJarAll testAll`
> This is a compromise that doesn't slow down the PR job (which is executed 
> much more often) while still testing trunk in all of our supported JDK and 
> Scala versions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2580) Kafka Broker keeps file handles open for all log files (even if its not written to/read from)

[Vinoth Chandar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14941910#comment-14941910
 ] 

Vinoth Chandar commented on KAFKA-2580:
---

A LRU file handle is something very commonly employed in databases, which works 
pretty well in practice. (considering that it involves random access).  So +1  
on that path. 

[~granthenke] would you have cycles for this? If no one is working on this 
currently, we (uber) can take a stab at this, later this quarter. 


> Kafka Broker keeps file handles open for all log files (even if its not 
> written to/read from)
> -
>
> Key: KAFKA-2580
> URL: https://issues.apache.org/jira/browse/KAFKA-2580
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2.1
>Reporter: Vinoth Chandar
>
> We noticed this in one of our clusters where we stage logs for a longer 
> amount of time. It appears that the Kafka broker keeps file handles open even 
> for non active (not written to or read from) files. (in fact, there are some 
> threads going back to 2013 
> http://grokbase.com/t/kafka/users/132p65qwcn/keeping-logs-forever) 
> Needless to say, this is a problem and forces us to either artificially bump 
> up ulimit (its already at 100K) or expand the cluster (even if we have 
> sufficient IO and everything). 
> Filing this ticket, since I could find anything similar. Very interested to 
> know if there are plans to address this (given how Samza's changelog topic is 
> meant to be a persistent large state use case).  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2566) Improve Jenkins set-up

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14941923#comment-14941923
 ] 

Guozhang Wang commented on KAFKA-2566:
--

Agree, will change accordingly.

BTW, current default scala version specified in build.gradle is 2.10.5, shall 
we upgrade to 2.11.7 now?

> Improve Jenkins set-up
> --
>
> Key: KAFKA-2566
> URL: https://issues.apache.org/jira/browse/KAFKA-2566
> Project: Kafka
>  Issue Type: Task
>Reporter: Ismael Juma
>
> There are currently two Jenkins jobs:
> https://builds.apache.org/job/Kafka-trunk
> https://builds.apache.org/job/kafka-trunk-git-pr
> They both run with Java 7 and execute the following gradle command:
> ./gradlew -PscalaVersion=2.10.1 test
> There are a few issues with this:
> * We don't test Java 8 even though that's the only stable release of the JDK 
> that still receives security fixes
> * We are testing with Scala 2.10.1 even though we should be testing with 
> Scala 2.10.5
> * We are not testing with Scala 2.11.x
> * We are not doing clean builds
> I suggest the following:
> 1. Change the `kafka-trunk-git-pr` job to use the `./gradlew clean test` 
> command.
> 2. Change the `Kafka-trunk` job to use the `./gradlew clean jarAll docsJarAll 
> testAll` command.
> 3. Introduce a kafka-trunk-jdk8 job with the command `./gradlew clean jarAll 
> docsJarAll testAll`
> This is a compromise that doesn't slow down the PR job (which is executed 
> much more often) while still testing trunk in all of our supported JDK and 
> Scala versions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2597: Add to .gitignore the Eclipse IDE ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/259


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2597) Add Eclipse directories to .gitignore

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14941927#comment-14941927
 ] 

ASF GitHub Bot commented on KAFKA-2597:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/259


> Add Eclipse directories to .gitignore
> -
>
> Key: KAFKA-2597
> URL: https://issues.apache.org/jira/browse/KAFKA-2597
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Randall Hauch
>Assignee: Randall Hauch
>Priority: Trivial
> Fix For: 0.9.0.0
>
>
> Add to {{.gitignore}} the Eclipse IDE directories {{.metadata}} and 
> {{.recommenders}}. These store state of the IDE's workspace, and should not 
> be checked in.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2597) Add Eclipse directories to .gitignore

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-2597:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Issue resolved by pull request 259
[https://github.com/apache/kafka/pull/259]

> Add Eclipse directories to .gitignore
> -
>
> Key: KAFKA-2597
> URL: https://issues.apache.org/jira/browse/KAFKA-2597
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Randall Hauch
>Assignee: Randall Hauch
>Priority: Trivial
> Fix For: 0.9.0.0
>
>
> Add to {{.gitignore}} the Eclipse IDE directories {{.metadata}} and 
> {{.recommenders}}. These store state of the IDE's workspace, and should not 
> be checked in.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2566) Improve Jenkins set-up

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14941928#comment-14941928
 ] 

Ismael Juma commented on KAFKA-2566:


It's the same reason as I just said. We build by default with 2.10.5 to ensure 
we don't use 2.11 features and APIs. I do think 2.11.7 should be the default 
distribution though.

> Improve Jenkins set-up
> --
>
> Key: KAFKA-2566
> URL: https://issues.apache.org/jira/browse/KAFKA-2566
> Project: Kafka
>  Issue Type: Task
>Reporter: Ismael Juma
>
> There are currently two Jenkins jobs:
> https://builds.apache.org/job/Kafka-trunk
> https://builds.apache.org/job/kafka-trunk-git-pr
> They both run with Java 7 and execute the following gradle command:
> ./gradlew -PscalaVersion=2.10.1 test
> There are a few issues with this:
> * We don't test Java 8 even though that's the only stable release of the JDK 
> that still receives security fixes
> * We are testing with Scala 2.10.1 even though we should be testing with 
> Scala 2.10.5
> * We are not testing with Scala 2.11.x
> * We are not doing clean builds
> I suggest the following:
> 1. Change the `kafka-trunk-git-pr` job to use the `./gradlew clean test` 
> command.
> 2. Change the `Kafka-trunk` job to use the `./gradlew clean jarAll docsJarAll 
> testAll` command.
> 3. Introduce a kafka-trunk-jdk8 job with the command `./gradlew clean jarAll 
> docsJarAll testAll`
> This is a compromise that doesn't slow down the PR job (which is executed 
> much more often) while still testing trunk in all of our supported JDK and 
> Scala versions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2566) Improve Jenkins set-up

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14941931#comment-14941931
 ] 

Guozhang Wang commented on KAFKA-2566:
--

Cool, is there an existing ticket for upgrading the default scala distribution 
version? If not let's create one.

> Improve Jenkins set-up
> --
>
> Key: KAFKA-2566
> URL: https://issues.apache.org/jira/browse/KAFKA-2566
> Project: Kafka
>  Issue Type: Task
>Reporter: Ismael Juma
>
> There are currently two Jenkins jobs:
> https://builds.apache.org/job/Kafka-trunk
> https://builds.apache.org/job/kafka-trunk-git-pr
> They both run with Java 7 and execute the following gradle command:
> ./gradlew -PscalaVersion=2.10.1 test
> There are a few issues with this:
> * We don't test Java 8 even though that's the only stable release of the JDK 
> that still receives security fixes
> * We are testing with Scala 2.10.1 even though we should be testing with 
> Scala 2.10.5
> * We are not testing with Scala 2.11.x
> * We are not doing clean builds
> I suggest the following:
> 1. Change the `kafka-trunk-git-pr` job to use the `./gradlew clean test` 
> command.
> 2. Change the `Kafka-trunk` job to use the `./gradlew clean jarAll docsJarAll 
> testAll` command.
> 3. Introduce a kafka-trunk-jdk8 job with the command `./gradlew clean jarAll 
> docsJarAll testAll`
> This is a compromise that doesn't slow down the PR job (which is executed 
> much more often) while still testing trunk in all of our supported JDK and 
> Scala versions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2566) Improve Jenkins set-up

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14941939#comment-14941939
 ] 

Ismael Juma commented on KAFKA-2566:


Not yet, I can do it tomorrow.

> Improve Jenkins set-up
> --
>
> Key: KAFKA-2566
> URL: https://issues.apache.org/jira/browse/KAFKA-2566
> Project: Kafka
>  Issue Type: Task
>Reporter: Ismael Juma
>
> There are currently two Jenkins jobs:
> https://builds.apache.org/job/Kafka-trunk
> https://builds.apache.org/job/kafka-trunk-git-pr
> They both run with Java 7 and execute the following gradle command:
> ./gradlew -PscalaVersion=2.10.1 test
> There are a few issues with this:
> * We don't test Java 8 even though that's the only stable release of the JDK 
> that still receives security fixes
> * We are testing with Scala 2.10.1 even though we should be testing with 
> Scala 2.10.5
> * We are not testing with Scala 2.11.x
> * We are not doing clean builds
> I suggest the following:
> 1. Change the `kafka-trunk-git-pr` job to use the `./gradlew clean test` 
> command.
> 2. Change the `Kafka-trunk` job to use the `./gradlew clean jarAll docsJarAll 
> testAll` command.
> 3. Introduce a kafka-trunk-jdk8 job with the command `./gradlew clean jarAll 
> docsJarAll testAll`
> This is a compromise that doesn't slow down the PR job (which is executed 
> much more often) while still testing trunk in all of our supported JDK and 
> Scala versions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: Kafka-trunk #655

[Apache Jenkins Server]

See 

Changes:

[wangguoz] KAFKA-2597: Add to .gitignore the Eclipse IDE directories

--
[...truncated 844 lines...]
kafka.coordinator.ConsumerCoordinatorResponseTest > 
testJoinGroupInconsistentPartitionAssignmentStrategy PASSED

kafka.coordinator.ConsumerCoordinatorResponseTest > 
testJoinGroupUnknownConsumerNewGroup PASSED

kafka.coordinator.ConsumerCoordinatorResponseTest > testValidJoinGroup PASSED

kafka.integration.SslTopicMetadataTest > testTopicMetadataRequest PASSED

kafka.coordinator.ConsumerCoordinatorResponseTest > 
testHeartbeatUnknownConsumerExistingGroup PASSED

kafka.coordinator.ConsumerCoordinatorResponseTest > testValidHeartbeat PASSED

kafka.admin.DeleteTopicTest > testResumeDeleteTopicOnControllerFailover PASSED

kafka.admin.DeleteConsumerGroupTest > 
testGroupTopicWideDeleteInZKForGroupConsumingMultipleTopics PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.MinIsrConfigTest > testDefaultKafkaConfig PASSED

kafka.admin.AdminTest > testPartitionReassignmentWithLeaderInNewReplicas PASSED

kafka.api.ConsumerTest > testUnsubscribeTopic PASSED

kafka.api.ConsumerBounceTest > testSeekAndCommitWithBrokerFailures PASSED

kafka.api.QuotasTest > testProducerConsumerOverrideUnthrottled PASSED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.common.ZkNodeChangeNotificationListenerTest > testProcessNotification 
PASSED

kafka.admin.AddPartitionsTest > testManualAssignmentOfReplicas PASSED

kafka.integration.PlaintextTopicMetadataTest > testTopicMetadataRequest PASSED

kafka.admin.DeleteConsumerGroupTest > testGroupWideDeleteInZK PASSED

kafka.integration.SslTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.admin.DeleteTopicTest > testResumeDeleteTopicWithRecoveredFollower PASSED

kafka.admin.AdminTest > testShutdownBroker PASSED

kafka.admin.AdminTest > testTopicCreationWithCollision PASSED

kafka.admin.AdminTest > testTopicCreationInZK PASSED

kafka.api.ConsumerTest > testAutoCommitOnClose PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[0] PASSED

kafka.api.ProducerSendTest > testCloseWithZeroTimeoutFromSenderThread PASSED

kafka.metrics.KafkaTimerTest > testKafkaTimer PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.common.ConfigTest > testInvalidGroupIds PASSED

kafka.common.ConfigTest > testInvalidClientIds PASSED

kafka.admin.AddPartitionsTest > testReplicaPlacement PASSED

kafka.controller.ControllerFailoverTest > testMetadataUpdate PASSED

kafka.network.SocketServerTest > testMaxConnectionsPerIp PASSED

kafka.network.SocketServerTest > simpleRequest PASSED

kafka.network.SocketServerTest > testSessionPrincipal PASSED

kafka.network.SocketServerTest > testSocketsCloseOnShutdown PASSED

kafka.admin.DeleteTopicTest > testDeleteTopicAlreadyMarkedAsDeleted PASSED

kafka.network.SocketServerTest > testMaxConnectionsPerIPOverrides PASSED

kafka.network.SocketServerTest > testSSLSocketServer PASSED

kafka.network.SocketServerTest > tooBigRequestIsRejected PASSED

kafka.integration.FetcherTest > testFetcher PASSED

kafka.zk.ZKEphemeralTest > testZkWatchedEphemeralRecursive PASSED

kafka.zk.ZKEphemeralTest > testOverlappingSessions PASSED

kafka.zk.ZKEphemeralTest > testEphemeralNodeCleanup PASSED

kafka.zk.ZKEphemeralTest > testZkWatchedEphemeral PASSED

kafka.zk.ZKEphemeralTest > testSameSession PASSED

kafka.api.QuotasTest > testThrottledProducerConsumer PASSED

kafka.integration.UncleanLeaderElectionTest > testUncleanLeaderElectionEnabled 
PASSED

kafka.api.ConsumerTest > testListTopics PASSED

kafka.utils.ReplicationUtilsTest > testUpdateLeaderAndIsr PASSED

kafka.utils.ReplicationUtilsTest > testGetLeaderIsrAndEpochForPartition PASSED

kafka.admin.DeleteTopicTest > testPartitionReassignmentDuringDeleteTopic PASSED

kafka.api.ConsumerTest > testExpandingTopicSubscriptions PASSED

kafka.admin.DeleteTopicTest > testDeleteNonExistingTopic PASSED

kafka.api.ConsumerTest > testPatternUnsubscription PASSED

kafka.admin.DeleteTopicTest > testRecreateTopicAfterDeletion PASSED

kafka.api.ConsumerBounceTest > testConsumptionWithBrokerFailures PASSED

kafka.utils.ByteBoundedBlockingQueueTest > testByteBoundedBlockingQueue PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[1] PASSED

kafka.integration.UncleanLeaderElectionTest > 
testCleanLeaderElectionDisabledByTopicOverride PASSED

kafka.api.ConsumerTest > testGroupConsumption PASSED

kafka.admin.DeleteTopicTest > testAddPartitionDuringDeleteTopic PASSED

kafka.api.ConsumerTest > testPartitionsFor PASSED

kafka.admin.DeleteTopicTest > testDeleteTopicWithAllAliveReplicas PASSED

kafka.api.ConsumerTest > testAutoCommitOnRebalance PASSED

kafka.admin.DeleteTopicTest > testDeleteTopicDuringAddPartition PASSED

kafka.api

[jira] [Commented] (KAFKA-2606) Remove kafka.utils.Time in favour of o.a.kafka.common.utils.Time

[Aditya Auradkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14941984#comment-14941984
 ] 

Aditya Auradkar commented on KAFKA-2606:


[~ijuma] This is a duplicate of https://issues.apache.org/jira/browse/KAFKA-2247
If you plan to work on this, I'll close the other one. Otherwise, I can close 
this.

> Remove kafka.utils.Time in favour of o.a.kafka.common.utils.Time
> 
>
> Key: KAFKA-2606
> URL: https://issues.apache.org/jira/browse/KAFKA-2606
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.2
>Reporter: Ismael Juma
>Priority: Minor
>  Labels: newbie
>
> They duplicate each other at the moment and some server classes actually need 
> an instance of both types, which is annoying.
> It's worth noting that `kafka.utils.MockTime` includes a `scheduler` that is 
> used by some tests while `o.a.kafka.common.utils.Time` does not. We either 
> need to add this functionality or change the tests not to need it anymore.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2606) Remove kafka.utils.Time in favour of o.a.kafka.common.utils.Time

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14941998#comment-14941998
 ] 

Ismael Juma commented on KAFKA-2606:


Sorry, my bad, I couldn't find it when I searched for it. We can close this but 
maybe mention the scheduler aspect on the other issue?

> Remove kafka.utils.Time in favour of o.a.kafka.common.utils.Time
> 
>
> Key: KAFKA-2606
> URL: https://issues.apache.org/jira/browse/KAFKA-2606
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.2
>Reporter: Ismael Juma
>Priority: Minor
>  Labels: newbie
>
> They duplicate each other at the moment and some server classes actually need 
> an instance of both types, which is annoying.
> It's worth noting that `kafka.utils.MockTime` includes a `scheduler` that is 
> used by some tests while `o.a.kafka.common.utils.Time` does not. We either 
> need to add this functionality or change the tests not to need it anymore.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2477) Replicas spuriously deleting all segments in partition

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14941999#comment-14941999
 ] 

Jun Rao commented on KAFKA-2477:


[~hakon], thanks a lot for the update. This seems like a real issue. Also, the 
point that you made on  "Log.read() can potentially read nextOffsetMetadata 
multiple times" is also relevant. In Log.read(), we have the following code:

// check if the offset is valid and in range
val next = nextOffsetMetadata.messageOffset
if(startOffset == next)
  return FetchDataInfo(nextOffsetMetadata, MessageSet.Empty)

This seems wrong. If nextOffsetMetadata changes after the if test, we could 
return a larger fetchOffsetMetadata in FetchDataInfo that we should. This will 
potentially affect the computation of things like isr. In this case, we should 
get a reference of nextOffsetMetadata first and use that to do the if test and 
as the return value.

Log.read() also references nextOffsetMetadata again in the last line. I am not 
sure if the comment is correct. The last message will never be deleted, so it 
seems that we can never reach the last statement.

// okay we are beyond the end of the last segment with no data fetched 
although the start offset is in range,
// this can happen when all messages with offset larger than start offsets 
have been deleted.
// In this case, we will return the empty set with log end offset metadata
FetchDataInfo(nextOffsetMetadata, MessageSet.Empty)

[~becket_qin], do you want to fix nextOffsetMetadata in your patch too?

> Replicas spuriously deleting all segments in partition
> --
>
> Key: KAFKA-2477
> URL: https://issues.apache.org/jira/browse/KAFKA-2477
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
>Reporter: Håkon Hitland
>Assignee: Jiangjie Qin
> Fix For: 0.9.0.0
>
> Attachments: kafka_log.txt, kafka_log_trace.txt
>
>
> We're seeing some strange behaviour in brokers: a replica will sometimes 
> schedule all segments in a partition for deletion, and then immediately start 
> replicating them back, triggering our check for under-replicating topics.
> This happens on average a couple of times a week, for different brokers and 
> topics.
> We have per-topic retention.ms and retention.bytes configuration, the topics 
> where we've seen this happen are hitting the size limit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2566) Improve Jenkins set-up

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14942034#comment-14942034
 ] 

Jun Rao commented on KAFKA-2566:


The trunk Jenkins job actually runs test with both scala 2.10 and 2.11. We can 
change it to run testAll though.

> Improve Jenkins set-up
> --
>
> Key: KAFKA-2566
> URL: https://issues.apache.org/jira/browse/KAFKA-2566
> Project: Kafka
>  Issue Type: Task
>Reporter: Ismael Juma
>
> There are currently two Jenkins jobs:
> https://builds.apache.org/job/Kafka-trunk
> https://builds.apache.org/job/kafka-trunk-git-pr
> They both run with Java 7 and execute the following gradle command:
> ./gradlew -PscalaVersion=2.10.1 test
> There are a few issues with this:
> * We don't test Java 8 even though that's the only stable release of the JDK 
> that still receives security fixes
> * We are testing with Scala 2.10.1 even though we should be testing with 
> Scala 2.10.5
> * We are not testing with Scala 2.11.x
> * We are not doing clean builds
> I suggest the following:
> 1. Change the `kafka-trunk-git-pr` job to use the `./gradlew clean test` 
> command.
> 2. Change the `Kafka-trunk` job to use the `./gradlew clean jarAll docsJarAll 
> testAll` command.
> 3. Introduce a kafka-trunk-jdk8 job with the command `./gradlew clean jarAll 
> docsJarAll testAll`
> This is a compromise that doesn't slow down the PR job (which is executed 
> much more often) while still testing trunk in all of our supported JDK and 
> Scala versions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2247) Merge kafka.utils.Time and kafka.common.utils.Time

[Aditya Auradkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aditya Auradkar updated KAFKA-2247:
---
Description: 
We currently have 2 different versions of Time in clients and core. These need 
to be merged.

It's worth noting that `kafka.utils.MockTime` includes a `scheduler` that is 
used by some tests while `o.a.kafka.common.utils.Time` does not. We either need 
to add this functionality or change the tests not to need it anymore.

  was:We currently have 2 different versions of Time in clients and core. These 
need to be merged


> Merge kafka.utils.Time and kafka.common.utils.Time
> --
>
> Key: KAFKA-2247
> URL: https://issues.apache.org/jira/browse/KAFKA-2247
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Aditya Auradkar
>Assignee: Aditya Auradkar
>Priority: Minor
>
> We currently have 2 different versions of Time in clients and core. These 
> need to be merged.
> It's worth noting that `kafka.utils.MockTime` includes a `scheduler` that is 
> used by some tests while `o.a.kafka.common.utils.Time` does not. We either 
> need to add this functionality or change the tests not to need it anymore.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-2606) Remove kafka.utils.Time in favour of o.a.kafka.common.utils.Time

[Aditya Auradkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aditya Auradkar resolved KAFKA-2606.

Resolution: Duplicate

Duplicate of: https://issues.apache.org/jira/browse/KAFKA-2247

Ismael - I copied the piece about the scheduler from this ticket onto 2247. 
Thanks!

> Remove kafka.utils.Time in favour of o.a.kafka.common.utils.Time
> 
>
> Key: KAFKA-2606
> URL: https://issues.apache.org/jira/browse/KAFKA-2606
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.2
>Reporter: Ismael Juma
>Priority: Minor
>  Labels: newbie
>
> They duplicate each other at the moment and some server classes actually need 
> an instance of both types, which is annoying.
> It's worth noting that `kafka.utils.MockTime` includes a `scheduler` that is 
> used by some tests while `o.a.kafka.common.utils.Time` does not. We either 
> need to add this functionality or change the tests not to need it anymore.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2566) Improve Jenkins set-up

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14942071#comment-14942071
 ] 

Guozhang Wang commented on KAFKA-2566:
--

[~junrao] Yeah, we plan to run testAll for trunk-jdk7/8, for trunk-git-pr I 
think it is sufficient to run with the default Scala version; and at the same 
time we may upgrade default version to 2.11.7 now.

> Improve Jenkins set-up
> --
>
> Key: KAFKA-2566
> URL: https://issues.apache.org/jira/browse/KAFKA-2566
> Project: Kafka
>  Issue Type: Task
>Reporter: Ismael Juma
>
> There are currently two Jenkins jobs:
> https://builds.apache.org/job/Kafka-trunk
> https://builds.apache.org/job/kafka-trunk-git-pr
> They both run with Java 7 and execute the following gradle command:
> ./gradlew -PscalaVersion=2.10.1 test
> There are a few issues with this:
> * We don't test Java 8 even though that's the only stable release of the JDK 
> that still receives security fixes
> * We are testing with Scala 2.10.1 even though we should be testing with 
> Scala 2.10.5
> * We are not testing with Scala 2.11.x
> * We are not doing clean builds
> I suggest the following:
> 1. Change the `kafka-trunk-git-pr` job to use the `./gradlew clean test` 
> command.
> 2. Change the `Kafka-trunk` job to use the `./gradlew clean jarAll docsJarAll 
> testAll` command.
> 3. Introduce a kafka-trunk-jdk8 job with the command `./gradlew clean jarAll 
> docsJarAll testAll`
> This is a compromise that doesn't slow down the PR job (which is executed 
> much more often) while still testing trunk in all of our supported JDK and 
> Scala versions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2591: Fix StreamingMetrics

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/265


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-2591) Remove Persistent Data before Restoringafter a Fault

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-2591:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Issue resolved by pull request 265
[https://github.com/apache/kafka/pull/265]

> Remove Persistent Data before Restoringafter a Fault
> 
>
> Key: KAFKA-2591
> URL: https://issues.apache.org/jira/browse/KAFKA-2591
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
> Fix For: 0.9.0.0
>
>
> When the checkpoint is missing upon starting up, it should be indicating that 
> the previous run is stopped uncleanly (or it is the first time running this 
> process job), hence any persisted data is not reliable any more and we need 
> to restore from the beginning.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2591) Remove Persistent Data before Restoringafter a Fault

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14942076#comment-14942076
 ] 

ASF GitHub Bot commented on KAFKA-2591:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/265


> Remove Persistent Data before Restoringafter a Fault
> 
>
> Key: KAFKA-2591
> URL: https://issues.apache.org/jira/browse/KAFKA-2591
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
> Fix For: 0.9.0.0
>
>
> When the checkpoint is missing upon starting up, it should be indicating that 
> the previous run is stopped uncleanly (or it is the first time running this 
> process job), hence any persisted data is not reliable any more and we need 
> to restore from the beginning.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: KAFKA-2364 migrate docs from SVN to git

[Manikumar Reddy]

Thanks Gwen,  i will
update the next steps.
On Oct 3, 2015 1:08 AM, "Gwen Shapira"  wrote:

> Hi,
>
> I created asf-git under https://git-wip-us.apache.org/repos/asf/kafka-site
> .
> git and pushed our existing docs in there.
> What do we need to do to get infra to show this in our website?
>
> Next steps:
> 1) Minor fix to PR 171
> 2) Merge PR 171
> 3) Get Apache to show our git site
> 4) Update wiki with "contributing to docs" process
>
> Gwen
>
>
>
> On Tue, Sep 15, 2015 at 8:40 AM, Manikumar Reddy 
> wrote:
>
> > Hi Gwen,
> >
> > We need to create new branch named "asf-site"  in new git repository[1].
> > This is requirement from Apache Infra
> > for git based websites [2].  After creating new branch, we will the copy
> > the existing to svn repo contents to
> > new branch.
> >
> >
> > 1. https://git-wip-us.apache.org/repos/asf/kafka-site.git
> > 2. https://issues.apache.org/jira/browse/INFRA-10143
> >
> >
> > Kumar
> >
> > On Tue, Sep 15, 2015 at 2:19 AM, Gwen Shapira  wrote:
> >
> > > Hi Manikumar,
> > >
> > > Sorry for huge delay!
> > >
> > > 1) This looks good, I'll get it in
> > >
> > > 2) I'm confused - do we need a new branch or a new repository? it looks
> > > like you already got a new repository, so why do we need a branch as
> > well?
> > >
> > >
> > >
> > > On Wed, Sep 2, 2015 at 8:11 AM, Manikumar Reddy 
> > > wrote:
> > >
> > > > Jun/Gwen/Guozhang,
> > > >Need your help to complete this.
> > > >
> > > >   (1) Copy latest docs to kafka repo:
> > > > https://github.com/apache/kafka/pull/171
> > > >
> > > >   (2) svn site repo -> git site repo migration : need committer help
> to
> > > > create a branch "asf-site".
> > > >
> > > >new git site repo :
> > > > https://git-wip-us.apache.org/repos/asf/kafka-site.git
> > > >
> > > > Kumar
> > > >
> > > > On Wed, Aug 26, 2015 at 7:43 PM, Manikumar Reddy <
> ku...@nmsworks.co.in
> > >
> > > > wrote:
> > > >
> > > > > Hi Guozhang,
> > > > >
> > > > >   Our plan is to follow Gwen's suggested approach and migrate the
> > > > existing
> > > > > svn site repo to new git repo.
> > > > >
> > > > >   (1) Gwen's suggestion will help to us maintain latest docs in
> Kafka
> > > > repo
> > > > > itself.  We periodically need to copy these latest docs to site
> > repo. I
> > > > > will submit patch for this.
> > > > >
> > > > >   (2)  svn repo -> git repo  migration will help us to integrate
> site
> > > > repo
> > > > > to git tooling/github. It will be easy to maintain the site repo
> and
> > > > > changes.  So we have created new git repo for docs and need
> committer
> > > > help
> > > > > to create a branch "asf-site".
> > > > >
> > > > >new git repo:
> > > https://git-wip-us.apache.org/repos/asf/kafka-site.git
> > > > >
> > > > >   Hope this clears the confusion.
> > > > >
> > > > > Kumar
> > > > > I thought Gwen's suggestion was to us a separate folder in the same
> > > repo
> > > > > for docs instead of a separate branch, Gwen can correct me if I was
> > > > wrong?
> > > > >
> > > > > Guozhang
> > > > >
> > > > > On Mon, Aug 24, 2015 at 10:31 AM, Manikumar Reddy <
> > > ku...@nmsworks.co.in>
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > >Infra team created git repo for kafka site docs.
> > > > > >
> > > > > >Gwen/Guozhang,
> > > > > >Need your help to create a branch "asf-site" and copy the
> > > > exiting
> > > > > > svn contents to that branch.
> > > > > >
> > > > > > git repo:
> > https://git-wip-us.apache.org/repos/asf/kafka-site.git
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/INFRA-10143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14709630#comment-14709630
> > > > > >
> > > > > > Kumar
> > > > > >
> > > > > > On Fri, Aug 21, 2015 at 6:16 PM, Ismael Juma 
> > > > wrote:
> > > > > >
> > > > > > > My preference would be to do `2` because it reduces the number
> of
> > > > tools
> > > > > > we
> > > > > > > need to know. If we want to clone the repo for the generated
> > site,
> > > we
> > > > > can
> > > > > > > use the same tools as we do for the code repo and we can watch
> > for
> > > > > > changes
> > > > > > > on GitHub, etc.
> > > > > > >
> > > > > > > Ismael
> > > > > > >
> > > > > > > On Fri, Aug 21, 2015 at 1:34 PM, Manikumar Reddy <
> > > > ku...@nmsworks.co.in
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi All,
> > > > > > > >
> > > > > > > > Can we finalize the  approach? So that we can proceed
> further.
> > > > > > > >
> > > > > > > > 1. Gwen's suggestion + existing svn repo
> > > > > > > > 2. Gwen's suggestion + new git repo for docs
> > > > > > > >
> > > > > > > > kumar
> > > > > > > >
> > > > > > > > On Thu, Aug 20, 2015 at 11:48 PM, Manikumar Reddy <
> > > > > > ku...@nmsworks.co.in>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > >   Also can we migrate svn repo to git repo? This will help
> us
> > > to
> > > > > fix
> > > > > > > > > occasional  doc ch

Build failed in Jenkins: Kafka-trunk #656

[Apache Jenkins Server]

See 

Changes:

[wangguoz] KAFKA-2591: Fix StreamingMetrics

--
[...truncated 4001 lines...]
kafka.coordinator.CoordinatorMetadataTest > 
testUnbindNonexistentGroupFromTopics PASSED

kafka.coordinator.CoordinatorMetadataTest > 
testAddGroupReturnsPreexistingGroupIfItAlreadyExists PASSED

kafka.coordinator.CoordinatorMetadataTest > 
testUnbindGroupFromTopicsListenedOnByNoOtherGroup PASSED

kafka.coordinator.CoordinatorMetadataTest > testRemoveNonexistentGroup PASSED

kafka.coordinator.CoordinatorMetadataTest > 
testUnbindGroupFromTopicsNotListenedOn PASSED

kafka.coordinator.CoordinatorMetadataTest > 
testUnbindGroupFromTopicsListenedOnByOtherGroups PASSED

kafka.coordinator.CoordinatorMetadataTest > 
testRemoveGroupWithNoOtherGroupsBoundToItsTopics PASSED

kafka.coordinator.CoordinatorMetadataTest > 
testRemoveGroupWithOtherGroupsBoundToItsTopics PASSED

kafka.coordinator.CoordinatorMetadataTest > testGetNonexistentGroup PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[0] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[1] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[2] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[3] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[4] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[5] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[6] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[7] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[8] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[9] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[10] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[11] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[12] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[13] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[14] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[15] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[16] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[17] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[18] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[19] PASSED

kafka.log.FileMessageSetTest > testTruncate PASSED

kafka.log.FileMessageSetTest > testIterationOverPartialAndTruncation PASSED

kafka.log.FileMessageSetTest > testRead PASSED

kafka.log.FileMessageSetTest > testFileSize PASSED

kafka.log.FileMessageSetTest > testIteratorWithLimits PASSED

kafka.log.FileMessageSetTest > testPreallocateTrue PASSED

kafka.log.FileMessageSetTest > testIteratorIsConsistent PASSED

kafka.log.FileMessageSetTest > testIterationDoesntChangePosition PASSED

kafka.log.FileMessageSetTest > testWrittenEqualsRead PASSED

kafka.log.FileMessageSetTest > testWriteTo PASSED

kafka.log.FileMessageSetTest > testPreallocateFalse PASSED

kafka.log.FileMessageSetTest > testPreallocateClearShutdown PASSED

kafka.log.FileMessageSetTest > testSearch PASSED

kafka.log.FileMessageSetTest > testSizeInBytes PASSED

kafka.log.CleanerTest > testBuildOffsetMap PASSED

kafka.log.CleanerTest > testSegmentGrouping PASSED

kafka.log.CleanerTest > testCleanSegmentsWithAbort PASSED

kafka.log.CleanerTest > testSegmentGroupingWithSparseOffsets PASSED

kafka.log.CleanerTest > testRecoveryAfterCrash PASSED

kafka.log.CleanerTest > testCleaningWithDeletes PASSED

kafka.log.CleanerTest > testCleanSegments PASSED

kafka.log.CleanerTest > testCleaningWithUnkeyedMessages PASSED
:core:copyDependantLibs
:core:jar
:examples:compileJava
:examples:processResources UP-TO-DATE
:examples:classes
:examples:compileTestJava UP-TO-DATE
:examples:processTestResources UP-TO-DATE
:examples:testClasses UP-TO-DATE
:examples:test UP-TO-DATE
:clients:javadoc
:clients:javadocJar
:clients:srcJar
:clients:testJar
:clients:signArchives SKIPPED
:log4j-appender:checkstyleMain
:log4j-appender:compileTestJava
:log4j-appender:processTestResources UP-TO-DATE
:log4j-appender:testClasses
:log4j-appender:checkstyleTest
:log4j-appender:test

org.apache.kafka.log4jappender.KafkaLog4jAppenderTest > testLog4jAppends PASSED

org.apache.kafka.log4jappender.KafkaLog4jAppenderTest > testKafkaLog4jConfigs 
PASSED
:streams:compileJava
Download 
https://repo1.maven.org/maven2/org/rocksdb/rocksdbjni/3.10.1/rocksdbjni-3.10.1.pom
Download 
https://repo1.maven.org/maven2/org/rocksdb/rocksdbjni/3.10.1/rocksdbjni-3.10.1.jar
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
:streams:processResources UP-TO-DATE
:streams:classes
:streams:checkstyleMain
:streams:compileTestJavaNote: Some input files use unchecked or unsafe 
operations.
Note: Recompile with -Xlint:unchecked for details.

:str

Should the KafkaProducer callback get the record batch?

[Gwen Shapira]

Hi,

Currently the callback of Producer gets either Metadata (partition +
offset) or Exception.
In case of an exception, it doesn't get any information other than the
exception. Which means that if I want to write failures to a "dead letter
queue" of some kind, I have to wait on the Future.

I'd like to add RecordBatch to onComplete, so in case of errors we can do
something with those records.

Questions:
1. Does it seem necessary to anyone else? What do all of you do when the
callback receives an exception?
2. Any thoughts on how to add a parameter without causing compatibility
issues? I think it will require a new interface (in addition to Callback),
but hope there's a better way.

Gwen