date:20170514

[jira] [Updated] (KAFKA-5191) Autogenerate Consumer Fetcher metrics

2017-05-14 Thread James Cheng (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-5191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Cheng updated KAFKA-5191:
---
Attachment: docs_now_include_partition_level_metrics.png

The generated docs now include the partition-level lag metrics. I still think 
that the metrics are named incorrectly, and should be renamed, but I won't try 
to address that in this PR.

> Autogenerate Consumer Fetcher metrics
> -
>
> Key: KAFKA-5191
> URL: https://issues.apache.org/jira/browse/KAFKA-5191
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: James Cheng
>Assignee: James Cheng
> Attachments: docs_now_include_partition_level_metrics.png, 
> generated_fetcher_docs.png, generated_fetcher_docs_with_alternate_css.png, 
> generated_fetcher_docs_with_css.png
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-3480) Autogenerate metrics documentation

2017-05-14 Thread James Cheng (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009633#comment-16009633
 ] 

James Cheng commented on KAFKA-3480:


I tried rebasing this PR, and it was way too hard (9-months worth of changes). 
I'm going to close this PR, and open a series of newer, smaller ones. The first 
one is in KAFKA-5191

> Autogenerate metrics documentation
> --
>
> Key: KAFKA-3480
> URL: https://issues.apache.org/jira/browse/KAFKA-3480
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jason Gustafson
>Assignee: James Cheng
> Attachments: sample_metrics.html, Screen Shot 2016-04-07 at 6.52.19 
> PM.png
>
>
> Metrics documentation is done manually, which means it's hard to keep it 
> current. It would be nice to have automatic generation of this documentation 
> as we have for configs.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #3049: MINOR: removed code duplicates from several files ...

2017-05-14 Thread wlsc

GitHub user wlsc opened a pull request:

https://github.com/apache/kafka/pull/3049

MINOR: removed code duplicates from several files (KafkaStreams)

This PR offers following to KafkaStreams project:
* removed code duplication from **TopologyBuilder**'s function 
**makeNodeGroups** (extract function)
* removed code duplication from **GlobalProcessorContextImpl** and from 
**ProcessorContextImpl** to parent class **AbstractProcessorContext** (move 
method to parent)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/wlsc/kafka refactoring/remove-duplicates

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3049.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3049


commit fa90d68f7d547f54ae48a8254acac4759ddd4580
Author: Wladimir Schmidt 
Date:   2017-05-14T08:33:28Z

removed code duplicate from GlobalProcessorContextImpl and
from ProcessorContextImpl to parent AbstractProcessorContext

commit 6c70019a2d8f6b98b29366576f3b9f531ec0aea4
Author: Wladimir Schmidt 
Date:   2017-05-14T08:48:37Z

removed code duplicate from TopologyBuilder's function "makeNodeGroups"




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

RE: [DISCUSS] KIP-148: Add a connect timeout for client

2017-05-14 Thread ????????

Hi Guozhang,

Sorry for the delay, thanks for the question.  It seems two different 
parameters to me:
connect.timeout.ms: only work for the connecting phrase, after connected phrase 
this parameter is not used.
connections.max.idle.ms: currently not work in the connecting phrase (only 
select return readyKeys >0) will add to the expired manager, after connected 
will check if the connection is still alive in some time.

Even if we change the connections.max.idle.ms to work including the connecting 
phrase, we can not set this parameter to a small value, such as 5 seconds. 
Because the client is maybe busy sending message to other node, it will be 
disconnected in 5 seconds, so the default value of connections.max.idle.ms is 
setting to a larger time. We should have two parameters to control the 
connecting phrase behavior and the connected phrase behavior, do you think so?

Thanks,

David

--  --
??: "Guozhang Wang";;
: 2017??5??6??(??) 7:52
??: "dev@kafka.apache.org"; 

: Re: [DISCUSS] KIP-148: Add a connect timeout for client

Hello David,

Thanks for the KIP. For the described issue, I'm wondering if it can be
resolved by tuning the CONNECTIONS_MAX_IDLE_MS_CONFIG (
connections.max.idle.ms) on the client side? Default is 9 minutes.

Guozhang

On Tue, May 2, 2017 at 8:22 AM,  <254479...@qq.com> wrote:

> Hi all,
>
> Currently in our test environment, we found that after one of the broker
> node crash (reboot or os crash), the client may still be connecting to the
> crash node to send metadata request or other request, and it needs several
> minutes to be aware that the connection is timeout then try another node to
> connect to send the request. Then the client may still not be aware of the
> metadata change after several minutes.
>
>
> So I want to add a connect timeout on the  client,  please take a look at??
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 148%3A+Add+a+connect+timeout+for+client
>
> Regards,
>
> David

-- 
-- Guozhang

[jira] [Updated] (KAFKA-5232) Kafka broker fails to start if a topic containing dot in its name is marked for delete but hasn't been deleted during previous uptime

2017-05-14 Thread Ismael Juma (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-5232:
---
Fix Version/s: 0.10.2.2

> Kafka broker fails to start if a topic containing dot in its name is marked 
> for delete but hasn't been deleted during previous uptime
> -
>
> Key: KAFKA-5232
> URL: https://issues.apache.org/jira/browse/KAFKA-5232
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.2.0, 0.10.2.1
>Reporter: jaikiran pai
> Fix For: 0.11.0.0, 0.10.2.2
>
>
> We are using 0.10.2.0 (but this is reproducible even with 0.10.2.1 and latest 
> upstream) in our environments. Our topic names contain (one or more) dot 
> characters in their name. So we have topics like {{foo.bar-testtopic}}. Topic 
> deletion is enabled on the broker(s) and our application does delete the 
> topics as and when necessary.
> We just ran into a case today where for some reason the Kafka broker had 
> either to be taken down (or went down on its own). Some of the topics which 
> were deleted (i.e. a deletion marker folder was created for them) were left 
> around after Kafka broker had gone down. So the Kafka logs dir had 
> directories like 
> {{foo.bar-testtopic-0.bb7981c216b845648edfe6e2b0a5c050-delete}}.
> When we restarted the Kafka broker, it refused to start and kept shutting 
> down and running into this exception:
> {code}
> [2017-05-12 21:36:27,876] ERROR There was an error in one of the threads 
> during logs loading: java.lang.StringIndexOutOfBoundsException: String index 
> out of range: -1 (kafka.log.LogManager)
> [2017-05-12 21:36:27,900] FATAL [Kafka Server 0], Fatal error during 
> KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
> java.lang.StringIndexOutOfBoundsException: String index out of range: -1
>   at java.lang.String.substring(String.java:1967)
>   at kafka.log.Log$.parseTopicPartitionName(Log.scala:1146)
>   at kafka.log.LogManager.$anonfun$loadLogs$10(LogManager.scala:153)
>   at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:57)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> [2017-05-12 21:36:27,950] INFO [Kafka Server 0], shutting down 
> (kafka.server.KafkaServer)
> {code}
> The only way we could get past this is pointing Kafka broker to a different 
> Kafka logs directory which effectively meant a lot our topics were no longer 
> accessible to the application.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-5232) Kafka broker fails to start if a topic containing dot in its name is marked for delete but hasn't been deleted during previous uptime

2017-05-14 Thread Ismael Juma (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-5232:
---
Fix Version/s: 0.11.0.0

> Kafka broker fails to start if a topic containing dot in its name is marked 
> for delete but hasn't been deleted during previous uptime
> -
>
> Key: KAFKA-5232
> URL: https://issues.apache.org/jira/browse/KAFKA-5232
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.2.0, 0.10.2.1
>Reporter: jaikiran pai
> Fix For: 0.11.0.0, 0.10.2.2
>
>
> We are using 0.10.2.0 (but this is reproducible even with 0.10.2.1 and latest 
> upstream) in our environments. Our topic names contain (one or more) dot 
> characters in their name. So we have topics like {{foo.bar-testtopic}}. Topic 
> deletion is enabled on the broker(s) and our application does delete the 
> topics as and when necessary.
> We just ran into a case today where for some reason the Kafka broker had 
> either to be taken down (or went down on its own). Some of the topics which 
> were deleted (i.e. a deletion marker folder was created for them) were left 
> around after Kafka broker had gone down. So the Kafka logs dir had 
> directories like 
> {{foo.bar-testtopic-0.bb7981c216b845648edfe6e2b0a5c050-delete}}.
> When we restarted the Kafka broker, it refused to start and kept shutting 
> down and running into this exception:
> {code}
> [2017-05-12 21:36:27,876] ERROR There was an error in one of the threads 
> during logs loading: java.lang.StringIndexOutOfBoundsException: String index 
> out of range: -1 (kafka.log.LogManager)
> [2017-05-12 21:36:27,900] FATAL [Kafka Server 0], Fatal error during 
> KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
> java.lang.StringIndexOutOfBoundsException: String index out of range: -1
>   at java.lang.String.substring(String.java:1967)
>   at kafka.log.Log$.parseTopicPartitionName(Log.scala:1146)
>   at kafka.log.LogManager.$anonfun$loadLogs$10(LogManager.scala:153)
>   at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:57)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> [2017-05-12 21:36:27,950] INFO [Kafka Server 0], shutting down 
> (kafka.server.KafkaServer)
> {code}
> The only way we could get past this is pointing Kafka broker to a different 
> Kafka logs directory which effectively meant a lot our topics were no longer 
> accessible to the application.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-5232) Kafka broker fails to start if a topic containing dot in its name is marked for delete but hasn't been deleted during previous uptime

2017-05-14 Thread Ismael Juma (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-5232:
---
Priority: Critical  (was: Major)

> Kafka broker fails to start if a topic containing dot in its name is marked 
> for delete but hasn't been deleted during previous uptime
> -
>
> Key: KAFKA-5232
> URL: https://issues.apache.org/jira/browse/KAFKA-5232
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.2.0, 0.10.2.1
>Reporter: jaikiran pai
>Priority: Critical
> Fix For: 0.11.0.0, 0.10.2.2
>
>
> We are using 0.10.2.0 (but this is reproducible even with 0.10.2.1 and latest 
> upstream) in our environments. Our topic names contain (one or more) dot 
> characters in their name. So we have topics like {{foo.bar-testtopic}}. Topic 
> deletion is enabled on the broker(s) and our application does delete the 
> topics as and when necessary.
> We just ran into a case today where for some reason the Kafka broker had 
> either to be taken down (or went down on its own). Some of the topics which 
> were deleted (i.e. a deletion marker folder was created for them) were left 
> around after Kafka broker had gone down. So the Kafka logs dir had 
> directories like 
> {{foo.bar-testtopic-0.bb7981c216b845648edfe6e2b0a5c050-delete}}.
> When we restarted the Kafka broker, it refused to start and kept shutting 
> down and running into this exception:
> {code}
> [2017-05-12 21:36:27,876] ERROR There was an error in one of the threads 
> during logs loading: java.lang.StringIndexOutOfBoundsException: String index 
> out of range: -1 (kafka.log.LogManager)
> [2017-05-12 21:36:27,900] FATAL [Kafka Server 0], Fatal error during 
> KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
> java.lang.StringIndexOutOfBoundsException: String index out of range: -1
>   at java.lang.String.substring(String.java:1967)
>   at kafka.log.Log$.parseTopicPartitionName(Log.scala:1146)
>   at kafka.log.LogManager.$anonfun$loadLogs$10(LogManager.scala:153)
>   at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:57)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> [2017-05-12 21:36:27,950] INFO [Kafka Server 0], shutting down 
> (kafka.server.KafkaServer)
> {code}
> The only way we could get past this is pointing Kafka broker to a different 
> Kafka logs directory which effectively meant a lot our topics were no longer 
> accessible to the application.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (KAFKA-5232) Kafka broker fails to start if a topic containing dot in its name is marked for delete but hasn't been deleted during previous uptime

2017-05-14 Thread Ismael Juma (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma reassigned KAFKA-5232:
--

Assignee: jaikiran pai

> Kafka broker fails to start if a topic containing dot in its name is marked 
> for delete but hasn't been deleted during previous uptime
> -
>
> Key: KAFKA-5232
> URL: https://issues.apache.org/jira/browse/KAFKA-5232
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.2.0, 0.10.2.1
>Reporter: jaikiran pai
>Assignee: jaikiran pai
>Priority: Critical
> Fix For: 0.11.0.0, 0.10.2.2
>
>
> We are using 0.10.2.0 (but this is reproducible even with 0.10.2.1 and latest 
> upstream) in our environments. Our topic names contain (one or more) dot 
> characters in their name. So we have topics like {{foo.bar-testtopic}}. Topic 
> deletion is enabled on the broker(s) and our application does delete the 
> topics as and when necessary.
> We just ran into a case today where for some reason the Kafka broker had 
> either to be taken down (or went down on its own). Some of the topics which 
> were deleted (i.e. a deletion marker folder was created for them) were left 
> around after Kafka broker had gone down. So the Kafka logs dir had 
> directories like 
> {{foo.bar-testtopic-0.bb7981c216b845648edfe6e2b0a5c050-delete}}.
> When we restarted the Kafka broker, it refused to start and kept shutting 
> down and running into this exception:
> {code}
> [2017-05-12 21:36:27,876] ERROR There was an error in one of the threads 
> during logs loading: java.lang.StringIndexOutOfBoundsException: String index 
> out of range: -1 (kafka.log.LogManager)
> [2017-05-12 21:36:27,900] FATAL [Kafka Server 0], Fatal error during 
> KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
> java.lang.StringIndexOutOfBoundsException: String index out of range: -1
>   at java.lang.String.substring(String.java:1967)
>   at kafka.log.Log$.parseTopicPartitionName(Log.scala:1146)
>   at kafka.log.LogManager.$anonfun$loadLogs$10(LogManager.scala:153)
>   at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:57)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> [2017-05-12 21:36:27,950] INFO [Kafka Server 0], shutting down 
> (kafka.server.KafkaServer)
> {code}
> The only way we could get past this is pointing Kafka broker to a different 
> Kafka logs directory which effectively meant a lot our topics were no longer 
> accessible to the application.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-3356) Remove ConsumerOffsetChecker, deprecated in 0.9, in 0.11

2017-05-14 Thread Mickael Maison (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009693#comment-16009693
 ] 

Mickael Maison commented on KAFKA-3356:
---

Features of ConsumerOffsetChecker:
- For a specified group, list the current offset, log size, lag and owner per 
topic/partition
- For a specified group, filter output for a list of topics
- List all the brokers
- Allows to specify retry.backoff.ms and socket.timeout.ms

Features of ConsumerGroupCommand (when using old-consumer):
- For a specified group, list the current offset, log size, lag and owner per 
topic/partition
- List all groups
- Delete consumer group information
- Allows to specify retry.backoff.ms and socket.timeout.ms via a configuration 
file

So the new tool is missing a way to list all brokers and a way to filter group 
information using a list of topics.


> Remove ConsumerOffsetChecker, deprecated in 0.9, in 0.11
> 
>
> Key: KAFKA-3356
> URL: https://issues.apache.org/jira/browse/KAFKA-3356
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 0.10.2.0
>Reporter: Ashish Singh
>Assignee: Mickael Maison
>Priority: Blocker
> Fix For: 0.11.0.0
>
>
> ConsumerOffsetChecker is marked deprecated as of 0.9, should be removed in 
> 0.11.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #3050: (trunk) KAFKA-5232 Fix Log.parseTopicPartitionName...

2017-05-14 Thread jaikiran

GitHub user jaikiran opened a pull request:

https://github.com/apache/kafka/pull/3050

(trunk) KAFKA-5232 Fix Log.parseTopicPartitionName to take into account dot 
character in topic names of deleted topics

The commit here contains a fix and a test case for the issue reported in 
https://issues.apache.org/jira/browse/KAFKA-5232. This PR is meant for `trunk` 
branch and a corresponding PR for `0.10.2` branch has been raised here 
https://github.com/apache/kafka/pull/3043 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jaikiran/kafka KAFKA-5232-trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3050.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3050


commit 8695ce9a64ce27b329468c63288aff17cd1c2d2e
Author: Jaikiran Pai 
Date:   2017-05-13T12:15:05Z

KAFKA-5232 Fix the bug where the Log.parseTopicPartitionName wasn't taking 
into account the potential presence of '.' character in topic names while 
looking for topics that are marked for deletion




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-5232) Kafka broker fails to start if a topic containing dot in its name is marked for delete but hasn't been deleted during previous uptime

2017-05-14 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009752#comment-16009752
 ] 

ASF GitHub Bot commented on KAFKA-5232:
---

GitHub user jaikiran opened a pull request:

https://github.com/apache/kafka/pull/3050

(trunk) KAFKA-5232 Fix Log.parseTopicPartitionName to take into account dot 
character in topic names of deleted topics

The commit here contains a fix and a test case for the issue reported in 
https://issues.apache.org/jira/browse/KAFKA-5232. This PR is meant for `trunk` 
branch and a corresponding PR for `0.10.2` branch has been raised here 
https://github.com/apache/kafka/pull/3043 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jaikiran/kafka KAFKA-5232-trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3050.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3050


commit 8695ce9a64ce27b329468c63288aff17cd1c2d2e
Author: Jaikiran Pai 
Date:   2017-05-13T12:15:05Z

KAFKA-5232 Fix the bug where the Log.parseTopicPartitionName wasn't taking 
into account the potential presence of '.' character in topic names while 
looking for topics that are marked for deletion




> Kafka broker fails to start if a topic containing dot in its name is marked 
> for delete but hasn't been deleted during previous uptime
> -
>
> Key: KAFKA-5232
> URL: https://issues.apache.org/jira/browse/KAFKA-5232
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.2.0, 0.10.2.1
>Reporter: jaikiran pai
>Assignee: jaikiran pai
>Priority: Critical
> Fix For: 0.11.0.0, 0.10.2.2
>
>
> We are using 0.10.2.0 (but this is reproducible even with 0.10.2.1 and latest 
> upstream) in our environments. Our topic names contain (one or more) dot 
> characters in their name. So we have topics like {{foo.bar-testtopic}}. Topic 
> deletion is enabled on the broker(s) and our application does delete the 
> topics as and when necessary.
> We just ran into a case today where for some reason the Kafka broker had 
> either to be taken down (or went down on its own). Some of the topics which 
> were deleted (i.e. a deletion marker folder was created for them) were left 
> around after Kafka broker had gone down. So the Kafka logs dir had 
> directories like 
> {{foo.bar-testtopic-0.bb7981c216b845648edfe6e2b0a5c050-delete}}.
> When we restarted the Kafka broker, it refused to start and kept shutting 
> down and running into this exception:
> {code}
> [2017-05-12 21:36:27,876] ERROR There was an error in one of the threads 
> during logs loading: java.lang.StringIndexOutOfBoundsException: String index 
> out of range: -1 (kafka.log.LogManager)
> [2017-05-12 21:36:27,900] FATAL [Kafka Server 0], Fatal error during 
> KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
> java.lang.StringIndexOutOfBoundsException: String index out of range: -1
>   at java.lang.String.substring(String.java:1967)
>   at kafka.log.Log$.parseTopicPartitionName(Log.scala:1146)
>   at kafka.log.LogManager.$anonfun$loadLogs$10(LogManager.scala:153)
>   at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:57)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> [2017-05-12 21:36:27,950] INFO [Kafka Server 0], shutting down 
> (kafka.server.KafkaServer)
> {code}
> The only way we could get past this is pointing Kafka broker to a different 
> Kafka logs directory which effectively meant a lot our topics were no longer 
> accessible to the application.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Please subscribe me

2017-05-14 Thread Dimple Patel

Please subscribe me

[jira] [Created] (KAFKA-5234) GetOffsetShell: retrieve offsets for multiple topics with single request

2017-05-14 Thread Arseniy Tashoyan (JIRA)

Arseniy Tashoyan created KAFKA-5234:
---

 Summary: GetOffsetShell: retrieve offsets for multiple topics with 
single request
 Key: KAFKA-5234
 URL: https://issues.apache.org/jira/browse/KAFKA-5234
 Project: Kafka
  Issue Type: Improvement
  Components: tools
Reporter: Arseniy Tashoyan
 Fix For: 0.11.0.0


At present, GetOffsetShell is able to retrieve offsets for one topic only:

--topic  REQUIRED: The topic to get offsets from.

If user wants to get offsets for several topics, he has to call GetOffsetShell 
as many times as the number of topics to explore. Some solutions may have 
dozens of topics. Monitoring of a large Kafka cluster with GetOffsetShell 
requires additional scripting efforts and produces visible performance drawback 
due to multiple requests to the broker.

Instead, GetOffsetShell should support multiple topics, for example:

--topics topic1,topic2,topic3

Moreover, GetOffsetShell should be able to retrieve offsets for _all_ topics, 
when user specified none topics in command line.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (KAFKA-5235) GetOffsetShell: retrieve offsets for all given topics and partitions with single request to the broker

2017-05-14 Thread Arseniy Tashoyan (JIRA)

Arseniy Tashoyan created KAFKA-5235:
---

 Summary: GetOffsetShell: retrieve offsets for all given topics and 
partitions with single request to the broker
 Key: KAFKA-5235
 URL: https://issues.apache.org/jira/browse/KAFKA-5235
 Project: Kafka
  Issue Type: Improvement
  Components: tools
Reporter: Arseniy Tashoyan
 Fix For: 0.11.0.0


GetOffsetShell is implemented on old SimpleConsumer. It needs Zookeeper to 
retrieve metadata about topics and partitions. At present, GetOffsetShell does 
the following:
- get metadata from Zookeeper
- iterate over partitions
- for each partition, connect to its leader broker and request offsets
Instead, GetOffsetShell can use new KafkaConsumer and retrieve offsets by means 
of endOffsets(), beginningOffsets() and offsetsForTimes() methods. One request 
is sufficient for all topics and partitions.
As far as GetOffsetShell is re-implemented with new KafkaConsumer API, it will 
not depend on obsolete API: SimpleConsumer, old producer API.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5235) GetOffsetShell: retrieve offsets for all given topics and partitions with single request to the broker

2017-05-14 Thread Arseniy Tashoyan (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-5235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009792#comment-16009792
 ] 

Arseniy Tashoyan commented on KAFKA-5235:
-

Both improvements can be done in scope of switching to KafkaConsumer API

> GetOffsetShell: retrieve offsets for all given topics and partitions with 
> single request to the broker
> --
>
> Key: KAFKA-5235
> URL: https://issues.apache.org/jira/browse/KAFKA-5235
> Project: Kafka
>  Issue Type: Improvement
>  Components: tools
>Reporter: Arseniy Tashoyan
>  Labels: tool
> Fix For: 0.11.0.0
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> GetOffsetShell is implemented on old SimpleConsumer. It needs Zookeeper to 
> retrieve metadata about topics and partitions. At present, GetOffsetShell 
> does the following:
> - get metadata from Zookeeper
> - iterate over partitions
> - for each partition, connect to its leader broker and request offsets
> Instead, GetOffsetShell can use new KafkaConsumer and retrieve offsets by 
> means of endOffsets(), beginningOffsets() and offsetsForTimes() methods. One 
> request is sufficient for all topics and partitions.
> As far as GetOffsetShell is re-implemented with new KafkaConsumer API, it 
> will not depend on obsolete API: SimpleConsumer, old producer API.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5235) GetOffsetShell: retrieve offsets for all given topics and partitions with single request to the broker

2017-05-14 Thread Arseniy Tashoyan (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-5235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009800#comment-16009800
 ] 

Arseniy Tashoyan commented on KAFKA-5235:
-

How can I assign this issue on myself? I don't see any button like 'assign' on 
this page.

> GetOffsetShell: retrieve offsets for all given topics and partitions with 
> single request to the broker
> --
>
> Key: KAFKA-5235
> URL: https://issues.apache.org/jira/browse/KAFKA-5235
> Project: Kafka
>  Issue Type: Improvement
>  Components: tools
>Reporter: Arseniy Tashoyan
>  Labels: tool
> Fix For: 0.11.0.0
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> GetOffsetShell is implemented on old SimpleConsumer. It needs Zookeeper to 
> retrieve metadata about topics and partitions. At present, GetOffsetShell 
> does the following:
> - get metadata from Zookeeper
> - iterate over partitions
> - for each partition, connect to its leader broker and request offsets
> Instead, GetOffsetShell can use new KafkaConsumer and retrieve offsets by 
> means of endOffsets(), beginningOffsets() and offsetsForTimes() methods. One 
> request is sufficient for all topics and partitions.
> As far as GetOffsetShell is re-implemented with new KafkaConsumer API, it 
> will not depend on obsolete API: SimpleConsumer, old producer API.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-4850) RocksDb cannot use Bloom Filters

2017-05-14 Thread Bharat Viswanadham (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-4850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bharat Viswanadham updated KAFKA-4850:
--
Status: In Progress  (was: Patch Available)

> RocksDb cannot use Bloom Filters
> 
>
> Key: KAFKA-4850
> URL: https://issues.apache.org/jira/browse/KAFKA-4850
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Eno Thereska
>Assignee: Bharat Viswanadham
> Fix For: 0.11.0.0
>
>
> Bloom Filters would speed up RocksDb lookups. However they currently do not 
> work in RocksDb 5.0.2. This has been fixed in trunk, but we'll have to wait 
> until that is released and tested. 
> Then we can add the line in RocksDbStore.java in openDb:
> tableConfig.setFilter(new BloomFilter(10));



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #3045: MINOR: Handle nulls in NonEmptyListValidator

2017-05-14 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3045


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: kafka-trunk-jdk7 #2191

2017-05-14 Thread Apache Jenkins Server

See 


Changes:

[ismael] MINOR: Handle nulls in NonEmptyListValidator

--
[...truncated 858.64 KB...]
kafka.log.BrokerCompressionTest > testBrokerSideCompression[4] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[5] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[5] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[6] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[6] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[7] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[7] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[8] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[8] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[9] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[9] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[10] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[10] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[11] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[11] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[12] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[12] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[13] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[13] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[14] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[14] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[15] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[15] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[16] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[16] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[17] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[17] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[18] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[18] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[19] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[19] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[0] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[0] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[0] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[0] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[0] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerTest[0] PASSED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[0] STARTED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[0] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[1] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[1] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[1] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[1] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[1] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerTest[1] PASSED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[1] STARTED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[1] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[2] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[2] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[2] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[2] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[2] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerTest[2] PASSED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[2] STARTED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[2] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[3] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[3] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[3] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[3] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[3] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerTest[3] PASSED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessa

[jira] [Commented] (KAFKA-5235) GetOffsetShell: retrieve offsets for all given topics and partitions with single request to the broker

2017-05-14 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-5235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009814#comment-16009814
 ] 

ASF GitHub Bot commented on KAFKA-5235:
---

GitHub user tashoyan opened a pull request:

https://github.com/apache/kafka/pull/3051

KAFKA-5235: GetOffsetShell: new KafkaConsumer API, support for multiple 
topics, minimized number of requests to server

This PR addresses two improvements:

[KAFKA-5235 GetOffsetShell: retrieve offsets for all given topics and 
partitions with single request to the 
broker](https://issues.apache.org/jira/browse/KAFKA-5235)
[KAFKA-5234 GetOffsetShell: retrieve offsets for multiple topics with 
single request](https://issues.apache.org/jira/browse/KAFKA-5234)

1. Previous implementation used SimpleConsumer to get offsets and old 
Producer API to get topic/partition metadata. Previous implementation 
determined a leader broker for each partition and then requested the leader for 
offsets. In total, it did as many requests to the broker as the number of 
partitions (plus a request to Zookeeper for metadata).
New implementation `kafka-get-offsets.sh` uses KafkaConsumer API. It makes 
at most two requests to the broker: 1) to query existing topics and partitions, 
2) to grab all requested offsets. New implementation correctly handles 
non-existing topics and partitions asked by user:

> kafka-get-offsets.sh --bootstrap-servers vm:9092 --topics AAA,ZZZ 
--partitions 0,1
> AAA:0:7
> AAA:1:Partition not found
> ZZZ:0:Topic not found

2. Previously, user could get offsets for one topic only. Now user can get 
offsets for many topics at once:
`kafka-get-offsets.sh --bootstrap-servers vm:9092 --topics AAA,ZZZ`
Moreover, now user is able to retrieve offsets for _all_ topics - this is 
the default when no topics specified:
`kafka-get-offsets.sh --bootstrap-servers vm:9092`
Thanks to this feature, there is no need anymore to retrieve all topics by 
means of `kafka-topics.sh`.
When no topics specified, the new `kafka-get-offsets.sh` tool takes into 
account only user-level topics and ignores Kafka-internal topics (i.e. consumer 
offsets). This behavior can be altered via a special command line argument:
`kafka-get-offsets.sh --bootstrap-servers vm:9092 --include-internal-topics`

3. New `kafka-get-offsets.sh` tool is consistent with other console tools 
with respect to command line argument names. In addition, 
`kafka-get-offsets.sh` tool gives the possibility to pass an arbitrary setting 
to KafkaConsumer via `--consumer-property` argument.

I hope, now `kafka-get-offsets.sh` is easier in use and gives performance 
improvement.
@junrao I suppose you may want to review.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tashoyan/kafka trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3051.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3051


commit ec16d064aac4dfba164aeefaae3950db7b2e35af
Author: Arseniy Tashoyan 
Date:   2017-05-01T19:52:26Z

Add a testable method that retrieves offsets: getOffsets(). Cover it with 
tests.

commit df60a30da3a5c3d0d95c85df2c5eb32a6eeae107
Author: Arseniy Tashoyan 
Date:   2017-05-01T19:55:13Z

Fix some trivial warnings

commit 5aa9639666d32f7e036cee4cb42ac9b7223def2e
Author: Arseniy Tashoyan 
Date:   2017-05-01T21:07:35Z

Switch the implementation to getOffsets().

commit 3d772b8fac0c45cbe7631064a57361b7928b9bc2
Author: Arseniy Tashoyan 
Date:   2017-05-01T22:01:19Z

Add a test for a replicated partition

commit 15e8b1a83471919c40d56f32ac858a38b7ad7b31
Author: Arseniy Tashoyan 
Date:   2017-05-10T21:42:33Z

Add the implementation based on new KafkaConsumer. Switch tests to this new 
implementation. Now it works for replicated topics.

commit 1cdfce266c217b080410235c671f2764068dc96c
Author: Arseniy Tashoyan 
Date:   2017-05-11T20:33:34Z

Implement support for non-existing topics and partitions

commit 7a4bdc4deed9987f348ad76c027bf891d7ef3257
Author: Arseniy Tashoyan 
Date:   2017-05-11T21:08:26Z

Refactor: rename the method. Add documentation.

commit d473adb73434b7d4347cf62fbf29e73615ea8a82
Author: Arseniy Tashoyan 
Date:   2017-05-12T07:39:13Z

Refine the doc

commit a21bfab73bac67d2441472f8f73d7a9c956dcc5c
Author: Arseniy Tashoyan 
Date:   2017-05-12T09:11:19Z

Add more tests for non-existing partitions. Refactor tests.

commit 0b8e6f8e02802a5e3fb40156efa656bb91f2211d
Author: Arseniy Tashoyan 
Date:   2017-05-12T09:26:34Z

Add a test for a non-existing topic.

commit 6a65574ac28f86712894e25d6edff1051bbc50ac
Author: Arseniy Tashoyan 
Date:   2017-05-12T10:42:08Z

Add the abil

[GitHub] kafka pull request #3051: KAFKA-5235: GetOffsetShell: new KafkaConsumer API,...

2017-05-14 Thread tashoyan

GitHub user tashoyan opened a pull request:

https://github.com/apache/kafka/pull/3051

KAFKA-5235: GetOffsetShell: new KafkaConsumer API, support for multiple 
topics, minimized number of requests to server

This PR addresses two improvements:

[KAFKA-5235 GetOffsetShell: retrieve offsets for all given topics and 
partitions with single request to the 
broker](https://issues.apache.org/jira/browse/KAFKA-5235)
[KAFKA-5234 GetOffsetShell: retrieve offsets for multiple topics with 
single request](https://issues.apache.org/jira/browse/KAFKA-5234)

1. Previous implementation used SimpleConsumer to get offsets and old 
Producer API to get topic/partition metadata. Previous implementation 
determined a leader broker for each partition and then requested the leader for 
offsets. In total, it did as many requests to the broker as the number of 
partitions (plus a request to Zookeeper for metadata).
New implementation `kafka-get-offsets.sh` uses KafkaConsumer API. It makes 
at most two requests to the broker: 1) to query existing topics and partitions, 
2) to grab all requested offsets. New implementation correctly handles 
non-existing topics and partitions asked by user:

> kafka-get-offsets.sh --bootstrap-servers vm:9092 --topics AAA,ZZZ 
--partitions 0,1
> AAA:0:7
> AAA:1:Partition not found
> ZZZ:0:Topic not found

2. Previously, user could get offsets for one topic only. Now user can get 
offsets for many topics at once:
`kafka-get-offsets.sh --bootstrap-servers vm:9092 --topics AAA,ZZZ`
Moreover, now user is able to retrieve offsets for _all_ topics - this is 
the default when no topics specified:
`kafka-get-offsets.sh --bootstrap-servers vm:9092`
Thanks to this feature, there is no need anymore to retrieve all topics by 
means of `kafka-topics.sh`.
When no topics specified, the new `kafka-get-offsets.sh` tool takes into 
account only user-level topics and ignores Kafka-internal topics (i.e. consumer 
offsets). This behavior can be altered via a special command line argument:
`kafka-get-offsets.sh --bootstrap-servers vm:9092 --include-internal-topics`

3. New `kafka-get-offsets.sh` tool is consistent with other console tools 
with respect to command line argument names. In addition, 
`kafka-get-offsets.sh` tool gives the possibility to pass an arbitrary setting 
to KafkaConsumer via `--consumer-property` argument.

I hope, now `kafka-get-offsets.sh` is easier in use and gives performance 
improvement.
@junrao I suppose you may want to review.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tashoyan/kafka trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3051.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3051


commit ec16d064aac4dfba164aeefaae3950db7b2e35af
Author: Arseniy Tashoyan 
Date:   2017-05-01T19:52:26Z

Add a testable method that retrieves offsets: getOffsets(). Cover it with 
tests.

commit df60a30da3a5c3d0d95c85df2c5eb32a6eeae107
Author: Arseniy Tashoyan 
Date:   2017-05-01T19:55:13Z

Fix some trivial warnings

commit 5aa9639666d32f7e036cee4cb42ac9b7223def2e
Author: Arseniy Tashoyan 
Date:   2017-05-01T21:07:35Z

Switch the implementation to getOffsets().

commit 3d772b8fac0c45cbe7631064a57361b7928b9bc2
Author: Arseniy Tashoyan 
Date:   2017-05-01T22:01:19Z

Add a test for a replicated partition

commit 15e8b1a83471919c40d56f32ac858a38b7ad7b31
Author: Arseniy Tashoyan 
Date:   2017-05-10T21:42:33Z

Add the implementation based on new KafkaConsumer. Switch tests to this new 
implementation. Now it works for replicated topics.

commit 1cdfce266c217b080410235c671f2764068dc96c
Author: Arseniy Tashoyan 
Date:   2017-05-11T20:33:34Z

Implement support for non-existing topics and partitions

commit 7a4bdc4deed9987f348ad76c027bf891d7ef3257
Author: Arseniy Tashoyan 
Date:   2017-05-11T21:08:26Z

Refactor: rename the method. Add documentation.

commit d473adb73434b7d4347cf62fbf29e73615ea8a82
Author: Arseniy Tashoyan 
Date:   2017-05-12T07:39:13Z

Refine the doc

commit a21bfab73bac67d2441472f8f73d7a9c956dcc5c
Author: Arseniy Tashoyan 
Date:   2017-05-12T09:11:19Z

Add more tests for non-existing partitions. Refactor tests.

commit 0b8e6f8e02802a5e3fb40156efa656bb91f2211d
Author: Arseniy Tashoyan 
Date:   2017-05-12T09:26:34Z

Add a test for a non-existing topic.

commit 6a65574ac28f86712894e25d6edff1051bbc50ac
Author: Arseniy Tashoyan 
Date:   2017-05-12T10:42:08Z

Add the ability to retrieve offsets for all partition of all topics. Add 
the ability to exclude internal topics.

commit f23c85305868b2f4f0a00c318cb3a2d0786b467b
Author: Arseniy Tashoyan 
Date:   2017-05-12T14:54:55Z

Switch the tool to new implementation

commit

Jenkins build is back to normal : kafka-trunk-jdk8 #1521

2017-05-14 Thread Apache Jenkins Server

See

[jira] [Created] (KAFKA-5236) Regression in on-disk log size when using Snappy compression with 0.8.2 log message format

2017-05-14 Thread Nick Travers (JIRA)

Nick Travers created KAFKA-5236:
---

 Summary: Regression in on-disk log size when using Snappy 
compression with 0.8.2 log message format
 Key: KAFKA-5236
 URL: https://issues.apache.org/jira/browse/KAFKA-5236
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.10.2.1
Reporter: Nick Travers


We recently upgraded our brokers in our production environments from 0.10.1.1 
to 0.10.2.1 and we've noticed a sizable regression in the on-disk .log file 
size. For some deployments the increase was as much as 50%.

We run our brokers with the 0.8.2 log message format version. The majority of 
our message volume comes from 0.10.x Java clients sending messages encoded with 
the Snappy codec.

Some initial testing only shows a regression between the two versions when 
using Snappy compression with a log message format of 0.8.2.

I also tested 0.10.x log message formats as well as Gzip compression. The log 
sizes do not differ in this case, so the issue seems confined to 0.8.2 message 
format and Snappy compression.

A git-bisect lead me to this commit, which modified the server-side 
implementation of `Record`:
https://github.com/apache/kafka/commit/67f1e5b91bf073151ff57d5d656693e385726697

Here's the PR, which has more context:
https://github.com/apache/kafka/pull/2140

Here is a link to the test I used to re-producer this issue:
https://github.com/nicktrav/kafka/commit/68e8db4fa525e173651ac740edb270b0d90b8818

cc: [~hachikuji] [~junrao] [~ijuma] [~guozhang] (tagged on original PR)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5236) Regression in on-disk log size when using Snappy compression with 0.8.2 log message format

2017-05-14 Thread Ismael Juma (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-5236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009863#comment-16009863
 ] 

Ismael Juma commented on KAFKA-5236:


Thanks for the report. The root cause is that the block size for Snappy was 
changed from 32 KB to 1 KB in the broker:

https://github.com/apache/kafka/pull/2140#discussion_r90383989

This is the same block size used by the producer and with the 0.10.x format, 
the broker won't recompress the messages in the common case.

KAFKA-5148 and KAFKA-3704 are related. We should probably use the default block 
size by default (32 KB) in both broker and producer and allow the block size to 
be configurable as per those JIRAs.

> Regression in on-disk log size when using Snappy compression with 0.8.2 log 
> message format
> --
>
> Key: KAFKA-5236
> URL: https://issues.apache.org/jira/browse/KAFKA-5236
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.2.1
>Reporter: Nick Travers
>
> We recently upgraded our brokers in our production environments from 0.10.1.1 
> to 0.10.2.1 and we've noticed a sizable regression in the on-disk .log file 
> size. For some deployments the increase was as much as 50%.
> We run our brokers with the 0.8.2 log message format version. The majority of 
> our message volume comes from 0.10.x Java clients sending messages encoded 
> with the Snappy codec.
> Some initial testing only shows a regression between the two versions when 
> using Snappy compression with a log message format of 0.8.2.
> I also tested 0.10.x log message formats as well as Gzip compression. The log 
> sizes do not differ in this case, so the issue seems confined to 0.8.2 
> message format and Snappy compression.
> A git-bisect lead me to this commit, which modified the server-side 
> implementation of `Record`:
> https://github.com/apache/kafka/commit/67f1e5b91bf073151ff57d5d656693e385726697
> Here's the PR, which has more context:
> https://github.com/apache/kafka/pull/2140
> Here is a link to the test I used to re-producer this issue:
> https://github.com/nicktrav/kafka/commit/68e8db4fa525e173651ac740edb270b0d90b8818
> cc: [~hachikuji] [~junrao] [~ijuma] [~guozhang] (tagged on original PR)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (KAFKA-5237) SimpleConsumerShell logs terminating message to stdout instead of stderr

2017-05-14 Thread Daniel Einspanjer (JIRA)

Daniel Einspanjer created KAFKA-5237:


 Summary: SimpleConsumerShell logs terminating message to stdout 
instead of stderr
 Key: KAFKA-5237
 URL: https://issues.apache.org/jira/browse/KAFKA-5237
 Project: Kafka
  Issue Type: Improvement
  Components: clients
Affects Versions: 0.10.2.1
Reporter: Daniel Einspanjer
Priority: Trivial


The SimpleConsumerShell has one big advantage over the standard 
kafka-console-consumer client, it supports the --no-wait-at-logend parameter 
which lets you script its use without having to rely on a timeout or dealing 
with the exception and stacktrace thrown by said timeout.

Unfortunately, when you use this option, it will write a termination message to 
stdout when it is finished.  This means if you are using it to dump the 
contents of a topic to a file, you get an extra line.

This pull request just changes that one line to call System.err.println instead 
of println.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #3052: KAFKA-5237 - change logging of termination message...

2017-05-14 Thread deinspanjer

GitHub user deinspanjer opened a pull request:

https://github.com/apache/kafka/pull/3052

KAFKA-5237 - change logging of termination message to use stderr instead of 
stdout



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/deinspanjer/kafka 
improvement/simple_consumer_shell_logging

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3052.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3052


commit 01a8fef379bd83e38e4938b34a1272d3347bb075
Author: Daniel Einspanjer 
Date:   2017-05-14T22:09:34Z

KAFKA-5237 - change logging of termination message to use stderr instead of 
stdout




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-5237) SimpleConsumerShell logs terminating message to stdout instead of stderr

2017-05-14 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-5237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009868#comment-16009868
 ] 

ASF GitHub Bot commented on KAFKA-5237:
---

GitHub user deinspanjer opened a pull request:

https://github.com/apache/kafka/pull/3052

KAFKA-5237 - change logging of termination message to use stderr instead of 
stdout



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/deinspanjer/kafka 
improvement/simple_consumer_shell_logging

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3052.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3052


commit 01a8fef379bd83e38e4938b34a1272d3347bb075
Author: Daniel Einspanjer 
Date:   2017-05-14T22:09:34Z

KAFKA-5237 - change logging of termination message to use stderr instead of 
stdout




> SimpleConsumerShell logs terminating message to stdout instead of stderr
> 
>
> Key: KAFKA-5237
> URL: https://issues.apache.org/jira/browse/KAFKA-5237
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.10.2.1
>Reporter: Daniel Einspanjer
>Priority: Trivial
>
> The SimpleConsumerShell has one big advantage over the standard 
> kafka-console-consumer client, it supports the --no-wait-at-logend parameter 
> which lets you script its use without having to rely on a timeout or dealing 
> with the exception and stacktrace thrown by said timeout.
> Unfortunately, when you use this option, it will write a termination message 
> to stdout when it is finished.  This means if you are using it to dump the 
> contents of a topic to a file, you get an extra line.
> This pull request just changes that one line to call System.err.println 
> instead of println.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (KAFKA-5238) BrokerTopicMetrics can be recreated after topic is deleted

2017-05-14 Thread Ismael Juma (JIRA)

Ismael Juma created KAFKA-5238:
--

 Summary: BrokerTopicMetrics can be recreated after topic is deleted
 Key: KAFKA-5238
 URL: https://issues.apache.org/jira/browse/KAFKA-5238
 Project: Kafka
  Issue Type: Bug
Reporter: Ismael Juma


As part of KAFKA-3258, we added code to remove metrics during topic deletion. 
This works fine as long as there are no fetch requests in the purgatory. If 
there are, however, we'll recreate the metrics when we call 
`ReplicaManager.appendToLocalLog`.

This can be reproduced by updating 
MetricsTest.testBrokerTopicMetricsUnregisteredAfterDeletingTopic() in the 
following way:

{code}
@Test
  def testBrokerTopicMetricsUnregisteredAfterDeletingTopic() {
val topic = "test-broker-topic-metric"
AdminUtils.createTopic(zkUtils, topic, 2, 1)
// Produce a few messages and consume them to create the metrics
TestUtils.produceMessages(servers, topic, nMessages)
TestUtils.consumeTopicRecords(servers, topic, nMessages)
assertTrue("Topic metrics don't exist", topicMetricGroups(topic).nonEmpty)
assertNotNull(BrokerTopicStats.getBrokerTopicStats(topic))
AdminUtils.deleteTopic(zkUtils, topic)
TestUtils.verifyTopicDeletion(zkUtils, topic, 1, servers)
Thread.sleep(1)
assertEquals("Topic metrics exists after deleteTopic", Set.empty, 
topicMetricGroups(topic))
  }
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #3044: KAFKA-5230: Fix conversion of Class configs to han...

2017-05-14 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3044


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-5230) Recommended values for Connect transformations contain the wrong class name

2017-05-14 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009883#comment-16009883
 ] 

ASF GitHub Bot commented on KAFKA-5230:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3044


> Recommended values for Connect transformations contain the wrong class name
> ---
>
> Key: KAFKA-5230
> URL: https://issues.apache.org/jira/browse/KAFKA-5230
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.10.2.0, 0.10.2.1
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
>  Labels: newbie
> Fix For: 0.11.0.0, 0.10.2.2
>
>
> If you try to validate a connector config with a transformation, it includes 
> suggested values for that config:
> {code}
> curl 
> 'http://localhost:8083/connector-plugins/org.apache.kafka.connect.file.FileStreamSourceConnector/config/validate'
>  -X PUT -H 'Content-Type: application/json' -H 'Accept: */*' --data-binary 
> '{"connector.class":"org.apache.kafka.connect.file.FileStreamSourceConnector","name":"blah-blah","transforms":"something","file":"/tmp/blah.what","topic":"test-topic","tasks.max":1}'
> {code}
> However, some of those recommendations do not contain the correct value:
> {code}
> {
>   "definition": {
> "name": "transforms.something.type",
> "type": "CLASS",
> "required": true,
> "default_value": null,
> "importance": "HIGH",
> "documentation": "Class for the 'something' transformation.",
> "group": "Transforms: something",
> "width": "LONG",
> "display_name": "Transformation type for something",
> "dependents": [],
> "order": 0
>   },
>   "value": {
> "name": "transforms.something.type",
> "value": null,
> "recommended_values": [
>   "org.apache.kafka.connect.transforms.ExtractField.Key",
>   "org.apache.kafka.connect.transforms.ExtractField.Value",
>   "org.apache.kafka.connect.transforms.HoistField.Key",
>   "org.apache.kafka.connect.transforms.HoistField.Value",
>   "org.apache.kafka.connect.transforms.InsertField.Key",
>   "org.apache.kafka.connect.transforms.InsertField.Value",
>   "org.apache.kafka.connect.transforms.MaskField.Key",
>   "org.apache.kafka.connect.transforms.MaskField.Value",
>   "org.apache.kafka.connect.transforms.RegexRouter",
>   "org.apache.kafka.connect.transforms.ReplaceField.Key",
>   "org.apache.kafka.connect.transforms.ReplaceField.Value",
>   "org.apache.kafka.connect.transforms.SetSchemaMetadata.Key",
>   "org.apache.kafka.connect.transforms.SetSchemaMetadata.Value",
>   "org.apache.kafka.connect.transforms.TimestampRouter",
>   "org.apache.kafka.connect.transforms.ValueToKey"
> ],
> "errors": [
>   "Missing required configuration \"transforms.something.type\" which 
> has no default value.",
>   "Invalid value null for configuration transforms.something.type: 
> Not a Transformation"
> ],
> "visible": true
>   }
> {code}
> In particular, nested classes for Key and Value transformations are being 
> returned as, e.g.
> org.apache.kafka.connect.transforms.ReplaceField.Key
> instead of
> org.apache.kafka.connect.transforms.ReplaceField$Key
> It seems this is the difference between the canonical and regular name.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-5230) Recommended values for Connect transformations contain the wrong class name

2017-05-14 Thread Ismael Juma (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-5230:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Recommended values for Connect transformations contain the wrong class name
> ---
>
> Key: KAFKA-5230
> URL: https://issues.apache.org/jira/browse/KAFKA-5230
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.10.2.0, 0.10.2.1
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
>  Labels: newbie
> Fix For: 0.11.0.0, 0.10.2.2
>
>
> If you try to validate a connector config with a transformation, it includes 
> suggested values for that config:
> {code}
> curl 
> 'http://localhost:8083/connector-plugins/org.apache.kafka.connect.file.FileStreamSourceConnector/config/validate'
>  -X PUT -H 'Content-Type: application/json' -H 'Accept: */*' --data-binary 
> '{"connector.class":"org.apache.kafka.connect.file.FileStreamSourceConnector","name":"blah-blah","transforms":"something","file":"/tmp/blah.what","topic":"test-topic","tasks.max":1}'
> {code}
> However, some of those recommendations do not contain the correct value:
> {code}
> {
>   "definition": {
> "name": "transforms.something.type",
> "type": "CLASS",
> "required": true,
> "default_value": null,
> "importance": "HIGH",
> "documentation": "Class for the 'something' transformation.",
> "group": "Transforms: something",
> "width": "LONG",
> "display_name": "Transformation type for something",
> "dependents": [],
> "order": 0
>   },
>   "value": {
> "name": "transforms.something.type",
> "value": null,
> "recommended_values": [
>   "org.apache.kafka.connect.transforms.ExtractField.Key",
>   "org.apache.kafka.connect.transforms.ExtractField.Value",
>   "org.apache.kafka.connect.transforms.HoistField.Key",
>   "org.apache.kafka.connect.transforms.HoistField.Value",
>   "org.apache.kafka.connect.transforms.InsertField.Key",
>   "org.apache.kafka.connect.transforms.InsertField.Value",
>   "org.apache.kafka.connect.transforms.MaskField.Key",
>   "org.apache.kafka.connect.transforms.MaskField.Value",
>   "org.apache.kafka.connect.transforms.RegexRouter",
>   "org.apache.kafka.connect.transforms.ReplaceField.Key",
>   "org.apache.kafka.connect.transforms.ReplaceField.Value",
>   "org.apache.kafka.connect.transforms.SetSchemaMetadata.Key",
>   "org.apache.kafka.connect.transforms.SetSchemaMetadata.Value",
>   "org.apache.kafka.connect.transforms.TimestampRouter",
>   "org.apache.kafka.connect.transforms.ValueToKey"
> ],
> "errors": [
>   "Missing required configuration \"transforms.something.type\" which 
> has no default value.",
>   "Invalid value null for configuration transforms.something.type: 
> Not a Transformation"
> ],
> "visible": true
>   }
> {code}
> In particular, nested classes for Key and Value transformations are being 
> returned as, e.g.
> org.apache.kafka.connect.transforms.ReplaceField.Key
> instead of
> org.apache.kafka.connect.transforms.ReplaceField$Key
> It seems this is the difference between the canonical and regular name.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to normal : kafka-trunk-jdk7 #2192

2017-05-14 Thread Apache Jenkins Server

See

[jira] [Commented] (KAFKA-5200) Deleting topic when one broker is down will prevent topic to be re-creatable

2017-05-14 Thread huxi (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-5200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009914#comment-16009914
 ] 

huxi commented on KAFKA-5200:
-

[~ecomar] Seems there is no convenient tool to do this. However, you could 
follow the instructions below to complete the deletion:
1. Stop all healthy brokers
2. Manually delete `/brokers/topics/` znode in Zookeeper
3. Delete log files for the topic in the file systems
4. Remove `/admin/delete_topics` znode in Zookeeper
5. Start all healthy brokers

> Deleting topic when one broker is down will prevent topic to be re-creatable
> 
>
> Key: KAFKA-5200
> URL: https://issues.apache.org/jira/browse/KAFKA-5200
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Reporter: Edoardo Comar
>
> In a cluster with 5 broker, replication factor=3, min in sync=2,
> one broker went down 
> A user's app remained of course unaware of that and deleted a topic that 
> (unknowingly) had a replica on the dead broker.
> The topic went in 'pending delete' mode
> The user then tried to recreate the topic - which failed, so his app was left 
> stuck - no working topic and no ability to create one.
> The reassignment tool fails to move the replica out of the dead broker - 
> specifically because the broker with the partition replica to move is dead :-)
> Incidentally the confluent-rebalancer docs say
> http://docs.confluent.io/current/kafka/post-deployment.html#scaling-the-cluster
> > Supports moving partitions away from dead brokers
> It'd be nice to similarly improve the opensource reassignment tool



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (KAFKA-5239) Producer buffer pool allocates memory inside a lock.

2017-05-14 Thread Sean McCauliff (JIRA)

Sean McCauliff created KAFKA-5239:
-

 Summary: Producer buffer pool allocates memory inside a lock.
 Key: KAFKA-5239
 URL: https://issues.apache.org/jira/browse/KAFKA-5239
 Project: Kafka
  Issue Type: Bug
  Components: clients
Reporter: Sean McCauliff


KAFKA-4840 placed the ByteBuffer allocation inside the critical section.  
Previously byte buffer allocation happened outside of the critical section.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #3053: KAFKA-5239: Producer buffer pool allocates memory ...

2017-05-14 Thread smccauliff

GitHub user smccauliff opened a pull request:

https://github.com/apache/kafka/pull/3053

KAFKA-5239: Producer buffer pool allocates memory inside a lock

Move byte buffer allocation out of lock.
Add unit test for restoring count when OOM is thrown from byte buffer 
allocation.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/smccauliff/kafka kafka-5239

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3053.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3053


commit f8e1afdb9f5650583ddd6b56ffe64fed834f15aa
Author: Sean McCauliff 
Date:   2017-05-15T04:24:52Z

Move byte buffer allocation out of lock.
Add unit test for restoring count when OOM is thrown from byte buffer 
allocation.
i




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-5239) Producer buffer pool allocates memory inside a lock.

2017-05-14 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-5239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009981#comment-16009981
 ] 

ASF GitHub Bot commented on KAFKA-5239:
---

GitHub user smccauliff opened a pull request:

https://github.com/apache/kafka/pull/3053

KAFKA-5239: Producer buffer pool allocates memory inside a lock

Move byte buffer allocation out of lock.
Add unit test for restoring count when OOM is thrown from byte buffer 
allocation.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/smccauliff/kafka kafka-5239

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3053.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3053


commit f8e1afdb9f5650583ddd6b56ffe64fed834f15aa
Author: Sean McCauliff 
Date:   2017-05-15T04:24:52Z

Move byte buffer allocation out of lock.
Add unit test for restoring count when OOM is thrown from byte buffer 
allocation.
i




> Producer buffer pool allocates memory inside a lock.
> 
>
> Key: KAFKA-5239
> URL: https://issues.apache.org/jira/browse/KAFKA-5239
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Reporter: Sean McCauliff
>
> KAFKA-4840 placed the ByteBuffer allocation inside the critical section.  
> Previously byte buffer allocation happened outside of the critical section.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4273) Streams DSL - Add TTL / retention period support for intermediate topics and state stores

2017-05-14 Thread Guozhang Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-4273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16010005#comment-16010005
 ] 

Guozhang Wang commented on KAFKA-4273:
--

[~vinubarro] I'm wondering if your use case can be handled by Streams' session 
window aggregations? Details can be found at:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-94+Session+Windows

In general, you can keep your aggregate results for as long as X time units 
after there are no more updates on the aggregated keys.

> Streams DSL - Add TTL / retention period support for intermediate topics and 
> state stores
> -
>
> Key: KAFKA-4273
> URL: https://issues.apache.org/jira/browse/KAFKA-4273
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.1
>Reporter: Davor Poldrugo
>
> Hi!
> I'm using Streams DSL (0.10.0.1), which can only use RocksDB for local state 
> as far as I know - it's not configurable.
> In my use case my data has TTL / retnetion period. It's 48 hours. After that 
> - data can be discarded.
> I join two topics: "messages" and "prices" using windowed inner join.
> The two intermediate Kafka topics for this join are named:
>  * messages-prices-join-this-changelog
>  * messages-prices-join-other-changelog
> Since these topics are created as compacted by Kafka Streams, and I don't 
> wan't to keep data forever, I have altered them to not use compaction. Right 
> now my RocksDB state stores grow indefinitely, and I don't have any options 
> to define TTL, or somehow periodically clean the older data.
> A "hack" that I use to keep my disk usage low - I have schedulled a job to 
> periodically stop Kafka Streams instances - one at the time. This triggers a 
> rebalance, and partitions migrate to other instances. When the instance is 
> started again, there's another rebalance, and sometimes this instance starts 
> processing partitions that wasn't processing before the stop - which leads to 
> deletion of the RocksDB state store for those partitions 
> (state.cleanup.delay.ms). In the next rebalance the local store is recreated 
> with a restore consumer - which reads data from - as previously mentioned - a 
> non compacted topic. And this effectively leads to a "hacked TTL support" in 
> Kafka Streams DSL.
> Questions:
>  * Do you think would be reasonable to add support in the DSL api to define 
> TTL for local store?
>  * Which opens another question - there are use cases which don't need the 
> intermediate topics to be created as "compact". Could also this be added to 
> the DSL api? Maybe only this could be added, and this flag should also be 
> used for the RocksDB TTL. Of course in this case another config would be 
> mandatory - the retention period or TTL for the intermediate topics and the 
> state stores. I saw there is a new cleanup.policy - compact_and_delete - 
> added with KAFKA-4015.
>  * Which also leads to another question, maybe some intermediate topics / 
> state stores need different TTL, so a it's not as simple as that. But after 
> KAFKA-3870, it will be easier.
> RocksDB supports TTL:
>  * 
> https://github.com/apache/kafka/blob/0.10.0.1/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBStore.java#L166
>  * https://github.com/facebook/rocksdb/wiki/Time-to-Live
>  * 
> https://github.com/facebook/rocksdb/blob/master/java/src/main/java/org/rocksdb/TtlDB.java
> A somehow similar issue: KAFKA-4212



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

37 matches

Mail list logo