[jira] [Created] (KAFKA-3339) org.apache.kafka.clients.consumer.KafkaConsumer javadoc for seekToBeginning, seekToEnd incomplete
Harald Kirsch created KAFKA-3339: Summary: org.apache.kafka.clients.consumer.KafkaConsumer javadoc for seekToBeginning, seekToEnd incomplete Key: KAFKA-3339 URL: https://issues.apache.org/jira/browse/KAFKA-3339 Project: Kafka Issue Type: Improvement Components: clients Affects Versions: 0.9.0.1 Reporter: Harald Kirsch The api documentation for these methods should remark that all subscribed partitions are seeked if no TopicPartition is provided. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-3339) org.apache.kafka.clients.consumer.KafkaConsumer javadoc for seekToBeginning, seekToEnd incomplete
[ https://issues.apache.org/jira/browse/KAFKA-3339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harald Kirsch updated KAFKA-3339: - Description: The api documentation for seekToBeginning, seekToEnd in these methods should remark that all subscribed partitions are seeked if no TopicPartition is provided. (was: The api documentation for these methods should remark that all subscribed partitions are seeked if no TopicPartition is provided.) > org.apache.kafka.clients.consumer.KafkaConsumer javadoc for seekToBeginning, > seekToEnd incomplete > - > > Key: KAFKA-3339 > URL: https://issues.apache.org/jira/browse/KAFKA-3339 > Project: Kafka > Issue Type: Improvement > Components: clients >Affects Versions: 0.9.0.1 >Reporter: Harald Kirsch > > The api documentation for seekToBeginning, seekToEnd in these methods should > remark that all subscribed partitions are seeked if no TopicPartition is > provided. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-3339) org.apache.kafka.clients.consumer.KafkaConsumer javadoc for seekToBeginning, seekToEnd incomplete
[ https://issues.apache.org/jira/browse/KAFKA-3339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harald Kirsch updated KAFKA-3339: - Description: The api documentation for seekToBeginning and seekToEnd in org.apache.kafka.clients.consumer.KafkaConsumer these methods should remark that all subscribed partitions are seeked if no TopicPartition is provided. (was: The api documentation for seekToBeginning, seekToEnd in these methods should remark that all subscribed partitions are seeked if no TopicPartition is provided.) > org.apache.kafka.clients.consumer.KafkaConsumer javadoc for seekToBeginning, > seekToEnd incomplete > - > > Key: KAFKA-3339 > URL: https://issues.apache.org/jira/browse/KAFKA-3339 > Project: Kafka > Issue Type: Improvement > Components: clients >Affects Versions: 0.9.0.1 >Reporter: Harald Kirsch > > The api documentation for seekToBeginning and seekToEnd in > org.apache.kafka.clients.consumer.KafkaConsumer these methods should remark > that all subscribed partitions are seeked if no TopicPartition is provided. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3339) org.apache.kafka.clients.consumer.KafkaConsumer javadoc for seekToBeginning, seekToEnd incomplete
[ https://issues.apache.org/jira/browse/KAFKA-3339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182210#comment-15182210 ] Ashish K Singh commented on KAFKA-3339: --- [~haraldk] you are correct. Do you want to create a PR for this, as you already know the fix. If not, I can provide a quick patch. > org.apache.kafka.clients.consumer.KafkaConsumer javadoc for seekToBeginning, > seekToEnd incomplete > - > > Key: KAFKA-3339 > URL: https://issues.apache.org/jira/browse/KAFKA-3339 > Project: Kafka > Issue Type: Improvement > Components: clients >Affects Versions: 0.9.0.1 >Reporter: Harald Kirsch > > The api documentation for seekToBeginning and seekToEnd in > org.apache.kafka.clients.consumer.KafkaConsumer these methods should remark > that all subscribed partitions are seeked if no TopicPartition is provided. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-3340) Add support for rebalance and adding concurrently records with MockConsumer
Florian Hussonnois created KAFKA-3340: - Summary: Add support for rebalance and adding concurrently records with MockConsumer Key: KAFKA-3340 URL: https://issues.apache.org/jira/browse/KAFKA-3340 Project: Kafka Issue Type: Improvement Affects Versions: 0.9.0.1 Reporter: Florian Hussonnois Priority: Minor The MockConsumer class should support adding records concurrently. This allow to implement more complex test scenarios in which records are added concurrently with the records are polled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (KAFKA-2914) Kafka Connect Source connector for HBase
[ https://issues.apache.org/jira/browse/KAFKA-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava resolved KAFKA-2914. -- Resolution: Won't Fix [~apurtell] Thanks for the link. I'm going to mark this "Won't Fix" since the Kafka project is taking a federated approach to connector development and won't be including connectors within the project itself. Hopefully HBase connector development can be picked up at the link you've provided. > Kafka Connect Source connector for HBase > - > > Key: KAFKA-2914 > URL: https://issues.apache.org/jira/browse/KAFKA-2914 > Project: Kafka > Issue Type: New Feature > Components: copycat >Reporter: Niels Basjes >Assignee: Ewen Cheslack-Postava > > In many cases I see HBase being used to persist data. > I would like to listen to the changes and process them in a streaming system > (like Apache Flink). > Feature request: A Kafka Connect "Source" that listens to the changes in a > specified HBase table. These changes are then stored in a 'standardized' form > in Kafka so that it becomes possible to process the observed changes in > near-realtime. I expect this 'standard' to be very HBase specific. > Implementation suggestion: Perhaps listening to the HBase WAL like the "HBase > Side Effects Processor" does? > https://github.com/NGDATA/hbase-indexer/tree/master/hbase-sep -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-3341) Improve error handling on invalid requests
Ismael Juma created KAFKA-3341: -- Summary: Improve error handling on invalid requests Key: KAFKA-3341 URL: https://issues.apache.org/jira/browse/KAFKA-3341 Project: Kafka Issue Type: Bug Components: network Affects Versions: 0.9.0.1 Reporter: Ismael Juma Assignee: Ismael Juma Fix For: 0.10.0.0 We should include the request id in the error message if parsing of `RequestHeader` fails and we should not try to mute the selector after closing the connection due to an error (as it causes a second exception to be thrown). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] kafka pull request: KAFKA-3341: Improve error handling on invalid ...
GitHub user ijuma opened a pull request: https://github.com/apache/kafka/pull/1017 KAFKA-3341: Improve error handling on invalid requests * Include request id when parsing of request header fails * Don't mute selector on a connection that was closed due to an error (otherwise a second exception is thrown) * Throw appropriate exception from `ApiKeys.fromId` if invalid id is passed I ran into the top two issues while trying to figure out why a connection from a producer to a broker was failing (and it made things harder than necessary). While fixing them, I noticed the third issue. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ijuma/kafka kafka-3341-improve-error-handling-invalid-requests Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1017.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1017 commit 4508da56e132b2e9b09fb03b3a1215608b539171 Author: Ismael Juma Date: 2016-03-06T16:16:48Z Throw more informative exception if wrong id is passed to `ApiKeys.forId` commit 7530e001e97d34f1d55f217617f99e264d91f119 Author: Ismael Juma Date: 2016-03-06T16:20:16Z Don't try to mute selector if it has already been closed commit 0f93c33b29ad77677bf7dda72012f15fc0346ad9 Author: Ismael Juma Date: 2016-03-06T23:57:47Z Include request id when parsing of request header fails Similar to what we do when creating the request fails. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (KAFKA-3341) Improve error handling on invalid requests
[ https://issues.apache.org/jira/browse/KAFKA-3341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182412#comment-15182412 ] ASF GitHub Bot commented on KAFKA-3341: --- GitHub user ijuma opened a pull request: https://github.com/apache/kafka/pull/1017 KAFKA-3341: Improve error handling on invalid requests * Include request id when parsing of request header fails * Don't mute selector on a connection that was closed due to an error (otherwise a second exception is thrown) * Throw appropriate exception from `ApiKeys.fromId` if invalid id is passed I ran into the top two issues while trying to figure out why a connection from a producer to a broker was failing (and it made things harder than necessary). While fixing them, I noticed the third issue. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ijuma/kafka kafka-3341-improve-error-handling-invalid-requests Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1017.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1017 commit 4508da56e132b2e9b09fb03b3a1215608b539171 Author: Ismael Juma Date: 2016-03-06T16:16:48Z Throw more informative exception if wrong id is passed to `ApiKeys.forId` commit 7530e001e97d34f1d55f217617f99e264d91f119 Author: Ismael Juma Date: 2016-03-06T16:20:16Z Don't try to mute selector if it has already been closed commit 0f93c33b29ad77677bf7dda72012f15fc0346ad9 Author: Ismael Juma Date: 2016-03-06T23:57:47Z Include request id when parsing of request header fails Similar to what we do when creating the request fails. > Improve error handling on invalid requests > -- > > Key: KAFKA-3341 > URL: https://issues.apache.org/jira/browse/KAFKA-3341 > Project: Kafka > Issue Type: Bug > Components: network >Affects Versions: 0.9.0.1 >Reporter: Ismael Juma >Assignee: Ismael Juma > Fix For: 0.10.0.0 > > > We should include the request id in the error message if parsing of > `RequestHeader` fails and we should not try to mute the selector after > closing the connection due to an error (as it causes a second exception to be > thrown). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-3341) Improve error handling on invalid requests
[ https://issues.apache.org/jira/browse/KAFKA-3341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-3341: --- Status: Patch Available (was: Open) > Improve error handling on invalid requests > -- > > Key: KAFKA-3341 > URL: https://issues.apache.org/jira/browse/KAFKA-3341 > Project: Kafka > Issue Type: Bug > Components: network >Affects Versions: 0.9.0.1 >Reporter: Ismael Juma >Assignee: Ismael Juma > Fix For: 0.10.0.0 > > > We should include the request id in the error message if parsing of > `RequestHeader` fails and we should not try to mute the selector after > closing the connection due to an error (as it causes a second exception to be > thrown). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3018) Kafka producer hangs on producer.close() call if the producer topic contains single quotes in the topic name
[ https://issues.apache.org/jira/browse/KAFKA-3018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182416#comment-15182416 ] Chi Hoang commented on KAFKA-3018: -- Coming back to this. Seems like the more appropriate design would be to split server from core, and start moving code that will be shared by both client and server to core. The suggestion to wait until the server rejects the topic means we have to wait and have more complex interactions to get the true nature of the problem. I would still prefer to have the failure occur the moment the topic is referenced, and in this case, it is during instantiation of a ProducerRecord. I'm also in favor of the server being the ultimate enforcer of the topic name. To save a few cycles, Topic can be a formal object which owns the validation, and the ProducerRecord would take a Topic object instead of a String, so the ProducerRecord constructor would not be responsible for validating the topic. Splitting out another module would probably require a KIP, so I'm not going to take another shot at this until we establish a direction. > Kafka producer hangs on producer.close() call if the producer topic contains > single quotes in the topic name > > > Key: KAFKA-3018 > URL: https://issues.apache.org/jira/browse/KAFKA-3018 > Project: Kafka > Issue Type: Bug > Components: producer >Affects Versions: 0.8.2.0 >Reporter: kanav anand >Assignee: Jun Rao > > While creating topics with quotes in the name throws a exception but if you > try to close a producer configured with a topic name with quotes the producer > hangs. > It can be easily replicated and verified by setting topic.name for a producer > with a string containing single quotes in it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3018) Kafka producer hangs on producer.close() call if the producer topic contains single quotes in the topic name
[ https://issues.apache.org/jira/browse/KAFKA-3018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182423#comment-15182423 ] Ismael Juma commented on KAFKA-3018: The path Kafka has taken is actually a bit different. There is a `common` package in the `clients` module which is shared by both the clients and the broker. One of the things to keep in mind is that newer brokers support older clients so even with shared logic, it can happen that they may not be the same due to different versions. I think everyone agrees that validation on the broker side is required either way, so it may make sense to start with that and then decide whether it's worth doing it on the client-side too. > Kafka producer hangs on producer.close() call if the producer topic contains > single quotes in the topic name > > > Key: KAFKA-3018 > URL: https://issues.apache.org/jira/browse/KAFKA-3018 > Project: Kafka > Issue Type: Bug > Components: producer >Affects Versions: 0.8.2.0 >Reporter: kanav anand >Assignee: Jun Rao > > While creating topics with quotes in the name throws a exception but if you > try to close a producer configured with a topic name with quotes the producer > hangs. > It can be easily replicated and verified by setting topic.name for a producer > with a string containing single quotes in it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3018) Kafka producer hangs on producer.close() call if the producer topic contains single quotes in the topic name
[ https://issues.apache.org/jira/browse/KAFKA-3018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182433#comment-15182433 ] Chi Hoang commented on KAFKA-3018: -- The server is already correctly validating the topic name, and the message coming from the server is also pretty clear. The problem described in this ticket revolve around how the client handles it. [~granthenke]'s original suggestion was to move the topic validation to the common client code, and I thought it was sensible and would have made things simpler. However, that meant that core would continue to depend on client, and that part was not so sensible to me. I don't know the history behind making core dependent on client - git blame shows that KAFKA-1237 introduced this relationship - so if we want to maintain this dependency chain, I would go back to Grant's original suggestion and move the logic to clients-->common. I have enough context to make that change and push a new pr. > Kafka producer hangs on producer.close() call if the producer topic contains > single quotes in the topic name > > > Key: KAFKA-3018 > URL: https://issues.apache.org/jira/browse/KAFKA-3018 > Project: Kafka > Issue Type: Bug > Components: producer >Affects Versions: 0.8.2.0 >Reporter: kanav anand >Assignee: Jun Rao > > While creating topics with quotes in the name throws a exception but if you > try to close a producer configured with a topic name with quotes the producer > hangs. > It can be easily replicated and verified by setting topic.name for a producer > with a string containing single quotes in it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3018) Kafka producer hangs on producer.close() call if the producer topic contains single quotes in the topic name
[ https://issues.apache.org/jira/browse/KAFKA-3018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182447#comment-15182447 ] Jiangjie Qin commented on KAFKA-3018: - [~chi_groupon] The situation we are trying to avoid here is that the client and server has different judgement of whether a topic is valid or not. e.g. the clients said the topic is invalid and the broker said it is valid. This will be confusing to the user. The issue we have today is that the invalid topic error is only shown in the log but not thrown as an exception to the user. Instead user thread will block. We can fix that and propagate the exception properly. But I think we should avoid validating the topic on both client an server side because the code might be different as [~ijuma] mentioned. > Kafka producer hangs on producer.close() call if the producer topic contains > single quotes in the topic name > > > Key: KAFKA-3018 > URL: https://issues.apache.org/jira/browse/KAFKA-3018 > Project: Kafka > Issue Type: Bug > Components: producer >Affects Versions: 0.8.2.0 >Reporter: kanav anand >Assignee: Jun Rao > > While creating topics with quotes in the name throws a exception but if you > try to close a producer configured with a topic name with quotes the producer > hangs. > It can be easily replicated and verified by setting topic.name for a producer > with a string containing single quotes in it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] kafka pull request: KAFKA-2960: Clear purgatory for partitions bef...
GitHub user becketqin opened a pull request: https://github.com/apache/kafka/pull/1018 KAFKA-2960: Clear purgatory for partitions before becoming follower You can merge this pull request into a Git repository by running: $ git pull https://github.com/becketqin/kafka KAFKA-2960 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1018.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1018 commit 6ee590bc8f65217227c8bda98644dce35ed0d701 Author: Jiangjie Qin Date: 2016-03-07T04:04:45Z KAFKA-2960: Clear purgatory for partition before becoming follower --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (KAFKA-2960) DelayedProduce may cause message lose during repeatly leader change
[ https://issues.apache.org/jira/browse/KAFKA-2960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiangjie Qin updated KAFKA-2960: Status: Patch Available (was: Open) > DelayedProduce may cause message lose during repeatly leader change > --- > > Key: KAFKA-2960 > URL: https://issues.apache.org/jira/browse/KAFKA-2960 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.9.0.0 >Reporter: Xing Huang >Assignee: Jiangjie Qin > Fix For: 0.10.0.0 > > > related to #KAFKA-1148 > When a leader replica became follower then leader again, it may truncated its > log as follower. But the second time it became leader, its ISR may shrink and > if at this moment new messages were appended, the DelayedProduce generated > when it was leader the first time may be satisfied, and the client will > receive a response with no error. But, actually the messages were lost. > We simulated this scene, which proved the message lose could happen. And it > seems to be the reason for a data lose recently happened to us according to > broker logs and client logs. > I think we should check the leader epoch when send a response, or satisfy > DelayedProduce when leader change as described in #KAFKA-1148. > And we may need an new error code to inform the producer about this error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2960) DelayedProduce may cause message lose during repeatly leader change
[ https://issues.apache.org/jira/browse/KAFKA-2960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182552#comment-15182552 ] ASF GitHub Bot commented on KAFKA-2960: --- GitHub user becketqin opened a pull request: https://github.com/apache/kafka/pull/1018 KAFKA-2960: Clear purgatory for partitions before becoming follower You can merge this pull request into a Git repository by running: $ git pull https://github.com/becketqin/kafka KAFKA-2960 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1018.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1018 commit 6ee590bc8f65217227c8bda98644dce35ed0d701 Author: Jiangjie Qin Date: 2016-03-07T04:04:45Z KAFKA-2960: Clear purgatory for partition before becoming follower > DelayedProduce may cause message lose during repeatly leader change > --- > > Key: KAFKA-2960 > URL: https://issues.apache.org/jira/browse/KAFKA-2960 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.9.0.0 >Reporter: Xing Huang >Assignee: Jiangjie Qin > Fix For: 0.10.0.0 > > > related to #KAFKA-1148 > When a leader replica became follower then leader again, it may truncated its > log as follower. But the second time it became leader, its ISR may shrink and > if at this moment new messages were appended, the DelayedProduce generated > when it was leader the first time may be satisfied, and the client will > receive a response with no error. But, actually the messages were lost. > We simulated this scene, which proved the message lose could happen. And it > seems to be the reason for a data lose recently happened to us according to > broker logs and client logs. > I think we should check the leader epoch when send a response, or satisfy > DelayedProduce when leader change as described in #KAFKA-1148. > And we may need an new error code to inform the producer about this error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3339) org.apache.kafka.clients.consumer.KafkaConsumer javadoc for seekToBeginning, seekToEnd incomplete
[
https://issues.apache.org/jira/browse/KAFKA-3339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182722#comment-15182722
]
Harald Kirsch commented on KAFKA-3339:
--
[~singhashish] A bit too much for a pull request. For both methods I would just
add something along the lines of:
If no {@code TopicPartition} is provided, all topic/partition pairs returned by
{@link #assignment} are repositioned.
> org.apache.kafka.clients.consumer.KafkaConsumer javadoc for seekToBeginning,
> seekToEnd incomplete
> -
>
> Key: KAFKA-3339
> URL: https://issues.apache.org/jira/browse/KAFKA-3339
> Project: Kafka
> Issue Type: Improvement
> Components: clients
>Affects Versions: 0.9.0.1
>Reporter: Harald Kirsch
>
> The api documentation for seekToBeginning and seekToEnd in
> org.apache.kafka.clients.consumer.KafkaConsumer these methods should remark
> that all subscribed partitions are seeked if no TopicPartition is provided.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
