[jira] [Commented] (KAFKA-3615) Exclude test jars in CLASSPATH of kafka-run-class.sh
[ https://issues.apache.org/jira/browse/KAFKA-3615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255603#comment-15255603 ] Eno Thereska commented on KAFKA-3615: - [~liquanpei] Thanks for the JIRA. Could you provide some more description on why this is needed? Aren't there cases when one would want to run a Kafka test using kafka-run-class.sh? > Exclude test jars in CLASSPATH of kafka-run-class.sh > > > Key: KAFKA-3615 > URL: https://issues.apache.org/jira/browse/KAFKA-3615 > Project: Kafka > Issue Type: Improvement > Components: admin, build >Affects Versions: 0.10.0.0 >Reporter: Liquan Pei >Assignee: Liquan Pei > Labels: newbie > Fix For: 0.10.0.0 > > Original Estimate: 24h > Remaining Estimate: 24h > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3615) Exclude test jars in CLASSPATH of kafka-run-class.sh
[ https://issues.apache.org/jira/browse/KAFKA-3615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255611#comment-15255611 ] Ismael Juma commented on KAFKA-3615: This is needed because connect scans the classpath and I am assuming that the test jars are causing that to take a lot longer. I think it makes sense for `kafka-run-class.sh` not to include test jars by default. One could add an environment variable to change that behaviour in the rare cases where one wants to launch a class that is in the test folder. > Exclude test jars in CLASSPATH of kafka-run-class.sh > > > Key: KAFKA-3615 > URL: https://issues.apache.org/jira/browse/KAFKA-3615 > Project: Kafka > Issue Type: Improvement > Components: admin, build >Affects Versions: 0.10.0.0 >Reporter: Liquan Pei >Assignee: Liquan Pei > Labels: newbie > Fix For: 0.10.0.0 > > Original Estimate: 24h > Remaining Estimate: 24h > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-3616) Make kafka producers/consumers injectable for KafkaStreams
Yuto Kawamura created KAFKA-3616:
Summary: Make kafka producers/consumers injectable for KafkaStreams
Key: KAFKA-3616
URL: https://issues.apache.org/jira/browse/KAFKA-3616
Project: Kafka
Issue Type: Improvement
Components: streams
Affects Versions: 0.10.1.0
Reporter: Yuto Kawamura
Assignee: Yuto Kawamura
While playing with Kafka Streams I found that there's some cases that we want
to take control of kafka producer/consumers instantiation inside the
StreamThread.
Most significant case is that we have our own impl of kafka Producer which was
built to provide much reliable message delivery but there's no way to inject
our own instance into StreamTask ATM.
Another example is that we wanna observe the result of {{producer.send()}} that
is done inside the RecordCollector. We can provide our own Callback instance
but there's no way to inject that callback to RecordCollector again.
Here I'd like to suggest KafkaStreams giving an interface to inject these
clients. I considered various approaches to do this like passing them through
constructor or make instantiation methods overridable but eventually tried to
simply intorude another argument to the KafkaStreams constructor which is
responsible for supplying client instances.
Incomplete PR will be filled up to show changeset in my mind, so please give me
feedbacks. Will follow-up PR quickly if I get positive feedback.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] kafka pull request: KAFKA-3616: Make kafka producers/consumers inj...
GitHub user kawamuray opened a pull request: https://github.com/apache/kafka/pull/1264 KAFKA-3616: Make kafka producers/consumers injectable for KafkaStreams Ticket: https://issues.apache.org/jira/browse/KAFKA-3616 WIP. Just to show my mind. Will follow-up after positive feedback. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kawamuray/kafka kafka-3616-inject-clients Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1264.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1264 commit 8e608baa6670097d1bee896c9f82459a9aee4612 Author: Yuto Kawamura Date: 2016-04-24T14:54:53Z KAFKA-3616: Make kafka producers/consumers injectable for KafkaStreams --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (KAFKA-3616) Make kafka producers/consumers injectable for KafkaStreams
[
https://issues.apache.org/jira/browse/KAFKA-3616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuto Kawamura updated KAFKA-3616:
-
Status: Patch Available (was: Open)
> Make kafka producers/consumers injectable for KafkaStreams
> --
>
> Key: KAFKA-3616
> URL: https://issues.apache.org/jira/browse/KAFKA-3616
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Yuto Kawamura
>Assignee: Yuto Kawamura
>
> While playing with Kafka Streams I found that there's some cases that we want
> to take control of kafka producer/consumers instantiation inside the
> StreamThread.
> Most significant case is that we have our own impl of kafka Producer which
> was built to provide much reliable message delivery but there's no way to
> inject our own instance into StreamTask ATM.
> Another example is that we wanna observe the result of {{producer.send()}}
> that is done inside the RecordCollector. We can provide our own Callback
> instance but there's no way to inject that callback to RecordCollector again.
> Here I'd like to suggest KafkaStreams giving an interface to inject these
> clients. I considered various approaches to do this like passing them through
> constructor or make instantiation methods overridable but eventually tried to
> simply intorude another argument to the KafkaStreams constructor which is
> responsible for supplying client instances.
> Incomplete PR will be filled up to show changeset in my mind, so please give
> me feedbacks. Will follow-up PR quickly if I get positive feedback.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-3616) Make kafka producers/consumers injectable for KafkaStreams
[
https://issues.apache.org/jira/browse/KAFKA-3616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255615#comment-15255615
]
Yuto Kawamura commented on KAFKA-3616:
--
https://github.com/apache/kafka/pull/1264
[~guozhang] PTAL if you don't mind :)
> Make kafka producers/consumers injectable for KafkaStreams
> --
>
> Key: KAFKA-3616
> URL: https://issues.apache.org/jira/browse/KAFKA-3616
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Yuto Kawamura
>Assignee: Yuto Kawamura
>
> While playing with Kafka Streams I found that there's some cases that we want
> to take control of kafka producer/consumers instantiation inside the
> StreamThread.
> Most significant case is that we have our own impl of kafka Producer which
> was built to provide much reliable message delivery but there's no way to
> inject our own instance into StreamTask ATM.
> Another example is that we wanna observe the result of {{producer.send()}}
> that is done inside the RecordCollector. We can provide our own Callback
> instance but there's no way to inject that callback to RecordCollector again.
> Here I'd like to suggest KafkaStreams giving an interface to inject these
> clients. I considered various approaches to do this like passing them through
> constructor or make instantiation methods overridable but eventually tried to
> simply intorude another argument to the KafkaStreams constructor which is
> responsible for supplying client instances.
> Incomplete PR will be filled up to show changeset in my mind, so please give
> me feedbacks. Will follow-up PR quickly if I get positive feedback.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-3606) Traverse CLASSPATH during herder start to list connectors
[ https://issues.apache.org/jira/browse/KAFKA-3606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish K Singh updated KAFKA-3606: -- Fix Version/s: (was: 0.10.1.0) 0.10.0.0 > Traverse CLASSPATH during herder start to list connectors > - > > Key: KAFKA-3606 > URL: https://issues.apache.org/jira/browse/KAFKA-3606 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Affects Versions: 0.10.0.0 >Reporter: Liquan Pei >Assignee: Liquan Pei >Priority: Blocker > Fix For: 0.10.0.0 > > Original Estimate: 24h > Remaining Estimate: 24h > > List connectors API requires CLASSPATH traversal, which can takes up to 30s > to return available connectors. To work around this, we traverse the > CLASSPATH when staring herder and cache the result. Also we should guard > against concurrent CLASSPATH traversal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] kafka pull request: KAFKA-3128; Add metrics for ZooKeeper events z...
GitHub user ijuma opened a pull request: https://github.com/apache/kafka/pull/1265 KAFKA-3128; Add metrics for ZooKeeper events zookeeper metrics You can merge this pull request into a Git repository by running: $ git pull https://github.com/ijuma/kafka kafka-3128-zookeeper-metrics Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1265.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1265 commit fcd5dd7fad5bd6d21787e1143cc297205d38c333 Author: Ismael Juma Date: 2016-04-24T20:06:59Z Introduce metrics for ZK session expire listener commit edfb9601fa11b892bfb77813fe09394441e58fbf Author: Ismael Juma Date: 2016-04-24T20:07:26Z Use `timeMs` instead of `time.milliseconds` in `Sensor.record` commit 9f7c7b34bc43f3fe891c90ab614f32c6cbd35f82 Author: Ismael Juma Date: 2016-04-24T20:14:48Z Several test clean-ups --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (KAFKA-3128) Add metrics for ZooKeeper events
[ https://issues.apache.org/jira/browse/KAFKA-3128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-3128: --- Reviewer: Jun Rao Status: Patch Available (was: Open) > Add metrics for ZooKeeper events > > > Key: KAFKA-3128 > URL: https://issues.apache.org/jira/browse/KAFKA-3128 > Project: Kafka > Issue Type: Improvement > Components: core, zkclient >Reporter: Flavio Junqueira >Assignee: Ismael Juma > Fix For: 0.10.0.0 > > > It would be useful to report via Kafka metrics the number of ZK event > notifications, such as connection loss events, session expiration events, > etc., as a way of spotting potential issues with the communication with the > ZK ensemble. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3599) Minor code cleanup
[
https://issues.apache.org/jira/browse/KAFKA-3599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255733#comment-15255733
]
Matthias J. Sax commented on KAFKA-3599:
Removing the `equals()` issue from this JIRA because this is covered by
[~miguno] PRs for KAFKA-3613 already.
> Minor code cleanup
> --
>
> Key: KAFKA-3599
> URL: https://issues.apache.org/jira/browse/KAFKA-3599
> Project: Kafka
> Issue Type: Sub-task
> Components: streams
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
>Priority: Minor
> Labels: api
>
> - Clean up {{Window.equals()}} and {{Window.equalsTo()}}. Both do the same.
> - remove {{WindowStoreUtils}} from public API, ie, move to sub-package
> {{internals}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] kafka pull request: KAKFA-3599: Minor code cleanup
GitHub user mjsax opened a pull request: https://github.com/apache/kafka/pull/1266 KAKFA-3599: Minor code cleanup You can merge this pull request into a Git repository by running: $ git pull https://github.com/mjsax/kafka kafka-3599-minorCodeCleanup Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1266.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1266 commit e68cfde7f52d8c86b89b8425143f5e042c223313 Author: Matthias J. Sax Date: 2016-04-24T22:02:55Z KAKFA-3599: Minor code cleanup --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (KAFKA-3599) Minor code cleanup
[
https://issues.apache.org/jira/browse/KAFKA-3599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-3599:
---
Status: Patch Available (was: In Progress)
> Minor code cleanup
> --
>
> Key: KAFKA-3599
> URL: https://issues.apache.org/jira/browse/KAFKA-3599
> Project: Kafka
> Issue Type: Sub-task
> Components: streams
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
>Priority: Minor
> Labels: api
>
> - remove {{WindowStoreUtils}} from public API, ie, move to sub-package
> {{internals}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-3599) Minor code cleanup
[
https://issues.apache.org/jira/browse/KAFKA-3599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-3599:
---
Description:
- remove {{WindowStoreUtils}} from public API, ie, move to sub-package
{{internals}}
was:
- Clean up {{Window.equals()}} and {{Window.equalsTo()}}. Both do the same.
- remove {{WindowStoreUtils}} from public API, ie, move to sub-package
{{internals}}
> Minor code cleanup
> --
>
> Key: KAFKA-3599
> URL: https://issues.apache.org/jira/browse/KAFKA-3599
> Project: Kafka
> Issue Type: Sub-task
> Components: streams
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
>Priority: Minor
> Labels: api
>
> - remove {{WindowStoreUtils}} from public API, ie, move to sub-package
> {{internals}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-3599) Move WindowStoreUtils to package "internals"
[
https://issues.apache.org/jira/browse/KAFKA-3599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-3599:
---
Summary: Move WindowStoreUtils to package "internals" (was: Minor code
cleanup)
> Move WindowStoreUtils to package "internals"
>
>
> Key: KAFKA-3599
> URL: https://issues.apache.org/jira/browse/KAFKA-3599
> Project: Kafka
> Issue Type: Sub-task
> Components: streams
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
>Priority: Minor
> Labels: api
>
> - remove {{WindowStoreUtils}} from public API, ie, move to sub-package
> {{internals}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-3615) Exclude test jars in CLASSPATH of kafka-run-class.sh
[ https://issues.apache.org/jira/browse/KAFKA-3615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255747#comment-15255747 ] Grant Henke commented on KAFKA-3615: This is only true when running from source right? Not from an actual release where all the jars are under /lib. > Exclude test jars in CLASSPATH of kafka-run-class.sh > > > Key: KAFKA-3615 > URL: https://issues.apache.org/jira/browse/KAFKA-3615 > Project: Kafka > Issue Type: Improvement > Components: admin, build >Affects Versions: 0.10.0.0 >Reporter: Liquan Pei >Assignee: Liquan Pei > Labels: newbie > Fix For: 0.10.0.0 > > Original Estimate: 24h > Remaining Estimate: 24h > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3615) Exclude test jars in CLASSPATH of kafka-run-class.sh
[ https://issues.apache.org/jira/browse/KAFKA-3615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255749#comment-15255749 ] Ismael Juma commented on KAFKA-3615: Yeah, I think so although I haven't checked. > Exclude test jars in CLASSPATH of kafka-run-class.sh > > > Key: KAFKA-3615 > URL: https://issues.apache.org/jira/browse/KAFKA-3615 > Project: Kafka > Issue Type: Improvement > Components: admin, build >Affects Versions: 0.10.0.0 >Reporter: Liquan Pei >Assignee: Liquan Pei > Labels: newbie > Fix For: 0.10.0.0 > > Original Estimate: 24h > Remaining Estimate: 24h > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-3616) Make kafka producers/consumers injectable for KafkaStreams
[
https://issues.apache.org/jira/browse/KAFKA-3616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-3616:
-
Reviewer: Guozhang Wang
> Make kafka producers/consumers injectable for KafkaStreams
> --
>
> Key: KAFKA-3616
> URL: https://issues.apache.org/jira/browse/KAFKA-3616
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Yuto Kawamura
>Assignee: Yuto Kawamura
>
> While playing with Kafka Streams I found that there's some cases that we want
> to take control of kafka producer/consumers instantiation inside the
> StreamThread.
> Most significant case is that we have our own impl of kafka Producer which
> was built to provide much reliable message delivery but there's no way to
> inject our own instance into StreamTask ATM.
> Another example is that we wanna observe the result of {{producer.send()}}
> that is done inside the RecordCollector. We can provide our own Callback
> instance but there's no way to inject that callback to RecordCollector again.
> Here I'd like to suggest KafkaStreams giving an interface to inject these
> clients. I considered various approaches to do this like passing them through
> constructor or make instantiation methods overridable but eventually tried to
> simply intorude another argument to the KafkaStreams constructor which is
> responsible for supplying client instances.
> Incomplete PR will be filled up to show changeset in my mind, so please give
> me feedbacks. Will follow-up PR quickly if I get positive feedback.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-3616) Make kafka producers/consumers injectable for KafkaStreams
[
https://issues.apache.org/jira/browse/KAFKA-3616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-3616:
-
Labels: api (was: )
> Make kafka producers/consumers injectable for KafkaStreams
> --
>
> Key: KAFKA-3616
> URL: https://issues.apache.org/jira/browse/KAFKA-3616
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Yuto Kawamura
>Assignee: Yuto Kawamura
> Labels: api
> Fix For: 0.10.1.0
>
>
> While playing with Kafka Streams I found that there's some cases that we want
> to take control of kafka producer/consumers instantiation inside the
> StreamThread.
> Most significant case is that we have our own impl of kafka Producer which
> was built to provide much reliable message delivery but there's no way to
> inject our own instance into StreamTask ATM.
> Another example is that we wanna observe the result of {{producer.send()}}
> that is done inside the RecordCollector. We can provide our own Callback
> instance but there's no way to inject that callback to RecordCollector again.
> Here I'd like to suggest KafkaStreams giving an interface to inject these
> clients. I considered various approaches to do this like passing them through
> constructor or make instantiation methods overridable but eventually tried to
> simply intorude another argument to the KafkaStreams constructor which is
> responsible for supplying client instances.
> Incomplete PR will be filled up to show changeset in my mind, so please give
> me feedbacks. Will follow-up PR quickly if I get positive feedback.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-3616) Make kafka producers/consumers injectable for KafkaStreams
[
https://issues.apache.org/jira/browse/KAFKA-3616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang updated KAFKA-3616:
-
Fix Version/s: 0.10.1.0
> Make kafka producers/consumers injectable for KafkaStreams
> --
>
> Key: KAFKA-3616
> URL: https://issues.apache.org/jira/browse/KAFKA-3616
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Yuto Kawamura
>Assignee: Yuto Kawamura
> Labels: api
> Fix For: 0.10.1.0
>
>
> While playing with Kafka Streams I found that there's some cases that we want
> to take control of kafka producer/consumers instantiation inside the
> StreamThread.
> Most significant case is that we have our own impl of kafka Producer which
> was built to provide much reliable message delivery but there's no way to
> inject our own instance into StreamTask ATM.
> Another example is that we wanna observe the result of {{producer.send()}}
> that is done inside the RecordCollector. We can provide our own Callback
> instance but there's no way to inject that callback to RecordCollector again.
> Here I'd like to suggest KafkaStreams giving an interface to inject these
> clients. I considered various approaches to do this like passing them through
> constructor or make instantiation methods overridable but eventually tried to
> simply intorude another argument to the KafkaStreams constructor which is
> responsible for supplying client instances.
> Incomplete PR will be filled up to show changeset in my mind, so please give
> me feedbacks. Will follow-up PR quickly if I get positive feedback.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-3616) Make kafka producers/consumers injectable for KafkaStreams
[
https://issues.apache.org/jira/browse/KAFKA-3616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255755#comment-15255755
]
Guozhang Wang commented on KAFKA-3616:
--
Will review.
This will also benefit our unit tests, where mock producers / consumer can be
used.
> Make kafka producers/consumers injectable for KafkaStreams
> --
>
> Key: KAFKA-3616
> URL: https://issues.apache.org/jira/browse/KAFKA-3616
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Yuto Kawamura
>Assignee: Yuto Kawamura
> Labels: api
> Fix For: 0.10.1.0
>
>
> While playing with Kafka Streams I found that there's some cases that we want
> to take control of kafka producer/consumers instantiation inside the
> StreamThread.
> Most significant case is that we have our own impl of kafka Producer which
> was built to provide much reliable message delivery but there's no way to
> inject our own instance into StreamTask ATM.
> Another example is that we wanna observe the result of {{producer.send()}}
> that is done inside the RecordCollector. We can provide our own Callback
> instance but there's no way to inject that callback to RecordCollector again.
> Here I'd like to suggest KafkaStreams giving an interface to inject these
> clients. I considered various approaches to do this like passing them through
> constructor or make instantiation methods overridable but eventually tried to
> simply intorude another argument to the KafkaStreams constructor which is
> responsible for supplying client instances.
> Incomplete PR will be filled up to show changeset in my mind, so please give
> me feedbacks. Will follow-up PR quickly if I get positive feedback.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: [DISCUSS] KIP-43: Kafka SASL enhancements
Rajini, the KIP wiki is a bit unclear on the protocol changes. Could you document the proposed Kafka protocol requests&responses in the standard format (as on "A guide to the Kafka protocol"). This information should also be added to that page when the KIP is accepted. I think it would also be good to clarify what SASL handshake means, if that is the Kafka-leved SASL mechanism handshake or the opaque SASL data handshake performed by the SASL libraries. Thanks, Magnus 2016-04-19 17:20 GMT-07:00 Jun Rao : > Just to close the loop on this. Discussed with Magnus offline on how KIP-43 > and KIP-35 can play together. We agreed upon the following proposal. > > On a SASL port, > > client sends: > > ApiVersionRequest (optional), SaslHandshakeRequest, SASL tokens (size > delimited as being done now), regular api requests > > client receives: > > ApiVersionResponse (optional), SaslHandshakeResponse, SASL tokens (size > delimited as being done now), regular api responses > > The format of SaslHandshakeRequest is what's currently described in > KIP-43. There > will be some minor tweaks on ApiVersionResponse, which Magnus will follow > up in the KIP-35 thread itself. > > Thanks, > > Jun > > > On Wed, Apr 13, 2016 at 5:59 AM, Rajini Sivaram < > rajinisiva...@googlemail.com> wrote: > > > I have updated the PR (https://github.com/apache/kafka/pull/812) and > > KIP-43 > > to use standard Kafka format for the new request/response added by > KIP-43. > > I haven't changed the overall structure of the Java code. Feedback is > > appreciated. > > > > Thanks, > > > > Rajini > > > > On Tue, Apr 12, 2016 at 3:52 PM, Ismael Juma wrote: > > > > > Hi Jun, > > > > > > Comments inline. > > > > > > On Mon, Apr 11, 2016 at 1:57 AM, Jun Rao wrote: > > > > > > > Yes, that should be fine right? Since the new api key will start with > > a 0 > > > > byte, it actually guarantees that it's different from 0x60 (1st byte > in > > > the > > > > old protocol) even if we change the request version id in the future. > > > > > > > > > Yes, this is true. Also, the GSS API library will throw an exception if > > the > > > first byte is not 0x60 (for the case where newer clients connect to > older > > > brokers): > > > > > > > > > > > > https://github.com/frohoff/jdk8u-dev-jdk/blob/master/src/share/classes/sun/security/jgss/GSSHeader.java#L97 > > > > > > > > > And the DEFECTIVE_TOKEN status code is specified in both RFC 2743[1] > and > > > RFC 5653[2]. Section 3.1 of RFC 2743 specifies that the token tag > > consists > > > of the following elements, in order: > > > > > > 1. 0x60 -- Tag for [APPLICATION 0] SEQUENCE; indicates that > > > -- constructed form, definite length encoding follows. > > > > > > 2. Token length octets ... > > > > > > > > > Ismael > > > > > > [1] Generic Security Service Application Program Interface Version 2, > > > Update 1: https://tools.ietf.org/html/rfc2743 > > > [2] Generic Security Service API Version 2: Java Bindings Update: > > > https://tools.ietf.org/html/rfc5653 > > > > > > Ismael > > > > > > > > > > > -- > > Regards, > > > > Rajini > > >
[jira] [Commented] (KAFKA-3616) Make kafka producers/consumers injectable for KafkaStreams
[
https://issues.apache.org/jira/browse/KAFKA-3616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255857#comment-15255857
]
Guozhang Wang commented on KAFKA-3616:
--
About "reliable message delivery", out of curiosity could you elaborate a bit
on what's the flaw on the current clients, and how you fix it in your own
implementation?
About the callback in producer, recently an "interceptor" interface is added to
Kafka and will be released soon in 0.10.0.0, would that meet your needs?
https://cwiki.apache.org/confluence/display/KAFKA/KIP-42%3A+Add+Producer+and+Consumer+Interceptors
> Make kafka producers/consumers injectable for KafkaStreams
> --
>
> Key: KAFKA-3616
> URL: https://issues.apache.org/jira/browse/KAFKA-3616
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Yuto Kawamura
>Assignee: Yuto Kawamura
> Labels: api
> Fix For: 0.10.1.0
>
>
> While playing with Kafka Streams I found that there's some cases that we want
> to take control of kafka producer/consumers instantiation inside the
> StreamThread.
> Most significant case is that we have our own impl of kafka Producer which
> was built to provide much reliable message delivery but there's no way to
> inject our own instance into StreamTask ATM.
> Another example is that we wanna observe the result of {{producer.send()}}
> that is done inside the RecordCollector. We can provide our own Callback
> instance but there's no way to inject that callback to RecordCollector again.
> Here I'd like to suggest KafkaStreams giving an interface to inject these
> clients. I considered various approaches to do this like passing them through
> constructor or make instantiation methods overridable but eventually tried to
> simply intorude another argument to the KafkaStreams constructor which is
> responsible for supplying client instances.
> Incomplete PR will be filled up to show changeset in my mind, so please give
> me feedbacks. Will follow-up PR quickly if I get positive feedback.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-3615) Exclude test jars in CLASSPATH of kafka-run-class.sh
[ https://issues.apache.org/jira/browse/KAFKA-3615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255889#comment-15255889 ] Liquan Pei commented on KAFKA-3615: --- [~enothereska] Kafka Connect provides a REST API to list installed connectors. This involves traversing the CLASSPATH which is constructed in kafka-run-class.sh. In case that test jars are included in the CLASSPATH, some unwanted connectors purely for testing will be returned in the result of the API call. Also, this will make the API call take longer. [~granthenke] Yeah, this is for running from source. > Exclude test jars in CLASSPATH of kafka-run-class.sh > > > Key: KAFKA-3615 > URL: https://issues.apache.org/jira/browse/KAFKA-3615 > Project: Kafka > Issue Type: Improvement > Components: admin, build >Affects Versions: 0.10.0.0 >Reporter: Liquan Pei >Assignee: Liquan Pei > Labels: newbie > Fix For: 0.10.0.0 > > Original Estimate: 24h > Remaining Estimate: 24h > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
wiki privs
I'd like to create a new KIP page on the wiki, but I can't figure out how to create a new page or edit an existing page. Do I need certain privileges on my user account to be able to do this? Thanks, -Dana
