[
https://issues.apache.org/jira/browse/KAFKA-3616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuto Kawamura updated KAFKA-3616:
---------------------------------
Comment: was deleted
(was: > About "reliable message delivery", out of curiosity could you elaborate
a bit on what's the flaw on the current clients, and how you fix it in your own
implementation?
There's no problem on producer impl. It's about kafka cluster availability.
Basically our cluster doing pretty well but there's some possibility of broker
fail due to any reasons like network glitches, traffic bursting, or human error
during maintenance and actually there were all these cases happened.
We've solved these kind of problem by some solutions, like introducing another
cluster and transparently failover messages when the primary cluster fails. It
worked pretty well for topics that are not required to be provided with keeping
order but quite important and high throughput, but might not good to use with
KafkaStreams as it requires to manage multiple clusters.
Another solution is to buffer produced messages locally with persist them to
disk so it would be able to utilize with Streams.
> About the callback in producer, recently an "interceptor" interface is added
> to Kafka and will be released soon in 0.10.0.0, would that meet your needs?
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-42%3A+Add+Producer+and+Consumer+Interceptors
Oh I missed this great feature! Definitely it solves part of my needs so should
be utilized. Still I have another request to the Streams; allow processor to
supply callback for forwarded messages but this should be discussed in another
ticket. Will issue one.)
> Make kafka producers/consumers injectable for KafkaStreams
> ----------------------------------------------------------
>
> Key: KAFKA-3616
> URL: https://issues.apache.org/jira/browse/KAFKA-3616
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Affects Versions: 0.10.1.0
> Reporter: Yuto Kawamura
> Assignee: Yuto Kawamura
> Labels: api
> Fix For: 0.10.1.0
>
>
> While playing with Kafka Streams I found that there's some cases that we want
> to take control of kafka producer/consumers instantiation inside the
> StreamThread.
> Most significant case is that we have our own impl of kafka Producer which
> was built to provide much reliable message delivery but there's no way to
> inject our own instance into StreamTask ATM.
> Another example is that we wanna observe the result of {{producer.send()}}
> that is done inside the RecordCollector. We can provide our own Callback
> instance but there's no way to inject that callback to RecordCollector again.
> Here I'd like to suggest KafkaStreams giving an interface to inject these
> clients. I considered various approaches to do this like passing them through
> constructor or make instantiation methods overridable but eventually tried to
> simply intorude another argument to the KafkaStreams constructor which is
> responsible for supplying client instances.
> Incomplete PR will be filled up to show changeset in my mind, so please give
> me feedbacks. Will follow-up PR quickly if I get positive feedback.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
