Hi folks,
I need to consume from and produce to a topic. I have my own data pipeline
to process the data.
So I was wondering beside the stores and StreamDSL what does Kafka Streams
brings to the table
that might be useful to me ?
Sameer when you say that you need to "consume from and produce to a topic"
to me that seems like a good fit for Kafka Streams. Streaming your data
out of Kafka for a transform and back in has some fundamental costs and
operational challenges involved. Are the events in your stream stateless?
If i
Thanks Jacob. I'm using kafka as a distributed queue and my data pipeline
is a several components which connected together via a stream abstraction.
each component has a input and output stream. Basically the source of
this pipeline can be anything and the output can be anything to.
On Tue, Mar 13
If you are augmenting streaming data with dimensional data, or if you can
transform your data with a map, filter or join operation Streams will be a
good option. If your system is stateless and the transformations are not
interdependent, you may want to look into using one of the queue
technologie
"If your system is stateless and the transformations are not interdependent"
then I would just look at using Kafka Connect's Single Message Transform
(SMT) feature.
-hans
/**
* Hans Jespersen, Director Systems Engineering, Confluent Inc.
* h...@confluent.io (650)924-2670
*/
On Tue, Mar 13, 20
Hello Kafka users, developers and client-developers,
This is the third candidate for release of Apache Kafka 1.1.0.
This is minor version release of Apache Kakfa. It Includes 29 new KIPs.
Please see the release plan for more details:
https://cwiki.apache.org/confluence/pages/viewpage.action?page
Hi Andrew,
It seems the throughput of the new cluster is smaller than that of the old
cluster. And for this reason MirrorMaker cannot send messages fast enough
(i.e. they expire). I recommend comparing the configurations.
For the hanging MirrorMaker instances, I think looking at stack dumps would
Hm, the hardware should be mostly beefier in this new cluster, but there
are a couple of differences: Mainly we are now using RAID instead of JBOD,
and our high volume producers (that 150K / second I mentioned) use TLS.
Also, other producers seem have no problems, it is only MirrorMaker. Even
s
Beside state and DSL Kafka Stream also allow for fully fault-tolerant,
salable, and elastic deployment.
-Matthias
On 3/13/18 10:55 AM, Hans Jespersen wrote:
> "If your system is stateless and the transformations are not interdependent"
> then I would just look at using Kafka Connect's Single Mes
Hi Alexander,
I don't (yet) have any real suggestions for what you are trying to
achieve, but I'd be interested to understand if you looked at the
alternative forms of authentication instead of SSL. Namely Kerberos
which is already available and delegation tokens which will be
available in 1.1
>Fr
Hi Damian,
Given release plan link in earlier mail is about 1.0 release. You may want
to replace that with 1.1.0 release plan link[1].
1 -
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75957546
Thanks,
Satish.
On Wed, Mar 14, 2018 at 12:47 AM, Damian Guy wrote:
> Hello Kafka
I agree with Sönke that kerberos is a better choice here. SSL and JKS is
probably NOT the right choice of this short lived (48 hours) scenario.
Typically certs are valid in order of years.
On Tue, Mar 13, 2018 at 7:21 PM, Sönke Liebau <
soenke.lie...@opencore.com.invalid> wrote:
> Hi Alexander,
>
Hi Damian,
Thanks for starting vote thread for 1.1.0 release.
There may be a typo on the tag to be voted upon for this release candidate.
I guess it should be https://github.com/apache/kafka/tree/1.1.0-rc2 instead
of https://github.com/apache/kafka/tree/1.1.0-rc.
On Wed, Mar 14, 2018 at 8:27 AM,
13 matches
Mail list logo
