many Kafka-specific metrics, and I'm
not sure if this is the best approach in general.
If anyone has experience with this setup or can recommend a better way to
monitor Kafka Connect in Kubernetes using OpenTelemetry, I'd really appreciate
your suggestions.
[1] https://opentelemetry.io
Are there any garbage collection suggestions specific to Kafka or are the
standard java suggestions good?
This e-mail may contain information that is privileged or confidential. If you
are not the intended recipient, please delete the e-mail and any attachments
and notify us immediately.
Hi Folks,
Canarying traffic is an excellent way of reducing the impact when
releasing a new release with a bug. Such canarying is somewhat easier
with a few queueing backends like sqs & redis. In sqs for example each
application container/instance of canary can self regulate how much
throughput th
Hi,
I'm just working through the Quickstart page.
https://kafka.apache.org/quickstart
As I'm working on Windows, I use MinGW to emulate Linux.
1) On MinGW, I get this error:
me@computer MINGW64 /c/java/kafka_2.12
$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
log4j:ER
o the DR issues?
>
> Thank you for your time
>
> Henning Røigaard-Petersen
>
> -Original Message-
> From: Ryanne Dolan
> Sent: 3. september 2018 20:27
> To: users@kafka.apache.org
> Subject: Re: Official Kafka Disaster Recovery is insufficient - Suggestions
>
gt; not using Kafka as Source of Truth due to the DR issues?
>
> Thank you for your time
>
> Henning Røigaard-Petersen
>
> -Original Message-
> From: Ryanne Dolan
> Sent: 3. september 2018 20:27
> To: users@kafka.apache.org
> Subject: Re: Official Kafka Disas
yanne Dolan
Sent: 3. september 2018 20:27
To: users@kafka.apache.org
Subject: Re: Official Kafka Disaster Recovery is insufficient - Suggestions
needed
Sorry to have misspelled your name Henning.
On Mon, Sep 3, 2018, 1:26 PM Ryanne Dolan wrote:
> Hanning,
>
> In mission-critical (a
Sorry to have misspelled your name Henning.
On Mon, Sep 3, 2018, 1:26 PM Ryanne Dolan wrote:
> Hanning,
>
> In mission-critical (and indeed GDPR-related) applications, I've ETL'd
> Kafka to a secondary store e.g. HDFS, and built tooling around recovering
> state back into Kafka. I've had situati
Hanning,
In mission-critical (and indeed GDPR-related) applications, I've ETL'd
Kafka to a secondary store e.g. HDFS, and built tooling around recovering
state back into Kafka. I've had situations where data is accidentally or
incorrectly ingested into Kafka, causing downstream systems to process
I am looking for advice on how to handle disasters not covered by the official
methods of replication, whether intra-cluster replication (via replication
factor and producer acks) or multi-cluster replication (using Confluent
Replicator).
We are looking into using Kafka not only as a broker for
looking for your suggestions.
Thanks
Sukumar N
On 2017-06-29 20:34 (+0530), Hans Jespersen wrote:
> Request quotas was just added to 0.11. Does that help in your use case?
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-124+-+Request+rate+quotas
>
> -hans
>
> >
ill result in hang the user-thread or app-servers.
>
>
>
> So we don't want to go for applying quota for our use case. Can you please
> share some suggestions to handle this use-case in our Kafka broker. Like,
> before messages get appended to log, it should validate f
.
So we don't want to go for applying quota for our use case. Can you please
share some suggestions to handle this use-case in our Kafka broker. Like,
before messages get appended to log, it should validate for throttling and if
it requires being throttled. Throttling mechanism should be e
Martini
__
From: Vincenzo D'Amore
Sent: Saturday, December 3, 2016 12:14 PM
To: users@kafka.apache.org
Subject: Re: Suggestions
I found what's wrong, well... finally! Given that consumer
application shoul loa
I found what's wrong, well... finally! Given that consumer
application shoul load received data into a Solr instance.
Incidentally my version of Solr is SolrCloud, and Solrj client use
zookeeper, a different version of zookeeper...
Now I specified in my pom.xml the same version of zookeeper 3.4.8 I
>
> then, the strange thing is that the consumer on
> the second topic which stay in poll forever, *without receive any message*.
How long is 'forever'? Did you wait more than 5 minutes?
On Fri, Dec 2, 2016 at 2:55 AM, Vincenzo D'Amore wrote:
> Hi Kafka Gurus :)
>
> I'm creating process betwe
Can you use the console consumer to see the messages on the other topics?
> On Dec 2, 2016, at 04:56, Vincenzo D'Amore wrote:
>
> Hi Kafka Gurus :)
>
> I'm creating process between few applications.
>
> First application create a producer and then write a message into a main
> topic (A), within t
Hi Kafka Gurus :)
I'm creating process between few applications.
First application create a producer and then write a message into a main
topic (A), within the message there is the name of a second topic (B). Then
promptly create a second producer and write few message into the new topic
(B).
I
@kafka.apache.org
Subject: 0.10 Metrics Reporter Suggestions
Hello,
We’re looking into the functionality of Metrics Reporters for producers and
consumers in Kafka 0.10. Are there any projects that can be recommended that
seem promising; specifically involving sending metrics to either StatsD or
Graphite?
As
Hello,
We’re looking into the functionality of Metrics Reporters for producers and
consumers in Kafka 0.10. Are there any projects that can be recommended that
seem promising; specifically involving sending metrics to either StatsD or
Graphite?
As always, thank you for your help!
Lawrence We
Kafka Connect can definitely be used for this -- it's one of the reasons we
designed it with standalone mode (
http://docs.confluent.io/3.0.0/connect/userguide.html#workers). For the
specific connector, we include a very simple File connector with Kafka
which will just take each line and send it to
As a newbie, just setup my first Kafka 3 node, each in its own host cluster
and its own ZK. Everything went fine, I could see three brokers registered
in all three ZK /brokers/ids until I created a topic with this exception:
~/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181
--replica
On Tue, May 3, 2016 at 1:00 PM Banias H wrote:
> I should add Flume is not an option for various reasons.
>
> On Tue, May 3, 2016 at 2:49 PM, Banias H wrote:
>
> > We use Kafka (0.9.x) internally in our pipeline and now we would like to
> > ingest application logs sitting in local file system of
I should add Flume is not an option for various reasons.
On Tue, May 3, 2016 at 2:49 PM, Banias H wrote:
> We use Kafka (0.9.x) internally in our pipeline and now we would like to
> ingest application logs sitting in local file system of servers external to
> the Kafka cluster.
>
> We could writ
We use Kafka (0.9.x) internally in our pipeline and now we would like to
ingest application logs sitting in local file system of servers external to
the Kafka cluster.
We could write a producer program running on the application servers to
push files to Kafka. However we wonder if we can leverage
Hi All,
When Kafka is running on kerberoized cluster/ SSL. Can we add an option
security.protocol. So, that user can given PLAINTEXT, SSL, SASL_PLAINTEXT,
SASL_SSL. This will be helpful during running console producer and console
consumer.
./bin/kafka-console-producer.sh --broker-list --topic --
ion is
> what
> > technology stack to use would be most suitable so that I can generate the
> > highest amount of load with the least amount of load generating
> > machines/hardware. Using threads, processes, or whatever. node.js,
> python,
> > scala, ruby, java, .net, etc.
> >
> > thoughts, suggestions appreciated,
> > David
> >
>
ck to use would be most suitable so that I can generate the
> highest amount of load with the least amount of load generating
> machines/hardware. Using threads, processes, or whatever. node.js, python,
> scala, ruby, java, .net, etc.
>
> thoughts, suggestions appreciated,
> David
>
east amount of load generating
machines/hardware. Using threads, processes, or whatever. node.js, python,
scala, ruby, java, .net, etc.
thoughts, suggestions appreciated,
David
I could go either way. I think that if ZK is up, then Kafka's going to go
crazy trying to figure out who's the master of what, but maybe I'm not thinking
the problem through clearly.
That does beg the issue: it seems like it'd be good to have something
written down somewhere to say how o
I would have thought you'd want ZK up before Kafka started, but I don't
have any strong data to back that up.
On Sat, 8 Aug 2015 at 7:59 AM Steve Miller wrote:
>So... we had an extensive recabling exercise, during which we had to
> shut down and derack and rerack a whole Kafka cluster. Then
So... we had an extensive recabling exercise, during which we had to shut
down and derack and rerack a whole Kafka cluster. Then when we brought it back
up, we discovered the hard way that two hosts had their "rebuild on reboot"
flag set in Cobbler.
Everything on those hosts is gone as a
> > > Guozhang
> > >
> > >
> > > On Tue, Jun 17, 2014 at 9:56 PM, Daniel Compton > > (mailto:d...@danielcompton.net)
> > (mailto:d...@danielcompton.net)>
> > > wrote:
> > >
> > > > Hi
> > > >
> > &g
gt; >
> > Guozhang
> >
> >
> > On Tue, Jun 17, 2014 at 9:56 PM, Daniel Compton (mailto:d...@danielcompton.net)>
> > wrote:
> >
> > > Hi
> > >
> > > I was following the instructions for Kafka mirroring and had two
> > > sugge
instructions for Kafka mirroring and had two
> > suggestions for improving the documentation at
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330:
> > 1. Move "Note that the --zkconnect argument should point to the source
> > cluster's
Thanks Daniel for the findings, please feel free to update the wiki.
Guozhang
On Tue, Jun 17, 2014 at 9:56 PM, Daniel Compton
wrote:
> Hi
>
> I was following the instructions for Kafka mirroring and had two
> suggestions for improving the documentation at
> https://cwiki.apache
Hi
I was following the instructions for Kafka mirroring and had two
suggestions for improving the documentation at
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330:
1. Move "Note that the --zkconnect argument should point to the source
cluster's ZooKeeper...”
apache.org
> Subject: Re: Design questions and your opinion and suggestions are
> appreciated
>
> 1) In 0.7, a topic exists on every broker. So new partitions will be
> automatically added on the new broker.
> 2) It's possible, if you use the partitioner. However, be aware in
like and what version will
have this new behavior?
Thanks again.
Jamie
-Original Message-
From: Jun Rao [mailto:jun...@gmail.com]
Sent: Wednesday, December 12, 2012 10:10 PM
To: users@kafka.apache.org
Subject: Re: Design questions and your opinion and suggestions are appreciated
1) In
m]
> Sent: Wednesday, December 12, 2012 2:26 PM
> To: users@kafka.apache.org
> Subject: Design questions and your opinion and suggestions are appreciated
>
> Hi,
>
> We are incorporating Kafka as part of our centralized log aggregation
> service for a clustered server system.
within the topic.
>
> Couple of design questions we would like to hear your opinion and any
> suggestions are appreciated!
>
> 1) If we dynamically adds another node into the cluster, base on our
> design above, we would want to create a new partition and add to each of
> th
bhash Agrawal
-Original Message-
From: Jamie Wang [mailto:jamie.w...@actuate.com]
Sent: Wednesday, December 12, 2012 2:26 PM
To: users@kafka.apache.org
Subject: Design questions and your opinion and suggestions are appreciated
Hi,
We are incorporating Kafka as part of our centralize
42 matches
Mail list logo
