If I understand it correctly each consumer should have it's 'own' thread,
and should not be accessible from other threads. But you could
(dynamically) create enough threads to cover all the partitions, so each
consumer only reads from one partition. You could also let all those
consumers access som
I using Kafka v 0.9 with TLS enabled, including client auth.
In
http://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption,
it is mentioned that "We need to generate a key and certificate for each
broker and client in the cluster. The common name (CN) of the broker
In config/connect-standalone.properties and
config/connect-distributed.properties, there are the following
configuration entries:
> key.converter.schemas.enable=false
> value.converter.schemas.enable=false
But there is no Java source code which uses these two configuration
entries. I am talking a
We use almost the same properties (the same if you account for defaults),
and have not seen any check whether the FQDN matches the CN, as it's al
working without matching names. It seems the requirement is only needed if
you use SASL_SSL as security protocol, which from you config you don't seem
to
Hi guys,
it´s possible to change an existing topic name?
Thanks and best wishes
Johannes
The server configuration in
http://stackoverflow.com/questions/37536259/broker-replication-error-not-authorized-to-access-topics-topic-authorization
specifies security.inter.broker.protocol=PLAINTEXT. This would result in
the principal "anonymous" to be used for inter-broker communication. Looks
l
There's no way to do that. If you're trying to maintain data, you'll need
to read all the data from the existing topic and produce it to the new one.
-Todd
On Wednesday, June 1, 2016, Johannes Nachtwey <
johannes.nachtweyatw...@gmail.com> wrote:
> Hi guys,
>
> it´s possible to change an existing
Not necessarily "admin" any name is ok, we use the CN stored in the
keystore, but we don't use sasl, and that's how the brokers communicate to
each other. You need some way of allowing them to communicate.
On Wed, Jun 1, 2016 at 10:33 AM Rajini Sivaram
wrote:
> The server configuration in
>
> ht
Hello there,
Pretty efficient way to monitor consumers and producers :
https://github.com/shunfei/DCMonitor
You will need to install https://prometheus.io/ as DCMonitor is based on it.
2016-05-27 11:24 GMT+02:00 Florian Hussonnois :
> Hi,
>
> You can also use InfluxDB instead of Graphite. Influ
Hello Jayesh,
Thank you very much for such a good description. My further questions are
(just to be my self clear about the concept).
1. If I have only one partition in a 'Topic' in a Kafka with following
configuration,
bin/kafka-topics.sh --create --zookeeper localhost:2181
--replication-factor
Hi Phil,
You are right that the check is not done by default. We have a couple of
JIRAs tracking that:
https://issues.apache.org/jira/browse/KAFKA-3665
https://issues.apache.org/jira/browse/KAFKA-3667
Enabling the check is a matter of setting
`ssl.endpoint.identification.algorithm`
to `https`, b
Hi Hafa
If you create a topic with replication-factor = 2, you can lose one of them
without losing data, so long as they were "in sync". Replicas can fall out of
sync if one of the machines runs slow. The system tracks in sync replicas.
These are exposed by JMX too. Check out the docs on replic
What is your use case ? Why do you want such persistence in Kafka ? For
such persistence I think you should use cassandra /mongo db type of no sql
db.
thanks
Ashutosh
On Wed, Jun 1, 2016 at 9:06 AM, VG wrote:
> Hi,
>
> There are number of messages floating on the internet suggesting that Kafka
Does creating new topic with new name and the same settings as the original one
and directly copying files from kafka log directory into the new topic folder
work? It would be nice if it would. I don't know if there is anything in log
file format info or Zoo info attached to original topic which
Hey Danny
Currently the bootstrap servers are only used when the client initialises
(there’s a bit of discussion around the issue in the jira below if you’re
interested). To implement failover you’d need to catch a timeout exception in
your client code, consulting your service discovery mechani
So what is the best way to hook up kafka with cassandra ?
Also is there any study or implementation of using HDFS as the storage for
Kafka ?
thanks
On Wed, Jun 1, 2016 at 4:24 PM, Ashutosh Kumar
wrote:
> What is your use case ? Why do you want such persistence in Kafka ? For
> such persistenc
Nope. You should upgrade to Kafka 0.9, assuming that your
ActiveControllerCount across all brokers is 0 or more than 1 (which is
typically the failure case we see).
Thanks
Tom Crayford
Heroku Kafka
On Wed, Jun 1, 2016 at 3:22 AM, Fredo Lee wrote:
> we use 0.8.2.2. is this version ok?
>
> 2016-
Ok. I'd recommend upgrading to 0.9 asap to fix the known bugs in 0.8 here.
Thanks
Tom Crayford
Heroku Kafka
On Wed, Jun 1, 2016 at 3:27 AM, Fredo Lee wrote:
> we use 0.8.2.2.
>
> 2016-05-31 20:14 GMT+08:00 Tom Crayford :
>
> > Is this under 0.8? There are a few known bugs in 0.8 that can lead
So, it means that I should create topics with at least replication-factor=2
inspite of how many servers in a kafka cluster. If any server goes down or
slows down then zookeeper will not go out-of-sync.
Currently, my all topics are with eplication-factor= 1 and I got an issue
that Zookeeper goes out
I do not see why this is a limitation. Any data storage application you use
will be limited by physical capacity of the nodes.
Distributed applications like Kafka (Distributed message broker), HDFS (
Distributed file system), Cassandra ( distributed key value dB), by design
allow to store huge amou
Pretty much. It’s not actually related to zookeeper.
Generalising a bit, replication factor 2 means Kafka can lose 1 machine and be
ok.
B
> On 1 Jun 2016, at 12:46, Hafsa Asif wrote:
>
> So, it means that I should create topics with at least replication-factor=2
> inspite of how many servers
Just for more info:
If I have 10 servers in a cluster, so for the most tolerant cluster, do we
need replication-factor = 10?
That is also the issue for rebalancing the scaling of kafka cluster, that
when we need to add server in a cluster then we also need to increase
partitions in topics as well ?
https is a protocol NOT an algorithm..no wonder this causes
confusion!ssl.endpoint.identification.algorithmhttps://en.wikipedia.org/wiki/Transport_Layer_Security
ismael can you change ssl.endpoint.identification.algorithm property to
ssl.endpoint.identification.protocolso the property matches wh
Hi Alex,
I use Kafka within Hortonworks HDP stack and din't find anything in Kafka
configuration marking a broker as bootstrap. I also nothing for bootstrap
configuration in API I use. Onlly Zookeeper(s) or broker(s).
What do I loose if I use Zookeeper in producer configuration?
On Tue, May 31,
Hi,
I am unable to see the images. But I use Kafka with HDP right now without
any problem.
On Tue, May 31, 2016 at 9:33 PM, Shaolu Xu wrote:
> Hi All,
>
>
> Anyone used HDP to run kafka, I used it and face a problem.The following
> is the error info:
>
> [image: Inline image 2]
>
>
> The follow
Hello everyone,
I am trying to setup a MirrorMaker between my company's local cluster and
another cluster in AWS to have replication over clusters. We have setup a
VPN between these two clusters, and as far as I can see, everything works
correctly, meaning that I can ping the nodes and telnet into
You are wrong Unmesh. Kafka design forces a partition to be on a single
node only.
My question is around the scalability of the partition itself.
How to overcome the restriction of a single node for a partition ?
Any clues anyone...
On Wed, Jun 1, 2016 at 5:24 PM, Unmesh Joshi wrote:
> I do n
Hi Marco,
We use the public DNS hostname that you can get from the AWS metadata
service.
Thanks,
Ben
On Wed, Jun 1, 2016 at 1:54 PM, Marco B. wrote:
> Hello everyone,
>
> I am trying to setup a MirrorMaker between my company's local cluster and
> another cluster in AWS to have replication ove
Gerard is correct.
The unit of parallelization in kafka is the topic and topic partition. A
single thread/consumer consumes each partition in a topic (even if multiple
topics). KafkaConsumer is NOT thread safe and should not be shared between
threads.
On Wed, Jun 1, 2016 at 12:11 AM, Gerard Klijs
To clarify, we are using a property name that matches the JDK name.
Changing the property name would be an incompatible change, so it would
require a period of deprecation. And for people familiar with JDK, it would
be confusing. So, not clear it would be a good thing.
Ismael
On Wed, Jun 1, 2016
Martin,
What I said is correct, see:
The JDK 8 release supports endpoint identification algorithms for TLS 1.2.
The algorithm name can be passed to the
setEndpointIdentificationAlgorithm() method
of javax.net.ssl.SSLParameters. The following table shows the currently
recognized names.
Endpoint Id
You can’t. I have file a wish for something like this:
https://issues.apache.org/jira/browse/KAFKA-3726.
–
Best regards,
Radek Gruchalski
ra...@gruchalski.com
de.linkedin.com/in/radgruchalski
Confidentiality:
This communication is intended for the above-named person and may be
confidential a
Apologies. Misinterpreted your question. I am curious to know if people
have faced issues with single partition on node in practice. Or because
most uses do not keep logs forever, this is generally not an issue? Was
this faced at LinkedIn?
On 1 Jun 2016 6:28 p.m., "VG" wrote:
You are wrong Unmesh
Hi,
Criteo has open sourced its Kafka .NET client. The driver focuses on the
producer but a simple consumer is provided as well. A lot of defensive
options are provided to resist cluster crashes on a large scale.
Source is on github: https://github.com/criteo/kafka-sharp
--
Serge Danzanvilliers
Hi Hafsa,
Not sure by what you mean "most tolerant cluster".
If you mean that you want the cluster to be able to tolerate 9 of 10 servers to
be down, then yes.
But I would question - is your traffic activity, system load and storage
requirement so low that it can be served by a single server?
If
Hi,
As per the description in KIP-32 the timestamp of Kafka message is
unchanged mirrored from one cluster to another if createTime is used. But I
tested with mirror maker in Kafka-0.10 this doesn't seem the case. The
timestamp of the same message is different in source and target. I checked
the l
Hi Ben,
Thanks for your answer. What if the instance does not have a public DNS
hostname?
These are all private nodes without public/elastic IP, therefore I don't
know what to set.
Marco
2016-06-01 15:09 GMT+02:00 Ben Davison :
> Hi Marco,
>
> We use the public DNS hostname that you can get fro
Hi all,
Is it possible to use the Kafka Streams DSL to build a topology that has a
source and/or sink that is/are not Kafka Topic(s)?
As an example, I would like to consume some events from an API via WebSockets
and use that as a source in a Kafka Streams topology — ideally one defined with
th
The intent was definitely as you described, but I think we forgot to
actually modify the code accordingly.
Do you mind opening a JIRA on the issue?
Gwen
On Wed, Jun 1, 2016 at 4:13 PM, tao xiao wrote:
> Hi,
>
> As per the description in KIP-32 the timestamp of Kafka message is
> unchanged mirr
Currently this is not part of the DSL and needs to be done separately
through KafkaConnect. Here's an example:
http://www.confluent.io/blog/hello-world-kafka-connect-kafka-streams
In the future we want to integrate Connect and Streams better, so you could
do something like "builder.stream(..).from
Here we have a scenario that we are considering to use Kafka as a
fault-tolerant message system.
There is an external source that keeps generating data. We are thinking of
deploying Kafka to store the produced data. It is assumed that the external
source is safe enough and would not crash or any b
Hi,
We have 5 brokers running on 0.9.0.1 with 5 ZK. This morning, multiple
topics were having "unavailable-partitions" (whose leader is not
available). After looking at multiple logs, forums and google results, we
finally restarted all the brokers one by one and issue seems to be
resolved.
Howeve
You might consider the approach we are using in Hermes (a pubsub system
with HTTP interface on top of Kafka):
http://hermes-pubsub.readthedocs.io/en/latest/configuration/buffer-persistence/
We use Chronicle Map to persist things that go into Kafka producer into
memory mapped file. In case of proc
Although I think it should have been an included option, it's very easy to
create and use your own message handler with the mirror maker. You can
simply copy the timestamp and type from the consumerecord to the
producerecord.
On Wed, Jun 1, 2016 at 5:48 PM Gwen Shapira wrote:
> The intent was de
Hello,
How do I build a KTable from two topics such that key is in one topic and
value in other?
Ex,
topic1 has a key called basekey and userId as value.
topic2 has same basekey and locationId as value
topic1 = {"basekey":1,"userId":111}
topic1 = {"basekey":2,"userId":222}
topic2 = {"basekey":1
Hi,
I have a 3 node cluster with kafka version 0.9.0.1 with many topics having
replication factor 3 and isr 2.
Node 0 is running multiple mirrormakers.
Node 1 in this cluster has an issue and lost all data from disk (All kafka
data logs got deleted from disk).
Kafka broker is down but zookeeper is
I have 4 node kafka broker with following configuration :
Default Number of Partitions : num.partitions : 1
Default Replication Factor : default.replication.factor : 1
Maximum Message Size : message.max.bytes : 10 MB
Replica Maximum Fetch Size : replica.fetch.max.bytes : 10 MB
Right now I have
On 6/1/16, 11:53, "Gwen Shapira" wrote:
> Currently this is not part of the DSL and needs to be done separately
> through KafkaConnect. Here's an example:
> http://www.confluent.io/blog/hello-world-kafka-connect-kafka-streams
Ah, I see! Thank you! And thanks for the super-fast reply!
> In the f
This would be Michael and Guozhang job to answer, but I'd look at two
options if I were you:
1) If the connector you need exists (
http://www.confluent.io/product/connectors), then you just need to run it.
It is just a simple REST API (submit job, job status, etc), so I wouldn't
count it as "learn
On 6/1/16, 18:59, "Gwen Shapira" wrote:
>This would be Michael and Guozhang job to answer
I look forward to hearing from them ;)
> but I'd look at two options if I were you:
>
> 1) If the connector you need exists (
> http://www.confluent.io/product/connectors), then you just need to run it.
>
Hmm, then is it doable assigning non-overlapped different topics to each
thread while implementing the Kafka Consumer with multi-threading?
2016-06-01 22:14 GMT+09:00 Christian Posta :
> Gerard is correct.
>
> The unit of parallelization in kafka is the topic and topic partition. A
> single threa
What I'm seeing is the consumer sometimes doesn't read the first message.
There are times when it does as well.
Any clues what might cause it?
Hi All,
I used the latest HDP 2.4 version.
Did you do some configuration before used HDP? I searched a solution that
is http://www.cnblogs.com/i2u9/p/ambari-kafka-multiip.html, but it not
works for me.
Attachment errorInfo and HDPConfig snapshot.
Thanks in advance!
Thanks，
Nicole
On Wed, Jun
Hi,
I've read up a little on MirrorMaker (e.g. the wiki [1] and KIP-3 [2]), but
haven't yet found a definitive answer to the following question.
Let's assume I am producing a certain topic to a Kafka cluster in
datacenter A.
I set up MirrorMaker in datacenter B and C to mirror the topic from
dat
I donot think you need public hostname.I have a similarsetup and its perfectly
fine.
What I would suggest you to change the hostname,make it persistent,and use FQDN
everywhere with /etc/hosts entry locally and on AWS machines.Your problem will
get fixed.
On 6/1/16, 8:54 PM, "Marco B." wrote
It works indeed but encrypting individual messages really influences the batch
compression done by Kafka.
Performance drops to about 1/3 of what it is without (even if we prepare the
encrypted samples upfront).
In the end what we going for is only encrypting what we really really need to
encrypt
I have an issue where my brokers would randomly shut itself down.
I turned on debug in log4j.properties but still do not see a reason why the
shutdown is happening.
Anyone seen this behavior before?
version 0.10.0
log4j.properties
log4j.rootLogger=DEBUG, kafkaAppender
* I tried TRACE level bu
What changes need to be done in kafka if we use file system encryption?
How consumer will impact?, do we need to any changes in configuration ?
Does it require SSL based communication?
- Original Message -
From: "Bruno Rassaerts"
To: users@kafka.apache.org
Sent: Thursday, June 2, 20
You could add a header to every message, with information whether it's
encrypted or not, then you don't have to encrypt all the messages, or you
only do it for some topics.
On Thu, Jun 2, 2016 at 6:36 AM Bruno Rassaerts
wrote:
> It works indeed but encrypting individual messages really influence
No you can't because:
- because of producer failures some messages may be duplicated.
- your not sure the cluster your copying from hasn't some already removed
data.
We try to solve the same problem, and are probably going to solve it by
copying the timestamps with the mirror maker, and on the swi
Hello,
My project is using kafka ,and I want register a user in the forum,what can I
do ?
Tong SS
61 matches
Mail list logo
