Some more things to think about:What is the data volume you are dealing with?Do
you need to have multiple partitions to support the data/throughput?Are you
looking at each partition to be dedicated to a single user or a group of
users?Is the data balanced across all your users or is it skewed?Ho
Technically/conceptually it is possible to have 200,000 topics, but do you
really need it like that?What do you intend to do with those messages - i.e.
how do you forsee them being processed downstream? And are those topics really
there to segregate different kinds of processing or different ids
Just wondering Mukesh - the reason you want this feature is because your value
payload is not small (tens of kb). Don't know if that is the right usage of
kafka. It might be worthwhile to store the avro files in a filesystem (regular,
cluster fs, hdfs or even hbase) and the value in your kafka m
Have you tried using the built-in stress test scripts?
bin/kafka-producer-perf-test.sh
bin/kafka-consumer-perf-test.sh
Here's how I stress tested them -
nohup ${KAFKA_HOME}/bin/kafka-producer-perf-test.sh --broker-list
${KAFKA_SERVERS} --topic ${TOPIC_NAME} --new-producer --threads 16 --messages
When I try running the Java Consumer example at
https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+ExampleI get
the following zookeeper connection error.
I have verified zookeeper connectivity using a variety fo means (using
Zookeeper built-in client, sending 4-letter commands to
r the wiki page I was referring to -
https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
From: Jayesh Thakrar
To: "users@kafka.apache.org"
Sent: Tuesday, January 6, 2015 11:09 AM
Subject: Zookeeper Connection When Using High Level Sample Consumer Code
I do see the Windows based scripts in the tar file - but haven't them
though.You should find them under bin/windows.
Also you can always use other Windows stress testing tools/suites to check your
local I/O performance..
From: Shlomi Hazan
To: users@kafka.apache.org; Jayesh Th
Does one also need to set the config parameter "delete.topic.enable" to true ?I
am using 8.2 beta and I had to set it to true to enable topic deletion.
From: Armando Martinez Briones
To: users@kafka.apache.org
Sent: Wednesday, January 14, 2015 11:33 AM
Subject: Re: Delete topic
than
Another option is to copy data from each topic (of interest/concern) to a "flat
file on a periodic basis".E.g. say you had a queue that only contained "textual
data".Periodically I would run the bundled console-consumer to read data from
the queue and dump to a file/directory and then backup it
Hi,
I am trying to write a consumer using the KafkaConsumer class from
https://github.com/apache/kafka/blob/0.8.2/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java.
My code is pretty simple with the snippet show below.However what I am seeing
is that I am not seeing any
My apologies in advance - for cross-posting the question in both Dev and User
user-groups
Hi,
I have been using Kafka as DevOps user for a few months now and am now looking
at "building/testing" Kafka from source code.My gradle build is failing (source
from GitHub) with the error show below.Th
cluster.
Thank you,Jayesh Thakrar
I am looking at ways how one might have data loss and duplication in a Kafka
cluster and need some help/pointers/discussions.
So far, here's what I have come up with:
Loss at producer-sideSince the data send call is actually adding data to a
cache/buffer, a crash of the producer can potentially r
below?
From: R Krishna
To: users@kafka.apache.org; Jayesh Thakrar
Sent: Tuesday, August 30, 2016 2:02 AM
Subject: Re: Question: Data Loss and Data Duplication in Kafka
Experimenting with kafka myself, and found timeouts/batch expiry (valid and
invalid configurations), and
14 matches
Mail list logo
