Re: Kafka performance when it comes to throughput

Hi Marisa, I am going to be running it in a Kubernetes cluster on Azure Kubernetes Service using the set up scripts available in my github repo https://github.com/izzyacademy/kafka-in-a-box https://youtu.be/TDw3tDAiBBM I will review the recommendations from the EventSizer.io tool as well to make

Re: Kafka performance when it comes to throughput

Hi Israel, Great job! It looks great and promising. I really like your YouTube channel and the way you present the material. A couple of things that you might want to consider for your benchmarks experiments: 1) What machine are you going to use? Is it a fast machine with enough cpu cores? I woul

Re: Kafka performance when it comes to throughput

Marisa, I have kicked off the video series on performance optimization for the Kafka setup. I will be working on the various configurations for latency, throughput, availability and durability. https://youtu.be/aPlbG349cXg The first ones will be on latency and throughput which is what you are i

Re: Kafka performance when it comes to throughput

Hi Alex, > Furthermore, setting up a localhost pub/sub demo on a single machine (your laptop?) is so far removed from a real-world scenario I can't imagine how any numbers derived from that would be useful. I can't imagine either. That's why I'm planning to run this on a lab Linux machine with 8

Re: Kafka performance when it comes to throughput

Wow, that's awesome! I wasn't expecting that. I truly appreciate your help and professionalism. > Let me find some time soon and I will do a video on that scenario optimized primarily for low latency and throughput. I will also compare how this performs when adjusted for durability and high availa

Re: Kafka performance when it comes to throughput

Marisa, you might consider engaging someone at Confluent, maybe they can give you some case studies or whitepapers from similar use-cases in the financial industry. (and yes, Kafka is used in the financial industry) . A client asking you to "prove that Kafka performs/scales" seems like an unusual

Re: Kafka performance when it comes to throughput

Thanks for your response Marisa. This has been a very interesting discussion and I appreciate it. It is a bit of a challenge in the sense that I wish I had a demo ready to go with similar use case and expectations to easily explain what I have been trying to convey I am always ready for a chall

Re: Kafka performance when it comes to throughput

Hi Israel, > You can achieve any performance benchmark you are willing to pay for. Thanks for your email. Allow me to respectfully disagree. I believe that some systems are better than others when it comes to performance. The idea that I can just take a slow system, multiply by 1 million, and the

Re: Kafka performance when it comes to throughput

Marisa, I do not agree with your assessment. There are several factors that could influence your performance numbers even with localhost. Your project should be configured based on your own needs. Your throughput could go up or lower depending on how you are configured based on what is important

Re: Kafka performance when it comes to throughput

Hi Joris, Thank you so much, friend! > I appreciate that setting up everything on localhost will be easier and lead to big numbers, but bear in mind that it's typically all the other real-life stuff (remote connections, replication, at-least once, ...) that causes massive slowdowns compared to lo

Re: Kafka performance when it comes to throughput

These tutorials - though quite a bit outdated - seem quite useful: http://cloudurable.com/blog/kafka-tutorial-kafka-producer/index.html (and the follow-ups). Ends up being close to how I write this in Java, and tutorial 13 talks about batching and acks etc, which you'll need in order to tune to max

Re: Kafka performance when it comes to throughput

Hi Joris, Thank you so much. I plan to write a Java Consumer and a Java Producer, for my benchmark. Do you recommend an example that I can use as a reference to write my basic Java producer and simple Java consumer? I'll for sure share the through number I get with the community. Maybe even write

Re: Kafka performance when it comes to throughput

I'd just follow the instructions in https://kafka.apache.org/quickstart to set up Kafka and Zookeeper on a single node, by running the Java processes directly. Or can run in Docker. For the producer and consumer I'd personally use Python, as it's the easiest to get going. You may want to look at h

Re: Kafka performance when it comes to throughput

Hi Joris, I've spoken to him. His answers are below: On Thu, Jan 6, 2022 at 1:37 PM Joris Peeters wrote: > There's a few unknown parameters here that might influence the answer, > though. From the top of my head, at least > - How much replication of the data is needed (for high availability),

Re: Kafka performance when it comes to throughput

Hi Okada, Thanks for your reply. Finally I see some numbers! I love numbers :) I've shown your email to my boss (I hope he will hire me to do this project) and he said the following: "I would like to see this 833k/sec number for myself. Am I asking too much? :) Can you set up a very basic and si

Re: Kafka performance when it comes to throughput

There's a few unknown parameters here that might influence the answer, though. From the top of my head, at least - How much replication of the data is needed (for high availability), and how many acks for the producer? (If fire-and-forget it can be faster, if need to replicate and ack from 3 broker

Re: Kafka performance when it comes to throughput

Hi Israel, Your email is great, but I'm afraid to forward it to my customer because it doesn't answer his question. I'm hoping that other members from this list will be able to give me a more NUMERIC answer, let's wait to see. Just to give you some follow up on your answer, when you say: > 30 p

Re: Kafka performance when it comes to throughput

Hi, Marisa. Kafka is well-designed to make full use of system resources, so I think calculating based on machine's spec is a good start. Let's say we have servers with 10Gbps full-duplex NIC. Also, let's say we set the topic's replication factor to 3 (so the cluster will have minimum 3 servers),

Re: Kafka performance when it comes to throughput

Hi Marisa I think there may be some confusion about the throughput for each partition and I want to explain briefly using some analogies Using transportation for example if we were to pick an airline or ridesharing organization to describe the volume of customers they can support per day we would

Kafka performance when it comes to throughput

Cheers from NYC! I'm trying to give a performance number to a potential client (from the financial market) who asked me the following question: *"If I have a Kafka system setup in the best way possible for performance, what is an approximate number that I can have in mind for the throughput of th

Kafka Performance Producer

Hello, Following is an issue we are trying to debug at the moment : Running org.apache.kafka.tools.ProducerPerformance code to performance test the kafka cluster. As a trial cluster has only one broker and zookeeper with 12GB of heap space. Running 6 producers on 3 machines with same transacti

Re: Kafka performance tests

Hi, Thanks a lot for the help, I have found a couple of studies too: https://events.static.linuxfound.org/sites/events/files/slides/HTKafka2.pdf https://cwiki.apache.org/confluence/display/KAFKA/Performance+testing https://cwiki.apache.org/confluence/display/KAFKA/An+analysis+of+the+impact+of

Re: Kafka performance tests

Hi I am working on this too and I still try to search nice sources how to optimize results. I can recommend you this article Benchmarking and this presentation

Kafka performance tests

Hi, I am working on testing the performance of Kafka. I am looking for articles, publications about performance tests, methods, results and configurations, but so far I have not found many. Do you know any useful studies? Has any of you done something similar? If yes, would you mind sharing those

[HELP] Guidelines/tools to test kafka performance with application layer involved

Hello, We are using kafka for pub sub and want test performance of entire system. Is there any tool readily available in kafka world which can simulate multiple publishers and subscribers tp measure latency and throughput considering custom application layer? Any guidelines around this would be

Kafka Performance metrics

Hello everyone. I am trying to get the Kafka related metrics using Jolokia. I am successful to extract the metrics for zookeeper but can't figure it out for the broker metrics retrieval. Like when I use this url http://localhost:8778/jolokia/list I get metrics related to zookeeper but not fo

Re: Kafka Performance testing

Hi, Depending on what you want, you can go : 1) https://cwiki.apache.org/confluence/display/KAFKA/Performance+testing where its explain the procedure to initiates performances tests and how to use *kafka-consumer-perf-test.sh* and *kafka-producer-perf-test.sh* among others useful scripts. 2

Kafka Performance testing

I am new to Kafka world, wanted to do performance testing on kafka. I googled around, but dint find any useful document from which i can start with. Could you explain or point me to document it would be great. Also which performance loading tool would be best to test kafka, I prefer Jmeter though.

Kafka performance

Hello, I am a new kafka user, trying to reach kafka limits :1 000 000 messages per second. Therefore, I loaded three Azure VMs having this configuration: 16 CPU Cores 56 Go Ram All disks are premium LRS and minimum capacity 512 Go. Download speed 2696.28 Mbit/s Upload speed 1121.13 Mbi

Re: Kafka performance on an ordinary machine

> As to what procedures I'd propose is spinning up a cluster and running some tests on it. This is really good advice. Capacity planning for your Kafka cluster is not particularly straightforward - especially if you intend to run a multi-tenant cluster. (If someone has a general model, please shar

Re: Kafka performance on an ordinary machine

Thank you. I have red these articles. Do you have any other documentation? On Wed, Nov 9, 2016 at 12:07 PM Karolis Pocius wrote: > There is no 'best' configuration, it all depends on your use cases, > expected load, etc. > > Documentation is a good place to start > https://kafka.apache.org/docum

Re: Kafka performance on an ordinary machine

There is no 'best' configuration, it all depends on your use cases, expected load, etc. Documentation is a good place to start https://kafka.apache.org/documentation.html#hwandos There are a few good benchmark articles. They're a bit dated by now, but still hold true in most cases: * https:

Kafka performance on an ordinary machine

Hello We want to use Kafka in our production environment but we don't have any information about what's the best server configuration and the best benchmark can be achieve in our production environment (Based on our hardware and VMs) I'm really rookie in this area. For getting to the best configura

Re: Benchmarking kafka performance

t into Docker and run a bunch of them. ;-) > > https://github.com/Eventador/evtools/tree/master/yesbench > > In terms of metrics, we measure through JMX: > - BytesInPerSec/BytesOutPerSec > - TotalTimeMs > > This is also a good post: > https://www.datadoghq.com/blog/monitori

Re: Benchmarking kafka performance

of metrics, we measure through JMX: - BytesInPerSec/BytesOutPerSec - TotalTimeMs This is also a good post: https://www.datadoghq.com/blog/monitoring-kafka-performance-metrics/ Thanks Kenny Gorman http://www.eventador.io > On Sep 22, 2016, at 3:21 PM, Vadim Keylis wrote: > > Good

Re: Benchmarking kafka performance

> Good morning. Which benchmarking tools we should use to compare performance > of 0.8 and 0.10 versions? Which metrics should we monitor ? It dependents on your use-case/requirements (if any). I suggest you take a look at a general high-level benchmark made by LinkedIn. They’ve focused onto

Re: Benchmarking kafka performance

Good afternoon. Any suggestions regarding benchmark tool would be greatly appreciated. Thanks On Mon, Sep 19, 2016 at 8:18 AM, Vadim Keylis wrote: > Good morning. Which benchmarking tools we should use to compare > performance of 0.8 and 0.10 versions? Which metrics should we monitor ? > > Tha

Benchmarking kafka performance

Good morning. Which benchmarking tools we should use to compare performance of 0.8 and 0.10 versions? Which metrics should we monitor ? Thanks in advance, Vadim

Debug Kafka performance

Hi, We have setup a 3-broker Kafka 0.9.0.0 cluster in Cloudera. The bokers are ec2 d2.xlarge instances. What we are trying to do is to run some producer perf tests on this cluster. >From the client host, we did a bandwidth test to the brokers using iperf3 and we are getting 900Mbps connections. F

Re: Reactive Kafka performance

David, you wrote: > Each task would effectively block on DB-io for every history-set retrieval; > obviously we would use a TTL cache (KTable could be useful here, but it > wouldn’t be able to hold “all” of the history for every user) Can you elaborate a bit why you think a KTable wouldn't be abl

Re: Reactive Kafka performance

Hey Guozhang, our basic road block is asynchronous processing (this is actually related to my previous post about asynchronous processing). Here is the simplest use-case: The streaming job receives messages. Each message is a user-event and needs to randomly look up that user’s history (for 1

Re: Reactive Kafka performance

Hello David, I'd love to learn details about the "flexibility" of Reactive Kafka compared with KStreams in order to see if KStreams can improve on that end. Would you like to elaborate a bit more on that end? Guozhang On Thu, Jul 28, 2016 at 12:16 PM, David Garcia wrote: > Our team is evaluat

Reactive Kafka performance

Our team is evaluating KStreams and Reactive Kafka (version 0.11-M4) on a confluent 3.0 cluster. Our testing is very simple (pulling from one topic, doing a simple transform) and then writing out to another topic. The performance for the two systems is night and day. Both applications were ru

Re: Kafka performance issue

Hey Tony, Are you connecting to Remote machine for Consumer and Producer? If yes try to check the network bandwidth. Do check the cluster configuration, memory, cpu resources, topic partitions, replication factor, you would get some trace. Thanks! VM On Apr 4, 2016 5:09 AM, "LeiNing" wrote: >

Kafka performance issue

Hi guys, Hope you are doing well. I'm encountering a performance issue and would be highly appreciate if you can shed some light. There is no problem by only running producer or consumer, but when I run both of them at the same time, the performance is getting very slow. What p

Kafka performance test failed

Hi, I am using kafka_2.9.2-0.8.1.1 Intended to run performance test using run-simulator.sh file. I started zookeeper and kafka server and finally run the below command /run-simulator.sh -kafkaServer=localhost -numTopic=10 -reportFile=report-html/data -time=15 -numConsumer=20 -numProducer=40 -xa

Re: Spark Kafka Performance

yandwindow in >> spark. >> >> - Messaggio originale - >> Da: "Gwen Shapira" >> Inviato: ‎03/‎11/‎2014 21:08 >> A: "users@kafka.apache.org" >> Cc: "u...@spark.incubator.apache.org" >> Oggetto: Re: Spark Kafka Perfor

Re: Spark Kafka Performance

uot; > Cc: "u...@spark.incubator.apache.org" > Oggetto: Re: Spark Kafka Performance > > Not sure about the throughput, but: > > "I mean that the words counted in spark should grow up" - The spark > word-count example doesn't accumulate. > It gets an

R: Spark Kafka Performance

Hi Gwen, I have changed the java code kafkawordcount to use reducebykeyandwindow in spark. - Messaggio originale - Da: "Gwen Shapira" Inviato: ‎03/‎11/‎2014 21:08 A: "users@kafka.apache.org" Cc: "u...@spark.incubator.apache.org" Oggetto: Re: Spark Kafka P

Re: Spark Kafka Performance

Not sure about the throughput, but: "I mean that the words counted in spark should grow up" - The spark word-count example doesn't accumulate. It gets an RDD every n seconds and counts the words in that RDD. So we don't expect the count to go up. On Mon, Nov 3, 2014 at 6:57 AM, Eduardo Costa Al

Spark Kafka Performance

Hi Guys, Anyone could explain me how to work Kafka with Spark, I am using the JavaKafkaWordCount.java like a test and the line command is: ./run-example org.apache.spark.streaming.examples.JavaKafkaWordCount spark://192.168.0.13:7077 computer49:2181 test-consumer-group unibs.it 3 and like a pro

Re: 答复: kafka performance question

per block. > > > -邮件原件- > 发件人: robairrob...@gmail.com [mailto:robairrob...@gmail.com] 代表 Robert > Turner > 发送时间: 2014年5月16日 13:45 > 收件人: users@kafka.apache.org > 主题: Re: kafka performance question > > A couple of thoughts spring to mind, are you sending the wh

答复: kafka performance question

: Re: kafka performance question A couple of thoughts spring to mind, are you sending the whole file as 1 message? and is your producer code using sync or async mode? Cheers Rob. On 14 May 2014 15:49, Jun Rao wrote: > How many brokers and partitions do you have? You may try increas

Re: kafka performance question

nt. > > > > In our cluster,there is a provider and a broker. We use a one thread read > > file from local disk of provider and send it to broker. The average > > throughput is only 3 MB/S~4MB/S. > > But if we just use java NIO API to send file ,the throughput can exceed

Re: kafka performance question

ghput can exceed > 200MB/S. > Why the kafka performance is so bad in our test, are we missing something?? > > > > Our server: > Cpu: Intel(R) Xeon(R) CPU E5-4650 0 @ 2.70GHz*4 > Mem:300G > Disk:600G 15K RPM SAS*8 > > Configuration of provider: > props

kafka performance question

our version is kafka_2.10-0.8.1 发件人: Zhujie (zhujie, Smartcare) 发送时间: 2014年5月14日 8:56 收件人: 'users@kafka.apache.org'; 'd...@kafka.apache.org' 主题: kafka performance question Dear all, We want to use kafka to collect and dispatch data file, but the performance is maybe lower t

kafka performance question

if we just use java NIO API to send file ,the throughput can exceed 200MB/S. Why the kafka performance is so bad in our test, are we missing something?? Our server: Cpu: Intel(R) Xeon(R) CPU E5-4650 0 @ 2.70GHz*4 Mem:300G Disk:600G 15K RPM SAS*8 Configuration of provider: props.put

Re: Kafka Performance Tuning

6 others are > failing with the above exception. > > Yashika > > > > > From: Timothy Chen > Sent: Friday, April 25, 2014 11:40:33 AM > To: users@kafka.apache.org > Subject: Re: Kafka Performance Tuning > > Hi Yashika, &g

RE: Kafka Performance Tuning

are able to read the LOGLINE10 topic but the 6 others are failing with the above exception. Yashika From: Timothy Chen Sent: Friday, April 25, 2014 11:40:33 AM To: users@kafka.apache.org Subject: Re: Kafka Performance Tuning Hi Yashika, No logs in bro

Re: Kafka Performance Tuning

Sent: Friday, April 25, 2014 9:03 AM > To: users@kafka.apache.org > Subject: Re: Kafka Performance Tuning > > Which version of Kafka are you using? Any error in the controller and > state-change log? > > Thanks, > > Jun > > > On Thu, Apr 24, 2014 at 7:37 PM, Yashik

RE: Kafka Performance Tuning

, 2014 9:03 AM To: users@kafka.apache.org Subject: Re: Kafka Performance Tuning Which version of Kafka are you using? Any error in the controller and state-change log? Thanks, Jun On Thu, Apr 24, 2014 at 7:37 PM, Yashika Gupta wrote: > I am running a single broker and the leader column ha

Re: Kafka Performance Tuning

> > Source) > > > at kafka.metrics.KafkaTimer.time(Unknown Source) > > > at > > > kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(Unknown > > Source) > > > at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown >

Re: Kafka Performance Tuning

; Source) > > at kafka.metrics.KafkaTimer.time(Unknown Source) > > at kafka.consumer.SimpleConsumer.fetch(Unknown Source) > > at kafka.server.AbstractFetcherThread.processFetchRequest(Unknown > > Source) > > at kafka.server.AbstractFetcherTh

Re: Kafka Performance Tuning

r.AbstractFetcherThread.doWork(Unknown Source) > at kafka.utils.ShutdownableThread.run(Unknown Source) > > > Regards, > Yashika > > From: Jun Rao > Sent: Thursday, April 24, 2014 10:49 PM > To: users@kafka.apache.org >

Re: Kafka Performance Tuning

e) > > at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown > > Source) > > at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown > > Source) > > at kafka.metrics.KafkaTimer.time(Unknown Source) > > at kafka

Re: Kafka Performance Tuning

er.AbstractFetcherThread.processFetchRequest(Unknown > Source) > at kafka.server.AbstractFetcherThread.doWork(Unknown Source) > at kafka.utils.ShutdownableThread.run(Unknown Source) > > > Regards, > Yashika > > From: Jun Rao > Sent: T

Re: Kafka Performance Tuning

fka.consumer.SimpleConsumer.fetch(Unknown Source) > at kafka.server.AbstractFetcherThread.processFetchRequest(Unknown > Source) > at kafka.server.AbstractFetcherThread.doWork(Unknown Source) > at kafka.utils.ShutdownableThread.run(Unknown Source) > > > Regards, > Yashik

RE: Kafka Performance Tuning

wn Source) Regards, Yashika From: Jun Rao Sent: Thursday, April 24, 2014 10:49 PM To: users@kafka.apache.org Subject: Re: Kafka Performance Tuning Before that error messge, the log should tell you the cause of the error. Could you dig that out? Thanks, Jun On

Re: Kafka Performance Tuning

Before that error messge, the log should tell you the cause of the error. Could you dig that out? Thanks, Jun On Thu, Apr 24, 2014 at 10:12 AM, Yashika Gupta wrote: > Hi, > > I am working on a POC where I have 1 Zookeeper and 2 Kafka Brokers on my > local machine. I am running 8 sets of Kafka

Kafka Performance Tuning

Hi, I am working on a POC where I have 1 Zookeeper and 2 Kafka Brokers on my local machine. I am running 8 sets of Kafka consumers and producers running in parallel. Below are my configurations: Consumer Configs: zookeeper.session.timeout.ms=12 zookeeper.sync.time.ms=2000 zookeeper.connecti

Re: Kafka performance test: "--request-num-acks -1" kills throughput

gt; From: Jun Rao [mailto:jun...@gmail.com] > Sent: Monday, February 3, 2014 9:11 PM > To: users@kafka.apache.org > Subject: Re: Kafka performance test: "--request-num-acks -1" kills > throughput > > Michael, > > Your understanding is mostly correct. For 2, the fol

RE: Kafka performance test: "--request-num-acks -1" kills throughput

users@kafka.apache.org Subject: Re: Kafka performance test: "--request-num-acks -1" kills throughput Michael, Your understanding is mostly correct. For 2, the follower will issue another fetch request as soon as it finishes processing the response of the previous fetch (by adding data, if any,

Re: Kafka performance test: "--request-num-acks -1" kills throughput

d of requests from follower brokers and timeout > settings will impact the throughput of a synchronized producer when > acksNum=-1? > > Thank you, > Michael Popov > > -Original Message- > From: Jun Rao [mailto:jun...@gmail.com] > Sent: Monday, February 3, 2014 10:09

RE: Kafka performance test: "--request-num-acks -1" kills throughput

Sent: Monday, February 3, 2014 10:09 AM To: users@kafka.apache.org Subject: Re: Kafka performance test: "--request-num-acks -1" kills throughput What's replica.fetch.min.bytes? Is it the default value of 1? If not, could you try to set it to 1? If yes, could you try using a small

Re: Kafka performance test: "--request-num-acks -1" kills throughput

gt; numbers are not as high as in other people's tests, that's understandable. > The main concern is why throughput drops 4-10 times when a number of > expected acks is not 1. > > Should we wait for newer versions of Kafka and settle with acks=1 for now? > > Thank you

RE: Kafka performance test: "--request-num-acks -1" kills throughput

fka and settle with acks=1 for now? Thank you, Michael Popov -Original Message- From: Jun Rao [mailto:jun...@gmail.com] Sent: Friday, January 31, 2014 8:57 PM To: users@kafka.apache.org Subject: Re: Kafka performance test: "--request-num-acks -1" kills throughput The request log

Re: Kafka performance test: "--request-num-acks -1" kills throughput

of the throughput in > tests with acksNum!=1. > > What may cause such long responses? Is there a way to address this issue? > > Thank you, > Michael Popov > > > > -Original Message- > From: Jun Rao [mailto:jun...@gmail.com] > Sent: Friday, January 31, 2014 7:52 AM > To:

RE: Kafka performance test: "--request-num-acks -1" kills throughput

. What may cause such long responses? Is there a way to address this issue? Thank you, Michael Popov -Original Message- From: Jun Rao [mailto:jun...@gmail.com] Sent: Friday, January 31, 2014 7:52 AM To: users@kafka.apache.org Subject: Re: Kafka performance test: "--request-

Re: Kafka performance test: "--request-num-acks -1" kills throughput

d4111,0] > (kafka.producer.async.DefaultEventHandler) > java.net.SocketTimeoutException > [2014-01-30 18:33:39,740] WARN Failed to send producer request with > correlation id 81147 to broker 1 with data for partitions [d4111,0] > (kafka.producer.async.DefaultEventHandler) > java.net.SocketTimeoutE

RE: Kafka performance test: "--request-num-acks -1" kills throughput

nuary 29, 2014 5:15 PM To: users@kafka.apache.org Subject: Re: Kafka performance test: "--request-num-acks -1" kills throughput Michael, The producer client in 0.8 has a single thread that does blocking data sends one broker at a time. However, we are working on a rewrite of the produce

RE: Kafka performance test: "--request-num-acks -1" kills throughput

.DefaultEventHandler) java.net.SocketTimeoutException Thank you, Michael Popov -Original Message- From: Jun Rao [mailto:jun...@gmail.com] Sent: Wednesday, January 29, 2014 9:01 PM To: users@kafka.apache.org Subject: Re: Kafka performance test: "--request-num-acks -1" kills throughput Does

Re: Kafka performance test: "--request-num-acks -1" kills throughput

Does the result change with just 1 partition? Thanks, Jun On Wed, Jan 29, 2014 at 4:06 PM, Michael Popov wrote: > Hi, > > We need a reliable low-latency message queue that can scale. Kafka looks > like a right system for this role. > > I am running performance tests on multiple platforms: Lin

Re: Kafka performance test: "--request-num-acks -1" kills throughput

Michael, The producer client in 0.8 has a single thread that does blocking data sends one broker at a time. However, we are working on a rewrite of the producer with an improved design that can support higher throughput compared to the current one. We don't have any performance numbers to share ye

Kafka performance test: "--request-num-acks -1" kills throughput

Hi, We need a reliable low-latency message queue that can scale. Kafka looks like a right system for this role. I am running performance tests on multiple platforms: Linux and Windows. For test purposes I create topics with 2 replicas and multiple partitions. In all deployments running test pr

Re: Kafka performance testing

, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> / On Wed, Jan 29, 2014 at 4:34 PM, Michael Popov wrote: > Hi, > > I try to run Kafka performa

Kafka performance testing

Hi, I try to run Kafka performance tests on my hosts. I get this error message: [User@Client1 kafka_2.8.0-0.8.0]$ ./bin/kafka-producer-perf-test.sh Exception in thread "main" java.lang.NoClassDefFoundError: kafka/perf/ProducerPerformance Caused by: java.lang.ClassNotFoun

Re: kafka performance test

The results there are obtained using our ProducerPerformance tool. Thanks, Jun On Wed, Jan 15, 2014 at 1:14 AM, 陈小军 wrote: > I was just reading this: > https://cwiki.apache.org/confluence/display/KAFKA/Performance+testingHasthis > been implemented yet? I can't seem tofind it

kafka performance test

I was just reading this:https://cwiki.apache.org/confluence/display/KAFKA/Performance+testingHas this been implemented yet? I can't seem tofind it in the source. Best Regards Jerry

Re: [ANNOUNCEMENT] Kafka Performance Monitoring in SPM

This is really cool. We have an ecosystem page that lists projects that integrate with kafka. It would be good to list this there so people can find out about it: https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem -Jay On Tue, Oct 22, 2013 at 6:31 AM, Otis Gospodnetic < otis.gospodne...

Re: [ANNOUNCEMENT] Kafka Performance Monitoring in SPM

This is great! Thanks for sharing. On Oct 22, 2013 6:31 AM, "Otis Gospodnetic" wrote: > Hi, > > Just a quick mention here that we've recently added support for > monitoring Kafka to SPM. This is in addition to monitoring Hadoop, > HBase, SenseiDB, Solr, Elasticseach, JVM, OS metrics, custom metri

[ANNOUNCEMENT] Kafka Performance Monitoring in SPM

Hi, Just a quick mention here that we've recently added support for monitoring Kafka to SPM. This is in addition to monitoring Hadoop, HBase, SenseiDB, Solr, Elasticseach, JVM, OS metrics, custom metrics, etc. See: http://blog.sematext.com/2013/10/16/announcement-spm-performance-monitoring-for-ka

Re: Kafka performance

013 at 11:49 AM, Francis Dallaire < >> > francis.dalla...@ubisoft.com> wrote: >> > >> >> +1 ... very good suggestion. >> >> >> >> >> >> Sent from Samsung Mobile >> >> >> >> >> >> >>

Re: Kafka performance

ote: > > > >> +1 ... very good suggestion. > >> > >> > >> Sent from Samsung Mobile > >> > >> > >> > >> Original message > >> From: Stone > >> Date: 05-17-2013 2:16 AM (GMT-05:00

Re: Kafka performance

; >> Sent from Samsung Mobile >> >> >> >> Original message ---- >> From: Stone >> Date: 05-17-2013 2:16 AM (GMT-05:00) >> To: users@kafka.apache.org >> Subject: Re: Kafka performance >> >> >> Hi Kishore, >> &g

Re: Kafka performance

lla...@ubisoft.com> wrote: > +1 ... very good suggestion. > > > Sent from Samsung Mobile > > > > Original message > From: Stone > Date: 05-17-2013 2:16 AM (GMT-05:00) > To: users@kafka.apache.org > Subject: Re: Kafka performance > > > Hi

Re: Kafka performance

+1 ... very good suggestion. Sent from Samsung Mobile Original message From: Stone Date: 05-17-2013 2:16 AM (GMT-05:00) To: users@kafka.apache.org Subject: Re: Kafka performance Hi Kishore, Since a message is only exposed to the consumers after it is flushed. Maybe you

Re: Kafka performance

Hi Kishore, Since a message is only exposed to the consumers after it is flushed. Maybe you can try lower log.default.flush.interval.ms which controls the maximum time that a message in any topic is kept in memory before flushed to disk ( by default it's 3000ms ) ref - http://kafka.apache.org/07/

Kafka performance

Hello All, I am evaluating Kafka for use in a real time financial product. I am tying to calculate the time it takes for a message to be fetched by the consumer after it has been sent by a producer. I am finding that this takes a long time of between 900-1200 ms per message on Windows 7 single cor