Regarding the issue that adding more partitions kill the performance: I would suspect it maybe due to not-sufficient batching. Note that in the new producer batching is done per-partition, and if linger.ms setting low, partition data may not be batched enough before they got sent to the brokers. Also since the new producer will drain all partitions that belongs to the same broker, when one of them hits either linger time or batch size, when you only have one or a few brokers this will further exaggerate the not-sufficient-batching issue. So monitoring on average batch size would be a good idea.
Guozhang On Wed, May 13, 2015 at 7:47 PM, Jay Kreps <jay.kr...@gmail.com> wrote: > Hey Garry, > > Super interesting. We honestly never did a ton of performance tuning on the > producer. I checked the profiles early on in development and we fixed a few > issues that popped up in deployment, but I don't think anyone has done a > really scientific look. If you (or anyone else) want to dive into things I > suspect it could be improved. > > Becket is exactly right. There are two possible bottlenecks you can hit in > the producer--the single background sender thread and the per-partition > lock. You can check utilization on the background thread with jvisualvm > (it's named something like kafka-producer-network-thread). The locking is > fairly hard to improve. > > It's a little surprising that adding partitions caused a large decrease in > performance. Generally this is only the case if you override the flush > settings on the broker to force fsync more frequently. > > The ISR issues under heavy load are probably fixable, the issue is > discussed a bit here: > > http://blog.confluent.io/2015/04/07/hands-free-kafka-replication-a-lesson-in-operational-simplicity/ > > The producer settings that may matter for performance are: > acks > batch.size (though beyond 32k I didn't see much improvement) > linger.ms (setting >= 1 may help a bit) > send.buffer.bytes (maybe, but probably not) > > Cheers, > > -Jay > > On Wed, May 13, 2015 at 3:42 PM, Jiangjie Qin <j...@linkedin.com.invalid> > wrote: > > > Thanks for sharing this, Garry. I actually did similar tests before but > > unfortunately lost the test data because my laptop rebooted and I forgot > > to save the dataŠ > > > > Anyway, several things to verify: > > > > 1. Remember KafkaProducer holds lock per partition. So if you have only > > one partition in the target topic and many application threads. Lock > > contention could be an issue. > > > > 2. It matters that how frequent the sender thread wake up and runs. You > > can take a look at the following sensors to further verify whether the > > sender thread really become a bottleneck or not: > > Select-rate > > Io-wait-time-ns-avg > > Io-time-ns-avg > > > > 3. Batch size matters, so take a look at the sensor batch-size-avg and > see > > if the average batch size makes sense or not. > > > > Looking forward to your further profiling. My thinking is that unless you > > are sending very small messages to a small number of partitions. You > don¹t > > need to worry about to use more than one producer. > > > > Thanks. > > > > Jiangjie (Becket) Qin > > > > > > > > On 5/13/15, 2:40 PM, "Garry Turkington" <g.turking...@improvedigital.com > > > > wrote: > > > > >Hi, > > > > > >I talked with Gwen at Strata last week and promised to share some of my > > >experiences benchmarking an app reliant on the new producer. I'm using > > >relatively meaty boxes running my producer code (24 core/64GB RAM) but I > > >wasn't pushing them until I got them on the same 10GB fabric as the > Kafka > > >cluster they are using (saturating the prior 1GB NICs was just too > easy). > > >There are 5 brokers, 24 core/192GB RAM/8*2TB disks, running 0.8.2.1. > > > > > >With lots of cores and a dedicated box the question was then how to > > >deploy my application. In particular how many worker threads and how > many > > >instances of the KafkaProducer to share amongst them. I also wanted to > > >see how things would change as I scale up the thread count. > > > > > >I ripped out the data retrieval part of my app (it reads from S3) and > > >instead replaced it with some code to produce random records of average > > >size 500 bytes but varying between 250 and 750. I started the app > > >running, ignored the first 25m messages then measured the timing for the > > >next 100m and calculated the average messages/sec written to Kafka > > >across that run. > > > > > >Starting small I created 4 application threads with a range of > approaches > > >to sharing KafkaProducer instances. The records written to the Kafka > > >cluster per second were as follows: > > > > > >4 threads all sharing 1 client: 332K > > >4 threads sharing 2 clients: 333K > > >4 threads, dedicated client per thread: 310K > > > > > >Note that when I had 2 KafkaProducer clients as in the second line above > > >each was used by 2 threads. Similar below, number of threads/number of > > >clients is the max number of threads per KafkaProducer instance. > > > > > >As can be seen from the above there's not much in it. Scaling up to 8 > > >application threads the numbers looked like: > > > > > >8 threads sharing 1 client: 387K > > >8 threads sharing 2 clients: 609K > > >8 threads sharing 4 clients: 628K > > >8 threads with dedicated client per thread: 527K > > > > > >This time sharing a single producer client across all threads has by > far > > >the worse performance and isn't much better than when using 4 > > >application threads. The 2 and 4 client options are much better and are > > >in the ballpark of 2x the 4 thread performance. A dedicated client per > > >thread isn't quite as good but isn't so far off to be unusable. So then > > >taking it to 16 application threads: > > > > > >16 threads sharing 1 client: 380K > > >16 threads sharing 2 clients: 675K > > >16 threads sharing 4 clients: 869K > > >16 threads sharing 8 clients: 733K > > >16 threads with a dedicated client per thread: 491K > > > > > >This gives a much clearer performance curve. The 16 thread/4 producer > > >client is by far the best performance but it is still far from 4x the > > >4-thread or 2x the 8-thread mark. At this point I seem to be hitting > some > > >limiting factor. On the client machine memory was still lightly used, > > >network was peaking just over 4GB/sec but CPU load was showing 1 minute > > >load averages around 18-20. CPU load did seem to increase with as did > the > > >number of KafkaProducer instances but that is more a conclusion from > > >memory and not backed by hard numbers. > > > > > >For completeness sake I did do a 24 thread test but the numbers are as > > >you'd expect. 1 client and 24 both showed poor performance. 4,6 or 8 > > >clients (24 has more ways of dividing it by 2!) all showed performance > > >around that of the 16 thread/4 client run above. The other configs were > > >in-between. > > > > > >With my full application I've found the best deployment so far is to > have > > > multiple instances running on the same box. I can get much better > > >performance from 3 instances each with 8 threads than 1 instance with 24 > > >threads. This is almost certainly because when adding in my own > > >application logic and the AWS clients there is just a lot more > contention > > >- not to mention much more i/o waits -- in each application instance. > The > > >benchmark variant doesn't have as much happening but just to compare I > > >ran a few concurrent instances: > > > > > >2 copies of 8 threads sharing 4 clients: 780K total > > >2 copies of 8 threads sharing 2 clients: 870K total > > >3 copies of 8 threads sharing 2 clients: 945k total > > > > > >So bottom line - around 900K/sec is the max I can get from one of these > > >hosts for my application. At which point I brought a 2nd host to bear > and > > >ran 2 concurrent instances of the best performing config on each: > > > > > >2 copies of 16 threads sharing 4 clients on 2 hosts: 1458k total > > > > > >This doesn't quite give 2x the single box performance but it does show > > >that the cluster has capacity to spare beyond what the single client > host > > >can drive. This was also backed up by the metrics on the brokers, they > > >got busy but moderately so given the amount of work they were doing. > > > > > >At this point things did get a bit 'ragged edge'. I noticed a very high > > >rate of ISR churn on the brokers, it looked like the replicas were > having > > >trouble keeping up with the master and hosts were constantly being > > >dropped out then re-added to the ISR. I had set the test topic to have a > > >relatively low partition count (1 per spindle) so I doubled that to see > > >if it could help the ISRs remain stable. And my performance fell > through > > >the floor. So whereas I thought this was an equation involving > > >application threads and producer instances perhaps partition count is a > > >third. I need look into that some more but so far it looks like that for > > >my application - I'm not suggesting this is a universal truth -- sharing > > >a KafkaProducer instance amongst around 4 threads is the sweet spot. > > > > > >I'll be doing further profiling of my application so I'll flag to the > > >list anything that appears within the producer itself. And because 900K > > >messages/sec was so close to a significant number I modified my code > that > > >generates the messages to keep the key random for each message but to > use > > >repeated message bodies across multiple messages. At which point 1.05m > > >messages/sec was possible - from a single box. Nice. :) > > > > > >This turned out much longer than planned, I probably should have blogged > > >this somewhere. If anyone reads this far hope it is useful or of > > >interest, I'd be interested in hearing if the profiles I'm seeing are > > >expected and if any other tests would be useful. > > > > > >Regards > > >Garry > > > > > -- -- Guozhang
