Re: Optimal Message Size for Kafka

2018-09-08 Thread Matt Veitas
Take a look at the built in producer performance tools that you can do your own benchmarking with your dataset. If compression is an option consider that as well. Here's a helpful cheatsheet to use: https://github.com/JNachtwey/kafka-cheat-sheets/blob/master/kafka-performance.md On Fri, Sep 7, 20

Re: Proxying the Kafka protocol

2019-03-19 Thread Matt Veitas
You might follow along with the Envoy proxy team and the work they are doing to support the Kafka binary protocol: https://github.com/envoyproxy/envoy/issues/2852 On Tue, Mar 19, 2019 at 11:46 AM Peter Bukowinski wrote: > https://docs.confluent.io/3.0.0/kafka-rest/docs/intro.html > > The Kafka R

Offset commit failed on partition - The request timed out

2019-04-17 Thread Matt Veitas
We have been seeing a number of timeouts for offset commits: `Offset commit failed on partition mytopic-12 at offset 16578269: The request timed out.`. These failed requests are sporadic and not causing a problem "yet", but we would rather like to understand what could the causes of something like