Hi,
Kafka's performance all comes from batching. There's going to be a huge
perf impact from limiting your batching like that, and that's likely the
issue. I'd recommend designing your system around Kafka's batching model,
which involves large numbers of messages per fetch request.
Thanks
Tom Cr
Hi Tom,
Thank you for your help. I have only one broker. I used kafka production
server configuration listed in kafka's documentation page:
http://kafka.apache.org/documentation.html#prodconfig . I have increased
the flush interval and number of messages to prevent the disk from becoming
the bottle
What's your server setup for the brokers and consumers? Generally I'd
expect something to be exhausted here and that to end up being the
bottleneck.
Thanks
Tom Crayford
Heroku Kafka
On Mon, May 23, 2016 at 7:32 PM, Yazeed Alabdulkarim <
y.alabdulka...@gmail.com> wrote:
> Hi,
> I am running simp
Hi,
I am running simple experiments to evaluate the scalability of Kafka
consumers with respect to the number of partitions. I assign every consumer
to a specific partition. Each consumer polls the records in its assigned
partition and print the first one, then polls again from the offset of the
pr
Thank you for your reply. I am running tests using a simple application
with one broker. That is why I am running everything on a single machine.
For the scalability, my application's throughput scales by more than 2x
going from 1 consumer/partition to 2 consumers/partitions which is great.
However
Hi there,
Firstly, I'd recommend not running the consumers and the brokers on the
same machine. Are you running multiple brokers? If not, that'd be my first
recommendation (it sounds like you might not be).
Secondly, yes, consumers scale up with partitions. At most you can have the
same number of
Hi,
For Kafka consumers, is it expected that the throughput will scale linearly
as I increase the number of consumers/partitions?
Also, I keep getting this info message: "Kafka Consumer Marking the
coordinator 2147483647 dead." What is the problem? How can I fix it? My
program continues without a
