Re: Advice for Kafka project in Africa...

2020-09-03 Thread Gaurav Bajaj
Hello Cedric, My 2 cents : We use Kafka alot but mostly for messaging and event streaming purpose. Using Kafka as Database is not a usecase i would look Kafka for. Ofcourse you can use it to store some intermediate states but having it as "system of records" would be stretched use case for Kafka.

Relation between "fetch.max.bytes" and "max.partition.fetch.bytes"

2019-08-13 Thread Gaurav Bajaj
Hello Kafka lovers, I have one doubt regarding these 2 configs in kafka 1.3.0 consumer , 1- fetch.max.bytes(default 50MB) and 2- max.partition.fetch.bytes(default 1MB). Objective is to pull 500MB of data in single poll from Java consumer of Kafka. Problem is my consumer is getting stuck(not pollin

Re: Kafka Producer Partition Key Selection

2018-08-29 Thread Gaurav Bajaj
Hello Senthil, In our case we use NULL as message Key to achieve even distribution in producer. With that we were able to achieve very even distribution with that. Our Kafka client version is 0.10.1.0 and Kafka broker version is 1.1 Thanks, Gaurav On Wed, Aug 29, 2018 at 9:15 AM, SenthilKumar K