Kafka Batch and Fetch

2016-03-30 Thread manish jaiswal
Hi, I am new to Kafka ,I have a doubt in Kafka batching. In Producer 9.0 how batching works if I Set batchsize 10mb then all messages in 10 mb batch go to one offset in Kafka broker? In new high level consumer how fetchsize work? It's like per message fetchsize inside one offset or whole message

kafka broker sizing

2016-03-30 Thread manish jaiswal
Hi, Can we set Kafka broker size? Suppose my system space 500gb.can I set Kafka broker with 100gb in my system? Thanks Manish

flafka comsumer fetch size

2016-01-12 Thread manish jaiswal
i m trying to read more than 1mb msg from kafka using flume and i m getting fetch size error. where to define in flume config for fetch.message.max.bytes. error: kafka.common.MessageSizeTooLargeException: Found a message larger than the maximum fetch size of this consumer on topic test partiti