Hi,
I'm using Kafka
0.8.2.1 SimpleConsumer . Could somebody clarify the meaning of a few config
parameters for the SimpleConsumer and FetchREquestBuilder? My questions are
below:
-- Q1: in the signature of the SimpleConsumer constructor I see the Int
'soTimeout' parameter -
what is the meaning of this timeout? Is this a timeout to connect to the Kafka
broker? Timeout on getting a response from any [or specific??] request to Kafka
(like FetchRequest)? Something else?
kafka.javaapi.consumer.SimpleConsumer
(val host: String,
val port: Int,
val soTimeout: Int,
val bufferSize: Int,
val clientId: String)
-- Q2: also, SimpleConsumer constructor takes Int 'bufferSize' parameter. What
is the meaning
of it? Is this how many bytes SimpleConsumer will read when a fetchRequest is
issued?
-- When building FetchREquest via FetchRequestBuilder (see below), I also need
to specify 'fetchSize':
FetchRequest req= newFetchRequestBuilder ()
.clientId(kafkaGroupId)
.addFetch(topic, partition, offset, fetchSizeInBytes)
.build();
Looking at the source code of the FetchRequestBuilder , I think (I'm not a
Scala pro) those calls
translate into the below method calls - and there the final parameter passed
into the FetchRequest is called 'minBytes', hinting that this is not the exact
fetch size, possibly….
class FetchRequestBuilder():
def addFetch(topic: String, partition: Int, offset: Long, fetchSize: Int)
def build() = {
val fetchRequest= FetchRequest(versionId, correlationId.getAndIncrement,
clientId, replicaId, maxWait, minBytes, requestMap.toMap)
FetchRequest(versionId: Short = FetchRequest.CurrentVersion,
correlationId: Int = FetchRequest.DefaultCorrelationId,
clientId: String = ConsumerConfig.DefaultClientId,
replicaId: Int = Request.OrdinaryConsumerId,
maxWait: Int = FetchRequest.DefaultMaxWait,
minBytes: Int = FetchRequest.DefaultMinBytes,
...)
So, my final question is:
-- Q3: how do 'bufferSize' and 'fetchSize/minBytes' relate? What exactly do
they define? Do I have to make
sure one is smaller or grater than the other?
Thanks,
Marina
