Thanks Paul. Yes, I am using 0.8 beta1. I followed your suggestion to set
request.required.acks=1 and got the same result. No error message seen in
broker logs, the size of the partition files were after sending 1,000,000
events, the size of each event was 1KB :
00000000000000000000.index 10240 KB
00000000000000000000.log 0KB
The broker configurations:
num.partitions=5
log.flush.interval.messages=20000
log.flush.interval.ms=5000
log.flush.scheduler.interval.ms=1000
log.retention.hours=1
log.segment.bytes=1073741824
log.cleanup.interval.mins=30
queued.max.requests=16
fetch.purgatory.purge.interval.requests=100
producer.purgatory.purge.interval.requests=100
It works if I change the code to props.put("compression.codec", "0");
thanks,
xlu
On Thu, Aug 29, 2013 at 6:48 PM, Paul Mackles <pmack...@adobe.com> wrote:
> I assume this is kafka 0.8, right? Are there any corresponding errors in
> the broker logs? With the configuration below, I don't think any errors
> will be reported back to the producer.
>
> You could also try setting erquest.required.acks=1 to see if errors are
> reported back to the client.
>
> On 8/29/13 4:40 AM, "Lu Xuechao" <lux...@gmail.com> wrote:
>
> >Hi ,
> >
> >I am trying to enable gzip compression for my events. But after I switched
> >compression.codec to "1" I found the produced events were even not be
> >persisted to disk log file. Of course, the consumer could not receive any
> >compressed events. I sent 10,000 or more events but the broker's log file
> >not changed. Seems no events were actually send to broker? Below is my
> >producer's code:
> >
> > Properties props = new Properties();
> > props.put("serializer.class", "kafka.serializer.StringEncoder");
> > props.put("metadata.broker.list", "127.0.0.1:9092");
> > props.put("partitioner.class",
> >"kafka.producer.DefaultPartitioner");
> > props.put("queue.enqueue.timeout.ms", "-1");
> > props.put("request.required.acks", "0");
> > props.put("producer.type", "async");
> >
> > props.put("batch.num.messages", "100");
> >
> > props.put("compression.codec", "1");
> >
> > ProducerConfig config = new ProducerConfig(props);
> > producer = new Producer<String, String>(config);
> >
> > KeyedMessage<String, String> data = new KeyedMessage<String,
> >String>("topic1", messageStr, msg);
> > producer.send(data);
> >
> >
> >If I comment out this line of code : props.put("compression.codec", "1");
> >then everything works fine. Did I miss something?
> >
> >thanks,
> >xlu
>
>
