Re: Question on MessageSizeTooLargeException

2014-01-20 Thread Bae, Jae Hyeon
Nope, just packaging for Netflix cloud environment. The first one is, producer discovery(metadata.broker.list) is integrated with Netflix Eureka. The second one is, yammer metric library is connected with Netflix Servo. Except these two big things, I fixed a few lines to fit into our monitoring en

Kafka server occure java.nio.BufferUnderflowException

2014-01-20 Thread 陈小军
Hi all, in my kafka cluster, it alway ocuure following erro logs [2014-01-21 15:25:24,462] ERROR Closing socket for /10.99.116.54 because of error (kafka.network.Processor) java.nio.BufferUnderflowException at java.nio.Buffer.nextGetIndex(Buffer.java:498) at java.nio.HeapByte

Re: Question on MessageSizeTooLargeException

2014-01-20 Thread Jun Rao
What kind of customization are you performing? Are you changing the wire and on-disk protocols? Thanks, Jun On Mon, Jan 20, 2014 at 10:02 AM, Bae, Jae Hyeon wrote: > Due to short retention period, I don't have that log segment now. > > How I am developing kafka is, > > I forked apache/kafka i

Re: log.retention.bytes.per.topic does not trigger deletion

2014-01-20 Thread Guozhang Wang
Yes, once 0.8.1 is released we will add a separate quick-start page. On Sun, Jan 19, 2014 at 10:57 PM, Joe Stein wrote: > There are a lot of nuance changes to the QuickStart and other changes for > 0.8.1 that will need a separate page... Not just modifications. > > > /**

Re: Question on MessageSizeTooLargeException

2014-01-20 Thread Bae, Jae Hyeon
Due to short retention period, I don't have that log segment now. How I am developing kafka is, I forked apache/kafka into my personal repo and customized a little bit. I kept tracking 0.8 branch but you seems moved to trunk branch. I will update it to trunk branch or 0.8.0 tag. Thank you Best,

Re: Jruby kafka 0.8.0 producer and consumer

2014-01-20 Thread Joe Stein
Yup, thanks Joe! I updated the client page with it https://cwiki.apache.org/confluence/display/KAFKA/Clients /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop

Re: Jruby kafka 0.8.0 producer and consumer

2014-01-20 Thread Neha Narkhede
That's great. Thanks for sharing, Joe! On Fri, Jan 17, 2014 at 8:01 PM, Joseph Lawson wrote: > Hi everyone, > > I wanted to let you all know that my jruby-kafka library now has a working > producer. The consumer has been working quite well for some time but now > the producer is functional as w

Re: [Consumer code not working][Kafka Newbie]

2014-01-20 Thread Jun Rao
Could you check the following FAQ? https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whydoesmyconsumernevergetanydata ? https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Myconsumerseemstohavestopped,why ? Thanks, Jun On Mon, Jan 20, 2014 at 7:22 AM, Abhishek Bhattacharjee < abh

Re: Which properties should I sync if I want to increase message.max.bytes?

2014-01-20 Thread Jun Rao
Yes, you need to increase replica.message.max.bytes too since it controls the fetcher used for internal replication. Thanks, Jun On Sun, Jan 19, 2014 at 10:32 PM, Bae, Jae Hyeon wrote: > If I want to increase message.max.bytes up to 10MB *with compression*, Are > there any properties I sync e

Re: Question on MessageSizeTooLargeException

2014-01-20 Thread Jun Rao
Could you use our DumpLogSegment tool on the relevant log segment and see if the log is corrupted? Also, are you using the 0.8.0 release? Thanks, Jun On Sun, Jan 19, 2014 at 10:09 PM, Bae, Jae Hyeon wrote: > Hello > > I finally upgraded kafka 0.7 to kafka 0.8 and a few kafka 0.8 clusters are

Re: [Consumer code not working][Kafka Newbie]

2014-01-20 Thread Abhishek Bhattacharjee
Sorry I have sent both codes as consumer codes. This is the producer code. *Producer.java* package kafka.examples; import java.util.Properties; import kafka.producer.KeyedMessage; import kafka.producer.ProducerConfig; public class Producer/* extends Thread*/ { private final kafka.javaapi.pro

[Consumer code not working][Kafka Newbie]

2014-01-20 Thread Abhishek Bhattacharjee
Hello, I am new to kafka and facing some problem. My producer code works properly and sends data. But the consumer is not able to read it. Here are the codes for Producer and Consumer. Something is wrong with the Consumer.java code can someone please help with this. *Producer.java* package kafka

Re: Kafka not showing log/lag size even though there is data in logs

2014-01-20 Thread Gaurav Sehgal
actually was my fault finally figured it out. Made a mistake in creating the topic. On Mon, Jan 20, 2014 at 1:27 PM, Gaurav Sehgal wrote: > I am using Kafka version 0.8, I have data in Kafka logs and wanted to > replay that so I used the kafka tool to reset consumer offsetsto 0. > > Though now