Re: Kafka throw InvalidMessageException and lost data

2013-03-24 Thread 王国栋
Hi Neha, I am sorry, we couldn't find the detail reason of this bug. But we do believe that the bug is in java.nio package. Something bad may happen during sync-producer sends data in byteBuffer or broker receives data from the channel. The serialize/deserialize process is right, because we enabl

Re: Kafka throw InvalidMessageException and lost data

2013-03-24 Thread Neha Narkhede
That's great. Would you know the details of the bug though ? It will be helpful for the community to understand. Thanks, Neha On Sunday, March 24, 2013, 王国栋 wrote: > Hi Neha & Jun, > > I think we have found the reason of this bug. > It is related to jdk versions. In the beginning, we use jdk1.6.

Re: Kafka throw InvalidMessageException and lost data

2013-03-24 Thread 王国栋
Hi Neha & Jun, I think we have found the reason of this bug. It is related to jdk versions. In the beginning, we use jdk1.6.20 to run the test. Yesterday, we used jdk1.6.38. Everything is OK now. Thanks a lot for all your kind help and answer. On Thu, Mar 21, 2013 at 1:05 PM, Neha Narkhede wr

Re: Kafka throw InvalidMessageException and lost data

2013-03-20 Thread Neha Narkhede
Do you mind filing a bug and attaching the reproducible test case there ? Thanks, Neha On Wednesday, March 20, 2013, 王国栋 wrote: > Hi Jun, > > We use one thread with one sync produce to send data to broker > (QPS:10k-15k, each log is about 1k bytes). The problem is reproduced. > > We have used Pr

Re: Kafka throw InvalidMessageException and lost data

2013-03-20 Thread 王国栋
Hi Jun, We use one thread with one sync produce to send data to broker (QPS:10k-15k, each log is about 1k bytes). The problem is reproduced. We have used Producer and SyncProducer in our test. The same Exception appears. Thanks. On Thu, Mar 21, 2013 at 12:19 PM, Jun Rao wrote: > How many th

Re: Kafka throw InvalidMessageException and lost data

2013-03-20 Thread Jun Rao
How many threads are you using? Thanks, Jun On Wed, Mar 20, 2013 at 7:33 PM, Yang Zhou wrote: > Sorry, I made a mistake, we use many threads producing at same time. > > > 2013/3/20 Jun Rao > > > How many producer instances do you have? Can you reproduce the problem > with > > a single produce

Re: Kafka throw InvalidMessageException and lost data

2013-03-20 Thread Yang Zhou
Sorry, I made a mistake, we use many threads producing at same time. 2013/3/20 Jun Rao > How many producer instances do you have? Can you reproduce the problem with > a single producer? > > Thanks, > > Jun > > On Wed, Mar 20, 2013 at 12:29 AM, 王国栋 wrote: > > > Hi Jun, > > > > we do not use any

Re: Kafka throw InvalidMessageException and lost data

2013-03-20 Thread Yang Zhou
There was only one producer running in all our tests. Beside, we also tried the low-level java api, the problem still shows up. Thanks 2013/3/20 Jun Rao > How many producer instances do you have? Can you reproduce the problem with > a single producer? > > Thanks, > > Jun > > On Wed, Mar 20, 20

Re: Kafka throw InvalidMessageException and lost data

2013-03-20 Thread Jun Rao
How many producer instances do you have? Can you reproduce the problem with a single producer? Thanks, Jun On Wed, Mar 20, 2013 at 12:29 AM, 王国栋 wrote: > Hi Jun, > > we do not use any compression in our test. > > We deploy producer and broker in the same machine. The problem still > exists. We

Re: Kafka throw InvalidMessageException and lost data

2013-03-20 Thread 王国栋
Hi Neha, Our test is reproducible. We use a log replay tool to send the same part of log to kafka. We will check the log file latter and try to find some more hints. Till now, it seems that if the batch size in sync producer is small, the problem is hard to reproduce. Thanks. On Tue, Mar 19,

Re: Kafka throw InvalidMessageException and lost data

2013-03-20 Thread 王国栋
Hi Jun, we do not use any compression in our test. We deploy producer and broker in the same machine. The problem still exists. We use sync producer, and send one message at a time(no batch now). We find that when the qps reaches more than 40k, the exception appears. So I don't think it's the und

Re: Kafka throw InvalidMessageException and lost data

2013-03-19 Thread Jun Rao
It basically means that the broker is expecting to read certain number of bytes in a buffer received from socket, but there are fewer bytes than expected in the buffer. Possible causes are (1) a bug in Kafka request serialization/deserialization logic; (2) corruption in the underlying system such a

Re: Kafka throw InvalidMessageException and lost data

2013-03-18 Thread Neha Narkhede
Do you mind trying out the DumpLogSegment tool on the log segment for the corrupted topic. That will validate if the log data is corrupted. Also, Is your test reproducible ? We ran into a similar issue in production but could not reproduce it. Thanks, Neha On Monday, March 18, 2013, Helin Xiang w

Re: Kafka throw InvalidMessageException and lost data

2013-03-18 Thread Helin Xiang
thanks Jun. we are using java producer. does the last exception "java.lang.IllegalArgumentException at java.nio.Buffer.limit(Buffer.java:266) " also means the broker received corrupted messages ? sorry i am not familiar with java nio. On Tue, Mar 19, 2013 at 12:58 PM, Jun Rao wrote: > H

Re: Kafka throw InvalidMessageException and lost data

2013-03-18 Thread Jun Rao
Hmm, both log4j messages suggest that the broker received some corrupted produce requests. Are you using the java producer? Also, we have seen that network router problems caused corrupted requests before. Thanks, Jun On Mon, Mar 18, 2013 at 8:22 PM, Helin Xiang wrote: > Hi， > We were doing so