for example the issue can be as below:
from the producer side, say 1000 messages is sent, and no exception found from
the callback;
from the kafka broker side, the log size only increase 900;
as a result only 900 messages received from the consumer;
conclusion:
the problem is we cannot get the exception from the producer callback, thus we
don't know the sent failure;
At 2015-11-12 16:46:55, "Prabhjot Bharaj" <prabhbha...@gmail.com> wrote:
>Hi,
>
>Just to confirm that the number of messages produced are what you are
>seeing, What does GetOffsetShell report for this topic ?
>
>Regards,
>Prabhjot
>
>On Thu, Nov 12, 2015 at 2:13 PM, Hawin Jiang <hawin.ji...@gmail.com> wrote:
>
>> Hi Jinxing
>>
>> I don't think we can resolve this issue by increasing producers. if I
>> increased more producers, it should lost more messages.
>>
>> I just test two producers.
>> Thread Producer 1 has 83064 messages in producer side and 82273 messages in
>> consumer side
>> Thread Producer 2 has 89844 messages in producer side and 88892 messages in
>> consumer side.
>>
>> Thanks.
>>
>>
>>
>> Best regards
>> Hawin
>>
>>
>> On Thu, Nov 12, 2015 at 12:24 AM, jinxing <jinxing6...@126.com> wrote:
>>
>> > maybe there some changes in 0.9.0.0;
>> >
>> >
>> > but still you can try increase producer sending rate, and see if there
>> are
>> > message lost but no exception;
>> >
>> >
>> > note that, to increase the producer sending rate, you must have enough
>> > producer 'power';
>> >
>> >
>> > in my case, I have 50 producer sending message at the same time : )
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > At 2015-11-12 16:16:23, "Hawin Jiang" <hawin.ji...@gmail.com> wrote:
>> > >Hi Jinxing
>> > >
>> > >I am using kafka_2.10-0.9.0.0-SNAPSHOT. I have downloaded source code
>> and
>> > >installed it last week.
>> > >
>> > >I saw 97446 messages have been sent to kafka successfully.
>> > >
>> > >So far, I have not found any failed messages.
>> > >
>> > >
>> > >
>> > >Best regards
>> > >Hawin
>> > >
>> > >On Thu, Nov 12, 2015 at 12:07 AM, jinxing <jinxing6...@126.com> wrote:
>> > >
>> > >> Hi, what version are you using ?
>> > >>
>> > >>
>> > >> i am using 0.8.2.0, and I encountered this problem before;
>> > >>
>> > >>
>> > >> it seems that if the message rate of the producer side is to high,
>> some
>> > of
>> > >> the messages will lost;
>> > >>
>> > >>
>> > >> also, i found that the callback method of the producer 'send' API is
>> not
>> > >> reliable;
>> > >>
>> > >>
>> > >> only successful sent message will trigger the callback, but the failed
>> > >> ones don't;
>> > >>
>> > >>
>> > >> you saw this?
>> > >>
>> > >>
>> > >>
>> > >>
>> > >>
>> > >>
>> > >>
>> > >>
>> > >> At 2015-11-12 16:01:17, "Hawin Jiang" <hawin.ji...@gmail.com> wrote:
>> > >> >Hi All
>> > >> >
>> > >> >I have sent messages to Kafka for one minute. I found 97446 messages
>> > in
>> > >> >producer side and 96896 messages in consumer side for Case 1.
>> > >> >I also tried case 2. I have faced the same issues. The number is not
>> > >> match
>> > >> >between producer and consumer.
>> > >> >Can someone take a look at this issue?
>> > >> >Thanks.
>> > >> >
>> > >> >
>> > >> >Case 1:
>> > >> >
>> > >> >long startTime = System.currentTimeMillis();
>> > >> >long maxDurationInMilliseconds = 1 * 60 * 1000;
>> > >> >int messageNo = 0;
>> > >> >while (true) {
>> > >> >if (System.currentTimeMillis() <= startTime
>> > >> >+ maxDurationInMilliseconds) {
>> > >> >messageNo++;
>> > >> >String messageStr = "KAFKA_"+messageNo;
>> > >> >System.out.println("Message: "+messageNo);
>> > >> >producer.send(new KeyedMessage<Integer, String>(topic,messageStr));
>> > >> >} else {
>> > >> >producer.close();
>> > >> >System.out.println("Total kafka Message: "+messageNo);
>> > >> >break;
>> > >> >}
>> > >> >}
>> > >> >
>> > >> >
>> > >> >Case 2:
>> > >> >
>> > >> >for (int i=1;i<=12000;i++)
>> > >> >String messageStr = "KAFKA_"+i;
>> > >> >producer.send(new KeyedMessage<Integer, String>(topic,messageStr));
>> > >> >
>> > >> >
>> > >> >
>> > >> >Best regards
>> > >> >Hawin
>> > >>
>> >
>>
>
>
>
>--
>---------------------------------------------------------
>"There are only 10 types of people in the world: Those who understand
>binary, and those who don't"
