>>>> The bytes-in-rate reported on the broker is the post-compression rate.
Based on that fact, I am trying to calculate the volume by multiplying duration and rate, i.e, (end.time - start.time) X MB.sec , but it does not give me the total.data.sent.in.MB.I put the duration in the brackets below: Any thoughts? 2014-05-27 14:10:37:347, 2014-05-27 14:11:15:546, 0, 420, 1000, 2002.72, 52.4285, 5000000, 130893.4789 ( 34 sec ) 2014-05-27 13:33:31:493, 2014-05-27 13:34:11:862, 2, 420, 1000, 2002.72, 49.6102, 5000000, 123857.4153 ( 40 sec ) Thanks, Maung On May 28, 2014, at 7:37 AM, Jun Rao <jun...@gmail.com> wrote: > The bytes-in-rate reported on the broker is the post-compression rate. > > Thanks, > > Jun > > > On Tue, May 27, 2014 at 9:16 PM, Maung Than <maung_t...@apple.com> wrote: > >> Any idea when and which release that could be included in? >> >> We would like to have it sooner and can we do something about it? >> >> Thanks, >> Maung >> >> On May 27, 2014, at 5:10 PM, Guozhang Wang <wangg...@gmail.com> wrote: >> >>> Maung, >>> >>> Yes, the throughput computed here is based on pre-compression bytes. In >> the >>> old producer we do not have a metric exposing the compression ratio, in >> the >>> new producer available in current trunk we can easily add that metric. I >>> have just created a ticket for this a moment ago. >>> >>> https://issues.apache.org/jira/browse/KAFKA-1472 >>> >>> Guozhang >>> >>> >>> >>> On Tue, May 27, 2014 at 2:41 PM, Maung Than <maung_t...@apple.com> >> wrote: >>> >>>> >>>> Hi All, >>>> >>>> We are seeing total data sent below is the same with or without >>>> compression. >>>> Is it always raw data? >>>> If so is there a way we can get the compressed volume? >>>> >>>> start.time, end.time, compression, message.size, batch.size, >>>> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec >>>> [2014-05-27 14:10:37,471] WARN Property reconnect.interval is not valid >>>> (kafka.utils.VerifiableProperties) >>>> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". >>>> SLF4J: Defaulting to no-operation (NOP) logger implementation >>>> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for >> further >>>> details. >>>> [2014-05-27 14:10:37,547] WARN Property reconnect.interval is not valid >>>> (kafka.utils.VerifiableProperties) >>>> [2014-05-27 14:10:37,549] WARN Property reconnect.interval is not valid >>>> (kafka.utils.VerifiableProperties) >>>> [2014-05-27 14:10:37,552] WARN Property reconnect.interval is not valid >>>> (kafka.utils.VerifiableProperties) >>>> [2014-05-27 14:10:37,560] WARN Property reconnect.interval is not valid >>>> (kafka.utils.VerifiableProperties) >>>> 2014-05-27 14:10:37:347, 2014-05-27 14:11:15:546, 0, 420, 1000, 2002.72, >>>> 52.4285, 5000000, 130893.4789 >>>> >>>> >>>> -bash-4.1$ >>>> /Users/worun/kafkabuild/kafka-0.8.1-src/bin/kafka-producer-perf-test.sh >>>> --broker-list vp21q12ic-hpaj020921:9092 --messages 5000000 --topic >> imessage >>>> --threads 5 --message-size 420 --batch-size 1000 --compression-codec 2 >>>> start.time, end.time, compression, message.size, batch.size, >>>> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec >>>> [2014-05-27 13:33:31,616] WARN Property reconnect.interval is not valid >>>> (kafka.utils.VerifiableProperties) >>>> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". >>>> SLF4J: Defaulting to no-operation (NOP) logger implementation >>>> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for >> further >>>> details. >>>> [2014-05-27 13:33:31,692] WARN Property reconnect.interval is not valid >>>> (kafka.utils.VerifiableProperties) >>>> [2014-05-27 13:33:31,694] WARN Property reconnect.interval is not valid >>>> (kafka.utils.VerifiableProperties) >>>> [2014-05-27 13:33:31,697] WARN Property reconnect.interval is not valid >>>> (kafka.utils.VerifiableProperties) >>>> [2014-05-27 13:33:31,705] WARN Property reconnect.interval is not valid >>>> (kafka.utils.VerifiableProperties) >>>> 2014-05-27 13:33:31:493, 2014-05-27 13:34:11:862, 2, 420, 1000, 2002.72, >>>> 49.6102, 5000000, 123857.4153 >>>> >>>> Thanks, >>>> Maung >>>> >>> >>> >>> >>> -- >>> -- Guozhang >> >>
