If you start with an empty topic, the log size on the broker gives you the total bytes sent.
Thanks, Jun On Wed, May 28, 2014 at 10:38 AM, Maung Than <maung_t...@apple.com> wrote: > > >>>> The bytes-in-rate reported on the broker is the post-compression rate. > > Based on that fact, I am trying to calculate the volume by multiplying > duration and rate, i.e, (end.time - start.time) X MB.sec , > but it does not give me the total.data.sent.in.MB.I put the duration in > the brackets below: > > Any thoughts? > > > 2014-05-27 14:10:37:347, 2014-05-27 14:11:15:546, 0, 420, 1000, > 2002.72, 52.4285, 5000000, 130893.4789 ( 34 sec ) > > > 2014-05-27 13:33:31:493, 2014-05-27 13:34:11:862, 2, 420, 1000, > 2002.72, 49.6102, 5000000, 123857.4153 ( 40 sec ) > > Thanks, > Maung > > > On May 28, 2014, at 7:37 AM, Jun Rao <jun...@gmail.com> wrote: > > > The bytes-in-rate reported on the broker is the post-compression rate. > > > > Thanks, > > > > Jun > > > > > > On Tue, May 27, 2014 at 9:16 PM, Maung Than <maung_t...@apple.com> > wrote: > > > >> Any idea when and which release that could be included in? > >> > >> We would like to have it sooner and can we do something about it? > >> > >> Thanks, > >> Maung > >> > >> On May 27, 2014, at 5:10 PM, Guozhang Wang <wangg...@gmail.com> wrote: > >> > >>> Maung, > >>> > >>> Yes, the throughput computed here is based on pre-compression bytes. In > >> the > >>> old producer we do not have a metric exposing the compression ratio, in > >> the > >>> new producer available in current trunk we can easily add that metric. > I > >>> have just created a ticket for this a moment ago. > >>> > >>> https://issues.apache.org/jira/browse/KAFKA-1472 > >>> > >>> Guozhang > >>> > >>> > >>> > >>> On Tue, May 27, 2014 at 2:41 PM, Maung Than <maung_t...@apple.com> > >> wrote: > >>> > >>>> > >>>> Hi All, > >>>> > >>>> We are seeing total data sent below is the same with or without > >>>> compression. > >>>> Is it always raw data? > >>>> If so is there a way we can get the compressed volume? > >>>> > >>>> start.time, end.time, compression, message.size, batch.size, > >>>> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec > >>>> [2014-05-27 14:10:37,471] WARN Property reconnect.interval is not > valid > >>>> (kafka.utils.VerifiableProperties) > >>>> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". > >>>> SLF4J: Defaulting to no-operation (NOP) logger implementation > >>>> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for > >> further > >>>> details. > >>>> [2014-05-27 14:10:37,547] WARN Property reconnect.interval is not > valid > >>>> (kafka.utils.VerifiableProperties) > >>>> [2014-05-27 14:10:37,549] WARN Property reconnect.interval is not > valid > >>>> (kafka.utils.VerifiableProperties) > >>>> [2014-05-27 14:10:37,552] WARN Property reconnect.interval is not > valid > >>>> (kafka.utils.VerifiableProperties) > >>>> [2014-05-27 14:10:37,560] WARN Property reconnect.interval is not > valid > >>>> (kafka.utils.VerifiableProperties) > >>>> 2014-05-27 14:10:37:347, 2014-05-27 14:11:15:546, 0, 420, 1000, > 2002.72, > >>>> 52.4285, 5000000, 130893.4789 > >>>> > >>>> > >>>> -bash-4.1$ > >>>> > /Users/worun/kafkabuild/kafka-0.8.1-src/bin/kafka-producer-perf-test.sh > >>>> --broker-list vp21q12ic-hpaj020921:9092 --messages 5000000 --topic > >> imessage > >>>> --threads 5 --message-size 420 --batch-size 1000 --compression-codec 2 > >>>> start.time, end.time, compression, message.size, batch.size, > >>>> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec > >>>> [2014-05-27 13:33:31,616] WARN Property reconnect.interval is not > valid > >>>> (kafka.utils.VerifiableProperties) > >>>> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". > >>>> SLF4J: Defaulting to no-operation (NOP) logger implementation > >>>> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for > >> further > >>>> details. > >>>> [2014-05-27 13:33:31,692] WARN Property reconnect.interval is not > valid > >>>> (kafka.utils.VerifiableProperties) > >>>> [2014-05-27 13:33:31,694] WARN Property reconnect.interval is not > valid > >>>> (kafka.utils.VerifiableProperties) > >>>> [2014-05-27 13:33:31,697] WARN Property reconnect.interval is not > valid > >>>> (kafka.utils.VerifiableProperties) > >>>> [2014-05-27 13:33:31,705] WARN Property reconnect.interval is not > valid > >>>> (kafka.utils.VerifiableProperties) > >>>> 2014-05-27 13:33:31:493, 2014-05-27 13:34:11:862, 2, 420, 1000, > 2002.72, > >>>> 49.6102, 5000000, 123857.4153 > >>>> > >>>> Thanks, > >>>> Maung > >>>> > >>> > >>> > >>> > >>> -- > >>> -- Guozhang > >> > >> > >
