Hi, currently, Flink doesn't have code build in for throughput (rate) and latency measurements. Its a planned feature for the future and some first steps are done into that direction. Check out this code for measuring throughput: https://github.com/rmetzger/flink-state-watermarks/blob/master/src/main/java/com/dataartisans/utils/ThroughputLogger.java (It will log the rates to the log file)
For the latency, I would attach timestamps to your records and measure the delay by comparing the timestamps to the current system time (make sure the clocks on all machines are in sync) On Mon, May 16, 2016 at 11:43 PM, prateekarora <prateek.arora...@gmail.com> wrote: > thanks for the information . > I want to collect some performance metrics like throughput , latency etc > .thats why looking for these data to collect some information . > > how can i collect some performance data in flink ? > > Regards > Prateek > > > On Mon, May 16, 2016 at 2:32 PM, Fabian Hueske-2 [via Apache Flink User > Mailing List archive.] <[hidden email] > <http:///user/SendEmail.jtp?type=node&node=6945&i=0>> wrote: > >> Hi Prateek, >> >> the missing numbers are an artifact from how the stats are collected. >> ATM, Flink does only collect these metrics for data which is sent over >> connections *between* Flink operators. >> Since sources and sinks connect to external systems (and not Flink >> operators), the dash board does not show receiving byte for sources and >> sent bytes for sinks. >> >> Best, Fabian >> >> 2016-05-16 23:02 GMT+02:00 prateekarora <[hidden email] >> <http:///user/SendEmail.jtp?type=node&node=6944&i=0>>: >> >>> Hi >>> >>> In my flink kafka streaming application i am fetching data from one topic >>> and then process and sent to output topic . my application is working >>> fine >>> but flink dashboard shows Source [Bytes/records Received] and Sink >>> [Bytes/records sent] is zero. >>> >>> >>> Duration Name Bytes Records >>> Bytes >>> Records Status >>> Received Received >>> sent sent >>> 3m 15s Source: Custom Source 0 B 0 >>> 249 MB >>> 3,540 RUNNING >>> >>> >>> 3m 15s Map -> Sink: Unnamed 203 MB 2,883 0 B >>> 0 >>> RUNNING >>> >>> >>> >>> < >>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/n6941/flink-kafka.png >>> > >>> >>> what is the reason for not showing correct number ? >>> >>> Regards >>> Prateek >>> >>> >>> >>> >>> >>> >>> >>> -- >>> View this message in context: >>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-Kafka-Streaming-Source-Bytes-records-Received-and-Sink-Bytes-records-sent-show-zero-messages-tp6941.html >>> Sent from the Apache Flink User Mailing List archive. mailing list >>> archive at Nabble.com. >>> >> >> >> >> ------------------------------ >> If you reply to this email, your message will be added to the discussion >> below: >> >> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-Kafka-Streaming-Source-Bytes-records-Received-and-Sink-Bytes-records-sent-show-zero-messages-tp6941p6944.html >> To unsubscribe from Flink Kafka Streaming - Source [Bytes/records >> Received] and Sink [Bytes/records sent] show zero messages, click here. >> NAML >> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >> > > > ------------------------------ > View this message in context: Re: Flink Kafka Streaming - Source > [Bytes/records Received] and Sink [Bytes/records sent] show zero messages > <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-Kafka-Streaming-Source-Bytes-records-Received-and-Sink-Bytes-records-sent-show-zero-messages-tp6941p6945.html> > > Sent from the Apache Flink User Mailing List archive. mailing list archive > <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> at > Nabble.com. >
